FIU Data Science Bootcamp Application Deadline is Oct. 3 - Apply Now

Rabbit Holes, Red Herrings, and Rewards: Managing Curiosity

By Laura Montemayor • January 28, 2019

This post was written by Kerstin Frailey, Senior Data Scientist
on the 
Corporate Training team at Metis.

I recently wrote a post on Data Science at Work about a typical data science task: digging through someone else’s code for answers. Doing so is often unavoidable, sometimes critical, and frequently a time-suck. It’s also useful as an example of why curiosity ought to be intentionally managed. It got me thinking about how rarely managing curiosity is discussed and it inspired me to write about how I do it.

Curiosity is essential to good data science. It’s one of the most important characteristics to look for in a data scientist and to foster in your data team. However, jumping down a potential rabbit hole on the job is often viewed with suspicion or, at best, is reluctantly accepted. That's partly because the results of curiosity-driven diversions are unknown until achieved. And while it's true that some will be red herrings, many will have project-changing rewards. Pursuing curiously is dangerous – but entirely necessary – to good data science. Despite that, curiosity is rarely directly managed.

Why is managing curiosity particularly relevant to data science?

For one, data scientists are (hopefully) inherently curious. A data science team should be made of people who are excited about learning, solving problems, and hunting down answers.

But since data science is still a new field, and most companies have a bounty of potential projects to dive into, data scientists could be pursuing one project and be tempted by ten more exciting ones. On top of that, the once-popular tech mentality of move-fast-and-break-things has normalized us all to poorly scoped projects. Without proper scoping, we are left to face an unpredictable number of unknowns and are armed with no plan to attack them.

What's more, it’s often hard to differentiate between productive learning – a vital part of everyday data science – and aimless intellectual wandering.

Reining in Unbridled Pursuit

Many data scientists enter the field from research positions, academia, bootcamps, or through self-teaching. These are great environments to pursue a question relentlessly until a topic is mastered, a cafe kicks you out, or a dissertation is completed. This passion for learning is often what gets you the job in the first place.

But that same sort of time-unbounded adventure is not necessarily effective in a business environment. Pursuing curiosity is sometimes seen as lost productivity because it often does not yield a clear step forward in a project. However, effective data science demands the pursuit of curiosity, and curiosity-driven diversions can save a project from an unforeseen danger, can guide your data roadmap, and can keep data scientists interested in your project – and your company.

Curiosity often produces fruitful diversions. So how do we manage curiosity without crushing it?

Managing Your Team’s Curiosity

Manage, Don't Suppress

Enable data scientists to pursue questions, big and small, effectively. Equip your teams with the skills needed to manage their own curiosity and create an environment that helps them do so.

Manage Curiosity Appropriate to the Size of the Diversion

A 30-minute diversion does not merit a 45-minute process to manage it. That’s a classic example of over-engineering support. Allow your data scientists to Google, read books, whiteboard, close their eyes and think to themselves, or do whatever they need to do to make the most of these brief diversions.

On the other hand, a week-long diversion likely merits a formalized process to make the most of it. Whether that includes a short deck to share findings, creating a checklist of questions to answer, or establishing a temporary stand-up to track progress, the process should be discussed beforehand.

So, how do you know which type of diversion your team is faced with?

Properly Scope Projects

One of the best ways to manage diversions is to see them coming. You can do that by taking the time to properly scope your team’s projects. Good scoping is how you find out what you don’t know – and it’s exactly how you identify those big, juicy questions that could potentially derail your team’s progress.

Scoping is one of the most overlooked (and often altogether skipped) steps in a data science project. In an effort to prove worth or pursue an exciting idea, data science teams often rush to action without fully vetting a project. If your team is accepting more than one-third of the projects it’s considered, you need to revisit the scoping process.

Provide Structured Learning Opportunities

Curious people get bored. They get restless. They seek challenges. They leave companies for new ones. Providing structured learning opportunities like journal groups, lunch-and-learns, and dedicated training sessions can help feed the need to learn and build a stronger team at the same time.

Make Sharing Curiosity a Habit for Your Team

Sharing findings makes the time spent on a diversion ten times more productive. It builds a smarter team, precludes redundant diversions, and builds a culture of collaborative learning.

After a diversion, ask your team members what they found and ask them to share it. Make sharing standard. If your team has a standup, consider adding a spot to share there. If you're mostly remote, add a dedicated Slack channel. When the finding is a revelation, add a deck to your next team meeting. Make known spaces to share findings – and make them known. Make sharing curiosity easy and habitual.

Managing Your Own Curiosity

Pause Before You Dive In...

Before you go down a rabbit hole, think about it. Consider: What would be the ideal way to answer this question? What is the easiest way to answer this question? For example, Is there a teammate I can ask? Could I google it? Will I need to do original research?

Consider the minimum amount of information you need and what led you to ask this question in the first place. If you can’t find the answer, what can you change about your project, your assumptions, or your implementation so that you can move forward without this answer?

Write Down Your Question

It sounds simple, but actually writing down your point of curiosity will narrow your focus and keep your diversion on track. Writing down your question will serve as a guide and a reminder of what you’re actually searching for.

Assess the Implications of Not Knowing

We can’t know everything. The field is too big, the codebase too vast, Wikipedia too all-encompassing. Some answers are nice-to-know, some are need-to-know. Now is the time to evaluate the priority of your question’s answer.

Before jumping down the rabbit hole, ask yourself: What action will knowing this change? How will an answer change my behavior or decision? How would this compare to my alternatives?

Timebox Diversions

Now that you know your alternatives and the potential impact of this diversion, you can allot a reasonable amount of time to pursue it. Setting a deadline helps keep time spent proportionate to importance. This is helpful on minor pursuits, like tracking down implementation questions, but it can prove critical to larger questions that rely on coordinating among multiple parties, like digging up the details of data generation. Although it can be tempting to extend a timebox after it’s expired, remember that you chose that amount of time because that’s what the question’s priority merited.

Keep a List of Questions

What about diversions that arise within a diversion? Write them down. What about questions that aren’t fully answered within their timebox? Write them down. What about questions that don’t merit pursuit at all? Write them down.  

We’re often reluctant to let a question go unanswered. We feel that if we don’t answer it now, we never will. This doesn’t have to be the case. A list of questions allows you to return to them later and answer them as potential solutions arise during the course of your regular work.  If you happen to find some spare moments between projects, you can always return to this list for a discretionary diversion.

Share and Record What You Learned

No matter if it’s a 30-minute diversion to find out implementation details or a week-long project to uncover the origins of the data, one of the most important yet overlooked aspects of curiosity is sharing and recording what you learned. Discovery done at work should be shared. Make it a habit. Make your team smarter. Make that diversion ten times as productive by sharing your findings with others.  You’ll find that you understand it even better after you’ve shared the finding with your team.

Safeguard Curiosity – and Its Benefits

Curiosity is not a luxury, but a necessity for good data science. Managing curiosity allows us to pursue it without guilt and ensures that the fruits of its diversions are shared, which helps build healthy, engaged, and smarter teams. Sharing our learning allows us to capture the value of curiosity-driven diversions. In doing so, we protect our ability to follow our curiosity – and to continue reaping rewards.


Learn about Metis Corporate Training here

Similar Posts

business resource
VIDEO: Recorded Talk - How Machine Learning is Changing Finance with Javed Ahmed

By Carlos Russo • August 20, 2020

Watch a recording of Metis Sr. Data Scientist Javed Ahmed's talk on How Machine Learning is Changing Finance at the new Wake Forest University Financial Services and Fintech Hub.

business resource
VIDEO: An AI4 Panel Discussion on The State of AI in Banking

By Carlos Russo • September 23, 2020

Metis Sr. Data Scientist Javed Ahmed recently took part in a panel discussion about The State of AI in Banking during an online Ai4 event. He and the other panelists talked about upskilling, challenges related to COVID-19, and more. Watch the recorded panel discussion here.

business resource
Scoping Data Science Projects

By Damien Martin • July 07, 2021

In February, Metis Sr. Data Scientist Damien Martin wrote a post on how to foster a data literate and empowered workforce, which allows your data science team to then work on projects rather than ad hoc analyses. In this post, he explains how to carefully scope those data science projects for maximum impact and benefit.