This post was written by Kerstin Frailey, Senior Data Scientist
on the Corporate Training team at Metis.
I recently wrote a post on Data Science at Work about a typical data science task: digging through someone else’s code for answers. Doing so is often unavoidable, sometimes critical, and frequently a time-suck. It’s also useful as an example of why curiosity ought to be intentionally managed. It got me thinking about how rarely managing curiosity is discussed and it inspired me to write about how I do it.
Curiosity is essential to good data science. It’s one of the most important characteristics to look for in a data scientist and to foster in your data team. However, jumping down a potential rabbit hole on the job is often viewed with suspicion or, at best, is reluctantly accepted. That's partly because the results of curiosity-driven diversions are unknown until achieved. And while it's true that some will be red herrings, many will have project-changing rewards. Pursuing curiously is dangerous – but entirely necessary – to good data science. Despite that, curiosity is rarely directly managed.
Why is managing curiosity particularly relevant to data science?
For one, data scientists are (hopefully) inherently curious. A data science team should be made of people who are excited about learning, solving problems, and hunting down answers.
But since data science is still a new field, and most companies have a bounty of potential projects to dive into, data scientists could be pursuing one project and be tempted by ten more exciting ones. On top of that, the once-popular tech mentality of move-fast-and-break-things has normalized us all to poorly scoped projects. Without proper scoping, we are left to face an unpredictable number of unknowns and are armed with no plan to attack them.
What's more, it’s often hard to differentiate between productive learning – a vital part of everyday data science – and aimless intellectual wandering.
Reining in Unbridled Pursuit
Many data scientists enter the field from research positions, academia, bootcamps, or through self-teaching. These are great environments to pursue a question relentlessly until a topic is mastered, a cafe kicks you out, or a dissertation is completed. This passion for learning is often what gets you the job in the first place.
But that same sort of time-unbounded adventure is not necessarily effective in a business environment. Pursuing curiosity is sometimes seen as lost productivity because it often does not yield a clear step forward in a project. However, effective data science demands the pursuit of curiosity, and curiosity-driven diversions can save a project from an unforeseen danger, can guide your data roadmap, and can keep data scientists interested in your project – and your company.
Curiosity often produces fruitful diversions. So how do we manage curiosity without crushing it?
Managing Your Team’s Curiosity
Manage, Don't Suppress
Enable data scientists to pursue questions, big and small, effectively. Equip your teams with the skills needed to manage their own curiosity and create an environment that helps them do so.
Manage Curiosity Appropriate to the Size of the Diversion
A 30-minute diversion does not merit a 45-minute process to manage it. That’s a classic example of over-engineering support. Allow your data scientists to Google, read books, whiteboard, close their eyes and think to themselves, or do whatever they need to do to make the most of these brief diversions.
On the other hand, a week-long diversion likely merits a formalized process to make the most of it. Whether that includes a short deck to share findings, creating a checklist of questions to answer, or establishing a temporary stand-up to track progress, the process should be discussed beforehand.
So, how do you know which type of diversion your team is faced with?
Properly Scope Projects
One of the best ways to manage diversions is to see them coming. You can do that by taking the time to properly scope your team’s projects. Good scoping is how you find out what you don’t know – and it’s exactly how you identify those big, juicy questions that could potentially derail your team’s progress.
Scoping is one of the most overlooked (and often altogether skipped) steps in a data science project. In an effort to prove worth or pursue an exciting idea, data science teams often rush to action without fully vetting a project. If your team is accepting more than one-third of the projects it’s considered, you need to revisit the scoping process.
Provide Structured Learning Opportunities
Curious people get bored. They get restless. They seek challenges. They leave companies for new ones. Providing structured learning opportunities like journal groups, lunch-and-learns, and dedicated training sessions can help feed the need to learn and build a stronger team at the same time.
Make Sharing Curiosity a Habit for Your Team
Sharing findings makes the time spent on a diversion ten times more productive. It builds a smarter team, precludes redundant diversions, and builds a culture of collaborative learning.
After a diversion, ask your team members what they found and ask them to share it. Make sharing standard. If your team has a standup, consider adding a spot to share there. If you're mostly remote, add a dedicated Slack channel. When the finding is a revelation, add a deck to your next team meeting. Make known spaces to share findings – and make them known. Make sharing curiosity easy and habitual.
Managing Your Own Curiosity
Pause Before You Dive In...
Before you go down a rabbit hole, think about it. Consider: What would be the ideal way to answer this question? What is the easiest way to answer this question? For example, Is there a teammate I can ask? Could I google it? Will I need to do original research?
Consider the minimum amount of information you need and what led you to ask this question in the first place. If you can’t find the answer, what can you change about your project, your assumptions, or your implementation so that you can move forward without this answer?
Write Down Your Question
It sounds simple, but actually writing down your point of curiosity will narrow your focus and keep your diversion on track. Writing down your question will serve as a guide and a reminder of what you’re actually searching for.
Assess the Implications of Not Knowing
We can’t know everything. The field is too big, the codebase too vast, Wikipedia too all-encompassing. Some answers are nice-to-know, some are need-to-know. Now is the time to evaluate the priority of your question’s answer.
Before jumping down the rabbit hole, ask yourself: What action will knowing this change? How will an answer change my behavior or decision? How would this compare to my alternatives?
Now that you know your alternatives and the potential impact of this diversion, you can allot a reasonable amount of time to pursue it. Setting a deadline helps keep time spent proportionate to importance. This is helpful on minor pursuits, like tracking down implementation questions, but it can prove critical to larger questions that rely on coordinating among multiple parties, like digging up the details of data generation. Although it can be tempting to extend a timebox after it’s expired, remember that you chose that amount of time because that’s what the question’s priority merited.
Keep a List of Questions
What about diversions that arise within a diversion? Write them down. What about questions that aren’t fully answered within their timebox? Write them down. What about questions that don’t merit pursuit at all? Write them down.
We’re often reluctant to let a question go unanswered. We feel that if we don’t answer it now, we never will. This doesn’t have to be the case. A list of questions allows you to return to them later and answer them as potential solutions arise during the course of your regular work. If you happen to find some spare moments between projects, you can always return to this list for a discretionary diversion.
Share and Record What You Learned
No matter if it’s a 30-minute diversion to find out implementation details or a week-long project to uncover the origins of the data, one of the most important yet overlooked aspects of curiosity is sharing and recording what you learned. Discovery done at work should be shared. Make it a habit. Make your team smarter. Make that diversion ten times as productive by sharing your findings with others. You’ll find that you understand it even better after you’ve shared the finding with your team.
Safeguard Curiosity – and Its Benefits
Curiosity is not a luxury, but a necessity for good data science. Managing curiosity allows us to pursue it without guilt and ensures that the fruits of its diversions are shared, which helps build healthy, engaged, and smarter teams. Sharing our learning allows us to capture the value of curiosity-driven diversions. In doing so, we protect our ability to follow our curiosity – and to continue reaping rewards.
Learn about Metis Corporate Training here.