part of Course 121 Navigating a Data Science Career

The Landscape

Picture your many career options as a landscape. Every location is a potential goal. Getting there requires time and heart and you have finite quantity of both. There are many many places you could go. Which do you choose?

These aren't weekend trips. Most of these places take years to reach. Some of them, decades. The anxiety this provokes is intense. What if I spend most of my life getting to a place only to figure out I didn’t really want to go there? What if I loiter too long trying to make up my mind? What if I get distracted by a comfortable rock next to a creek, and by the time I decide to get up and move it’s too late to get where I wanted?

The fear of missing out is very real. Our peers can be brutal. Pressure to publish. Pressure to get first round funding. Pressure to make our first million, then our tenth, then our hundredth. Use blockchain. Learn large language models. Reform health care. Cure hunger. Fix homelessness. Get stock options. Get tenure. Get a Ph.D. Which way do we go? What can we bear to miss out on?

At its core this is an optimization problem. How do you allocate the scarce resource of your professional effort? You can’t solve an optimization problem until your optimization criterion is very precisely stated. In other words, what do you want more than anything else?

On the surface, it may appear that technical impact, business impact, and social impact go together. That wealth and intellect can be built at the same time. That fame and academic reputation are all different parts of the same thing. And that peace and companionship and personal satisfaction accompany them. But in my experience, this is not often the case. A great many professional decisions are between what you want and what you want more.

Knowing what you want most can sometimes be the hardest part. I once told a friend about an algorithm I was working on. He asked if I could magically have one of two things, would I rather a) have the algorithm be widely adopted or b) get recognized for it? It was a great exercise prioritization. Did I want to make something I am deeply proud of? Did I want other people to think that I’m smart? Did I want to be wealthy? Of course, there’s nothing wrong with wanting all of the above, but there is very likely a time when you get to choose one over the others. When that time comes which will it be?

The bad news is: No one can tell you.

The good news is: No one can tell you.

Most people don’t know what they’re talking about. Most people don’t have your interests at heart. What makes someone else happy and satisfied may not work for you. You might be looking for something different than everyone else. It's useful to gather ideas from everyone and everywhere, but your sense of rightness is the ultimate authority. No one can gainsay that. No one.

If you want to change the world, there are lots of goals you can choose from for focusing your technical career:

• Scientific
  • Understand intelligence
  • Explore the cosmos
  • Unify physical forces
• Environmental
  • Fight climate change
  • Sustainably clean water
  • Prevent extinctions
  • Reduce pollution
• Social
  • Hunger
  • Disease
  • Education
  • Energy
  • Race and gender equality
  • Human rights

Pitfalls

Letting someone else set your priorities

One way this happens is when the company you work for claims to be changing the world and asks you to give your all. Sometimes this means giving up your evenings and weekends. Giving up your peace of mind. Giving up your physical health. Giving up your confidence and your happiness.

Being swept along by the stampede

When everyone you talk to you and every post you read carries a message of hurry or publish or profit or blockchain, it is very difficult to keep your head. No matter how often you hear it from how many sources, don’t let someone else pick your goal.

Trying to do all the things

We have all heard the profiles of the M.D., Ph.D., J.D. entrepreneur humanitarian philanthropist celebrity. It is an appealing myth. Why choose? Do everything! Unfortunately, like Tony Stark and other archetypes of this path, it is a fiction. No one person has the time and mental capacity to do everything well. However, be on the lookout for those who spend all their effort appearing to do everything. If all you care about is the appearance, you can fool most of the people most of the time.

We only have so many days and so many heartbeats to navigate our professional landscape. We don’t get the option to climb every mountain and swim every sea. We have to choose.

Being indecisive

This isn’t quite like doing all the things, but it is related. If you set off toward the mountaintop and then early on in the journey change your mind and head for the lake, and then before you’ve gotten very far switch goals again for the glacier, you will end up never reaching any of them. In modest amounts, this is a fantastic way to discover new destinations that you didn’t even know existed. However, taken too far, this can prevent you from getting anywhere that you really want to be. When taken to the extreme it is indistinguishable from standing still.

Not stopping to smell the flowers

A relentless push for the far horizon will get you a lot of miles under your belt, but your attention is limited. At any moment you can either be focused on forward motion or on experiencing where you are. It’s easy to forget that the ultimate goal is really to experience the destination, not just to get there, and that there is a great deal of goodness and richness to experience along the way if we can allocate the attention to it.

Sunk cost fallacy

When it has become clear that direction you are headed in is wrong for you, choose a new one. It doesn't matter your age or how many years you've been doing what you're doing. It’s never too late to try change directions. It’s only too late when you’re dead.

My Own Plan

I can’t tell you what the best answer is for you. I can’t even tell you what the best answer is for me. Solving a problem like this confidently would require many lifetimes. I simply don’t have enough data. There are decisions that I have to make based on something other than data-driven experiments with tight confidence intervals. There’s still a lot of uncertainty. I end up having to place bets. Here are the bets I’m placing right now.

• I’m curious about machines that appear to think, especially machines that move. I want to know how every part of them works. I want to try out my own ideas to make them better. This has captivated me since I was six years old.
• I can best do this at work as an individual contributor. Management roles would prevent me from being immersed in the technical work. I’ve avoided them.
• When choosing between work that is likely to gather attention and big bonuses and work that I find technically intriguing, I’ll lean toward intriguing.
• When I hear chatter about a hot new subfield of AI, I take deep breaths, calm my FOMO, and turn back to my project on a thing that I’m curious about and feel satisfaction from.
• I say no to a lot of invitations to speak or write or collaborate. Many of them are amazing but not quite worth all the time away from my curiosity driven work.
• I invest time in sharing what I learn online. I’m so excited by understanding something new that I feel compelled to show others how cool it is. I want to mark the path to the place I found. Selfishly, explaining ideas helps me learn them on a deeper level too.

The best answer for you will almost certainly be different than the best answer for me. It all depends on what you enjoy. What you value. The people in your life. Health. Age. Money. Location. It would be a mistake to try to do what I’m doing. We are in different situations, and my approach is evolving all the time. What is most important, your North Star, is knowing what matters most to you. It’s OK if this changes over time. It probably will. But the clarity of your priorities is your strength and your direction.

Good luck on your journey.

Thank you to Aditya Jaishankar for the thoughtfully worded question that inspired this post.