Machine Learning products are built by teams, not unicorns

Over the last few years, many software organisations have started developing products which leverage machine learning.

Like every new technology, ML involves a certain learning curve, and managing this learning curve is critical for a successful ML adoption.

Interestingly, many organisations focus so much on the areas they need to learn, that they miss the opportunities to leverage assets they already have, which are critical for their success.

A great example of this phenomenon is the “Data Science Unicorn” myth.

The “Data Science Unicorn” myth

Some companies expect to hire a single data scientist, who will deliver their ML project practically single handedly.

This person is a “Data Science unicorn” - mythical being who can magically execute a Data Science project on her own

implementation often requires adaptation image source

A taste of what a non-trivial Data Science project involves

Here are some of the tasks you need to complete in order to deliver a successful ML project:

• Talk to business actors about the requirements, repeatedly. Communicate expectations, negotiate, and translate their needs into ML goals
• Get data, clean it and organise it
• Make decisions about which labels to create and their tradeoffs
• Supervise the labelling process if needed
• Develop and test features
• Set up and organise her development environment
• Run model experiments and track them
• Analyse model mistakes and figure out what to do about them
• Perform statistical analysis and prepare performance reports
• Make adjustments to the data and the requirements multiple times as she learns things about the domain
• Deal with packaging the model for deployment.
• Own the model when it runs in production
• Refresh and update it when feedback loop data comes in

Now, it’s true that good data scientists often like working alone — especially those who recently graduated from an advanced degree. They are also very analytical and like dealing with new challenges and unknowns.

However, there are very few people who can pull all this off by themselves, for anything beyond a POC, simply since they don’t have the skills or context, and most of them only have two hands.

What happens when you believe in the myth?
If you tell a Data Scientist you expect her to deliver the project by herself,
she will be flattered by the trust, and will do her best to meet your expectation.

However, after a while, these are some of the things you’ll start to notice:

• Data scientist is overwhelmed, stressed and feeling isolated
• She may not be aware of some of the areas she should be dealing with, or spend most of her time on the areas which interest more and neglect others
• No one can really supervise the work or validate assumptions. It’s hard to weigh in or help her make decisions — since no one else knows the details to a sufficient degree
• It’s hard add new people to the project at a late stage, since it’s built in a very unique and idiosyncratic style
• The data scientist becomes a single point of failure for the project. If she decides to quit you are back to square one
• Progress is slow and unpredictable
• There is a good chance she is actually trying to solve the wrong problem
• Everybody is frustrated

Ignoring the basic truth about ML projects

The unicorn myth is a good example for how companies do everything they can to avoid building ML teams, despite all the tasks and skills needed.

Even when many of these skills are not out of reach, to say the least…

Getting in touch with your soft(ware) side

Here’s some good news which you may have forgotten: Machine Learning is, at the end of the day, a type of software development effort.> If you’ve delivered sufficiently complex software projects to production before, you are probably already halfway to success (or more, in some cases).

Many of the activities in an ML project are still subject to core software development principles and best practices.

Caveat:
ML follows most Software Development principles — but their implementation often requires adaptation. So don’t try and forklift your scrum ceremonies, story points and time tracking to your new ML project quite yet…

Do you need convincing? Let’s go!

• Will your ML product have requirements? In most non-trivial projects, it certainly should! Sure, you will need to define them differently and expect to refine them on the go, but you still need to write them — and accept deliverables too. It should be someone who represents the user. You should have a Product Owner
• Will you need to build data pipelines and create features from the data? Data Engineers can help with that
• Will your model be serving decisions in production? Backend Engineers are good at performance tuning, defining API’s, and scaling services. Give them a call
• Talking about production — it’s obvious that before going live, you need to do sufficient testing — you will benefit from QA and automation engineers to help you test your model meets the requirements
• Do you think you’ll ever want to re-train your model with new data? Do you want to be able to test ideas quickly and reproducibly? Then you’ll need to engineer your model training pipeline, environment, experiment tracking system, and a slew of other tools which help debugging and exploring models easy (or, more likely — tolerable) You should assign a Research Engineer to build your infra

Now, these are logical roles , but in large enough projects you will need many of these roles to be **** carried out by separate individuals — because you
will need both a high level of expertise and enough capacity.

There’s another important benefit to involving your “software folk” early on — they have a good domain understanding, which is something new ML hires don’t usually have on day one.

If these insights seem obvious, that should be a relief.

But if this is so obvious, then why is it that so many companies do not introduce these practices and disciplines into their ML projects, and in the extreme case place their bets on unicorns?

I believe the issue is mainly a cultural one.

Understanding the cultural gap in ML projects

Data Scientists hail from a rather different culture, which is probably new to your organisation. With different belief systems, views of what professional success looks like, and of course day to day practices.

Some of them come from academia — which is radically different in many other, even more fundamental ways.

At the heart of the cultural clash are two different mindsets: the Research mindset and the Engineering mindset. Successful teams must overcome this clash and create harmony between them.

Here are some examples for what the clash looks like:

To make a very broad and slightly exaggerated generalisation:

When teams first form, Engineers often look at the Data Scientists and think that they are self important, that they over complicate, under commit and under deliver. They will also be shocked by their work habits, especially around code.

The Scientists, on the other hand, may initially see engineers as technicians who don’t understand data or machine learning, and hence don’t really “get” the project. Instead, they spend all their time obsessing about ceremonies and practices which seem far removed and irrelevant to the problem at hand. Do they really think committing code every five minutes is what’s going to solve this model bias?

What makes this cultural gap a bad one?

I am not an expert on the theory of multi-disciplinary teams, although I did spent 15 years of my career in multi-disciplinary environments.

Based on my experience, I’d say that the ML-Software cultural gap is an especially challenging one, compared to other cross-culture friction you often get in teams.

For example, Product-Engineering is a significant cultural gap, but still most team found a way to make it work, right?

Here are a few reasons why I think that’s so:

**It’s a new experience
**Product and engineering have likely worked together in previous projects, or other companies. They kind of know what to expect and that they can succeed together.

In ML projects that trust is not there yet.

**There are no clear ground rules
**Software teams usually follow a well articulated dev. methodology — usually a flavour of agile. The methodology defines accurately what are the different roles and responsibilities of different disciplines, and gives each of them their place without creating silos or disrespect.

In ML, however, this safety belt doesn’t exist yet, and teams need to find their own way to make it work. This is a big challenge for any team — even without a cultural gap.

**The similarities are misleading
**In the case of product people, or any non-technical SME (subject matter experts) which are a part of a team — it’s clear they come from a different background and there’s very little skills overlap. They usually do not attempt (and can’t) overstep the boundaries and tell engineers how to do their jobs, and the other way around.

With ML and Engineers, it’s a lot less obvious since on the surface, both disciplines write code, and in many cases — do many other similar activities.

As a result, the difference in opinion between ML and engineers remains undiscovered until a late stage, and when it surfaces, it surfaces right in the middle of people’s home turf — making it much worse

Nevertheless, if you want to build a successful ML project, it’s necessary to overcome the initial cultural clash and find a way to tap into the value of each of the disciplines in the project.

Conclusion

Machine learning is very different from software engineering, but also overlaps a lot.

Successful machine learning projects require a strong presence of software development professionals such as product managers and engineers.

Some companies overlook this fact and attempt to build “one man teams” a.k.a hire “data science unicorns”. However, this is not a good strategy for anything beyond a POC.

One of big challenges when setting up a team of multiple disciplines is the cultural clash between ML and software development cultures. By being aware of the origins and areas in which these cultures clash,
you can hopefully begin to address it.

There’s a lot more to be said about setting up successful ML teams who can work together effectively. I hope to cover some of this in future posts.

Cheers,

Assaf