From prototype to production - The journey of a successful data science project (Part 1)

A popular article from 2019 claimed that 87% of machine learning (ML) projects never make it to production, and this number is often cited in MLOps (Machine Learning Operations)  blogs. While this figure is not backed by data or reliable research, it is thought to be reasonable given the number of projects that stop at an early stage of proof-of-concept.

A more concerning issue is the number of projects committed to management or customers that are ultimately terminated before or after reaching production, despite significant investments of time and effort. According to research from MIT Sloan, the gap between organisations that are able to effectively utilise data science and those that struggle to do so is widening. One of the main reasons for this is a lack of understanding about how analytics and machine learning can be applied in the context of the business. Organisations may recognise the value of data science, but they may not know how to effectively incorporate it into their operations or how to measure the impact of their data science projects.

To bridge this gap, organisations need to take a strategic approach to data science. This involves identifying the key business challenges that data science can help solve, setting clear goals and objectives for data science projects, and building a team with the right skills and expertise to execute these projects. While development and optimisation of machine learning models are well-documented in technical literature, there is a lack of resources on how to effectively utilise machine learning in a business context. 

To address this gap, we present an eight-step process for approaching machine learning from a business perspective. The following steps have been successfully employed by our team at Sonalake in the creation and scaling of our machine-learning products. 

1. Defining the business objective
2. Taking a data-centric approach and assessing the available data
3. Developing a baseline 
4. Building an end-to-end prototype 
5. Testing and refining the model
6. Deploying the model
7. Post-production model monitoring through MLOps 
8. Ensuring model stability and performance

While these steps may seem straightforward, using them in combination provides a clear workflow for productionising machine learning models that drive significant impact for a business.

Business objective

The first challenge is to start with the problem and the objective rather than being driven by a technical solution.

As companies rush to incorporate machine learning into their organisations to keep up with the latest developments, leaders and machine learning experts tend to focus on building solutions that seek to automate or improve processes without first assessing whether the problems are actually suitable for ML to solve. 

Even if the business problems are a fit for machine learning, it is imperative that data science leaders and stakeholders clearly define the problem and agree upon what success might look like. Without a clear definition of what ‘success’ might look like, projects can languish in the experimentation phase for months, with stakeholders waiting for unrealistic levels of perfection and accuracy.

Machine learning isn't a silver bullet. It doesn't solve every problem. The inherent nature of an ML model is to evolve over time and this needs to be accounted for. Sometimes a model that achieves human-level performance is just what is required and stakeholders must understand that the model development is an ongoing process. Every model can evolve and become better with time.

At the very start of a data science project, ask the team or stakeholders: What business problem are we trying to solve? Why do we think machine learning is the best way to go? What is the measurable threshold of business value that this project is trying to achieve? What does "good enough" look like? 

If these clear and common definitions are not formulated from the start, many valuable machine learning projects will never reach production and valuable resources will be wasted. Instead of starting a machine learning project to deliver unrealistic goals, use machine learning to solve business problems. Once the business problem is solved by the model, then the team can gradually aim to raise the bar with machine learning.

Data-centric models and the significance of data collection

With a clearly defined business problem and targeted success metrics, the potential pitfalls get technical as we move ahead. It is essential to get the approach right from the very beginning of the project. Data and model (algorithm) together form the backbone of any AI system. 

Whenever companies talk about implementing AI at their organisation, the focus tends to be on using the latest, state-of-the-art machine learning model without realising that the data holds equal importance in the success of any Data science project. For a long time, working on ‘data’ has been thought of as a low-skill task and most data scientists prefer to work on models instead of the data. To improve a solution, we can either improve our code, improve the data or do both.

The latest developments in deep learning algorithms and advanced neural network architectures are paving the way for a paradigm shift toward data-driven AI. Here the code is the fixed part and the data entered in the code is refactored so that the model can produce better results. In general, the data is improved by improving the labelling order, removing noisy data, performing an in-depth error analysis, and performing many other data augmentation techniques appropriate to the problem.

The modified approach was empirically tested by a 2021 study titled "A Data-Driven Approach to the Design and Analysis of Deep-Knowledge Surface Inspection Systems in the Plastic Injection Molding Industry," which confirmed that data modification was more effective than code modification.

Baseline model

A baseline is a simple model that can be built without much experience or time, but it provides reasonable results for your work. Common underlying models include linear regression for continuous value prediction, logistic regression for structured data classification, pre-trained convolutional neural networks for vision-related tasks, recurrent neural networks for sequence modelling, and gradient boosting trees.

A baseline will take only 10% of the time to develop, but it will get us 80% of the results we want. A baseline is a simple model we can use to compare a more complex model. One advantage of starting with simple baseline models is that they are typically quicker and easier to implement, providing a benchmark to measure against if we later pursue a more complex solution. Additionally, working with these methods can help practitioners get familiar with the data and gain a deeper understanding of the problem they want to solve.

Building an end-to-end prototype

When developing the initial iteration of a machine learning model, it can be tempting to add unnecessary complexity in an effort to achieve the best possible solution. However, this approach can increase the time and effort required to build the model, as well as the overall complexity of the system, which can make it more difficult for new team members to understand and maintain. 

Additionally, starting with a simple model can help to quickly build out the entire pipeline, reduce the risk of unexpected surprises, and get the system working end-to-end. Simple models, such as shallow decision trees or linear regression, are often easier to understand and train quickly, which can be valuable for getting buy-in from stakeholders and non-technical partners. Starting with a basic set of features can also help to facilitate faster iteration and quickly get a model up and running. Therefore, it is essential to consider the costs of complexity when developing the first iteration of a model. While we can add complexity later if necessary, it is often more efficient to start with the simplest approach possible and see if you can improve upon the baseline performance.

Testing and refining the model

A common mistake is to optimise machine learning models before the model is tested using an appropriate performance testing metric, and before the overall performance of the model is understood in the overall context of the business. It is crucial to tune various parameters and hyperparameters in order to improve model performance, but this should not be the focus until the model architecture, objective functions, input features, and performance metrics have been thoroughly considered and established. Premature optimisation can be counterproductive and hinder the overall performance of the model. 

Before we attempt to optimise the model parameters, we need to have a well-defined performance metric. The performance metric should be based on the primary objectives of the business, and it is essential to understand the meaning of the metric value in the business context. For example, if the goal is to build a fraud detection system, accuracy may not be the most appropriate metric, as instances of fraud are typically rare. Instead, a metric such as precision or recall might be more suitable. On the other hand, if the goal is to predict customer churn, an accuracy rate of 98% might be considered an excellent performance. To establish a  benchmark value for the performance metric, it can be helpful to use the baseline model as a point of comparison.

To effectively measure the impact of a performance improvement on the business, it is important to have a strategy in place that quantifies the impact of each percentage point increment on the user of the product. For example, if the chosen performance metric is accuracy, it is necessary to consider whether an increase from 0.75 to 0.80 is significant to the user or barely noticeable and whether the potential added time and complexity required to achieve these additional points are worth the potential benefit. Understanding the tradeoff between performance and resources is crucial in deciding how to optimise the model and inform decisions about whether to continue investing time and resources in a particular model.

It is easier to optimise the various parameters of a machine learning model once we establish the performance metric and link it to a business impact. The optimisation process can be stopped when it no longer drives any incremental business impact. By being able to explain the tradeoffs involved in the model, errors that may not be immediately apparent due to high overall performance can be identified earlier. Understanding the tradeoffs made by the model allows for more informed decisions about how to optimise its parameters and improve performance.

Deploying the model

Deploying machine learning projects can be complex, as teams often have to contend with a wide range of ways to host and deploy models based on the specific business problem they are solving. This includes batched predictions for some projects, while others may require on-demand predictions through API requests. Furthermore, the resources required for some projects can be significant, which can make cross-functional teams consider the deployment of other projects that are deemed more critical. 

Given this complexity, teams need to have a clear understanding of the business problem and the desired outcomes at the start of the project. Teams should have a shared vision for the end-to-end solution and a clear understanding of the project's value compared to other priorities. Without a comprehensive strategy in place, it can be difficult to secure the necessary engineering resources to bring the project to production.

To help scale machine learning production, there are several practical steps that can be taken.

Adopting cloud-based solutions: By working in the cloud, teams can keep model training and deployment workflows in sync. Cloud-based solutions make it easier to automate processes, as many vendors and open-source tools are designed for cloud environments. The cloud also allows for a more consistent and repeatable process for testing, training, validation, and model deployment.

Implementing DevOps practices: Just as DevOps is used for data management, ML teams can adopt a similar approach by implementing a continuous integration (CI), continuous delivery (CD) and continuous training (CT) model. By using agile build cycles and tools, teams can quickly deliver changes to the codebase and improve overall performance. Containerisation can also be used to consistently run software across different devices, and make it easier for engineers to productionise their work.

Building an MLOps team and process: Investing in creating a strategic MLOps team and processes can reduce the risk of projects stalling out before production and make continuous improvements more feasible. This will set projects up for long-term success.

In Part 2, we’ll look at post-production model monitoring through MLOps and how to maintain model stability and performance.