Part 2: Guidelines of turning a PoC into an AI product

In the first part of this article, we saw the different types of data science projects that exist and how they are fundamentally different. In this part, we are focusing on the most interesting type of data science project, the one which turns into an AI solution or product. But how can the efforts it takes to build an AI product be explained? Isn´t it more or less the same as a PoC after all?

The concept car at an international auto show

I love to think of data science PoCs in similar terms to cars on exhibit at an international auto show. The car is standing on the turntable. It is nice, it is shiny, it raises interest, it gives a glimpse of what is feasible, and is there to amaze potential clients. But the car is also purpose-built for the exhibit. It may be too heavy, it surely is too expensive to produce in the same way, some parts are custom 3D prints, it may be too clumsy and not support all safety standards required. It may break down every 300km.

The challenge of turning “the car on exhibit into a product” means addressing these issues. How can we set up factories to build such cars, how do we produce the parts in a robust yet efficient way, is there a modular platform strategy this car can be tailored off? What does the distribution of spare parts look like and which training do the mechanics need to be able to maintain this car. After all, the client is looking for a “zero downtime” and “low maintenance” car. Similar considerations apply to data science projects.

From PoC to AI product

There is a considerable amount of work to be done to turn a proof of concept into a full AI product. Below I put together a nonextensive list of aspects you need to have on the radar. This list may not be complete, but all elements are key factors in successfully bridging the gap from PoC to product.

Build with the production environment in mind – not with your laptop

With the false assumption not to lose time and because “it is easier”, data scientists start building out models on their laptops. Whereas we agree this is the logical place to start, it is key to go beyond local environments and from early on build with the target environment in mind. This will reduce the number of surprises later on and help to scale as well.

Robust data pipelines

It’s called “data science” after all. Data is the new oil and data needs to be accurate and flow in properly. This is why it is of paramount importance that the data science team has access to the right data (and enough of it) from very early on. Having data engineering pipelines set up accordingly is important for automation. But data should not only be flowing in continuously, it should also be collected where it makes sense.

Collect feedback data

From early on, the collection of feedback data should be addressed. This can be as simple as a thumbs up or down from the user if the prediction was good. By continuously creating new labeled data, we ensure that feeding back data is possible and retraining of the models can happen. This advice is key for any company which is serious about going fully digital.

Model maintenance needs to be continuous

MLOps (Machine Learning Operations) aims to shorten the analytics development life cycle and increase model stability by automating repeatable steps in the workflows. A model usually is optimized for the moment it goes live. From that moment on, the world around is changing and data will be diverging from what the model was initially built from. If the model is not restrained with new raw or feedback data, its performance will decline over time. This is why it is important to collect metrics about model performance and make model retraining part of the maintenance activities.

Deploying AI at Scale

The deployment of AI at scale is one of the most important challenges in building an AI product. On one side, there are the technical challenges of training the models on an ever-changing and larger corpus of data. Here cloud computing can help overcome the bottlenecks of processing power by introducing the required elasticity. But obviously, this comes at a price of compute, so finding the right balance between cost, and retraining frequency are important. On the other side, there are also organizational challenges: Once deployments become larger, more people in the organization are kept busy keeping an eye on the components and metrics as more can go wrong. Different models need to be packaged and may coexist in production. Fallback scenarios need to be planned as more can go wrong – and it will have a larger impact. Here automation is the key to ensuring repeatability of deployments as well as stability of the system and the models.

Documentation

Most developers don’t like writing documentation – but they all like to read the documentation. It is obvious that documentation is an essential aspect of every (AI) product and needs to be kept up to date. The documentation should include information on APIs to access the model, but also internal documentation about how particular aspects of the model work. Last but not least, it should also explain as a kind of a user guide on how to use the AI product to non-technical people.

AI Ethics & Biases

AI systems are not exempt from making unfair decisions and sometimes this comes with serious consequences. Nowadays, AI models decide if we get a loan, get a scholarship, get a job, travel, or get arrested. Whereas research focuses a lot on the intended misuse of AI, the consequences of unintended AI usage can be equally damaging. While it may be impossible to ride AI systems of human bias, it is important to take caution to minimize its effects. This can be done by the careful selection of training data, close monitoring, continuous data governance, and a diverse population that covers a broad spectrum of inputs and offers a fair representation of our social structures. This is why it is important to understand the fairness of your models and closely monitor biases over time within your AI products.

Conclusion

In this post, I have given you my view on how I distinguish between three major types of data science projects. I have then focussed on the AI product and solution type and dug deeper into how to explain to senior managers and decision-makers what it takes to go from a simple PoC to a productive AI solution or product. As the main takeaway, I would like to stress that by having good processes and the right technology stack in mind, bridging the gap between PoC and productive AI products can be made predictably and robustly. It goes without saying that the right skills are key. Having data- and software engineers be part of the process early on, can help take the data science models into managed and deployable artifacts which work at scale.

Marc Giombetti

Marc Giombetti

Marc Giombetti is the Chief Product Architect at TwoImpulse. He has 15+ years of international experience in IT, mainly within the re/insurance sector. His professional interests include AI, Cloud, SaaS, IoT, Time Series, Software Quality, and Agile development.