Data Science (“DS”), Machine Learning (“ML”), and Artificial Intelligence (“AI”) applications are increasingly being used to analyze and develop deeper insights into everyday business activities. Having a functioning implementation of ML or AI model is a huge competitive edge in today’s highly connected world. For some, it is the key to survival itself! With the scaling power of the cloud, the availability of free libraries to rapidly develop and deploy models, and the stability of underlying mathematical algorithms for the last 50+ years, businesses can’t afford to get it wrong.
However, according to Gartner, “most data science and machine learning projects routinely fail to realize expected business value due to mistakes in project execution.” As many as 80% of all projects (estimates vary from 60% to 90% depending on which survey you choose) are never deployed into production. The projects that do make it, rarely deliver the promised business value, are expensive to maintain and require highly specialized resources, both human and technological. In short, the magic of ML or AI being a cost-effective competitive advantage remains elusive.
This article offers some practical solutions to increase the success of ML or AI projects. For folks interested in technical details, a case study analyzing one of the most common use cases in ML implementations – credit card fraud detection – is presented at the end.
Practical Solutions to Increase ML & AI Project Success
Institutions can realize increased benefits of ML project by implementing one or all the solutions listed below:
- Recognize that Data Science is a function by itself. ML projects use a blend of functional, business, and technical roles and require sponsorship from the highest levels of the organization – the C-Suite. Just like the positions of Chief Technology Officer, or Chief Information Officer were added to the C-Suite in the 80s and 90s, success in using the power of ML & AI requires a Chief Data Scientist executive-level position. This executive can help organizations cut through the hubris or sales pitches and guide projects undertaken to completion.
- Consider the fundamentals of the problem at hand. Most realistic ML algorithms compress the feature space into a manageable set thereby introducing error into the model. Incorporating fundamental analysis at the feature space level and the unobserved-but-known-patterns (known unknowns) level can typically improve accuracy beyond a certain threshold. This concept is best highlighted by the blow-up of Long Term Capital Management hedge fund in 1998 that incorrectly applied the mathematics (Brownian motion) of physics based systems to financial derivatives, which obeyed no such laws. In other words, what’s in the error space of the model can cost dearly sometimes.
It is said, “history doesn’t repeat itself, it only rhymes.” Learning only from known observations limits the capabilities to handle information from a different pattern. All models suffer from the “Turkey problem”
- Assess the stability of the patterns being modeled at the onset. Model tuning is computationally expensive, even in the cloud. This crucial upfront analysis can help in designing the proper solution that can deal with shifting patterns. In our credit card fraud detection example (see below), while really good ML models can be developed via training them on historical patterns, fraudsters, being tech-savvy too, tend to deliberately shift fraud patterns quite often to beat ML models implemented by the FIs.
- Develop a multi-model approach, where multiple independent ML models work together to form a complex dynamic model and solve the problem at hand. This strategy can separate features into natural groups and reduce the complexity of the implementation as well as ongoing model maintenance (addressing the problem in point #3). For example, to better detect credit card fraud, the FIs can create several mini ML models, each with the capability to flag a transaction based on its own feature set. Say a customer pattern model (customer identity, how they spend, typical amounts, frequency, etc.). A risk management model (is the customer new, is the transaction usual for the customer, size of the transaction, origin, etc.). A general fraud detection model (is this general transaction fraudulent). Then, an ensemble model can processes the results of each of the components to determine whether the transaction at hand is fraudulent or not.
- Use best-in-class infrastructure, preferably in the cloud. Models running on the research team’s computers are useful in providing point-in-time insights but not for adding operational excellence. A company whose DS team is able to access and use data in the cloud, especially large data sets in the petabyte or higher range, are far more likely to be successful in deploying ML models into production than others.
Conclusion
ML and AI implementations hold a lot of promise and can be existential for some companies. However, like any other business function, to add value they require robust operations, technical ability, and proper sponsorship from the C-Suite.