Although, optimization is not frequently highlighted in the most influential media channels about data science and is not headlined as important as artificial intelligence in massive news: it still has a very strong community around researching and applying these techniques to solve important problems in operation research; it is as old as calculus; it is deeply involved in the current daily task of many data scientists, and it will be more and more at the center of new technologies such as continuous intelligence.
Optimization is at the core of prescriptive analytics, that is why I will dive into this analytics, commonly used to solve a particular sort of problem.
The natural evolution of analytics (descriptive -> predictive -> prescriptive) is breaking into companies due to the power and value that smarter business decisions can have on KPIs. However, analytics overall is still dominated by descriptive (understand the past) and predictive (forecasting the future). In this context, the prescriptive analytics (which inherently involves optimization problems) must be better understood in how it fits into general analytic problems.
The full understanding of the optimization within the prescriptive analytics and its role in complex and variable-sensitive decision-making problems is the objective of this post. I will address the following topics:
- What are optimization and prescriptive analytics?
- Why is optimization important for solving a data science problem?
- Optimization in the core of data science
- When to bother with an optimization problem?
- Solvers and solutions providers
- Integration of AI problem with an optimization problem
Optimization or mathematical programming is the selection of the best element (according to some criteria) from a set of available alternatives. All the problems of this kind have the following components:
- Objective function
- Degrees of Freedom (Manipulable variables)
Some examples of daily optimization problems are:
- Optimal good and services acquisition
- Optimal expenditure on a limited budget
- Best route on commutes
It applies math programming models, to optimize a set of decisions for directing a given business situation to the best possible actions. The benefits of applying these techniques are:
- Exploring an astronomical number of possible combinations and find the proven best option.
- Applying a range of option constraints to maximize or minimize the criteria.
- Reducing decision-making risk.
- Freeing up time for higher-value efforts such as performing scenario analysis or considering larger strategic questions.
Some examples of prescriptive analytics applications are:
- Ride-sharing apps on dynamic pricing
- Airlines on ticket pricing revenue optimization management
- Cloud clients on cloud services optimization
- Ad-focused companies on ad revenue management
- Big retail companies on supply chain optimization
In the next chart, there is an exercise of Gurobi and end-to-end Analytics on when it is more likely to find an optimization problem in business: it should be a high degree of control over a certain quantifiable business problem.
Optimization in the core of Data Science
Optimization is on the everyday activities of a data scientist and an artificial intelligence professional. In statistical regression, they minimize the sum of the squared errors when fitting the model; on K-Means, they assign data points to clusters to minimize some aggregate distance metric; on Neural Networks, Stochastic Gradient Descent is used to minimize the average error over the training dataset.
The above are just examples of how optimization is not strange for the daily work of a data scientist.
When to bother with an optimization problem?
The key question that should be answered is: Is the cost of a single Sub-Optimal decision negligible or not. For instance, when building a book recommendation system, the fact that the customer does not receive the best songs possible but instead receives one or two “good” songs is enough, you definitely do not need to formulate an optimization problem. However, if you have a problem like efficient asset allocation and the customer received a good but not the best portfolio allocation, it could derive in huge less return on the assets taking more risk that the reasonably necessary.
Which are the best optimization platforms and solver providers for prescriptive analytics?
According to Gartner, the optimization platforms that are the drivers of the market are:
On the other hand, in terms of optimization solvers, the key players are:
- Arki Consulting and Development (Conopt)
- Fico (Xpress)
- Gurobi Optimizaion (Gurobi Optimizer)
- IBM (CPLEX Optimizers)
- SAS (SAS/OR)
- Ziena Optimization (Knitro)
Although there is too much effort from big companies to enhance the optimization solvers for their own problems, there is also an enthusiastic community on Python developers, eager to offer the best solver engines in open source terms. Some of the biggest initiatives on Python are:
There are also less important players (in market share), but still important within the industry:
From an AI problem to an optimization problem
Suppose, you have a problem where you need to generate the demand curve of a product, aka, extract the information of how the quantity sold affects the price of the product. The challenge is to use the historical data (which has multiple factors) to find the hidden patterns. A simple method could be used: Random Forest.
Quickly, there is another challenge, now you need to identify clusters of products having similar characteristics. For this new problem, you use K-Means.
Finally, the business asks you to maximize the revenue (and margin) of cluster subject to a min margin (revenue) constraint. Here the cumbersome is solving efficiently the problem so that Mixed Integer Linear Programming formulation and solver will be needed.
Other AI-problems that could be addressed are:
- Image classification and engagement prediction can be used to select the best picture to have the maximum engagement on social media.
- Text classification and tweet’s engagement prediction to get a recommendation of the best words combination for a tweet.
- Predictive maintenance on large scale companies.
- AI-driven logistics optimization reducing cost through real-time forecasting and behavioral coaching
- AI can be a valuable tool for customer service management and personalization challenges
- Combining customer demographic and past transaction data with social media monitoring can help generate individualized and customized product recommendations.
Future of optimization and prescriptive analytics
According to Gartner, the prescriptive analytics software market will reach $1.88 billion by 2020. In the next Hype Cycle chart, it can be seen that prescriptive analytics is exiting the peak of inflated expectations and it is expected to reach the plateau within 2 to 5 years.
Why prescriptive is important?
According to this paper Prescriptive Analytics Literature Review, this is the distribution of most recent papers (since 2014) based on application domains for prescriptive analytics methods. Some of the techniques applied to those are:
- Mathematical programming
- Evolutionary computation
- Logic-based models
- Machine Learning/Data Mining with logic-based models
- Probabilistic models
|Data Engineering/Information Systems||3|
In the next dashboard made by McKinsey Global Institute Analysis, you can see the estimated impact of some of these optimization techniques across the industries by type of problems that can be solved, type of data involved, and more. Notice there is a big correlation between what is being a hot research topic trend in recent years with the economic impact of those models applied to business.
According to Gartner, within the Top 10 Data and Analytics Technology Trends for 2019, it is continuous intelligence. This concept involves prescription analysis itself. It is about real-time analytics that is integrated into “business operations, processing current and historical data to prescribe actions in response to events” the article stated. This technology uses augmented analytics, event stream processing, optimization, machine learning, and business domain expertise. It could be seen as the ultimate in operational business intelligence.
- Jatherica Lepenioti, Alexandros Bousdekis, Dimitris Apostolou, Gregoris Mentzas. Prescriptive analytics: Literatura review and research challenges.