Demand and sales forecasting are of paramount importance in retail. Without this tool, companies encounter disruption of the inventory balance, through ordering too much or not enough products for a certain period of time. In the case of surplus, a company is forced to offer discounts to sell products. Otherwise, it may face inventory issues. A shortage, in turn, results in lost profits. However, these problems can be solved by applying demand and sales forecasting to increase the return on inventory and determine the intention of future consumers to buy a specific product at a specific price.

Let’s go through the process of implementing ML forecasting in retail together, discovering the key steps of building forecasting software.

Benefits of ML Demand Forecasting for Business

Looming uncertainty and changes in the market lead to highly volatile data. Unlike traditional methods, demand forecasting using machine learning is more flexible and allows the quick infusion of new information into models. That’s why ML models are adaptive and accurate enough to bring obvious benefits to the business:

• Increase in sales. All needed products will be available in the store, so customers can purchase them without waiting for long delivery times.
• Customer satisfaction maintenance. Warehouses will plan purchases in advance, so customers won’t face the problem of their favourite product’s absence.
• Higher inventory turnover. Thanks to proper planning of goods in warehouses, poorly selling goods will not be stale.
• Reduction in the number of spoilage products. Demand forecasting will help to competently plan the delivery of products, considering expiration dates.
• Reduced personnel costs. By analyzing and predicting future demand, we can plan an optimal number of employees for proper shift support.

ML demand forecasting methods, like other use cases of machine learning forecasting, can rely on a tremendous amount of data to make accurate predictions. However, the question of how to develop such models remains open, and we will consider it in the following section.

How to Develop an ML-Based Demand Forecasting Software

Before embarking on demand forecasting model development, you should understand the workflow of ML modelling. This offers a data-driven roadmap of how to optimize cooperation with software developers. Let’s review the process of how AI engineers at MobiDev approach ML demand forecasting tasks.

STEP 1. BRIEF DATA REVIEW

The first step when initiating the demand forecasting project is to provide the client with meaningful insights. The process includes the following steps:

1. Gather available data
2. Briefly review the data structure, accuracy, and consistency
3. Run a few data tests and pilots
4. Look through a statistical summary

In our experience, a few days is enough to understand the current situation and outline possible solutions.

STEP 2. SETTING BUSINESS GOALS AND SUCCESS METRICS

Each project is unique and has its own business goals. Therefore, this stage is key in creating an effective forecasting solution since it provides the starting point of the development process and outlines the following stages.

Before coming to the stage of developing a demand forecasting solution, a software development team needs to agree with the client/business owner on the success metrics for the model’s results evaluation. Success metrics offer a clear definition of what is “valuable” within demand forecasting. A typical message might state:

“I need a machine learning solution that predicts demand for […] products, for the next [week/month/a half-a-year/year], with […]% accuracy.”

This statement example will help you to identify what your success metrics will look like. You are expected to consider the following information:

• Product Types / Categories What types of products/product categories will you forecast?

Different products/services should be considered and predicted independently for most cases. For example, the demand forecast for perishable products and subscription services coming at the same time each month will likely be different.

• Time Frame What is the length of time for the demand forecast?

Short-term forecasts are commonly done for less than 12 months – 1 week/1 month/6 months.

These forecasts may have the following purposes:

• Uninterrupted supply of products/services
• Sales target setting and evaluating sales performance
• Optimization of prices according to market fluctuations and inflation

Long-term forecasts are completed for periods longer than a year. The main purposes of long-term forecasts may include the following:

• Long-term financial planning and funds acquisition
• Decision-making regarding the expansion of business
• Annual strategic planning
• Accuracy

What is the minimum expected percentage of demand forecast accuracy for making informed decisions?

Implementing retail software development projects, we were able to reach an average accuracy level of 95.96% for positions with enough data. The minimum required forecast accuracy level is set depending on your business goals.

Examples of metrics to measure the forecast accuracy are MAPE (Mean Absolute Percentage Error), MAE (Mean Absolute Error), or custom metrics.

STEP 3. DATA UNDERSTANDING & PREPARATION

Regardless of what we’d like to predict, data quality is a critical component of an accurate demand forecast. The following data could be used for building forecasting models:

When building a forecasting model, the data is evaluated according to the following parameters:

• Consistency
• Accuracy
• Validity
• Relevance
• Accessibility
• Completeness
• Detalization

In reality, the data collected by companies often isn’t ideal. It usually needs to be cleaned, analyzed for gaps and anomalies, checked for relevance, and restored. That’s why data science consultants can be involved at this stage.

Data understanding is the next task once preparation and structuring are completed. It’s not modelling yet but an excellent way to understand data by visualization. Below you can see how we visualized the data understanding process:

This visualization demonstrates data decomposition, extracting trends, and seasonal or other factors from input data. It’s divided into several graphs:

• The 1st graph is an original timeline (time series visualization)​_
• The 2nd, 3rd, and 4th graphs separately represent seasonality, trends, and noise for further analysis and forecasting

STEP 4. MACHINE LEARNING MODELS DEVELOPMENT

There are no “one-size-fits-all” forecasting algorithms. Often, demand forecasting features consist of several machine learning approaches. The choice of machine learning models depends on several factors, such as business goal, data type, data amount and quality, forecasting period, etc.

Here you’ll find those machine learning approaches when applied to our retail clients. These approaches can also be used for most demand forecasting cases:

• ARIMA/SARIMA
• Regression models
• XGBoost
• K-Nearest Neighbours Regression
• Random Forest
• Long Short-Term Memory (LSTM)

Below we would like to describe in more detail 3 ML approaches for working with time series data for real demand forecasting projects.

TIME SERIES APPROACH

Time series is a sequence of data points taken at successive, equally-spaced points in time. The major components to analyze include trends, seasonality, irregularity, and cyclicity.

In the retail field, the most applicable time series models are the following:

1. ARIMA (auto-regressive integrated moving average) models aim to describe the auto-correlations in the time series data. When planning short-term forecasts, ARIMA can make accurate predictions.
2. SARIMA (Seasonal Autoregressive Integrated Moving Average) models are the extension of the ARIMA model that supports uni-variate time series data involving backshifts of the seasonal period.
3. Exponential Smoothing models generate forecasts by using weighted averages of past observations to predict new values. The essence of these models is in combining Error, Trend, and Seasonal components into a smooth calculation.

You can also have an understanding from the visualization below of what prediction results usually look like when talking about working with time series prediction approaches.

Let’s say you want to forecast demand for vegetables in the next month. For a time series approach, you require historical sale transaction data for at least the previous six months. If you have historical data about seasonal products – vegetables in our case – the best choice will be the SARIMA model. The forecast error, in that case, can be around 10-15%.

REGRESSION MODELS

A regression model is a statistical model that estimates the relationship between one dependent variable and one or more independent variables using a line (or a plane in the case of two or more independent variables).

Regression models are also helpful in predicting future values from past ones. They can help determine underlying trends and deal with cases involving overstated prices.

While quite rare in real-life business cases, we can see a linear correlation between the target feature that needs to be predicted and the rest of the available variables. Because of this, it is important to select the proper regression model based on the custom client’s data.

A visual example of the different regression models is provided below:

Let’s say you want to calculate the demand for tomatoes based on their cost. Assuming that tomatoes grow in the summer and the price is lower because of the high tomato quantity, the demand indicator will increase by July and decrease by December.

The information required for such type of forecasting is historical transaction data, additional information about specific products (tomatoes in our case), discounts, average market cost, the amount in stock, etc. The forecast error can be 5-15%.

RANDOM FOREST

Random Forest is a well-known ensemble machine learning algorithm, done by constructing a multitude of decision trees at training time and outputting the mean/average prediction (regression) of the individual trees.

It can be used for both Classification and Regression problems in ML. However, it can also be used in time series forecasting, both univariate and multivariate datasets by creating lag variables and seasonal component variables manually.

Random Forest is the more advanced approach that takes multiple decision trees and merges them together. By taking an average of all individual decision tree estimates, the random forest model results in more reliable forecasts.

However, despite its versatility, Random Forest has some limitations. The model may be too slow for real-time predictions when analyzing a large number of trees.

If you have no information other than the quantity data about product sales, this method may not be as valuable. In such cases, the time series approach is superior.

STEP 5. TRAINING & DEPLOYMENT

Training

When training forecasting models, data scientists usually use historical data. By processing this data, algorithms provide ready-to-use trained model(s).

Validation

This step requires the optimization of the forecasting model parameters to achieve high performance. By using a cross-validation tuning method where the training dataset is split into several equal parts, data scientists train forecasting models with different sets of hyper-parameters. The goal of this step is to figure out which model’s parameters have the most accurate forecast.

Improvement

When researching the best business solutions, data scientists usually develop several machine learning models and then choose the ones that cover the project’s requirements the best. The improvement step involves the optimization of analytic results. For example, using model ensemble techniques, it’s possible to reach a more accurate forecast. In that case, the accuracy is calculated by combining the results of multiple forecasting models.

Deployment

This stage assumes the forecasting model(s) integration into production use. We also recommend setting a pipeline to aggregate new data to use for your next AI features. This can save you a lot of data preparation work in future projects. Doing this also increases the accuracy and variety of what you could be able to forecast.

Key Factors Affecting Demand Forecasting

Demand forecast tasks depend on a lot of obvious and non-obvious factors. Here are the ones with the most impact.

PRODUCT TYPES AND MODELlING ERRORS

The product type is an important factor to consider for the demand model. For example, for a perishable item that has an actual demand of 100 cases, the prediction of selling 90 cases is preferred over the prediction of 110 cases. Missing the sales of 10 cases is a better result than wasting 10 cases, even though the actual error is the same percentage.

REGIONAL IMPACTS ON MODEL PERFORMANCE

Predictive models are strongly influenced by regional factors that include customer behaviour and cultural determinants. They also include the following:

• Marketing campaigns may be regionally specific and have a different impact that depends on where a customer is located.
• Holidays may vary between regions, which might be a consideration for adjusting the model.
• Legal issues/laws may limit the use of certain data in different regions.

NEW COMPETITORS ON THE MARKET

Demand forecasting is a dynamic concept. The more competitors and product alternatives are present in the market, the harder demand forecasting becomes. The competition level contains sub-factors, such as the number of alternative products and competitors.

So, it is a very good idea to add this information dynamically to your demand forecasting model.

ECONOMIC SITUATION

The state of the economy influences businesses and demand forecasting models. To put it more bluntly: periods of economic decline are likely to cause lower demand for expensive products, though sales of low-priced goods may go up. Therefore, an economic situation as well as trends aren’t external factors and should be considered when building AI models.

Sales Forecasting For Retail During Uncertainty

When integrating demand forecasting systems, it’s essential to understand that they are vulnerable to anomalies or unpredictable situations. It means that machine learning models should be upgraded according to current reality.

As the demand forecasting model processes historical data, it can’t know that the demand has radically changed. For example, if last year, we had one demand indicator for medical face masks and antiviral drugs, this year, it would be completely different.

In that case, there might be several ways to get an accurate forecast. Here are the six most common ways:

1. Collect data about new market behaviour. Once the situation becomes more or less stable, develop a demand forecasting model from scratch.
2. Apply a feature engineering approach. By processing external data, news, a current market state, price index, exchange rates, and other economic factors, machine learning models are capable of making more up-to-date forecasts.
3. Upload the most recent data and provide it with the highest weights during model prediction. The period of a loadable dataset might vary from one to two months, depending on the products’ category. In this way, we can detect shifts in demand patterns and enhance forecast accuracy in a timely manner.
4. Apply the transfer learning approach. If there is any gathered historical data, we can use it to predict demand in the context of the current crisis.
5. Apply the information cascade modelling approach. We can forecast how people will make buying decisions according to the behaviour patterns of most people.
6. Apply the natural language processing (NLP) approach. NLP technology enables the processing of real comments from social networks, media platforms, and other available social sources. By utillizing text mining and sentiment analysis approaches, NLP models gather samples of customers’ conversations to detect people’s preferences, choices, sentiments, and behaviour shifts.

During AI app development, AI engineers analyze historical data for forecasting. This forecasting cannot predict the disruption caused by a global pandemic. Such an event requires the recalibration of the machine learning models. We met this challenge using machine learning models developed for a restaurant business prior to the pandemic.

But keep in mind that after the demand situation normalizes after the pandemic/war/ etc – you need to adjust your model back, since in other cases – the model can remember the pandemic’s pattern and predict it for the next short-time period (e.g. next year).