There’s a lot of talk about the role of AI in inventory. But with so many unknowns about how these tools are developed, how do you know whether your forecasting algorithm is really doing its job?
Here are three key considerations to take into account when measuring the accuracy of your AI-based forecasting. With clear guardrails and repeatable processes, you can make sure the AI in your forecasting is working like it should be.
1. Choose the right methodology
Measuring forecast accuracy is an art and science in its own right — book after book has been published on the topic.
Most inventory planning systems go the easy route by using simple regression to measure forecast accuracy. Using this standard process, you would run your algorithm as if today were, for example, 30 days ago. Then you would project the next 30 days and measure that against what actually happened during that period. If you projected to sell 1,200 units and you actually sold 1,080 units, your accuracy would be 90% (120 units off over 1,200 units projected).
By forecasting as if you were in a past state, you can see what the future forecast would be for a product, and match that against the actual sale. It’s a simple process and it works. But it’s far from the only one.
From Mean Absolute Percentage Error (MAPE) to Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE), there are hundreds of formulas you can use to measure the accuracy of your forecasts.
The important thing is to choose a method that fits the different forecasting algorithms you use.
Keep in mind, different products can have very different behaviors, even when they’re in the same category. Here are a few examples:
- Products with stable sales - You can use basically any method and timeframe to measure accuracy, as every period has the same behavior.
- Products with seasonal sales - This is the trickiest type because you might get very different results depending on the timeframe and method.
- Products with intermittent demand - With little sales, a day when you sold 3 items will be 3x bigger than a day when you sold 1 item, even though the absolute difference is only 2 items. This can make your accuracy metrics very volatile.
Whatever your product portfolio looks like, make sure your forecast accuracy formulas align with your top-performing algorithms on a given product.
2. Measure against the right time period
The time period you use to measure the accuracy of your forecasting algorithms matters more than you might think.
As mentioned, most retailers measure the accuracy of their forecasts for a pre-defined period (e.g. last 30 days, or last year of sales). But the more correct way to do it, is to measure for the period when you'll actually sell your product.
Let’s take a look at an example:
Say you're purchasing a product today that takes 90 days to arrive, and that the product will have 15 days of stock when the next batch arrives. The actual start of consumption of the next batch will be 105 days from today (90 days of lead time + 15 days of safety stock at arrival). In this example, let's say you're purchasing 30 extra days of inventory. So, the consumption period will be from day 105 to day 135 from today — this is the period that you should measure the accuracy of your forecast against.
Considering that different products have different lead times, and that each purchase or transfer has its own characteristics, in the ideal world you would have an adaptive forecast accuracy process that would give you the accuracy by product for that specific purchase or transfer.
Unfortunately, the correct time period isn’t the only complexity to account for when measuring forecast accuracy. On top of that, you also have different movements for different products. Here’s another example:
Let’s say you have a purchase that went straight from your supplier to a warehouse, and was then transferred from your warehouse to a fulfillment center. In this case, the accuracy formula you used to purchase the product from the supplier is different from the accuracy formula you’d need to use to ship the product to another location — e.g., 7-15 day lead time vs. 90-100 days out in your forecast.
It’s a lot to keep track of. Fortunately, AI-based automated systems can account for those intricacies for you, without the manual work that would normally eat up hours of your time.
With the right AI forecasting tools and the right parametrization, you can use one algorithm for when you're going to purchase the product and another for when you need to transfer the product. By having a second accuracy measurement in place, you’ll be able to gauge the real performance of your forecasting algorithms.
3. Account for bias
Bias is a part of every data analysis process, with or without AI. The key is to understand how to identify and adjust for it to avoid stockouts and overstocks.
As in the above example, you might find that your forecasting algorithms have a tendency to over- or under-forecast, creating a cycle where you’re consistently over- or understocked. Even a high accuracy forecast can include this type of bias.
The good news is, once you’ve pinpointed the bias, correcting it is relatively straightforward — simply adjust the forecast by the appropriate amount in the appropriate direction. In the case of an under-forecast bias, you would increase the forecast and decrease it in the case of over-forecast bias.
Again, it’s a simple method that gets the job done. But with the ability to use machine learning to pick up on the deeper nuances and movements driving each of your products, it is possible to train your AI to learn from any biases and account for them automatically in future forecasting.
The result could be countless hours saved in manual data analysis and a significant boost to your margins by way of increased sales and reduced holding costs.
Measure the real performance of your forecasting algorithms
If you're like most retailers using a 30-day moving average or your last year of sales for measuring your forecast accuracy, it’s easy to compare one versus the other. But the bigger question isn’t which one is better, it’s: ‘What am I missing?’
Past stockouts, viral campaigns, seasonality. Increased complexity means increased accuracy. And handling vast amounts of complex data is a challenge that’s uniquely fit for AI. But the truth is, it’s still kind of a black box.
With little insight into how your AI was developed, it’s critical to make sure you're measuring forecast accuracy correctly so you can understand the true strength of your forecasting algorithms, and adjust where needed. These three simple considerations will help you improve your forecast accuracy and prepare for a future where AI plays an even bigger role in inventory.
