Machine learning · forecasting and inventory optimisation · grocery retail
Demand Behind the Empty Shelf
When a fresh product sells out, sales stop telling the truth. We estimate the demand behind the empty shelf, with an interval, and turn it into tomorrow's order.
Overview
FreshRetailNet-50K is a public dataset of fresh-food sales: 50,000 product series from 898 stores, with hourly sales and an hourly flag for whether the product was in stock. In the data, 19.7 % of selling hours are out of stock, and 44 % of product-days run out at least once. During those hours the data records almost no sales. A forecast trained on recorded sales therefore treats an empty shelf as a quiet hour, and its forecasts run low.
We estimate the demand hidden by each stock-out, as a number with an 80 % interval. The model only learns from hours whose answer we know. We take a day that stayed in stock, copy onto it the stock-out pattern of a real sold-out day, hide those hours, and learn to recover them. It uses the rest of that day, the product's own history, the discount, the calendar, the weather and the category. On 167 stores it never saw, the interval held the true value 80.0 % of the time. Our first version held it only 71 % of the time on days that really sold out, because those days are busier than their open hours suggest. Training on the in-stock hours of sold-out days brought that test back to 79.5 %. Both tests hide hours whose sales we did observe. Demand after a sell-out can never be observed, so its interval rests on one assumption: that it behaves like the hours we can test. Across the dataset, our central estimate adds 17 % to recorded sales.
Next, probabilistic forecasts turn the recovered demand into orders. We tested them on the dataset's held-out week. A replenishment policy then orders up to a forecast quantile each evening, minus the stock that can still be sold. Stock ages first in, first out, and its shelf life is an assumption of 1, 2 or 3 days that you set. A simulated replay of the held-out week shows the waste and the lost sales next to ordering on recorded sales. In that replay, a lost sale costs twice a wasted unit, and demand includes our estimate. Ordering on our estimate then cut the combined cost of waste and lost sales by 4.6 %, 5.6 % and 10.9 % for a shelf life of 1, 2 and 3 days. If the stock-outs had hidden no demand at all, the two policies would land within 1.3 % of each other. The gain also depends on lost sales costing more than waste. If a lost sale costs the same as a wasted unit or less, and the stock-outs hid no demand, ordering on our estimate costs 1 to 5 % more. At a lost-sale cost of 1, the 3-day figure shrinks from -10.9 % to -0.2 %. At a 2- or 3-day shelf life, both policies order at the top of our forecast range (P95), so those two figures are the least certain. Sales are in the dataset's own scaled units. Every recovered number is our estimate, never an observation. The shelf in the film is a 3D render that replays one product's day from the data.
What we built
- Recovery of censored demand. LightGBM quantile models at P10 to P90, trained by hiding hours that were in stock. They are calibrated with split-conformal checks on separate stores and validated on 167 held-out stores. One of those checks uses only real sold-out days.
- Probabilistic forecasts for the held-out week, made one day ahead. They are trained on recorded sales and on recovered demand, and scored with CRPS, pinball loss, WAPE and bias against simple baselines.
- An order-up-to policy that tracks ageing. It sells first in, first out over an assumed shelf life of 1 to 3 days. The order quantile is tuned on two backtest weeks for each cost ratio, then frozen for the held-out week.
- A simulated replay that starts from an empty shelf. It brackets the result twice: once with demand that includes our estimate, and once with recorded sales only.
- An interactive replay of 2,000 series, all from held-out stores. In your browser you can pick a store and product, show recorded or estimated demand, move the cost of a lost sale against a wasted unit, and change the shelf life. The policy is re-tuned and replayed live.
Tech stack
Python, pandas, NumPy, LightGBM (quantile regression), split-conformal calibration · three.js (physically based rendering, accumulated depth of field and soft shadows) · vanilla JavaScript for the in-browser optimiser
Numbers
All numbers come from our own runs on FreshRetailNet-50K (Dingdong-Inc, CC BY 4.0). Sales are in the dataset's scaled units.
| Number | How we measured it |
|---|---|
| 19.7 % of selling hours out of stock | Hourly stock flags, 06:00–22:00, across all 50,000 series and 97 days. In 44.0 % of product-days the product ran out at least once. In 4.0 %, it was out all day. |
| 80.0 % interval coverage (target 80 %) | 444,393 product-days from 167 held-out stores. A real stock-out pattern was copied onto in-stock hours, those hours were hidden, and we checked whether the true total fell inside our P10–P90 interval. |
| 79.5 % coverage on real sold-out days | 222,549 held-out days that ran out at 12:00 or later. We hid the 3 in-stock hours that end 2 hours before the stock-out. Without training on sold-out days, coverage was 71.4 % and the estimate was 39 % low. |
| 40 % error on hidden hours (WAPE of the P50) | Same 444,393 days. The same hours on recent in-stock days give 49 % error. Scaling the open hours by the usual hourly pattern gives 60 %. |
| +17 % demand hidden by stock-outs | Sum of our P50 estimates across all 2.1 million product-days with a stock-out, compared with total recorded sales. A sum of medians understates the total. Summing the per-day P10 and P90 parts instead gives 8 % and 30 %. |
| 2.4 % lower CRPS on the held-out week | Forecast trained on recovered demand vs the same model trained on recorded sales: 0.192 vs 0.197 (scaled CRPS, lower is better). Scored on the 206,562 held-out product-days that stayed in stock, where recorded sales equal demand. A 28-day empirical-quantile baseline scores 0.240. |
| 42 % vs 56 % too-low forecasts on sold-out days | Share of held-out sold-out days where the forecast median is below the sales actually recorded, which is a floor on that day's demand. Recovered-demand forecast vs recorded-sales forecast. This check rewards forecasting high, so read it with the bias on in-stock days: +3.1 % vs -6.4 %. |
| -4.6 % / -5.6 % / -10.9 % replay cost at a shelf life of 1 / 2 / 3 days | Cost = (2 × lost sales + wasted units) per unit of demand, over the held-out week for all 50,000 series, starting from an empty shelf. Replay demand = recorded sales plus our P50 estimate for the stock-out hours. We compare ordering on our estimate with ordering on recorded sales. Each policy's order quantile was tuned on the two weeks before. Drawing the hidden demand from our interval instead (200 draws) gives -4.0 % / -7.2 % / -12.6 %. |
| Within 1.3 % if nothing was hidden | The same replay with recorded sales as the only demand: +1.3 % / -0.3 % / +0.8 %. |
| -5 % to +5 % at a lost-sale cost of 0.5 to 1 wasted unit | Same replay, other cost ratios. With our estimate as demand: -5.5 % to -0.2 %. With recorded sales only: +0.8 % to +5.4 %, so ordering on our estimate costs more when nothing was hidden and lost sales are cheap. At shelf lives of 2 and 3 days, both tuned policies sit at the P95 ceiling of our forecast range. |
| 20.4 % vs 23.0 % waste at a 1-day shelf life | Replay as above. Ordering on our estimate wastes 20.4 % of demand and loses 7.2 %. Ordering on recorded sales wastes 23.0 % and loses 6.7 %. Ordering yesterday's sales times a tuned factor costs 0.600 per unit of demand, against 0.347 for ours. |
| 42 ms to re-tune and replay in the browser | Median of 10 runs in headless Chromium on our M1 Pro laptop. Each run tunes 37 order quantiles × 2 policies on 14 backtest days, then replays the 7-day week, across 2,000 series. Phones will be slower. |
Stills
Try it live
An interactive view that runs in your browser. Nothing loads until you open it.
