Benchmark Algorithms for Forecasting

Point forecasts

These benchmarks come from the M5 forecasting competition that uses hierarchical intermittent sales data.

W.r.t. hierarchical data:

  • "bottom-up" method for "reconciliation", meaning these methods predict the lowest level series: 1-10, 12, 15, 17-20
  • top-down approach, meaning they predict top-level hierarchy time series: 11, 13, 14, 16
    • uses "reconciliation (based on historical proportions, estimated for the last 28 days)" in the m5 competition

What does reconciliation precisely mean? In case of a bottom-up approach, the benchmarks below make predictions on the lowest level of aggregation in the hierarchy. Vice versa for top-down. The point is that using reconciliation, you can use either the higher or the lowest level time series, and compute the rest. In case of the M5 combination this is the difference between having 30.490 inputs instead of all 42.840 levels. Because the data is hierarchical, you can compute predictions on the other levels of aggregation ("reconciliation"):

This means that the submitted forecasts must follow this hierarchical concept and, as a result, be coherent (forecasts at the lower levels have to sum up to the ones of the higher levels). In other words, it is assumed that the forecasting approach used for forecasting all 42,840 series of the competition derived coherent forecasts and, therefore, the forecasts of all levels can be automatically computed by aggregating (summing) the ones of the lowest level of the hierarchy.

Levels of aggregation:

  • Single: 1-9
  • Multiple: 10
  • Incomplete

Temporal aggregation: 9, 10

Types:

  • Statistical 1-16
  • Machine learning 17-20
  • Combination measures: 21-24

TODO: if I zoom in one one type, link to a new note.

  1. Naive random walk model
  2. Seasonal Naive
  3. Simple Exponential Smoothing (SES) (Gardner Jr. 1985)
    • Initialize with first observation
    • Choose smoothing parameter in [ 0.1, 0.3] by minimizing the "insample mean squared" (MSE)
  4. Moving Averages (MA)
    • Average last k observations
    • k in [2,5], chosen by minimizing insample MSE.
  5. Croston's method (CRO) (1972)
    • First method that takes into account intermittency
    • Decomposes time series into non-zero demand size $z_t$ and interdemand intervals $p_t$. Prediction is their ratio z/p.
    • Rationale seems to be that larger predicted intervals scale down the predicted non-zero demand $z_t$.
    • Both $z_t$ and $p_t$ are predicted using SES with a fixed smoothing parameter of 0.1
  6. Optimized Croston's Method (optCro).
    • Optimize the smoother parameter.
    • As a result, now the smoothing parameters may be different for z and p
  7. Syntetos-Boylan Approximation (SBA)
    • Adds a debiasing factor (e.g. 0.95) to Croston's method
  8. Teunter-Syntetos-Babai method (TSB)
    • Similar to optCro, but replaces the interdemand interval with demand probability $d_t$ (1 meaning demand occurs at time t) which is forecasted using SES.
  9. Aggregate-Disaggregate Intermittend Demand Approach (ADIDA)
    • Use time-buckets the size of the mean inter-demand interval and aggregate demand within those time buckets to minimize the variance caused by zero entries.
  10. Intermittent Multiple Aggegration Prediction Algorithm (iMAPA)
    • Similar to ADIDA temporal aggregation, but considers multiple aggregation levels to capture different dynamics in the data.
    • "The maximum aggregation level is set equal to the maximum inter-demand interval.": TODO I do not understand what this means.
  11. Exponential Smoothing - Top-Down (ES_td)
    • Select best exponential smoothing method for top-level series based on information criteria
  12. Exponential Smoothing - Bottom-Up (ES_bu)
    • Idem, but bottom-up
  13. Exponential Smoothing with eXplanatory variables (ESX)
    • In addition to ES, use explanatory variables as regressors
    • E.g. for the m5 sales data, how many states allow SNAP purchases on a given day,or does the given date include a special event, e.g. a holiday?
  14. AutoRegressive Integrated Moving Average - Top-Down (ARIMA_td)
  15. AutoRegressive Integrated Moving Average - Bottom-Up (ARIMA_td)
  16. AutoRegressive Integrated Moving Average with eXplanatory variables (ARIMAX)
    • Cf. ESX
  17. Multi-Layer Perceptron (MLP)
    • Have an input node for each day, so 14 for two past weeks
    • E.g. double the amount of hidden nodes, so 28
    • Scaled Conjugate Gradient method
    • Random weight initialization, so train multiple MLPs and average them using a median operator to compensate for bad initializations.
  18. Random Forest (RF)
    • Average predictions of multiple regression trees
    • E.g. at each split consider 500 non-pruned trees and e.g. four randomly sampled variables at each split
    • Bootstrap sampling with replacement
    • Consider N past days.
  19. Global Multi-Layer Perceptron (GMLP)
    • Instead of training one MLP per series, construct one bigger model to learn across all series.
    • Use last n observations of each series as inputs + information about:
    • 1) "the coefficient of variation of non-zero demands (CV^2)" and
    • 2) "the average number of time-periods between two successive non-zero demands (ADI)"
    • This may enable "cross learning" across series with different characteristics
    • TODO how exactly?
  20. Global Random Forest
    • Cf. GMLP but use a big random forest instead

Suggested combinations:

  1. Average ES and ARIMA, bottom up
  2. Average ES and ARIMA, top down
  3. Average ES bottom up and top down
  4. Average global and local MLPs