Root Mean Squared Scaled Error (RMSSE)

An @evaluation_metric for forecasting time series.

Used in the M5 forecasting competition.

This is a variant of the Mean Absolute Scaled Error (MASE) of Hyndman and Koehler (2006) that relies on squared errors instead of absolute ones. The main reason for this is dealing with intermittent data.

  1. Measures based on squared errors are more optimized for the mean than absolute errors, that are optimized for the median. If we have intermittent data (many zeros in the timeseries), then the problem with absolute errors would be that they assign lower errors to forecasts that are pessimistic and closer to zero. If instead the goal is to forecast an average value, e.g. average sales over some period, then departing from squared error is more desirable.
  2. Unlike absolute errors, RMSSE is scale-independent, so it also works for time series on different scales.
  3. RMSSE does not rely on division of equal- or close to zero values.
  4. The measure is symmetric: equally penalizes positive and negative forecast errors (because squaring) and large and small forecasts (because scale-independent, right?)

Formula for the regular Root Mean Squared Error for set differences:

$$ rmse = \sqrt{(\frac{1}{n})\sum_{i=1}^{n}(y_{i} - x_{i})^{2}} $$

And RMSSE for time series:

$$ rmsse = \sqrt{\frac{1}{h}\frac{\sum_{t=n+1}^{n+h}(y_{t}-\hat{y}{t})^{2}}{\frac{1}{n-1}\sum{t=2}^{n}(y_{t}-y_{t-1})^{2}}} $$