Model drift
Model drift indicates that model performance deteriorates over time. This is probably due to data drift i.e. the underlying data distribution becomes increasingly different from that of the training data.
More specifically, model drift could be due to:
- Changes in target variable, e.g. house prices becoming higher over time
- Changes in predictor variables, e.g. it becomes increasingly common to build houses with more bedrooms than the training data
- Changes in relation between predictors and target variable
Testing for model drift
"Raw" comparison
One could simply check if the new model score is worse than the previous score in absolute terms. This way you will probably detect model drift very often.
A less sensitive version could check if the new score is worse than all previous scores.
When doing ML in the wild, one expects some fluctuation in model performance. Both these approaches are probably too sensitive to these small changes.
So instead you want to use a statistical approach to check if there is a significant deterioration in model performance. You can do so using a parametric or non-parametric test.
Parametric test
If it is appropriate to make assumptions about the mean and standard deviation of the model scores, you can use parametric significance tests. Assuming a standard distribution, we may define model drift as scores that are more than two standard deviations away from the mean.
Pay attention, "worse" may be a higher or lower score than the mean, given the specific metric.
Non-parametric test
If we cannot make valid assumptions on the distribution of model scores, we can instead do an outlier test.
For example, you can define the interquartile range (IQR) over model scores, which contains the difference between the 75th and 25th percentile, i.e. q75% - q25%. The IQR is not sensitive to outliers and gives us a good estimation of the range where we expect the bulk of model scores to be.
Now we can define model drift as situations where new scores are more than 1.5xIQR above the 75th percentile or 1.5xIQR below the 25th percentile and the score is worse.
iqr = np.quantile(scores, 0.75) - np.quantile(scores, 0.25)
# For situation where higher is better, e.g. F1 scores
# we check whether the new score is an outlier "below" the 25th percentile
model_drift = new_score < np.quantile(scores, 0.25) - 1.5 * iqr
Comparing models using two sets of model scores
If you have two sets of model scores, you can instead do a t-test on both sets to see if one is significantly better than the other.
Sources
- https://rapidminer.com/blog/how-to-detect-drifting-models/