Gradient Boosting
Gradient boosting is an main ensemble technique (strictly speaking, it's an instance of the boosting approach to ensemble, but I currently do not know other approaches). Contrasted with bagging where you average different independent models.
General idea:
- Train a model on your data
- Train a second model that focuses on the features/cases where the first model's performance was poor, by using the previous model's error residuals as target variable.
- New prediction is made by running sequentially through the previous models.
- Rinse and repeat.
- Have some stopping condition; when stopped, weigh each individual predictor.
Gradient Boosting commonly works with decision trees, or regression models.
This is a great blog post, with the following nice quote:
This technique employs the logic in which the subsequent predictors learn from the mistakes of the previous predictors. Therefore, the observations have an unequal probability of appearing in subsequent models and ones with the highest error appear most. (So the observations are not chosen based on the bootstrap process, but based on the error)
In bagging techniques such as Random Forest models you build independent models by training each model on a random subset of the data. Conversely, with gradient boosting each next iteration prioritizes certain observations already trained upon by the previous model.
By focusing on high-error cases, you converge faster to a decent prediction. This is why this particular approach is called gradient boosting: the error gradient from the previous model is an indication of which cases performed poorly; or alternatively, which features will likely give the greatest increase in performance when you focus on them.
How you achieve this is by training on the error residuals of the previous network. Meaning: the second model will have the residuals (y_target - y_predicted) of the first model as its target variable. In case the previous model was spot on for a given case, the residual will be zero and this case will therefore not contribute to the trend of a regression line or the split of a decision tree. Conversely, cases with high residuals will strongly influence that model.
The main danger of this sequential approach is overfitting.
For an in-depth review, see How to explain gradient boosting by explained.ai .
Libraries
There are three major gradient boosting libraries, XGBoost, LightBM, CatBoost.
XGBoost has been around the longest and is slow compared to the other two. I do not see many people use it in this competition.
Advantage of LightBM and CatBoost is that they also handle categorical variables, which is handy for the explanatory variables in addition to the time series.
LightBM's main advantage is that it's fast.
CatBoost is also fast, but seems to be less prone to overfitting.
An additional advantage is that the CatBoost library offers functionality to asses feature importance.