Bagging (ML Ensemble technique)
One main ensemble technique used in machine learning; the other major approach is gradient boosting .
Random Forest models are a good example of this approach.
This is a great blogpost.
The bagging approach to ensemble learning is to train different independent models and to make a final prediction by averaging all predictions according to some scheme.
Possible averaging schemes:
- Weighted average
- Majority vote
- Normal average
The assumption here is that the models are independent (N.B. this is a main difference with the boosting approach). One way to stimulate independence is to train the model on random samples of the data.
The less correlated the individual models are, the better their chance of dealing with variance in the data.