What Is the “Boosting” Ensemble Method?
This series(“Bagging & Boosting Ensemble Methods and What is the Difference Between Them?”) consists of 6 separate articles and is the fourth article in this series. In this part, we will talk about What Is the “Boosting” Ensemble Method?
I have written about the bagging issue in my previous article. On top of that, we can see the difference between Boosting and Boosting above, but let’s try to see the system running in the background and finally share the pros and cons.
All the individual models formed by the boosting method in sequentially. This means that the result of the first model is passed on to the next model.
In the bagging method, the models are built at the same time, and therefore we do not know the error of each model. However, once the first model is created in the boosting method, we know the error of this model. So when we transfer this first model to the next model, our general intention is to further reduce the error. In some of the boosting algorithms, each model must have a minimum error reduction of 50%.
In contrast bagging, not all observations in the bootstrapping example are treated equally when it comes to boosting. Observations will have weight, and for a few observations, the weight will be lower for others.
Suppose we have created a binary classification model. First model does not predict the target correctly(accurately), and then the input for the next model comes in order and focuses more on predicting the target.
Some observations will have more weight when selecting the data samples from the boot sample. In this case, the data points will have a higher weight than the other data point and the destination may help to predict accurately.
In the final target, estimates from all models will be weighted. Therefore, the weighted average will be our final estimate.
What are the pros and cons shortly:
Boosting, all individual models will occur one after the other. The output of each model, along with the next model boot sample data, is transferred as input to the next model.
Boosting Algorithms
The following algorithm names are examples of the Boosting method. You can find many examples of these on my Kaggle account.
- AdaBoost
- XGBoost
- LightGBM
- CatBoost
- LPBoost
- GradientBoost
- BrownBoost
Let me put a stop to our topic here and say we will see you in our next topic, What is the Difference Between Bagging & Boosting in Tree-Based Methods?.
References
1. https://www.pluralsight.com/guides/ensemble-methods:-bagging-versus-boosting
2. https://www.researchgate.net/figure/Schematical-representation-of-gradient-boosting-regression-in-regards-to-algorithm_fig3_340524896
3. https://dataaspirant.com/ensemble-methods-bagging-vs-boosting-difference/#t-1599488265671
4. https://www.kaggle.com/mathchi/notebooks
5. http://www.plusxp.com/2011/02/back-to-the-future-the-game-episode-1-review/
