Machine Learning For Absolute Beginners: A Plain English Introduction (Second Edition) (Learn AI & Python for Beginners)

by Oliver Theobald

Another variant of multiple decision trees is the popular technique of boosting, which are a family of algorithms that convert “weak learners” to “strong learners.” The underlying principle of boosting is to add weights to iterations that were misclassified in earlier rounds.

A popular boosting algorithm is gradient boosting. Rather than selecting combinations of binary questions at random (like random forests), gradient boosting selects binary questions that improve prediction accuracy for each new tree. Decision trees are therefore grown sequentially, as each tree is created using information derived from the previous decision tree.

One drawback of using random forests and gradient boosting is that we return to a black-box technique and sacrifice the visual simplicity and ease of interpretation that comes with a single decision tree.

One of the most effective machine learning methodologies is ensemble modeling, also known as ensembles. Ensemble modeling combines statistical techniques to create a model that produces a unified prediction.

Ensemble models can also be generated using a single technique with numerous variations (known as a homogeneous ensemble) or through different techniques (known as a heterogeneous ensemble).

Four popular subcategories of ensemble modeling are bagging, boosting, a bucket of models, and stacking. Bagging, as we know, is short for “boosted aggregating” and is an example of a homogenous ensemble. This method draws upon randomly drawn datasets and combines predictions to design a unified model based on a voting process among the training data. Expressed in another way, bagging is a special process of model averaging. Random forest, as we know, is a popular example of bagging.

Boosting is a popular alternative technique that addresses error and data misclassified by the previous iteration to form a final model. Gradient boosting and AdaBoost are both popular examples of boosting.

A bucket of models trains numerous different algorithmic models using the same training data and then picks the one that perform...

Stacking runs multiple models simultaneously on the data and combines those results to produce a final model. This technique is currently very popular in machine learn...