Random Forests & Bagging Ensemble Methods

While a single decision tree suffers from high variance and instability, combining hundreds of decorrelated decision trees creates one of the most robust, accurate, and versatile algorithms in modern machine learning: the Random Forest.

The Principle of Ensemble Learning

Ensemble learning operates on the mathematical wisdom of the crowd: a group of diverse, moderately accurate individual models (weak learners) will collectively outperform any individual expert model, because individual errors cancel each other out when aggregated.

The Two Randomization Pillars of Random Forest

Invented by Leo Breiman in 2001, Random Forests combine two distinct sources of randomness to ensure individual trees are decorrelated:

1. Bootstrap Aggregation (Bagging)

Given a training set of N samples, each individual tree is trained on a distinct Bootstrap Sample generated by sampling N observations uniformly at random with replacement. On average, each bootstrap sample contains approximately 63.2% of the original data, leaving 36.8% unused (known as Out-of-Bag or OOB samples).

2. Random Feature Subspaces

If standard decision trees were trained on bootstrap samples, strong features would still be picked at the root of every tree, making the trees correlated. Random Forest solves this by randomly sampling a small subset of features (typically √p for classification, or p / 3 for regression) at each individual split node.

Out-of-Bag (OOB) Error: Free Cross-Validation

Because each tree never sees its ~36.8% Out-of-Bag samples during training, those OOB samples act as a built-in validation test set! By aggregating predictions across all trees where a given sample was left out, we compute the OOB Score, which accurately estimates generalization accuracy without requiring a separate validation split.

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import pandas as pd

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.25, random_state=42)

# Fit Random Forest with 150 trees and Out-of-Bag scoring enabled
rf = RandomForestClassifier(n_estimators=150, max_features='sqrt', oob_score=True, random_state=42)
rf.fit(X_train, y_train)

print(f"Out-of-Bag (OOB) Validation Score: {rf.oob_score_:.4f}")
print(f"Actual Test Set Accuracy:           {rf.score(X_test, y_test):.4f}")

# Extract MDI Feature Importance
feature_importance = pd.Series(rf.feature_importances_, index=iris.feature_names).sort_values(ascending=False)
print("
Feature Importance Rankings:
", feature_importance)