Skip to content
W3S101

Random Forest

Classification Algorithms - Random Forest

Section titled “Classification Algorithms - Random Forest”

Introduction to Random Forest

Section titled “Introduction to Random Forest”

Random Forest is a powerful and widely used ensemble learning method based on decision trees. It can be used for both classification and regression tasks, but it’s particularly popular for classification.

The core idea is simple: build a multitude (‘forest’) of decision trees during training and output the class that is the mode (most frequent) of the classes output by individual trees (for classification) or the mean prediction of the individual trees (for regression).

It leverages two key techniques to create diverse trees and improve upon the performance of single decision trees:

  • Bagging (Bootstrap Aggregating): Each tree in the forest is trained on a different random subset of the training data, sampled with replacement (bootstrap sample).
  • Random Feature Subsets: At each node split during the construction of a tree, only a random subset of the features is considered for finding the best split (instead of all features).

These two sources of randomness help to decorrelate the trees in the forest. While individual trees might be prone to overfitting, averaging their predictions reduces variance and leads to a more robust and accurate model that generalizes better to unseen data.

How Random Forest Works

Section titled “How Random Forest Works”

The algorithm for building a Random Forest classifier generally involves these steps:

  1. Specify Number of Trees (n_estimators): Decide how many decision trees to build in the forest (e.g., 100, 500).
  2. For each tree:
  3. a. Create Bootstrap Sample: Select a random subset of the original training data by sampling with replacement. This sample will typically be the same size as the original dataset but will contain duplicate instances and omit others.
  4. b. Build Decision Tree: Grow a decision tree on this bootstrap sample.
  5.   - **Random Feature Selection:** At each node, randomly select a subset of features (controlled by `max_features` hyperparameter).
  6.   - **Find Best Split:** Find the best split point among the *selected* features using a standard criterion (like Gini impurity or entropy).
  7.   - **Grow Fully (Typically):** Grow the tree to its maximum possible size without pruning (though hyperparameters like `max_depth` or `min_samples_leaf` can be used to control tree complexity).
  8. Prediction: To classify a new data point:
  9. a. Run the data point down each tree in the forest to get a prediction from each tree.
  10. b. Aggregate Predictions: Perform a majority vote: the class predicted by the highest number of trees is the final prediction of the Random Forest.

Conceptually: Imagine asking many slightly different experts (the individual trees, trained on different data subsets and considering different factors) for their opinion, and then going with the most popular opinion.

Implementation in Python (Scikit-learn)

Section titled “Implementation in Python (Scikit-learn)”

We’ll use Scikit-learn’s RandomForestClassifier on the Iris dataset.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score


# 1. Load Iris dataset
iris = load_iris()
X = iris.data
y = iris.target
feature_names = iris.feature_names
class_names = iris.target_names


# 2. Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)


# 3. Create Random Forest Classifier object
# Key Hyperparameters:
# n_estimators: Number of trees in the forest
# max_features: Number of features to consider at each split ('sqrt' or 'log2' are common defaults)
# max_depth: Maximum depth of individual trees (controls complexity)
# min_samples_split: Minimum samples required to split an internal node
# min_samples_leaf: Minimum samples required to be at a leaf node
# random_state: For reproducibility
# n_jobs: Number of CPU cores to use (-1 uses all available)


rf_clf = RandomForestClassifier(n_estimators=100, # Build 100 trees
                                random_state=42,
                                n_jobs=-1, # Use all cores
                                oob_score=True) # Enable Out-of-Bag score estimation


# 4. Train the Random Forest Classifier
rf_clf.fit(X_train, y_train)


# 5. Out-of-Bag (OOB) Score (Optional but useful)
# OOB score is an estimate of generalization accuracy using trees that didn't see a particular sample
# during their bootstrap sampling. It's like a built-in cross-validation.
print(f"Out-of-Bag (OOB) Score Estimate: {rf_clf.oob_score_:.4f}\n")


# 6. Predict the response for test dataset
y_pred = rf_clf.predict(X_test)


# 7. Evaluate Model
print("--- Model Evaluation on Test Set ---")
conf_mat = confusion_matrix(y_test, y_pred)
acc = accuracy_score(y_test, y_pred)
class_rep = classification_report(y_test, y_pred, target_names=class_names)


print(f"Test Accuracy: {acc:.4f}")
print("\nConfusion Matrix:")
plt.figure(figsize=(6, 4))
sns.heatmap(conf_mat, annot=True, fmt='d', cmap='Blues', xticklabels=class_names, yticklabels=class_names)
plt.xlabel('Predicted Label')
plt.ylabel('True Label')
plt.title('Confusion Matrix - Random Forest')
plt.show()


print("\nClassification Report:")
print(class_rep)


# 8. Feature Importance
importances = rf_clf.feature_importances_
indices = np.argsort(importances)[::-1] # Sort feature importances in descending order


print("\n--- Feature Importances ---")
for i in range(X.shape[1]):
    print(f"{i + 1}. Feature '{feature_names[indices[i]]}' ({importances[indices[i]]:.4f})")


# Plot feature importances
plt.figure(figsize=(10, 5))
plt.title("Feature Importances - Random Forest")
plt.bar(range(X.shape[1]), importances[indices], align="center")
plt.xticks(range(X.shape[1]), [feature_names[i] for i in indices], rotation=45)
plt.xlim([-1, X.shape[1]])
plt.ylabel('Importance')
plt.tight_layout()
plt.show()

Output (Illustrative)

Section titled “Output (Illustrative)”
Out-of-Bag (OOB) Score Estimate: 0.9429


--- Model Evaluation on Test Set ---
Test Accuracy: 1.0000


Confusion Matrix:
(Seaborn heatmap plot showing perfect classification on the test set for Iris)


Classification Report:
              precision    recall  f1-score   support


      setosa       1.00      1.00      1.00        15
  versicolor       1.00      1.00      1.00        15
   virginica       1.00      1.00      1.00        15


    accuracy                           1.00        45
   macro avg       1.00      1.00      1.00        45
weighted avg       1.00      1.00      1.00        45




--- Feature Importances ---
1. Feature 'petal width (cm)' (0.4445)
2. Feature 'petal length (cm)' (0.4233)
3. Feature 'sepal length (cm)' (0.1066)
4. Feature 'sepal width (cm)' (0.0256)

The output demonstrates training a Random Forest, getting an OOB score (a reliable estimate of performance without needing a separate validation set), evaluating on the test set (achieving perfect accuracy on this simple dataset), and examining feature importances. The petal features (‘petal width’ and ‘petal length’) are found to be significantly more important than the sepal features for classifying Iris species, according to this model.

Pros and Cons of Random Forest

Section titled “Pros and Cons of Random Forest”

Pros

Section titled “Pros”
  • High Accuracy: Generally achieves high prediction accuracy on many tasks.
  • Robust to Overfitting: Due to averaging predictions from multiple decorrelated trees, it’s much less prone to overfitting than single decision trees.
  • Handles Non-linearity: Inherits the ability of decision trees to capture complex non-linear relationships.
  • Handles Missing Values: Can handle missing data reasonably well (though imputation is still often recommended).
  • Feature Importance: Provides reliable estimates of feature importance.
  • Less Data Prep: Often works well without extensive feature scaling.
  • Parallelizable: Training individual trees can be parallelized, speeding up computation on multi-core systems (n_jobs=-1).
  • OOB Estimate: Provides an unbiased estimate of the generalization error without needing a separate validation set.

Cons

Section titled “Cons”
  • Less Interpretable: Can be seen as a ‘black box’ compared to a single decision tree. Understanding the exact reasoning behind a prediction is difficult.
  • Computationally Intensive: Can be slow to train and predict compared to simpler models like linear regression or Naive Bayes, especially with a large number of trees or large datasets.
  • Memory Usage: Can require significant memory to store the forest (many trees).
  • Can Still Overfit: While more robust, can still overfit on noisy datasets if hyperparameters (like tree depth) are not constrained.
  • Bias towards High Cardinality Features: For categorical features with many levels, Random Forests might be biased towards selecting those features.

Applications

Section titled “Applications”

Random Forests are used across many domains:

  • Finance: Credit scoring, fraud detection.
  • Healthcare: Disease prediction, patient outcome prediction.
  • E-commerce: Recommendation systems, customer churn prediction.
  • Remote Sensing: Land cover classification.
  • Biology: Gene expression analysis.

Further Resources

Section titled “Further Resources”