Randomized Decision Trees

Scikit-Learn - Randomized Decision Trees

This chapter explores randomized decision tree algorithms in Scikit-Learn, which are powerful ensemble methods known for their high accuracy and robustness against overfitting.

Introduction to Randomized Decision Tree Algorithms

Standard decision trees are prone to overfitting, especially if grown deep. Randomized decision tree algorithms, such as Random Forests and Extra-Trees (Extremely Randomized Trees), address this by building an ensemble of multiple decision trees. Randomness is introduced in two main ways: by training each tree on a different subsample of the data (bagging) and/or by considering only a random subset of features at each split point. The final prediction is typically an average (for regression) or a majority vote (for classification) of the predictions from individual trees.

The sklearn.ensemble module provides implementations for these algorithms.

The Random Forest Algorithm

Random Forest builds multiple decision trees. For each tree, it typically uses a bootstrap sample (random sample with replacement) of the training data. When splitting a node in a tree, it considers only a random subset of features to find the best split. This dual randomization helps to de-correlate the trees, reducing variance and improving generalization. Random Forests can be used for both classification and regression.

Random Forest Classifier (RandomForestClassifier)

sklearn.ensemble.RandomForestClassifier is used for classification tasks. Key parameters include n_estimators (number of trees), max_features (size of the random subsets of features), max_depth (maximum depth of each tree), and min_samples_split (minimum samples required to split a node).

Implementation Example 1: Synthetic Data

from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_blobs
from sklearn.ensemble import RandomForestClassifier

# Generate synthetic data for classification
X, y = make_blobs(n_samples=1000, n_features=10, centers=5, random_state=0)

# Initialize Random Forest Classifier
# Using fewer estimators for quicker example, in practice use more (e.g., 100+)
rf_clf = RandomForestClassifier(n_estimators=50, max_depth=None,
                                min_samples_split=2, random_state=0)

# Evaluate using cross-validation
scores = cross_val_score(rf_clf, X, y, cv=5) # 5-fold cross-validation
print(f"Cross-validation scores: {scores}")
print(f"Mean cross-validation score: {scores.mean():.4f}")

Output Example 1:

Cross-validation scores: [0.975 0.97  0.98  0.965 0.975]
Mean cross-validation score: 0.9730

Implementation Example 2: Iris Dataset

Using the Iris dataset for a practical classification example, including evaluation metrics.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score
from sklearn.datasets import load_iris

# Load Iris dataset
iris = load_iris()
X_iris = iris.data
y_iris = iris.target

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X_iris, y_iris, test_size=0.30, random_state=42)

# Initialize and train the Random Forest Classifier
rf_iris_clf = RandomForestClassifier(n_estimators=100, random_state=42)
rf_iris_clf.fit(X_train, y_train)

# Make predictions
y_pred = rf_iris_clf.predict(X_test)

# Evaluate the classifier
print("Confusion Matrix:")
print(confusion_matrix(y_test, y_pred))
print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=iris.target_names))
print(f"Accuracy Score: {accuracy_score(y_test, y_pred):.4f}")

Output Example 2:

Confusion Matrix:
[[19  0  0]
 [ 0 13  0]
 [ 0  0 13]]

Classification Report:
              precision    recall  f1-score   support

      setosa       1.00      1.00      1.00        19
  versicolor       1.00      1.00      1.00        13
   virginica       1.00      1.00      1.00        13

    accuracy                           1.00        45
   macro avg       1.00      1.00      1.00        45
weighted avg       1.00      1.00      1.00        45

Accuracy Score: 1.0000

Random Forest Regressor (RandomForestRegressor)

sklearn.ensemble.RandomForestRegressor is used for regression tasks. It uses similar parameters to the classifier, but the criterion for splitting nodes is typically based on reducing mean squared error (MSE) or mean absolute error (MAE).

Implementation Example

from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import make_regression
import numpy as np

# Generate synthetic data for regression
X_reg, y_reg = make_regression(n_features=10, n_informative=5, random_state=0, shuffle=False)

# Initialize and train Random Forest Regressor
rf_regr = RandomForestRegressor(n_estimators=100, max_depth=10, random_state=0)
rf_regr.fit(X_reg, y_reg)

# Make a prediction on a sample point (e.g., the first point from X_reg)
sample_point = X_reg[0].reshape(1, -1)
prediction = rf_regr.predict(sample_point)
print(f"Model parameters: {rf_regr.get_params()['n_estimators']} estimators, max_depth {rf_regr.get_params()['max_depth']}")
print(f"Prediction for sample point {sample_point}: {prediction}")
print(f"Actual value for sample point: {y_reg[0]}")

Output Example:

Model parameters: 100 estimators, max_depth 10
Prediction for sample point [[ 0.18500369  0.89837401  0.75398021 -0.40579343  0.01871914 -0.7394844
  -0.84078684 -0.46938239  0.17930832  0.00304159]]: [-47.6763054]
Actual value for sample point: -45.04868412282407

Extra-Trees (Extremely Randomized Trees)

Extra-Trees introduce more randomness compared to Random Forests. When choosing a split point, not only is a random subset of features considered, but the split threshold for each considered feature is also chosen randomly (from a uniform distribution within the feature’s range in the current sample). This often leads to slightly higher bias but can reduce variance further. They are generally faster to train because they don’t search for the optimal threshold.

Extra-Trees Classifier (ExtraTreesClassifier)

sklearn.ensemble.ExtraTreesClassifier uses the same key parameters as RandomForestClassifier.

Implementation Example 1: Synthetic Data

from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_blobs
from sklearn.ensemble import ExtraTreesClassifier

# Re-use synthetic data from Random Forest example
X_et, y_et = make_blobs(n_samples=1000, n_features=10, centers=5, random_state=0)

# Initialize Extra Trees Classifier
et_clf = ExtraTreesClassifier(n_estimators=50, max_depth=None,
                              min_samples_split=2, random_state=0)

# Evaluate using cross-validation
et_scores = cross_val_score(et_clf, X_et, y_et, cv=5)
print(f"Extra-Trees cross-validation scores: {et_scores}")
print(f"Extra-Trees mean cross-validation score: {et_scores.mean():.4f}")

Output Example 1:

Extra-Trees cross-validation scores: [0.98  0.97  0.98  0.97  0.98 ]
Extra-Trees mean cross-validation score: 0.9760

Implementation Example 2: Using a different dataset (e.g., make_classification)

Using make_classification for a more complex scenario.

from sklearn.datasets import make_classification
from sklearn.model_selection import KFold
# (KFold and cross_val_score already imported)
# (ExtraTreesClassifier already imported)

X_class, Y_class = make_classification(n_samples=1000, n_features=20, n_informative=15,
                                       n_redundant=5, random_state=7, n_classes=3)

# Setup KFold for cross-validation
kfold = KFold(n_splits=10, random_state=7, shuffle=True) # Added shuffle=True for reproducibility with random_state

num_trees = 150
max_feat = 7 # Example: consider 7 features at each split
et_clf_complex = ExtraTreesClassifier(n_estimators=num_trees, max_features=max_feat, random_state=7)
results_complex = cross_val_score(et_clf_complex, X_class, Y_class, cv=kfold)
print(f"Extra-Trees (complex data) mean accuracy: {results_complex.mean():.4f}")

Output Example 2:

Extra-Trees (complex data) mean accuracy: 0.8050

Extra-Trees Regressor (ExtraTreesRegressor)

sklearn.ensemble.ExtraTreesRegressor for regression tasks, with similar parameters to ExtraTreesClassifier.

Implementation Example

from sklearn.ensemble import ExtraTreesRegressor
# (make_regression and np already imported)

# Re-use synthetic regression data
X_reg_et, y_reg_et = make_regression(n_features=10, n_informative=5, random_state=0, shuffle=False)

# Initialize and train Extra Trees Regressor
et_regr = ExtraTreesRegressor(n_estimators=100, max_depth=10, random_state=0)
et_regr.fit(X_reg_et, y_reg_et)

# Make a prediction on the same sample point as RF Regressor example
sample_point_et = X_reg_et[0].reshape(1, -1)
prediction_et = et_regr.predict(sample_point_et)
print(f"ET Regressor parameters: {et_regr.get_params()['n_estimators']} estimators, max_depth {et_regr.get_params()['max_depth']}")
print(f"ET Prediction for sample point {sample_point_et}: {prediction_et}")
print(f"Actual value for sample point: {y_reg_et[0]}")

Output Example:

ET Regressor parameters: 100 estimators, max_depth 10
ET Prediction for sample point [[ 0.18500369  0.89837401  0.75398021 -0.40579343  0.01871914 -0.7394844
  -0.84078684 -0.46938239  0.17930832  0.00304159]]: [-44.47586897]
Actual value for sample point: -45.04868412282407

Randomized decision trees are versatile and often provide excellent performance with minimal hyperparameter tuning. For more details, refer to the Scikit-Learn documentation on ensemble methods.