Improving Performance of ML Models

Improving ML Model Performance with Ensembles

Why Use Ensemble Methods?

Ensemble methods are powerful techniques that combine the predictions of multiple individual machine learning models (called base learners or estimators) to produce a final prediction. The core idea is that by combining diverse models, the ensemble’s prediction is often more accurate, robust, and stable than any single base learner.

Think of it like seeking advice from multiple experts instead of just one. Each expert might have slightly different knowledge or biases, but combining their opinions often leads to a better overall decision. Ensembles work best when the base learners are diverse, meaning they make different kinds of errors.

Ensemble methods can be broadly categorized based on how the base learners are built:

Sequential Ensemble Methods (Boosting)

Base learners are generated sequentially. Each new learner focuses on correcting the mistakes made by the previous ones. This typically reduces bias and builds powerful predictive models. Examples include AdaBoost and Gradient Boosting.

Parallel Ensemble Methods (Bagging)

Base learners are generated independently and in parallel. Each learner is trained on a different random subset of the training data. This primarily reduces variance and improves stability. Examples include Bagged Decision Trees and Random Forests.

Common Ensemble Learning Strategies

Here are the most popular ways to combine predictions from base learners:

1. Bagging (Bootstrap Aggregating)

Bagging involves training multiple instances of the same base learner on different random subsets of the training data, typically selected with replacement (bootstrap sampling). The final prediction is obtained by averaging (for regression) or majority voting (for classification) the predictions of all base learners. Bagging is particularly effective for high-variance models like decision trees, helping to reduce overfitting.

2. Boosting

Boosting builds the ensemble sequentially. Each new model attempts to correct the errors made by the combined ensemble of previous models. Instances that were misclassified by earlier models are given higher weight in the training of subsequent models. This focuses the learning process on the ‘hardest’ examples. AdaBoost and Gradient Boosting Machines (GBM) are classic examples.

3. Voting

Voting combines predictions from multiple different types of models (e.g., Logistic Regression, SVM, Decision Tree). In hard voting, the final prediction is the class label predicted by the majority of the models. In soft voting, it averages the predicted probabilities from each model and selects the class with the highest average probability (often performs better if classifiers can output probabilities).

Another related technique is Stacking, where the predictions of multiple base learners are used as input features for a final ‘meta-learner’ which makes the ultimate prediction.

Bagging Ensemble Algorithms in Scikit-learn

Let’s implement some common bagging algorithms using the Pima Indians Diabetes dataset. We’ll use cross-validation to get a more reliable estimate of performance.

# Common imports for Bagging examples
import pandas as pd
from sklearn.model_selection import KFold, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# --- Load Data (Example using URL) ---
url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv'
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = pd.read_csv(url, names=names)
X = dataframe[names[:-1]]
y = dataframe[names[-1]]

# --- Setup Cross-Validation ---
# Use KFold for cross-validation. shuffle=True is recommended.
kfold = KFold(n_splits=10, shuffle=True, random_state=42)

Bagged Decision Trees

Bagging is often applied to Decision Trees because they tend to have high variance. We use BaggingClassifier with DecisionTreeClassifier as the base estimator.

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

# --- Define Base Estimator and Ensemble ---
base_estimator = DecisionTreeClassifier(random_state=42)
num_trees = 100 # Number of trees in the ensemble

# Create a pipeline to scale data then apply bagging
# Scaling is less critical for trees but good practice if comparing with other models
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('bagging', BaggingClassifier(estimator=base_estimator,
                                 n_estimators=num_trees,
                                 random_state=42))
])

# --- Evaluate Model ---
results = cross_val_score(pipeline, X, y, cv=kfold, scoring='accuracy')
print(f"Bagged Decision Trees Accuracy: {results.mean():.4f} (Std: {results.std():.4f})")

Output Explanation: This code trains 100 decision trees, each on a different bootstrap sample of the (scaled) data. The final accuracy is the average accuracy across 10 cross-validation folds. The standard deviation gives an idea of the variability of the score.

Random Forest

Random Forest is an enhancement over basic bagged trees. In addition to sampling data points (bagging), it also samples a random subset of features at each split point when building the trees. This further increases diversity among the trees and often leads to better performance.

from sklearn.ensemble import RandomForestClassifier

# --- Define and Evaluate Model ---
num_trees = 100
max_features = 'sqrt' # Common choice: sqrt(n_features)

pipeline = Pipeline([
    ('scaler', StandardScaler()), # Still included for consistency, though less vital for RF
    ('rf', RandomForestClassifier(n_estimators=num_trees,
                                  max_features=max_features,
                                  random_state=42))
])

results = cross_val_score(pipeline, X, y, cv=kfold, scoring='accuracy')
print(f"Random Forest Accuracy: {results.mean():.4f} (Std: {results.std():.4f})")

Output Explanation: Similar to bagged trees, but RandomForestClassifier inherently includes both data and feature sampling. max_features='sqrt' tells it to consider only a random subset (square root of total features) when looking for the best split at each node.

Extra Trees (Extremely Randomized Trees)

Extra Trees introduces even more randomness. It builds trees using the whole original sample (no bootstrap), but at each node, it considers a random subset of features (like Random Forest) and chooses the split point randomly among them, rather than finding the optimal split. This can sometimes reduce variance further and speed up training.

from sklearn.ensemble import ExtraTreesClassifier

# --- Define and Evaluate Model ---
num_trees = 100
max_features = 'sqrt'

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('et', ExtraTreesClassifier(n_estimators=num_trees,
                                max_features=max_features,
                                random_state=42))
])

results = cross_val_score(pipeline, X, y, cv=kfold, scoring='accuracy')
print(f"Extra Trees Accuracy: {results.mean():.4f} (Std: {results.std():.4f})")

Output Explanation: Trains an ensemble using the Extra Trees algorithm, known for its high level of randomization in both feature selection and split point determination.

Boosting Ensemble Algorithms in Scikit-learn

Boosting algorithms build models sequentially, with each new model focusing on the errors of the previous ones. Scaling data is generally recommended for boosting algorithms.

AdaBoost (Adaptive Boosting)

AdaBoost works by assigning weights to training instances. Initially, all instances have equal weight. After each base learner (often a shallow decision tree called a ‘stump’) is trained, the weights of misclassified instances are increased. Subsequent learners then focus more on these harder-to-classify instances.

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier # Often used as base estimator

# --- Define and Evaluate Model ---
# AdaBoost often uses shallow trees
base_estimator = DecisionTreeClassifier(max_depth=1, random_state=42)
num_estimators = 50

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('adaboost', AdaBoostClassifier(estimator=base_estimator, # Use 'estimator' instead of 'base_estimator' in newer versions
                                  n_estimators=num_estimators,
                                  random_state=42))
])

results = cross_val_score(pipeline, X, y, cv=kfold, scoring='accuracy')
print(f"AdaBoost Accuracy: {results.mean():.4f} (Std: {results.std():.4f})")

Output Explanation: Implements AdaBoost, typically using weak learners (decision stumps). It sequentially trains estimators, adjusting weights based on misclassifications.

Gradient Boosting Machine (GBM)

GBM builds models sequentially like AdaBoost, but instead of adjusting instance weights, each new model fits the residual errors (the difference between predictions and actual values) of the previous ensemble. It uses gradient descent to minimize a loss function.

from sklearn.ensemble import GradientBoostingClassifier

# --- Define and Evaluate Model ---
num_estimators = 100

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('gbm', GradientBoostingClassifier(n_estimators=num_estimators,
                                      random_state=42))
])

results = cross_val_score(pipeline, X, y, cv=kfold, scoring='accuracy')
print(f"Gradient Boosting Accuracy: {results.mean():.4f} (Std: {results.std():.4f})")

Output Explanation: Implements the standard Gradient Boosting algorithm. Each tree tries to correct the errors made by the preceding sequence of trees.

Note on Advanced Boosting: While Scikit-learn provides GradientBoostingClassifier, libraries like XGBoost, LightGBM, and CatBoost offer highly optimized and often superior implementations of gradient boosting with additional features like regularization and handling of categorical data. They are worth exploring for competitive performance:

• XGBoost: https://xgboost.readthedocs.io/
• LightGBM: https://lightgbm.readthedocs.io/
• CatBoost: https://catboost.ai/

Voting Ensemble Algorithm in Scikit-learn

The Voting Classifier combines predictions from multiple different machine learning models. It’s useful when you have several well-performing but diverse models.

Example: Combining Logistic Regression, Decision Tree, and SVM.

from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.ensemble import VotingClassifier

# --- Define Base Models ---
# Create pipelines for each model to include scaling
log_reg = Pipeline([('scaler', StandardScaler()),
                    ('lr', LogisticRegression(solver='liblinear', random_state=42))])

dtree = Pipeline([('scaler', StandardScaler()), # Scaling might not affect tree much, but keeps setup consistent
                   ('dt', DecisionTreeClassifier(random_state=42))])

svm_clf = Pipeline([('scaler', StandardScaler()),
                    # Enable probability estimates for soft voting
                    ('svc', SVC(probability=True, random_state=42))])

# --- Create Voting Ensemble ---
# List of (name, model) tuples
estimators = [
    ('lr', log_reg),
    ('dt', dtree),
    ('svc', svm_clf)
]

# Use soft voting (averages probabilities) - often better
ensemble = VotingClassifier(estimators=estimators, voting='soft')

# --- Evaluate Model ---
results = cross_val_score(ensemble, X, y, cv=kfold, scoring='accuracy')
print(f"Voting Classifier (Soft) Accuracy: {results.mean():.4f} (Std: {results.std():.4f})")

# --- Example for Hard Voting ---
ensemble_hard = VotingClassifier(estimators=estimators, voting='hard')
results_hard = cross_val_score(ensemble_hard, X, y, cv=kfold, scoring='accuracy')
print(f"Voting Classifier (Hard) Accuracy: {results_hard.mean():.4f} (Std: {results_hard.std():.4f})")

Output Explanation: This code trains three different models (Logistic Regression, Decision Tree, SVM) within pipelines that include scaling. It then combines their predictions using a VotingClassifier. Soft voting (averaging probabilities) and hard voting (majority vote) are both demonstrated. Note that probability=True is needed for SVC to participate in soft voting.

Ensemble methods are a cornerstone of modern machine learning, often providing significant performance boosts. Experimenting with different base learners and ensemble strategies is key to finding the best solution for a given problem. Remember that hyperparameter tuning of both the base learners and the ensemble method itself is also crucial for optimal results.