Improving Performance of ML Model (Contd…)

Improving Machine Learning Model Performance

Building an initial machine learning model is just the first step. Achieving optimal performance often requires iterative refinement. This involves various strategies, including data preprocessing, feature engineering, algorithm selection, and hyperparameter tuning (also known as algorithm tuning).

Performance Improvement via Hyperparameter Tuning

Most machine learning algorithms have settings, called hyperparameters, that are not learned from the data but are set before the training process begins. These hyperparameters control aspects of the learning process itself, such as model complexity or regularization strength.

Examples include:

• The C parameter in Support Vector Machines (SVM).
• The k in K-Nearest Neighbors (KNN).
• The alpha regularization parameter in Ridge or Lasso regression.
• The number of trees (n_estimators) or maximum depth (max_depth) in Random Forests.

In contrast, parameters (like the coefficients in linear regression or the weights in a neural network) are learned from the data during training.

Hyperparameter tuning (or optimization) is the process of systematically searching for the combination of hyperparameter values that yields the best model performance (evaluated on a validation set or through cross-validation) for a given problem and dataset.

Scikit-learn provides excellent tools for this, primarily Grid Search and Randomized Search.

Grid Search (GridSearchCV)

Grid Search performs an exhaustive search over a specified subset of the hyperparameter space. You define a ‘grid’ of hyperparameter values you want to try, and GridSearchCV trains and evaluates a model for every possible combination of these values using cross-validation.

Pros: Guaranteed to find the best combination within the specified grid.

Cons: Can be computationally very expensive, especially if the grid is large or involves many hyperparameters (‘curse of dimensionality’).

Example: Tuning alpha for Ridge Regression

We’ll tune the regularization strength (alpha) for Ridge Regression on the Diabetes dataset.

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Load the diabetes dataset
diabetes = load_diabetes()
X, y = diabetes.data, diabetes.target

# It's crucial to scale data when using regularization
# We use a Pipeline to combine scaling and the model
pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('ridge', Ridge())
])

# Define the grid of alpha values to search
# Using a logarithmic range is common for regularization parameters
alphas = np.logspace(-4, 2, 7) # Example: 7 values from 0.0001 to 100
param_grid = {'ridge__alpha': alphas} # Note the '__' notation for pipeline steps

# Set up GridSearchCV
# cv=5 means 5-fold cross-validation
# scoring='neg_mean_squared_error' - GridSearchCV maximizes scores, so use negative MSE for minimization
grid_search = GridSearchCV(estimator=pipe,
                           param_grid=param_grid,
                           cv=5,
                           scoring='neg_mean_squared_error',
                           verbose=1) # verbose shows progress

# Fit the grid search to the data
grid_search.fit(X, y)

# Print the best score (Mean CV score) and best parameters
print(f"Best Cross-Validation Score (Negative MSE): {grid_search.best_score_:.4f}")
print(f"Best Parameters found: {grid_search.best_params_}")

# Access the best estimator (the pipeline with the best alpha)
best_model = grid_search.best_estimator_
print(f"\nBest Ridge alpha: {best_model.named_steps['ridge'].alpha}")

Output (Illustrative)

Fitting 5 folds for each of 7 candidates, totalling 35 fits
...
Best Cross-Validation Score (Negative MSE): -2921.4895
Best Parameters found: {'ridge__alpha': 1.0}

Best Ridge alpha: 1.0

The output shows the best negative MSE achieved during cross-validation and the corresponding alpha value (1.0 in this example run) that yielded this score. A pipeline ensures scaling is done correctly within each cross-validation fold.

Randomized Search (RandomizedSearchCV)

Randomized Search samples a fixed number of hyperparameter combinations (n_iter) from specified statistical distributions (or lists). It doesn’t try every combination but rather explores a random subset of the parameter space.

Pros: Much more computationally efficient than Grid Search, especially with many hyperparameters. Often finds very good (if not the absolute best) hyperparameters faster.

Cons: Doesn’t guarantee finding the optimal combination; results depend on the number of iterations and the random sampling.

Example: Tuning alpha for Ridge Regression

We’ll use Randomized Search to explore a wider range of alpha values more efficiently.

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Ridge
from sklearn.model_selection import RandomizedSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from scipy.stats import uniform # For sampling from a uniform distribution

# Load the diabetes dataset
diabetes = load_diabetes()
X, y = diabetes.data, diabetes.target

# Set up the pipeline
pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('ridge', Ridge())
])

# Define the parameter distribution to sample from
# Sample alpha from a uniform distribution between 0 and 10
param_distributions = {
    'ridge__alpha': uniform(loc=0, scale=10) # Samples from U(0, 10)
}

# Set up RandomizedSearchCV
# n_iter controls how many parameter settings are sampled
# random_state ensures reproducibility
random_search = RandomizedSearchCV(estimator=pipe,
                               param_distributions=param_distributions,
                               n_iter=50, # Try 50 random combinations
                               cv=5,
                               scoring='neg_mean_squared_error',
                               random_state=42,
                               verbose=1)

# Fit the random search to the data
random_search.fit(X, y)

# Print the best score and best parameters
print(f"Best Cross-Validation Score (Negative MSE): {random_search.best_score_:.4f}")
print(f"Best Parameters found: {random_search.best_params_}")

# Access the best estimator
best_model = random_search.best_estimator_
print(f"\nBest Ridge alpha: {best_model.named_steps['ridge'].alpha:.4f}")

Output (Illustrative)

Fitting 5 folds for each of 50 candidates, totalling 250 fits
...
Best Cross-Validation Score (Negative MSE): -2921.4214
Best Parameters found: {'ridge__alpha': 0.9370821941623886}

Best Ridge alpha: 0.9371

Randomized Search explored 50 different alpha values sampled from the uniform distribution. The best score is very close to that found by Grid Search, but potentially achieved with less computation if the Grid Search space had been much larger. The best alpha found is approximately 0.937.

Other Performance Improvement Strategies

Besides hyperparameter tuning, consider:

• Feature Engineering: Creating new features from existing ones, selecting relevant features, or removing irrelevant/redundant ones.
• Data Augmentation/Collection: Obtaining more relevant training data.
• Handling Missing Data: Using appropriate imputation techniques.
• Addressing Class Imbalance: Using techniques like SMOTE, ADASYN, or adjusting class weights.
• Algorithm Selection: Trying different types of algorithms that might be better suited to the data’s structure.
• Ensemble Methods: Combining multiple models (e.g., Bagging, Boosting, Stacking) often leads to better performance and robustness than single models.

Advanced Tuning Techniques

For complex search spaces or computationally expensive models, more advanced techniques exist:

• Bayesian Optimization: Uses probability to guide the search towards more promising hyperparameter regions (e.g., using libraries like scikit-optimize or Hyperopt).
• Successive Halving / HyperBand: Iteratively allocate resources to promising configurations, quickly discarding poor ones (e.g., HalvingGridSearchCV, HalvingRandomSearchCV in newer scikit-learn versions).

Further Resources

• Scikit-learn Tuning Documentation: https://scikit-learn.org/stable/modules/grid_search.html
• Scikit-learn Pipeline Documentation: https://scikit-learn.org/stable/modules/compose.html#pipeline