Scikit Learn - Modelling Process
Scikit-learn - Modeling Process
Section titled “Scikit-learn - Modeling Process”This chapter covers the typical modeling process in Scikit-learn. We’ll explore each step, starting with dataset loading and preparation, then moving to model training, evaluation, and persistence. Understanding this workflow is crucial for effectively applying machine learning techniques.
Dataset Loading
Section titled “Dataset Loading”A dataset is a collection of data, typically organized into features and a target (or response). Key components include:
Features (X): These are the input variables or attributes used by the model to make predictions. Also known as predictors or independent variables.
- Feature Matrix: A 2D array-like structure (e.g., NumPy array or Pandas DataFrame) where rows represent samples and columns represent features. Shape:
[n_samples, n_features]. - Feature Names: A list of names corresponding to each column in the feature matrix, useful for understanding the data.
Response (y): This is the output variable that the model aims to predict. Also known as the target, label, or dependent variable.
- Response Vector: A 1D array-like structure representing the target values for each sample. Shape:
[n_samples]. - Target Names: For classification tasks, this represents the names of the different classes the response vector can take.
Scikit-learn includes several built-in datasets for practice, such as the Iris and Digits datasets for classification, and the California Housing dataset for regression. These are useful for quickly testing algorithms. You can find more under the sklearn.datasets module.
Example: Loading the Iris Dataset
Section titled “Example: Loading the Iris Dataset”Here’s how to load the classic Iris dataset. We use return_X_y=True for convenience to directly get the features (X) and target (y).
from sklearn.datasets import load_iris
# Load the datasetiris = load_iris()X = iris.datay = iris.target
# Alternatively, load features and target directly# X, y = load_iris(return_X_y=True)
feature_names = iris.feature_namestarget_names = iris.target_names
print(f"Feature names: {feature_names}")print(f"Target names: {target_names}")print(f"\nFirst 5 rows of X (Features):\n{X[:5]}")print(f"\nFirst 5 labels of y (Target):\n{y[:5]}")print(f"\nShape of X: {X.shape}")print(f"Shape of y: {y.shape}")Output
Section titled “Output”Feature names: ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']Target names: ['setosa' 'versicolor' 'virginica']
First 5 rows of X (Features):[[5.1 3.5 1.4 0.2] [4.9 3. 1.4 0.2] [4.7 3.2 1.3 0.2] [4.6 3.1 1.5 0.2] [5. 3.6 1.4 0.2]]
First 5 labels of y (Target):[0 0 0 0 0]
Shape of X: (150, 4)Shape of y: (150,)Splitting the Dataset
Section titled “Splitting the Dataset”To evaluate a model’s performance on unseen data, it’s essential to split the dataset into a training set and a testing set. The model learns from the training set, and its performance is then assessed on the testing set. This helps prevent overfitting, where a model performs well on training data but poorly on new data.
Example: Train-Test Split
Section titled “Example: Train-Test Split”The train_test_split function from sklearn.model_selection is commonly used. Here, we split the Iris dataset, allocating 30% for testing and 70% for training.
from sklearn.datasets import load_irisfrom sklearn.model_selection import train_test_split
# Load dataX, y = load_iris(return_X_y=True)
# Split data: 70% training, 30% testingX_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, random_state=42, stratify=y)
print(f"Shape of X_train: {X_train.shape}")print(f"Shape of X_test: {X_test.shape}")print(f"Shape of y_train: {y_train.shape}")print(f"Shape of y_test: {y_test.shape}")Output
Section titled “Output”Shape of X_train: (105, 4)Shape of X_test: (45, 4)Shape of y_train: (105,)Shape of y_test: (45,)Key parameters for train_test_split:
- X, y: The feature matrix and response vector to be split.
- test_size: Proportional size of the test set (e.g., 0.3 for 30%).
train_sizecan be used alternatively. - random_state: A seed for the random number generator. Setting this ensures that the split is the same every time you run the code, which is crucial for reproducibility. Any integer value can be used.
- stratify=y: (Recommended for classification) Ensures that the proportion of target classes in the train and test sets is similar to the original dataset. This is important for imbalanced datasets.
Train the Model
Section titled “Train the Model”After preparing the data, the next step is to train a machine learning model. Scikit-learn offers a wide array of algorithms, all following a consistent API: fit() to train, predict() to make predictions, and score() to evaluate.
Example: Training a K-Nearest Neighbors (KNN) Classifier
Section titled “Example: Training a K-Nearest Neighbors (KNN) Classifier”We’ll use the K-Nearest Neighbors (KNN) algorithm, a simple yet effective classifier. This example demonstrates the basic training and prediction workflow.
from sklearn.datasets import load_irisfrom sklearn.model_selection import train_test_splitfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.metrics import accuracy_scoreimport numpy as np
# Load and split data (as shown previously)X, y = load_iris(return_X_y=True)X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)
# Initialize the KNN classifier# n_neighbors specifies the 'K' in KNNknn_classifier = KNeighborsClassifier(n_neighbors=3)
# Train the model using the training dataknn_classifier.fit(X_train, y_train)
# Make predictions on the test datay_pred = knn_classifier.predict(X_test)
# Evaluate the model's accuracyaccuracy = accuracy_score(y_test, y_pred)print(f"Accuracy on test set: {accuracy:.4f}")
# Example of predicting on new, unseen samples# Note: New samples should have the same number of features as the training data (4 in this case)sample_new_data = np.array([ [5.0, 3.1, 1.6, 0.2], # Expected: setosa (0) [6.5, 3.0, 5.2, 2.0] # Expected: virginica (2)])predicted_species_indices = knn_classifier.predict(sample_new_data)iris = load_iris() # Reload for target_namespredicted_species_names = iris.target_names[predicted_species_indices]print(f"Predictions for new samples: {predicted_species_names}")Output
Section titled “Output”Accuracy on test set: 0.9778Predictions for new samples: ['setosa' 'virginica']For more details on KNN, refer to its dedicated chapter. Choosing the right model and tuning its parameters (like n_neighbors for KNN) often involves techniques like cross-validation, discussed later.
Model Persistence
Section titled “Model Persistence”Training a model can be time-consuming. Once trained, you’ll often want to save (persist) it for later use without retraining. The joblib library (part of the SciPy ecosystem and recommended by Scikit-learn) is excellent for this, especially for objects containing large NumPy arrays.
Let’s save the knn_classifier we trained earlier:
import joblib
# Assume knn_classifier is our trained model from the previous step# Save the model to a filemodel_filename = 'iris_knn_classifier.joblib'joblib.dump(knn_classifier, model_filename)print(f"Model saved to {model_filename}")
# ... Later, in another script or session ...
# Load the model from the fileloaded_knn_model = joblib.load(model_filename)print(f"Model loaded from {model_filename}")
# You can now use the loaded_knn_model to make predictions# Example: y_pred_loaded = loaded_knn_model.predict(X_test)# print(f"Accuracy of loaded model: {accuracy_score(y_test, y_pred_loaded):.4f}")This saves the entire model object, including its learned parameters. Ensure the environment where you load the model has the same versions of relevant libraries (like Scikit-learn) for compatibility.
Data Preprocessing
Section titled “Data Preprocessing”Raw data is often not suitable for direct input into machine learning algorithms. Preprocessing transforms raw data into a more usable format. Scikit-learn’s sklearn.preprocessing module offers various tools. Common preprocessing steps include scaling, normalization, and encoding categorical features.
Why preprocess? Many algorithms (e.g., SVMs, KNN, PCA, neural networks) are sensitive to feature scales. Features with larger values might dominate the learning process. Preprocessing helps to standardize the data.
Binarization
Section titled “Binarization”Binarization converts numerical features into boolean values (0 or 1) based on a threshold. This can be useful for creating binary features from continuous ones.
import numpy as npfrom sklearn.preprocessing import Binarizer
input_data = np.array([ [2.1, -1.9, 5.5], [-1.5, 2.4, 3.5], [0.5, -7.9, 5.6], [5.9, 2.3, -5.8]])
# Values > 0.5 become 1, others become 0binarizer = Binarizer(threshold=0.5)data_binarized = binarizer.fit_transform(input_data)print(f"\nBinarized data (threshold=0.5):\n{data_binarized}")Output
Section titled “Output”Binarized data (threshold=0.5):[[1. 0. 1.] [0. 1. 1.] [0. 0. 1.] [1. 1. 0.]]Mean Removal (Standardization)
Section titled “Mean Removal (Standardization)”Mean removal, more commonly known as standardization or Z-score normalization, transforms features to have zero mean and unit variance. This is achieved by subtracting the mean and dividing by the standard deviation for each feature.
import numpy as npfrom sklearn.preprocessing import StandardScaler
input_data = np.array([ [2.1, -1.9, 5.5], [-1.5, 2.4, 3.5], [0.5, -7.9, 5.6], [5.9, 2.3, -5.8]], dtype=np.float64)
print(f"Original Mean: {input_data.mean(axis=0)}")print(f"Original Std Dev: {input_data.std(axis=0)}")
scaler = StandardScaler()data_scaled = scaler.fit_transform(input_data)
print(f"\nScaled Data (StandardScaler):\n{data_scaled}")print(f"Mean of scaled data: {data_scaled.mean(axis=0)}")print(f"Std Dev of scaled data: {data_scaled.std(axis=0)}")Output
Section titled “Output”Original Mean: [ 1.75 -1.275 2.2 ]Original Std Dev: [2.71431391 4.20022321 4.69414529]
Scaled Data (StandardScaler):[[ 0.12896806 0.6318175 0.70299018] [-1.19734452 0.87470814 0.2769086 ] [-0.46049308 -1.57685687 0.72410327] [ 1.52886954 0.07033123 -1.7039976 ]]Mean of scaled data: [2.22044605e-17 0.00000000e+00 0.00000000e+00]Std Dev of scaled data: [1. 1. 1.]Note: It’s crucial to fit the scaler on the training data only and then use transform on both training and testing data to prevent data leakage from the test set.
Scaling (Min-Max Scaling)
Section titled “Scaling (Min-Max Scaling)”Min-Max scaling (or normalization) rescales features to a specific range, typically [0, 1] or [-1, 1]. This is done by subtracting the minimum value and dividing by the range (max - min) for each feature.
import numpy as npfrom sklearn.preprocessing import MinMaxScaler
input_data = np.array([ [2.1, -1.9, 5.5], [-1.5, 2.4, 3.5], [0.5, -7.9, 5.6], [5.9, 2.3, -5.8]], dtype=np.float64)
# Scale features to the range [0, 1]min_max_scaler = MinMaxScaler(feature_range=(0, 1))data_scaled_minmax = min_max_scaler.fit_transform(input_data)print(f"\nMin-Max Scaled Data (range [0, 1]):\n{data_scaled_minmax}")Output
Section titled “Output”Min-Max Scaled Data (range [0, 1]):[[0.48648649 0.58252427 0.99122807] [0. 1. 0.81578947] [0.27027027 0. 1. ] [1. 0.99029126 0. ]]Normalization
Section titled “Normalization”Normalization in Scikit-learn (using sklearn.preprocessing.Normalizer) rescales each sample (row) independently to have unit norm (L1 or L2). This is different from scaling features (columns) like StandardScaler or MinMaxScaler do. Normalization is often used for text classification or clustering.
L1 Normalization (Least Absolute Deviations)
Section titled “L1 Normalization (Least Absolute Deviations)”Rescales each sample so that the sum of the absolute values of its features is 1.
import numpy as npfrom sklearn.preprocessing import Normalizer
input_data = np.array([ [2.1, -1.9, 5.5], [-1.5, 2.4, 3.5], [0.5, -7.9, 5.6], [5.9, 2.3, -5.8]], dtype=np.float64)
normalizer_l1 = Normalizer(norm='l1')data_normalized_l1 = normalizer_l1.fit_transform(input_data)print(f"\nL1 Normalized Data:\n{data_normalized_l1}")print(f"Sum of absolute values per row (L1 norm): {np.sum(np.abs(data_normalized_l1), axis=1)}")Output
Section titled “Output”L1 Normalized Data:[[ 0.22105263 -0.2 0.57894737] [-0.2027027 0.32432432 0.47297297] [ 0.03571429 -0.56428571 0.4 ] [ 0.42142857 0.16428571 -0.41428571]]Sum of absolute values per row (L1 norm): [1. 1. 1. 1.]L2 Normalization (Least Squares)
Section titled “L2 Normalization (Least Squares)”Rescales each sample so that the sum of the squares of its features (Euclidean norm) is 1.
import numpy as npfrom sklearn.preprocessing import Normalizer
input_data = np.array([ [2.1, -1.9, 5.5], [-1.5, 2.4, 3.5], [0.5, -7.9, 5.6], [5.9, 2.3, -5.8]], dtype=np.float64)
normalizer_l2 = Normalizer(norm='l2')data_normalized_l2 = normalizer_l2.fit_transform(input_data)print(f"\nL2 Normalized Data:\n{data_normalized_l2}")print(f"Sum of squares per row (L2 norm squared): {np.sum(data_normalized_l2**2, axis=1)}")Output
Section titled “Output”L2 Normalized Data:[[ 0.33946114 -0.30713151 0.88906489] [-0.33325106 0.53320169 0.7775858 ] [ 0.05156558 -0.81473612 0.57753446] [ 0.68706914 0.26784051 -0.6754239 ]]Sum of squares per row (L2 norm squared): [1. 1. 1. 1.]Pipelines for Streamlining Workflows
Section titled “Pipelines for Streamlining Workflows”Scikit-learn’s Pipeline object allows you to chain multiple processing steps (like scaling and PCA) with a final estimator (like a classifier). This is highly recommended for:
- Convenience: Combines multiple steps into one object.
- Preventing Data Leakage: Ensures transformations are fit only on the training data during cross-validation.
- Reproducibility: Encapsulates the entire preprocessing and modeling workflow.
Example of a simple pipeline:
from sklearn.pipeline import Pipelinefrom sklearn.preprocessing import StandardScalerfrom sklearn.svm import SVCfrom sklearn.datasets import load_irisfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import accuracy_score
# Load and split dataX, y = load_iris(return_X_y=True)X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# Create a pipeline: StandardScaler then SVCpipe = Pipeline([ ('scaler', StandardScaler()), # Step 1: Scale data ('svc', SVC()) # Step 2: Support Vector Classifier])
# Fit the pipeline (applies scaler's fit_transform and then SVC's fit)pipe.fit(X_train, y_train)
# Make predictions (applies scaler's transform and then SVC's predict)y_pred_pipeline = pipe.predict(X_test)
accuracy_pipeline = accuracy_score(y_test, y_pred_pipeline)print(f"Accuracy with Pipeline: {accuracy_pipeline:.4f}")Output
Section titled “Output”Accuracy with Pipeline: 1.0000Pipelines are a powerful tool for building robust and maintainable machine learning models. You can learn more in the Scikit-learn documentation on sklearn.pipeline.Pipeline.