Data Feature Selection

ML with Python - Feature Selection

After preprocessing data, the next critical step is often feature selection. Not all features collected are equally useful for building an accurate machine learning model. Feature selection involves choosing a subset of the most relevant features from the original set.

Why is Feature Selection Important?

The features used to train a model directly impact its performance. Including irrelevant or redundant features can negatively affect the model in several ways:

• Improved Model Performance: Relevant features lead to better generalization and predictive accuracy, especially for simpler models.
• Reduced Overfitting: Fewer features can reduce model complexity, making it less likely to fit noise in the training data (overfitting).
• Faster Training Times: Training models with fewer features requires less computation.
• Improved Interpretability: Models with fewer features are often easier to understand and explain.
• Reduced Dimensionality: Addresses the ‘curse of dimensionality’ where performance degrades in high-dimensional spaces.

Feature selection is the process of automatically or manually selecting the features that contribute most to the prediction variable or output you are interested in. It differs from dimensionality reduction techniques like PCA, which transform features into a lower-dimensional space, potentially creating new, less interpretable features.

Common Feature Selection Techniques

Scikit-learn provides several methods for automatic feature selection:

1. Univariate Selection

This technique evaluates each feature independently based on statistical tests to determine the strength of its relationship with the target variable. Features with the highest scores are selected. Common tests include chi-squared (chi2) for categorical features and ANOVA F-value (f_classif for classification, f_regression for regression) for numerical features. Scikit-learn’s SelectKBest class implements this, allowing you to select the top ‘k’ features.

Example: Selecting the top 4 features from the Pima Indians Diabetes dataset using chi-squared.

import pandas as pd
import numpy as np
from sklearn.feature_selection import SelectKBest, chi2
from sklearn.preprocessing import MinMaxScaler # Chi2 requires non-negative features

# --- Load Data (Example using a URL for reproducibility) ---
# URL for the Pima Indians Diabetes dataset
url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv'
names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
dataframe = pd.read_csv(url, names=names)

# Separate features (X) and target (y)
X = dataframe[names[:-1]]
y = dataframe[names[-1]]

# Chi-squared requires non-negative values, scale data to [0, 1]
scaler = MinMaxScaler()
X_scaled = scaler.fit_transform(X)

# --- Feature Selection ---
# Select top 4 features based on chi-squared test
selector = SelectKBest(score_func=chi2, k=4)
fit = selector.fit(X_scaled, y)

# Get scores and selected feature indices
scores = fit.scores_
selected_indices = selector.get_support(indices=True)
selected_features = X.columns[selected_indices]

# --- Output Results ---
np.set_printoptions(precision=2)
print('Feature Scores:', scores)
print('\nSelected Feature Indices:', selected_indices)
print('Selected Features:', selected_features.tolist())

# Transform X to include only selected features
featured_data = fit.transform(X_scaled)
print(f"\nShape of data with selected features: {featured_data.shape}")
print("First 4 rows of featured data:\n", featured_data[0:4])

Output Explanation: The code loads the diabetes dataset, scales features to be non-negative (required by chi2), applies the SelectKBest method with the chi2 test to find the 4 features most strongly related to the ‘class’ label. It prints the statistical score for each feature and lists the names of the selected ones (‘plas’, ‘test’, ‘mass’, ‘age’). Finally, it shows the first few rows containing only these selected features.

Learn more about univariate selection: https://scikit-learn.org/stable/modules/feature_selection.html#univariate-feature-selection

2. Recursive Feature Elimination (RFE)

RFE works by recursively removing the least important features and building a model on the remaining ones. It uses an external estimator (like Logistic Regression or SVM) that assigns weights or importance scores to features. In each iteration, the feature with the lowest score is pruned until the desired number of features is reached.

Example: Using RFE with Logistic Regression to select the top 3 features.

import pandas as pd
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

# --- Load Data (Use previously loaded 'dataframe') ---
if 'dataframe' not in locals(): # Basic check if data is loaded
    url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv'
    names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
    dataframe = pd.read_csv(url, names=names)

# Separate features (X) and target (y)
X = dataframe[names[:-1]]
y = dataframe[names[-1]]

# --- Feature Selection ---
# Initialize the model (estimator) for RFE
model = LogisticRegression(solver='liblinear') # Using 'liblinear' solver

# Initialize RFE to select 3 features
# Note: Use 'n_features_to_select' instead of the old positional argument
rfe = RFE(estimator=model, n_features_to_select=3)
fit = rfe.fit(X, y)

# --- Output Results ---
selected_mask = fit.support_
ranking = fit.ranking_
selected_features = X.columns[selected_mask]

print(f"Number of Features Selected: {fit.n_features_}")
print(f"Selected Features Mask: {selected_mask}")
print(f"Selected Features Names: {selected_features.tolist()}")
print(f"Feature Ranking: {ranking}")

Output Explanation: This code applies RFE using Logistic Regression as the base model to identify the best 3 features. The output shows a boolean mask indicating which features were selected (True), the names of those features (preg, mass, pedi), and a ranking where ‘1’ indicates the selected features and higher numbers indicate features eliminated earlier.

Learn more about RFE: https://scikit-learn.org/stable/modules/feature_selection.html#recursive-feature-elimination

3. Principal Component Analysis (PCA) - Dimensionality Reduction

PCA is technically a dimensionality reduction technique, not feature selection. It doesn’t select original features but transforms the data into a new set of uncorrelated features called principal components. These components capture the maximum variance in the data. While it reduces dimensions, the new features are linear combinations of the old ones and might be less interpretable. Crucially, data should be scaled (e.g., using StandardScaler) before applying PCA.

Example: Applying PCA to reduce the dataset to 3 principal components.

import pandas as pd
import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# --- Load Data (Use previously loaded 'dataframe') ---
if 'dataframe' not in locals():
    url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv'
    names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
    dataframe = pd.read_csv(url, names=names)

# Separate features (X) and target (y)
X = dataframe[names[:-1]]
y = dataframe[names[-1]]

# --- Scale Data (Essential for PCA) ---
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# --- Apply PCA ---
# Initialize PCA to find 3 components
pca = PCA(n_components=3)
fit = pca.fit(X_scaled)

# --- Output Results ---
explained_variance_ratio = fit.explained_variance_ratio_
components = fit.components_

print(f"Explained Variance Ratio by component: {explained_variance_ratio}")
print(f"Total variance explained by 3 components: {explained_variance_ratio.sum():.4f}")
print(f"\nShape of PCA components: {components.shape}")
# print("PCA Components (Loadings):\n", components)

# Transform data to the new PCA space
X_pca = fit.transform(X_scaled)
print(f"\nShape of transformed data: {X_pca.shape}")
print("First 4 rows of PCA-transformed data:\n", X_pca[0:4])

Output Explanation: The code first scales the data, then applies PCA to reduce the 8 original features to 3 principal components. It shows the proportion of dataset variance captured by each component and the total variance captured by the selected 3. The output also shows the shape and first few rows of the data transformed into this new 3-dimensional space. Note that the transformed values don’t directly correspond to the original features.

Learn more about PCA: https://scikit-learn.org/stable/modules/decomposition.html#pca

4. Feature Importance from Tree-based Models

Ensemble tree-based models like Random Forest and Extra Trees calculate feature importances during training. This importance score reflects how much each feature contributes to reducing impurity (e.g., Gini impurity or entropy) across all the trees in the ensemble. Features with higher scores are considered more important.

Example: Using ExtraTreesClassifier to find feature importances.

import pandas as pd
import numpy as np
from sklearn.ensemble import ExtraTreesClassifier
import matplotlib.pyplot as plt # For visualization

# --- Load Data (Use previously loaded 'dataframe') ---
if 'dataframe' not in locals():
    url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv'
    names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
    dataframe = pd.read_csv(url, names=names)

# Separate features (X) and target (y)
X = dataframe[names[:-1]]
y = dataframe[names[-1]]

# --- Calculate Feature Importance ---
# Initialize the model
model = ExtraTreesClassifier(n_estimators=100, random_state=42)
model.fit(X, y)

# Get feature importances
importances = model.feature_importances_
feature_names = X.columns

# Create a Series for better display
importance_series = pd.Series(importances, index=feature_names).sort_values(ascending=False)

# --- Output Results ---
print("Feature Importances:\n", importance_series)

# --- Visualize (Optional) ---
# plt.figure(figsize=(10, 6))
# importance_series.plot(kind='bar')
# plt.title('Feature Importances from ExtraTreesClassifier')
# plt.ylabel('Importance Score')
# plt.show()
# Description: The code would generate a bar chart showing the importance score for each feature,
# allowing easy visual comparison. Features like 'plas', 'age', and 'mass' often show higher importance
# for this dataset.

Output Explanation: This example trains an ExtraTreesClassifier and extracts the calculated importance score for each feature. The scores are printed, sorted from most to least important. A higher score indicates that the feature was more useful in making predictions within the ensemble of trees. Visualization (commented out for text-only output) helps to quickly identify the top features.

Learn more about tree-based feature importance: https://scikit-learn.org/stable/modules/ensemble.html#feature-importance-evaluation

Other Techniques and Considerations

Other methods exist, including:

• L1 Regularization (Lasso): Linear models trained with L1 regularization tend to produce sparse coefficients, effectively setting the coefficients of less important features to zero, thus performing feature selection.
• Permutation Importance: A model-agnostic technique that measures the decrease in a model’s score when a single feature’s values are randomly shuffled.
• Sequential Feature Selection (SFS): Methods that iteratively add (forward selection) or remove (backward selection) features based on model performance.

Important Note on Data Leakage: To avoid overly optimistic results, feature selection should ideally be performed inside a cross-validation loop. This means selecting features based only on the training fold data in each iteration, preventing information from the validation fold from influencing the selection process. See Scikit-learn Pipeline documentation for integrating steps: https://scikit-learn.org/stable/modules/compose.html#pipeline