Dimensionality Reduction using PCA

Scikit-learn - Dimensionality Reduction using PCA

Dimensionality reduction is the process of reducing the number of random variables (features) under consideration, by obtaining a set of principal variables. It is a crucial technique in machine learning for tasks like visualization, computational efficiency, and mitigating the ‘curse of dimensionality’. Principal Component Analysis (PCA) is one of the most popular linear dimensionality reduction algorithms.

PCA aims to find the directions (principal components) that maximize the variance in the data and projects the data onto a new, lower-dimensional subspace formed by these components.

Exact PCA

Standard PCA performs a linear dimensionality reduction using Singular Value Decomposition (SVD) of the data to project it to a lower-dimensional space. By default, PCA centers the data (subtracts the mean from each feature) but does not scale it before applying SVD. Scaling (e.g., using StandardScaler) is often recommended as a preprocessing step if features have different units or scales.

Scikit-learn’s sklearn.decomposition.PCA module is implemented as a transformer object. It learns the principal components in its fit() method and can then be used to project data onto these components using transform().

Example: PCA on the Wine Dataset

The Wine dataset has 13 features. We’ll use PCA to reduce it to a smaller number of components and examine the explained variance.

from sklearn.datasets import load_wine
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import numpy as np

# Load the Wine dataset
wine = load_wine()
X, y = wine.data, wine.target

# It's good practice to scale data before PCA
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Apply PCA, let's reduce to 3 components
pca = PCA(n_components=3)
pca.fit(X_scaled) # Fit PCA on scaled data
X_pca = pca.transform(X_scaled) # Transform data to principal components

print(f"Original shape: {X_scaled.shape}")
print(f"Reduced shape after PCA: {X_pca.shape}")

print(f"\nExplained variance ratio by each component: {pca.explained_variance_ratio_}")
print(f"Total explained variance by 3 components: {np.sum(pca.explained_variance_ratio_):.4f}")

# The components themselves (eigenvectors)
# print(f"\nPrincipal components (eigenvectors):\n{pca.components_}")

Output (Explained variance can vary slightly based on implementation details but general trend holds)

Original shape: (178, 13)
Reduced shape after PCA: (178, 3)

Explained variance ratio by each component: [0.36198848 0.1920749  0.11123631]
Total explained variance by 3 components: 0.6653

The explained_variance_ratio_ attribute shows the percentage of variance explained by each selected component. If n_components is set to an integer, it specifies the number of components. If it’s a float between 0 and 1 (e.g., 0.95), PCA selects the number of components such that the amount of variance that needs to be explained is greater than the percentage specified.

Incremental PCA (IPCA)

Incremental Principal Component Analysis (IPCA) is useful for large datasets that do not fit in memory. It allows for partial computations on mini-batches of data, making it suitable for out-of-core learning. IPCA processes data in chunks using its partial_fit() method.

Scikit-learn provides sklearn.decomposition.IncrementalPCA. Similar to PCA, IPCA centers data but does not scale it.

Example: IPCA on Digits Dataset

We’ll use the Digits dataset to demonstrate IPCA.

from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import IncrementalPCA

# Load digits dataset
X, y = load_digits(return_X_y=True)

# Scale data
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Initialize IPCA
# n_components: target number of components
# batch_size: number of samples to use for each batch
ipca = IncrementalPCA(n_components=10, batch_size=200)

# Fit IPCA incrementally (or use fit() which handles batching)
# For demonstration, explicitly using partial_fit (not strictly necessary if using fit() later)
# for batch in np.array_split(X_scaled, len(X_scaled) // 200):
#     ipca.partial_fit(batch)

X_ipca = ipca.fit_transform(X_scaled) # fit() can also handle batching internally

print(f"Original shape: {X_scaled.shape}")
print(f"Reduced shape after IPCA: {X_ipca.shape}")
print(f"Total explained variance by 10 components (IPCA): {np.sum(ipca.explained_variance_ratio_):.4f}")

Output

Original shape: (1797, 64)
Reduced shape after IPCA: (1797, 10)
Total explained variance by 10 components (IPCA): 0.5338

Kernel PCA (KPCA)

Kernel Principal Component Analysis (KPCA) is an extension of PCA that uses kernels to achieve non-linear dimensionality reduction. By mapping data into a higher-dimensional feature space via a kernel function, KPCA can find non-linear relationships that linear PCA would miss.

Scikit-learn provides sklearn.decomposition.KernelPCA. Key parameters include n_components, kernel (e.g., ‘linear’, ‘rbf’, ‘poly’, ‘sigmoid’), and kernel-specific parameters like gamma or degree.

Example: Kernel PCA on Digits Dataset

from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import KernelPCA

# Load digits dataset
X, y = load_digits(return_X_y=True)

# Scale data
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Initialize Kernel PCA
# Using 'rbf' kernel, a common choice for non-linear mappings
kpca = KernelPCA(n_components=10, kernel='rbf', gamma=None, random_state=42)
# gamma=None will use 1/n_features

X_kpca = kpca.fit_transform(X_scaled)

print(f"Original shape: {X_scaled.shape}")
print(f"Reduced shape after Kernel PCA: {X_kpca.shape}")
# Note: explained_variance_ratio_ is not directly available for KernelPCA in the same way as PCA
# Eigenvalues can be accessed via kpca.eigenvalues_

Output

Original shape: (1797, 64)
Reduced shape after Kernel PCA: (1797, 10)

KPCA can be more computationally expensive than linear PCA, especially with large datasets. The choice of kernel and its parameters is crucial for good performance.

PCA using Randomized SVD

For very large datasets (many samples and/or many features), computing the full SVD can be slow. Randomized SVD provides an efficient way to approximate the top singular values and vectors. Scikit-learn’s PCA estimator can use randomized SVD by setting the svd_solver='randomized' parameter. This is particularly useful when the number of components to keep (n_components) is much smaller than the number of features or samples.

Example: Randomized PCA on Wine Dataset

from sklearn.datasets import load_wine
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import numpy as np

# Load the Wine dataset
wine = load_wine()
X, y = wine.data, wine.target

# Scale data
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Apply PCA with randomized SVD
# This is often faster for large datasets when n_components is small
r_pca = PCA(n_components=3, svd_solver='randomized', random_state=42)
r_pca.fit(X_scaled)
X_rpca = r_pca.transform(X_scaled)

print(f"Original shape: {X_scaled.shape}")
print(f"Reduced shape after Randomized PCA: {X_rpca.shape}")

print(f"\nExplained variance ratio (Randomized PCA): {r_pca.explained_variance_ratio_}")
print(f"Total explained variance by 3 components (Randomized PCA): {np.sum(r_pca.explained_variance_ratio_):.4f}")

Output (results should be very similar to exact PCA for smaller datasets)

Original shape: (178, 13)
Reduced shape after Randomized PCA: (178, 3)

Explained variance ratio (Randomized PCA): [0.36198848 0.1920749  0.11123631]
Total explained variance by 3 components (Randomized PCA): 0.6653

Other svd_solver options include ‘auto’ (default, chooses solver based on data size and n_components), ‘full’ (exact full SVD), and ‘arpack’ (for sparse data or when n_components is very small compared to min(n_samples, n_features)).