Scikit Learn - Extended Linear Modeling

Scikit-Learn - Extending Linear Models with Polynomial Features and Pipelines

This chapter focuses on enhancing the capabilities of linear models by incorporating non-linear features, specifically polynomial features, and streamlining workflows using Scikit-Learn’s Pipeline tools.

Introduction to Polynomial Features

Linear models, by definition, assume a linear relationship between features and the target variable. However, real-world data often exhibits non-linear patterns. One common way to allow linear models to fit non-linear data is to transform the input features into a higher-dimensional space that includes non-linear combinations of the original features. Polynomial features are a popular choice for this.

For example, a simple linear regression model for 1D data is y = w_0 + w_1*x. If the relationship is quadratic, we can create a new feature x^2 and fit a model like y = w_0 + w_1*x + w_2*x^2. This is still a linear model in terms of the coefficients w_0, w_1, w_2, but it can capture a quadratic relationship with x.

Mathematically, for 2D data (x_1, x_2), a standard linear model is:

ŷ(w, x) = w_0 + w_1*x_1 + w_2*x_2

By constructing second-order polynomial features, including interaction terms, the model becomes:

ŷ(w, x) = w_0 + w_1*x_1 + w_2*x_2 + w_3*x_1*x_2 + w_4*x_1^2 + w_5*x_2^2

This transformed model remains linear with respect to the new feature set (1, x_1, x_2, x_1*x_2, x_1^2, x_2^2) and can be solved using standard linear model techniques.

Scikit-Learn provides the PolynomialFeatures transformer in sklearn.preprocessing to generate these features.

Key Parameters for PolynomialFeatures

Parameter	Description
degree	int, default=2. The degree of the polynomial features. For example, degree=2 generates features up to x_i*x_j and x_i^2.
interaction_only	bool, default=False. If True, only interaction features are produced: features that are products of at most degree distinct input features (e.g., x_1*x_2 but not x_1^2 if degree=2).
include_bias	bool, default=True. If True, includes a bias column, i.e., the feature in which all polynomial powers are zero (the constant term, 1).
order	{‘C’, ‘F’}, default=‘C’. Order of output array in the dense case. ‘F’ order is faster to compute but may slow down subsequent estimators if they expect C-contiguous arrays.

Key Attributes for PolynomialFeatures

Attribute	Description
powers_	array of shape (n_output_features, n_input_features). powers_[i, j] is the exponent of the j-th input feature in the i-th output feature.
n_features_in_	int. Number of features seen during fit (original number of input features).
n_output_features_	int. The total number of polynomial output features. This is also the number of columns in the transformed data.

Implementation Example (PolynomialFeatures)

Transforming a 2D array into polynomial features of degree 2:

from sklearn.preprocessing import PolynomialFeatures
import numpy as np

# Sample input data: 4 samples, 2 features
X_original = np.arange(8).reshape(4, 2)
# X_original will be:
# [[0 1]
#  [2 3]
#  [4 5]
#  [6 7]]
print(f"Original X:\n{X_original}")

# Initialize PolynomialFeatures transformer for degree 2
poly = PolynomialFeatures(degree=2, include_bias=True)

# Fit and transform the data
X_poly = poly.fit_transform(X_original)
print(f"\nPolynomial features (degree 2):\n{X_poly}")
print(f"\nFeature names (if available, or use get_feature_names_out()): {poly.get_feature_names_out(['x1', 'x2'])}")
print(f"\nPowers matrix:\n{poly.powers_}")

Output Example:

Original X:
[[0 1]
 [2 3]
 [4 5]
 [6 7]]

Polynomial features (degree 2):
[[ 1.  0.  1.  0.  0.  1.]
 [ 1.  2.  3.  4.  6.  9.]
 [ 1.  4.  5. 16. 20. 25.]
 [ 1.  6.  7. 36. 42. 49.]]

Feature names (if available, or use get_feature_names_out()): ['1' 'x1' 'x2' 'x1^2' 'x1 x2' 'x2^2']

Powers matrix:
[[0 0]
 [1 0]
 [0 1]
 [2 0]
 [1 1]
 [0 2]]

The output features are [1, x1, x2, x1^2, x1*x2, x2^2].

Streamlining with Pipeline Tools (sklearn.pipeline.Pipeline)

Preprocessing steps like generating polynomial features are often combined with fitting a final estimator. Scikit-Learn’s Pipeline object allows chaining multiple estimators (transformers and a final predictor) into a single meta-estimator. This simplifies the workflow, prevents data leakage from the test set into the training process (e.g., fitting transformers only on training data), and makes grid searching over parameters of all steps easier.

Example: Polynomial Regression using a Pipeline

Let’s create a pipeline that first generates polynomial features and then fits a linear regression model. We’ll try to recover coefficients of a known polynomial function.

from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline # or make_pipeline for simpler construction
# (PolynomialFeatures, np already imported)

# Define the true polynomial relationship: y = 3 - 2*x + x^2 - x^3
np.random.seed(0)
x_true = np.arange(5)
y_true = 3 - 2 * x_true + x_true**2 - x_true**3
# Reshape x for Scikit-Learn (as a column vector)
X_true_reshaped = x_true[:, np.newaxis]

# Create a pipeline: 1. PolynomialFeatures, 2. LinearRegression
# We want degree 3 polynomial features.
# For LinearRegression, fit_intercept=False because PolynomialFeatures (with include_bias=True) already provides the intercept term.
poly_regression_pipeline = Pipeline([
    ('poly_features', PolynomialFeatures(degree=3, include_bias=True)),
    ('linear_regression', LinearRegression(fit_intercept=False))
])

# Fit the pipeline to the data
poly_regression_pipeline.fit(X_true_reshaped, y_true)

# Access the coefficients of the linear regression model within the pipeline
# The coefficients correspond to [1, x, x^2, x^3] features generated by PolynomialFeatures
learned_coefficients = poly_regression_pipeline.named_steps['linear_regression'].coef_
print(f"Learned coefficients: {learned_coefficients}")
# For a perfect fit on noiseless data, these should be close to [3., -2., 1., -1.]

# Make predictions using the pipeline
# y_pred = poly_regression_pipeline.predict(X_true_reshaped)
# print(f"Original y: {y_true}")
# print(f"Predicted y: {np.round(y_pred, 2)}")

Output Example:

Learned coefficients: [ 3. -2.  1. -1.]

The output shows that the linear model trained on polynomial features (within the pipeline) successfully recovered the exact coefficients of the input polynomial function. This demonstrates the power of combining feature engineering with linear models and the convenience of using pipelines.

Benefits of using Pipelines:

• Convenience and Encapsulation: Combines multiple processing steps into one object.
• Joint Parameter Selection: Allows grid searching over parameters from all steps in the pipeline simultaneously.
• Avoiding Data Leakage: Ensures that fitting of preprocessing steps (like scaling or PCA) happens only on the training data during cross-validation.

Using polynomial features can increase model complexity and the risk of overfitting, especially with high degrees. Regularization (e.g., Ridge or Lasso instead of LinearRegression in the pipeline) is often necessary. For more information, see the PolynomialFeatures documentation and the Pipeline documentation.