Linear Regression

Regression Algorithms - Linear Regression

Introduction to Linear Regression

Linear Regression is a fundamental supervised learning algorithm used for predicting a continuous target variable based on one or more predictor (independent) variables. It assumes a linear relationship between the predictors and the target.

Mathematically, this relationship is modeled as:

For Simple Linear Regression (one predictor):

Y = β₀ + β₁X + ε

For Multiple Linear Regression (multiple predictors):

Y = β₀ + β₁X₁ + β₂X₂ + ... + βₚXₚ + ε

Where:

• Y: The dependent (target) variable we want to predict.
• X, X₁, …, Xₚ: The independent (predictor) variables.
• β₀: The intercept (the value of Y when all X are 0).
• β₁, …, βₚ: The coefficients (slopes), representing the change in Y for a one-unit change in the corresponding X, holding other predictors constant.
• ε: The error term, representing the difference between the actual and predicted values (residuals).

The goal of linear regression is to find the optimal values for the coefficients (β₀, β₁, …) that minimize the error, typically measured by the sum of squared errors (SSE) or Residual Sum of Squares (RSS).

Types of Linear Relationship

• Positive Linear Relationship: As the predictor variable(s) increase, the target variable also tends to increase (positive slope coefficient).

• Negative Linear Relationship: As the predictor variable(s) increase, the target variable tends to decrease (negative slope coefficient).

Types of Linear Regression

Based on the number of predictor variables:

• Simple Linear Regression (SLR): Uses a single predictor variable (X) to predict the target (Y).
• Multiple Linear Regression (MLR): Uses two or more predictor variables (X₁, X₂, …) to predict the target (Y).

Simple Linear Regression (SLR)

SLR models the linear relationship between one independent variable and one dependent variable. It finds the best-fitting straight line through the data points.

Python Implementation (Conceptual - Manual Calculation)

While libraries like Scikit-learn are typically used, understanding the underlying calculations helps. This example demonstrates estimating coefficients manually using the least squares formulas.

import numpy as np
import matplotlib.pyplot as plt

def estimate_coefficients(x, y):
    n = np.size(x)

    # Calculate means
    mean_x, mean_y = np.mean(x), np.mean(y)

    # Calculate cross-deviation and deviation about x
    SS_xy = np.sum(y * x) - n * mean_y * mean_x
    SS_xx = np.sum(x * x) - n * mean_x * mean_x

    # Calculate regression coefficients (slope b1 and intercept b0)
    b1 = SS_xy / SS_xx
    b0 = mean_y - b1 * mean_x

    return (b0, b1)

def plot_regression_line(x, y, b):
    plt.figure(figsize=(8, 6))
    # Plot actual points as scatter plot
    plt.scatter(x, y, color="purple", marker="o", s=30, label='Data Points')

    # Predict response vector
    y_pred = b[0] + b[1] * x

    # Plot the regression line
    plt.plot(x, y_pred, color="green", label='Regression Line')

    plt.xlabel('Independent Variable (X)')
    plt.ylabel('Dependent Variable (Y)')
    plt.title('Simple Linear Regression Example')
    plt.legend()
    plt.grid(True)
    plt.show()

# --- Main execution ---
# Sample data
x = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
y = np.array([1, 3, 2, 5, 7, 8, 8, 9, 10, 12]) # Example data

# Estimate coefficients
b = estimate_coefficients(x, y)
print(f"Estimated coefficients:\nIntercept (b0) = {b[0]:.4f} \nSlope (b1) = {b[1]:.4f}")

# Plot the regression line
plot_regression_line(x, y, b)

Output (Illustrative)

Estimated coefficients:
Intercept (b0) = 1.2364
Slope (b1) = 1.1697

This code calculates the intercept and slope for the sample data and plots the resulting regression line against the data points. (The plot would show scattered points with a green line representing the best linear fit).

Python Implementation (Scikit-learn)

Using Scikit-learn is the standard and more practical approach. Here, we use the Diabetes dataset (loading only one feature for SLR).

import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score

# Load the diabetes dataset
diabetes = datasets.load_diabetes()

# Use only one feature (e.g., the third feature, BMI)
X = diabetes.data[:, np.newaxis, 2] # np.newaxis keeps it a 2D array
y = diabetes.target

# Split the data into training/testing sets
# Using only 30 samples for testing for demonstration
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=30, random_state=42)

# Create linear regression object
regr = linear_model.LinearRegression()

# Train the model using the training sets
regr.fit(X_train, y_train)

# Make predictions using the testing set
y_pred = regr.predict(X_test)

# The coefficients
print(f'Coefficient (Slope): {regr.coef_[0]:.2f}')
print(f'Intercept: {regr.intercept_:.2f}')
# The mean squared error
print(f'Mean squared error (MSE): {mean_squared_error(y_test, y_pred):.2f}')
# The coefficient of determination (R^2 score): 1 is perfect prediction
print(f'Coefficient of determination (R^2): {r2_score(y_test, y_pred):.2f}')

# Plot outputs
plt.figure(figsize=(8, 6))
plt.scatter(X_test, y_test, color='blue', label='Actual Data')
plt.plot(X_test, y_pred, color='red', linewidth=2, label='Predicted Line')
plt.xlabel('Feature (BMI)')
plt.ylabel('Target (Disease Progression)')
plt.title('Simple Linear Regression (Diabetes Dataset)')
# plt.xticks(()) # Optional: Hide x-axis ticks if desired
# plt.yticks(()) # Optional: Hide y-axis ticks if desired
plt.legend()
plt.grid(True)
plt.show()

Output (Illustrative)

Coefficient (Slope): 999.53
Intercept: 152.00
Mean squared error (MSE): 3260.93
Coefficient of determination (R^2): 0.48

This output shows the learned coefficient (slope), intercept, the MSE (average squared difference between actual and predicted values), and the R² score (proportion of variance in the target predictable from the feature). The plot would show the test data points and the regression line learned by the model.

Multiple Linear Regression (MLR)

MLR extends SLR by using multiple independent variables to predict a single dependent variable. The model fits a hyperplane (instead of a line) to the data in higher dimensions.

The model equation is: Y = β₀ + β₁X₁ + β₂X₂ + ... + βₚXₚ + ε

The interpretation of each coefficient βᵢ is the expected change in Y for a one-unit increase in Xᵢ, holding all other predictors constant.

Python Implementation (Scikit-learn)

We’ll use the California Housing dataset from Scikit-learn as the Boston dataset has ethical concerns and is deprecated.

import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score

# Load the California housing dataset
california = datasets.fetch_california_housing()
X = california.data
y = california.target
feature_names = california.feature_names

# Split the dataset into training and testing sets (e.g., 70% train, 30% test)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Create linear regression object
mlr = linear_model.LinearRegression()

# Train the model using the training sets
mlr.fit(X_train, y_train)

# Make predictions using the testing set
y_pred_train = mlr.predict(X_train)
y_pred_test = mlr.predict(X_test)

# The coefficients
print('Coefficients:')
for feature, coef in zip(feature_names, mlr.coef_):
    print(f'  {feature}: {coef:.4f}')
print(f'Intercept: {mlr.intercept_:.4f}')

# Evaluate the model
mse_test = mean_squared_error(y_test, y_pred_test)
r2_test = r2_score(y_test, y_pred_test)
print(f'\nTest Mean Squared Error (MSE): {mse_test:.4f}')
print(f'Test Coefficient of Determination (R^2): {r2_test:.4f}')

# --- Residual Plot ---
# Residuals are the difference between actual and predicted values
train_residuals = y_train - y_pred_train
test_residuals = y_test - y_pred_test

plt.figure(figsize=(10, 6))
plt.scatter(y_pred_train, train_residuals,
            color="steelblue", s=10, label='Train data', alpha=0.5)
plt.scatter(y_pred_test, test_residuals,
            color="orange", s=10, label='Test data', alpha=0.5)

plt.hlines(y=0, xmin=np.min(y_pred_train), xmax=np.max(y_pred_train),
           color='black', lw=2, linestyle='--')

plt.xlabel('Predicted Values')
plt.ylabel('Residuals')
plt.title('Residual Plot')
plt.legend()
plt.grid(True)
plt.show()

Output (Illustrative)

Coefficients:
  MedInc: 0.4444
  HouseAge: 0.0097
  AveRooms: -0.1134
  AveBedrms: 0.6992
  Population: -0.0000
  AveOccup: -0.0035
  Latitude: -0.4198
  Longitude: -0.4336
Intercept: -36.9112

Test Mean Squared Error (MSE): 0.5306
Test Coefficient of Determination (R^2): 0.5969

The output shows the coefficient for each feature, indicating its estimated effect on the target (median house value), holding others constant. The R² suggests the model explains about 59.7% of the variance in the test set. The residual plot helps diagnose model fit; ideally, residuals should be randomly scattered around zero without obvious patterns.

Assumptions of Linear Regression

Linear regression relies on several key assumptions for the results to be valid and reliable:

• Linearity: The relationship between predictors and the target variable is linear.
• Independence: Observations are independent of each other (especially important for time series data). Residuals should also be independent (no autocorrelation).
• Homoscedasticity: The variance of the residuals is constant across all levels of the predictor variables (i.e., the spread of errors is consistent).
• Normality of Residuals: The residuals are approximately normally distributed. This is more important for hypothesis testing and confidence intervals than for prediction accuracy.
• No Perfect Multicollinearity: Predictor variables should not be perfectly correlated with each other. High (but not perfect) multicollinearity can inflate the variance of coefficient estimates, making them unstable.

Violations of these assumptions can lead to biased or inefficient estimates and incorrect inferences. Diagnostic plots (like the residual plot shown above) and statistical tests are used to check these assumptions.

Regularization

When dealing with high dimensionality or multicollinearity, regularized linear models like Ridge, Lasso, and Elastic Net are often preferred. They add a penalty term to the cost function to shrink coefficients towards zero, which can improve model generalization and handle multicollinearity.

Further Resources

• Scikit-learn Linear Models Documentation: https://scikit-learn.org/stable/modules/linear_model.html
• StatQuest: Linear Regression Explained: https://statquest.org/video-index/ (Search for Linear Regression)