Logistic Regression

Classification Algorithms - Logistic Regression

Introduction to Logistic Regression

Despite its name, Logistic Regression is a fundamental classification algorithm used in supervised learning. It’s primarily used to predict the probability that an input belongs to a particular category or class. While it can be extended to multi-class problems, its most common application is in binary classification, where the target variable has only two possible outcomes (e.g., Yes/No, True/False, 0/1).

Think of predicting whether a customer will click on an ad (Yes/No) or if a tumor is malignant or benign (M/B). Logistic Regression models the probability of the ‘positive’ class (often denoted as 1) occurring.

Unlike Linear Regression which predicts a continuous value, Logistic Regression uses a specific transformation function – the logistic function (or sigmoid function) – to map any real-valued input (typically a linear combination of features) into a probability value between 0 and 1.

Mathematically, it models the probability P(Y=1 | X) as a function of the input features X.

The Sigmoid Function

The core of Logistic Regression is the sigmoid function:

σ(z) = 1 / (1 + e^(-z))

Where z is the linear combination of input features and weights (similar to Linear Regression): z = θ₀ + θ₁x₁ + θ₂x₂ + ... + θ<0xE2><0x82><0x99>x<0xE2><0x82><0x99> (or z = Xθ in vector notation).

The sigmoid function squashes the output z (which can range from -∞ to +∞) into the range (0, 1). This output can be interpreted as the estimated probability of the positive class.

Graphically, the sigmoid function is an ‘S’-shaped curve. For large positive values of z, σ(z) approaches 1. For large negative values of z, σ(z) approaches 0. When z=0, σ(z) = 0.5.

To make a class prediction, we typically set a threshold (often 0.5). If the predicted probability σ(z) is greater than or equal to the threshold, we classify the instance as positive (1); otherwise, we classify it as negative (0).

Loss Function and Optimization

To train the model (i.e., find the optimal weights θ), Logistic Regression uses a loss function called Log Loss (or Binary Cross-Entropy). This function measures how well the predicted probabilities match the actual class labels (0 or 1) in the training data.

The goal during training is to minimize this Log Loss function. This is typically achieved using optimization algorithms like Gradient Descent or more advanced methods (e.g., L-BFGS, commonly used in library implementations) which iteratively adjust the weights θ to reduce the loss.

Types of Logistic Regression

Based on the number of target categories:

Binary Logistic Regression

The most common type, where the target variable has only two possible outcomes (e.g., 0 or 1, Spam/Not Spam, Malignant/Benign).

Multinomial Logistic Regression

Used when the target variable has three or more possible outcomes that are unordered (nominal), meaning there’s no inherent ranking between categories (e.g., classifying news articles into ‘Sports’, ‘Politics’, ‘Technology’). It models the probability of each class.

Ordinal Logistic Regression

Used when the target variable has three or more possible outcomes that are ordered (ordinal), meaning there is a meaningful ranking between categories (e.g., customer satisfaction ratings like ‘Poor’, ‘Good’, ‘Excellent’). It takes the ordering into account.

Scikit-learn’s LogisticRegression class can handle Binary and Multinomial cases directly.

Key Assumptions of Logistic Regression

While relatively robust, Logistic Regression relies on certain assumptions for optimal performance:

• Binary Outcome: For binary logistic regression, the dependent variable must be binary.
• Independence of Observations: Observations should be independent of each other.
• No Severe Multicollinearity: Independent variables should not be highly correlated with each other. High multicollinearity can inflate variance and make coefficient interpretation unreliable.
• Linearity of Independent Variables and Log Odds: Assumes a linear relationship between the independent variables and the log-odds of the outcome. Violations can sometimes be addressed by transforming variables or adding interaction terms.
• Large Sample Size: Logistic regression typically requires a reasonably large sample size for stable and reliable coefficient estimates.

Feature Scaling: While not a strict assumption, performance (especially convergence speed during training) often improves significantly if features are scaled (e.g., using StandardScaler or MinMaxScaler).

Implementation in Python with Scikit-learn

Let’s implement Binary and Multinomial Logistic Regression using Scikit-learn.

1. Binary Logistic Regression Example (Iris Dataset)

We’ll use the Iris dataset but modify it for a binary task: classifying whether a flower is ‘setosa’ (class 0) or ‘not setosa’ (class 1, combining ‘versicolor’ and ‘virginica’). We’ll use only the first two features (sepal length and width) for easier visualization (though the original tutorial plot is removed).

import numpy as np
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

# --- Load and Prepare Data ---
iris = load_iris()
# Use only first two features for this example
X = iris.data[:, :2]
# Create binary target: 0 = setosa, 1 = not setosa
y = (iris.target != 0).astype(int)

feature_names = iris.feature_names[:2]
target_names = ['setosa', 'not setosa']

# --- Split Data ---
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)

# --- Scale Features ---
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# --- Train Logistic Regression Model ---
# C is the inverse of regularization strength; smaller values specify stronger regularization.
model_binary = LogisticRegression(C=1.0, random_state=42, solver='liblinear')
model_binary.fit(X_train_scaled, y_train)

# --- Make Predictions ---
y_pred_binary = model_binary.predict(X_test_scaled)
proba_binary = model_binary.predict_proba(X_test_scaled) # Get probabilities

# --- Evaluate Model ---
print("--- Binary Logistic Regression (Iris: Setosa vs. Not Setosa) ---")
accuracy_binary = accuracy_score(y_test, y_pred_binary)
print(f"Accuracy: {accuracy_binary:.4f}")

print("\nConfusion Matrix:")
print(confusion_matrix(y_test, y_pred_binary))

print("\nClassification Report:")
print(classification_report(y_test, y_pred_binary, target_names=target_names))

# print("\nPredicted Probabilities (first 5):\n", proba_binary[:5])
# Description: Shows the probability estimates for each class (setosa, not setosa) for the first 5 test samples.

Explanation: This code trains a binary logistic regression model. It loads Iris data, converts the target to binary, splits, scales features, trains the model, predicts classes and probabilities, and evaluates using standard metrics. The output shows high accuracy, indicating the model effectively separates setosa from the other two species based on sepal dimensions.

2. Multinomial Logistic Regression Example (Digits Dataset)

Now, let’s use the Digits dataset, which has 10 classes (digits 0-9), making it a multinomial classification problem.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

# --- Load Data ---
digits = load_digits()
X = digits.data
y = digits.target

# --- Split Data ---
# Using a larger test size as in the original example
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=1, stratify=y)

# --- Scale Features ---
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# --- Train Multinomial Logistic Regression Model ---
# Scikit-learn's LogisticRegression handles multinomial automatically
# when multi_class='auto' or 'multinomial' (with appropriate solvers like 'lbfgs')
# Using a solver suitable for multinomial like 'lbfgs'
model_multi = LogisticRegression(multi_class='multinomial', solver='lbfgs', max_iter=5000, random_state=42)
model_multi.fit(X_train_scaled, y_train)

# --- Make Predictions ---
y_pred_multi = model_multi.predict(X_test_scaled)

# --- Evaluate Model ---
print("\n--- Multinomial Logistic Regression (Digits Dataset) ---")
accuracy_multi = accuracy_score(y_test, y_pred_multi)
print(f"Accuracy: {accuracy_multi:.4f}")

# Confusion matrix is larger for multi-class
print("\nConfusion Matrix:")
cm_multi = confusion_matrix(y_test, y_pred_multi)
# print(cm_multi) # Can be large, maybe just print shape or specific parts
print(f"(Confusion Matrix Shape: {cm_multi.shape})")

print("\nClassification Report:")
# Get target names as strings for the report
target_names_digits = [str(i) for i in digits.target_names]
print(classification_report(y_test, y_pred_multi, target_names=target_names_digits))

Explanation: This code tackles a multi-class problem. It loads the digits dataset, splits, scales, and trains a LogisticRegression model configured for multinomial classification (multi_class='multinomial', solver='lbfgs'). The evaluation shows high accuracy and provides per-class metrics (precision, recall, f1-score) in the classification report, demonstrating the model’s effectiveness in recognizing handwritten digits.

Logistic Regression is a versatile and interpretable classification algorithm, serving as a great baseline model and often performing well on linearly separable or near-linearly separable data. Its output probabilities are also valuable in many applications.

Further reading: https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression