Scikit Learn - Linear Modeling
Scikit-Learn - Overview of Linear Models
Section titled “Scikit-Learn - Overview of Linear Models”This chapter provides an overview of various linear models available in Scikit-Learn. Linear models are a fundamental class of models in machine learning, used for both regression and classification tasks.
Linear models predict a target value based on a linear combination of input features. For regression, the model takes the form y_pred = w_1*x_1 + ... + w_p*x_p + b, where w are coefficients and b is the intercept. For classification, a similar linear combination is typically passed through a function (like sigmoid for logistic regression) or used to define a decision boundary (like in SVMs).
Scikit-Learn offers a rich set of tools for linear modeling, often found in the sklearn.linear_model module. Here’s a brief look at some common linear models:
| Model Name | Brief Description & Use Case |
|---|---|
| 1 | Linear Regression (LinearRegression)Fits a linear model to minimize the residual sum of squares between the observed targets in the dataset and the targets predicted by the linear approximation. Used for regression tasks where the relationship between features and target is assumed to be linear. |
| 2 | Logistic Regression (LogisticRegression)Despite its name, Logistic Regression is a linear model for classification (binary or multiclass). It uses a logistic (sigmoid) function to model the probability of a class. It’s widely used for binary classification problems and can be extended to multiclass settings (e.g., via one-vs-rest). |
| 3 | Ridge Regression (Ridge, RidgeCV)A linear regression model that includes L2 regularization (penalty on the square of the magnitude of coefficients). This helps to prevent overfitting by shrinking the coefficients, especially useful when dealing with multicollinearity (highly correlated features). RidgeCV includes built-in cross-validation to find the optimal regularization strength (alpha). |
| 4 | LASSO Regression (Lasso, LassoCV)Least Absolute Shrinkage and Selection Operator. A linear regression model that includes L1 regularization (penalty on the absolute value of coefficients). LASSO can shrink some coefficients to exactly zero, effectively performing feature selection. LassoCV includes cross-validation for alpha selection. |
| 5 | Elastic-Net (ElasticNet, ElasticNetCV)A linear regression model that combines L1 and L2 regularization. It’s useful when there are multiple correlated features (LASSO might pick one arbitrarily, while Elastic-Net tends to select groups) or when the number of features is much larger than the number of samples. ElasticNetCV includes cross-validation for alpha and l1_ratio. |
| 6 | Bayesian Ridge Regression (BayesianRidge)A Bayesian approach to Ridge Regression. Instead of finding a single best value for coefficients, it estimates a probability distribution for them. It can be robust to ill-posed problems and can provide uncertainty estimates for its predictions. |
| 7 | Stochastic Gradient Descent (SGDClassifier, SGDRegressor)Optimization algorithms that can fit various linear models (e.g., linear SVM, logistic regression, linear regression) by iteratively updating coefficients using one sample (or a mini-batch) at a time. Very efficient for large datasets. |
| 8 | Perceptron (Perceptron)One of the simplest linear classification algorithms. It’s an online learning algorithm that updates its weights if it misclassifies a training sample. |
| 9 | Passive Aggressive Algorithms (PassiveAggressiveClassifier, PassiveAggressiveRegressor)Online learning algorithms for classification and regression. They are ‘passive’ if the prediction is correct, and ‘aggressive’ in updating weights if a misclassification occurs or the prediction error exceeds a threshold. |
| 10 | Huber Regressor (HuberRegressor)A linear regression model that is robust to outliers. It uses a linear loss for samples classified as outliers and a squared loss for inliers. The epsilon parameter controls this transition. |
| 11 | Multi-task Lasso (MultiTaskLasso)A Lasso model that estimates sparse coefficients for multiple regression problems jointly, enforcing the selected features to be the same across all tasks. Useful when output variables are related. |
| 12 | Multi-task Elastic-Net (MultiTaskElasticNet)An Elastic-Net model that, like Multi-task Lasso, fits multiple regression problems jointly while sharing features. Combines L1 and L2 penalties for each task. |
Key Considerations for Linear Models:
- Feature Scaling: Many linear models, especially those using gradient descent (like SGD) or regularization, perform better when features are scaled to a similar range (e.g., using
StandardScalerorMinMaxScaler). - Regularization: Techniques like Ridge (L2), Lasso (L1), and Elastic-Net help prevent overfitting and can improve model generalization, particularly with high-dimensional data or multicollinearity.
- Interpretability: Linear models are often more interpretable than complex non-linear models, as the coefficients can indicate the strength and direction of each feature’s influence on the target.
- Assumptions: Classical linear regression has assumptions like linearity, independence of errors, homoscedasticity (constant variance of errors), and normality of errors. While Scikit-Learn models are more focused on predictive power, understanding these can be helpful for diagnostics.
Each of these models has its own specific parameters, use cases, and strengths. Subsequent chapters or specific documentation pages will delve into the details of individual linear models. For a comprehensive list and details, refer to the Scikit-Learn linear models documentation.