Scikit Learn - Anomaly Detection

Scikit-Learn - Anomaly Detection

This tutorial explores anomaly detection using Scikit-Learn, focusing on identifying data points that deviate significantly from the rest of the dataset.

Anomaly detection is a crucial technique for identifying unusual patterns or outliers in data. It finds applications in diverse fields like fraud detection (e.g., unusual credit card transactions), intrusion detection in networks, system health monitoring, medical diagnosis (e.g., identifying tumors in scans), and predictive maintenance. Anomalies, also known as outliers, can be broadly categorized into:

• Point anomalies: An individual data instance is anomalous with respect to the rest of the data. For example, a single, extremely high purchase amount.
• Contextual anomalies (or conditional anomalies): A data instance is anomalous in a specific context but not otherwise. For example, high spending on heating in summer would be anomalous.
• Collective anomalies: A collection of related data instances is anomalous with respect to the entire dataset, even if individual instances within the collection are not. For example, a human heartbeat electrocardiogram showing an unnaturally flat line for a period, where individual data points on that line might seem normal in isolation.

Approaches to Anomaly Detection

Scikit-Learn primarily supports two main approaches for anomaly detection:

Outlier Detection

In outlier detection, the training data is assumed to contain outliers that are far from the majority of the data. Outlier detection algorithms aim to fit a model to the regions where the training data is most concentrated, thereby identifying observations that deviate from this norm. This is often referred to as unsupervised anomaly detection because labels indicating normal or anomalous are not used during training.

Novelty Detection

Novelty detection is concerned with identifying new or unseen patterns in observations that were not present in the training data. In this scenario, the training data is assumed to be ‘clean’ (not polluted by outliers). This is often referred to as semi-supervised anomaly detection.

Scikit-Learn provides a suite of machine learning tools applicable to both outlier and novelty detection. These tools, like other Scikit-Learn estimators, learn from the data using the fit() method:

estimator.fit(X_train)

Once trained, new observations can be classified as inliers (typically labeled 1 or True) or outliers/novelties (typically labeled -1 or False) using the predict() method:

predictions = estimator.predict(X_test)

Many estimators also provide a score_samples() method, which returns a raw score for each sample (the higher, the more normal). The predict() method often applies a threshold to these raw scores. This threshold can sometimes be controlled by a contamination parameter, which represents the expected proportion of outliers in the dataset.

Alternatively, the decision_function() method can be used for some estimators. It typically returns positive values for inliers and negative values for outliers.

decision_scores = estimator.decision_function(X_test)

Scikit-Learn Algorithms for Anomaly Detection

Let’s explore some popular algorithms available in Scikit-Learn for anomaly detection.

Elliptic Envelope (covariance.EllipticEnvelope)

The Elliptic Envelope algorithm assumes that the regular, inlying data comes from a known distribution, typically a Gaussian distribution. It fits a robust covariance estimate to the data, effectively fitting an ellipse to the central data points while identifying points outside this ellipse as outliers. This method is best suited for normally distributed data.

Key Parameters

Common parameters for sklearn.covariance.EllipticEnvelope:

Parameter	Description
store_precision	Boolean, default=True. If True, the estimated precision matrix is stored. This attribute is precision_.
assume_centered	Boolean, default=False. If False, the robust location and covariance are computed using the FastMCD algorithm. If True, the data is assumed to be centered, and the FastMCD algorithm computes the support of the robust location and covariance from this centered data. Setting this to True can be useful if the data is already centered and can speed up computation.
support_fraction	float in (0., 1.], default=None. The proportion of points to be included in the support of the raw MCD estimates. If None, it is set to (n_samples + n_features + 1) / 2 / n_samples. Must be greater than contamination.
contamination	float in (0., 0.5), default=0.1. The proportion of outliers expected in the data set. This defines the threshold on the Mahalanobis distances to label points as outliers.
random_state	int, RandomState instance or None, default=None. Controls the pseudo-randomness of the robust covariance estimation. Pass an int for reproducible output across multiple function calls.

Key Attributes

Important attributes after fitting EllipticEnvelope:

Attribute	Description
location_	array-like, shape (n_features,). Estimated robust location (mean).
covariance_	array-like, shape (n_features, n_features). Estimated robust covariance matrix.
precision_	array-like, shape (n_features, n_features). Estimated pseudo-inverse of the covariance matrix (precision matrix).
support_	array-like, shape (n_samples,). A mask of the observations used to compute the robust estimates of location and shape.
offset_	float. Offset used to define the decision function from the raw scores. decision_function = score_samples - offset_.

Implementation Example:

import numpy as np
from sklearn.covariance import EllipticEnvelope
# Generate some data (for example purposes)
# True covariance matrix of the inliers
true_cov = np.array([[0.8, 0.3],
                     [0.3, 0.4]])
X_inliers = np.random.RandomState(0).multivariate_normal(mean=[0, 0], cov=true_cov, size=200)
# Add some outliers
X_outliers = np.random.RandomState(1).multivariate_normal(mean=[5, 5], cov=[[1,0],[0,1]], size=20)
X = np.vstack([X_inliers, X_outliers])

# Fit the Elliptic Envelope model
cov_model = EllipticEnvelope(random_state=0, contamination=0.1) # contamination approx 20/220
cov_model.fit(X)

# Predict outliers (1 for inlier, -1 for outlier)
# For example, test points [0,0] (expected inlier) and [6,6] (expected outlier)
predictions = cov_model.predict([[0, 0], [6, 6]])
print(f"Predictions: {predictions}")
# Mahalanobis distances (raw scores)
mahal_dist = cov_model.mahalanobis(X)
# print(f"First 5 Mahalanobis distances: {mahal_dist[:5]}")

Expected Output (may vary slightly due to randomness if not fully seeded in data generation):

Predictions: [ 1 -1]

Practical Tip: Elliptic Envelope works well when the inliers are concentrated and follow a Gaussian-like distribution. It might not perform well on multimodal data.

Isolation Forest (ensemble.IsolationForest)

Isolation Forest is an efficient algorithm for outlier detection, especially in high-dimensional datasets. It works by randomly partitioning the data and isolating observations. The underlying idea is that outliers are ‘few and different,’ making them easier to isolate compared to normal points. The number of splittings required to isolate a sample is equivalent to the path length from the root node to the terminating node in a tree structure.

Key Parameters

Common parameters for sklearn.ensemble.IsolationForest:

Parameter	Description
n_estimators	int, default=100. The number of base estimators (trees) in the ensemble.
max_samples	int or float, default=‘auto’. The number of samples to draw from X to train each base estimator. If ‘auto’, then max_samples=min(256, n_samples).
contamination	float or ‘auto’, default=‘auto’. The proportion of outliers in the data set. If ‘auto’, the threshold is determined as in the original paper, which corresponds to 0.1 for novelty=False. For novelty=True, the default is 0.0. When set to a float, it should be in the range (0, 0.5].
max_features	int or float, default=1.0. The number of features to draw from X to train each base estimator.
bootstrap	boolean, default=False. If True, individual trees are fit on random subsets of the training data sampled with replacement. If False, sampling without replacement is performed.
random_state	int, RandomState instance or None, default=None. Controls the pseudo-randomness of the selection of features and split points.
n_jobs	int, default=None. The number of jobs to run in parallel for both fit and predict. None means 1 unless in a joblib.parallel_backend context. -1 means using all processors.
warm_start	bool, default=False. When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new forest.

Key Attributes

Important attributes after fitting IsolationForest:

Attribute	Description
estimators_	list of DecisionTreeClassifier. The collection of fitted sub-estimators.
max_samples_	integer. The actual number of samples used for max_samples.
offset_	float. Offset used to define the decision function from the raw scores. decision_function = score_samples - offset_.

Implementation Example:

from sklearn.ensemble import IsolationForest
import numpy as np

# Generate some data
rng = np.random.RandomState(42)
X_inliers = 0.3 * rng.randn(100, 2)
X_inliers = np.r_[X_inliers + 2, X_inliers - 2]
X_outliers = rng.uniform(low=-4, high=4, size=(20, 2))
X = np.r_[X_inliers, X_outliers]

# Fit the Isolation Forest model
# For newer scikit-learn versions, contamination='auto' behaves like 0.1
# For explicit control, set contamination to a float e.g. 0.1 for 10% outliers
iso_forest = IsolationForest(n_estimators=100, contamination='auto', random_state=42)
iso_forest.fit(X)

# Predict outliers
y_pred = iso_forest.predict(X) # -1 for outliers, 1 for inliers

# Example: check one known inlier and one known outlier point from generation
# (adjust indices based on your data generation if different)
print(f"Prediction for a typical inlier-like point [2,2]: {iso_forest.predict([[2,2]])}")
print(f"Prediction for a typical outlier-like point [-3,-3]: {iso_forest.predict([[-3,-3]])}")

# To get anomaly scores:
# scores = iso_forest.decision_function(X)
# print(f"Decision scores for first 5 points: {scores[:5]}")

Example Output (the specific model parameters might vary in the output):

Prediction for a typical inlier-like point [2,2]: [1]
Prediction for a typical outlier-like point [-3,-3]: [-1]

Practical Tip: Isolation Forest is generally robust and scales well. The contamination parameter is key to tuning its sensitivity.

Local Outlier Factor (LOF) (neighbors.LocalOutlierFactor)

Local Outlier Factor (LOF) is an unsupervised algorithm that computes a score reflecting the degree of abnormality of an observation. It measures the local density deviation of a given data point with respect to its neighbors. Points that have a substantially lower density than their neighbors are considered outliers.

Key Parameters

Common parameters for sklearn.neighbors.LocalOutlierFactor:

Parameter	Description
n_neighbors	int, default=20. Number of neighbors to use by default for k-neighbors queries.
algorithm	{‘auto’, ‘ball_tree’, ‘kd_tree’, ‘brute’}, default=‘auto’. Algorithm used to compute the nearest neighbors.
leaf_size	int, default=30. Leaf size passed to BallTree or KDTree. Can affect the speed of the construction and query, as well as the memory required to store the tree.
metric	str or callable, default=‘minkowski’. Metric to use for distance computation. Default is ‘minkowski’, which results in the standard Euclidean distance when p=2.
p	int, default=2. Parameter for the Minkowski metric. When p=1, this is equivalent to using manhattan_distance (l1), and euclidean_distance (l2) for p=2.
contamination	float or ‘auto’, default=‘auto’. The proportion of outliers in the data set. When ‘auto’, it is set to 0.1. Range is (0, 0.5]. Used when novelty=False.
novelty	bool, default=False. If True, LocalOutlierFactor is used for novelty detection (trained on clean data and predicts on new unseen data). If False, it’s used for outlier detection (identifies outliers within the training data).
n_jobs	int, default=None. The number of parallel jobs to run for neighbors search. None means 1 unless in a joblib.parallel_backend context. -1 means using all processors.

Key Attributes

Important attributes after fitting LocalOutlierFactor:

Attribute	Description
negative_outlier_factor_	numpy array, shape (n_samples,). The opposite of the Local Outlier Factor of the training samples. The higher, the more normal. Inliers tend to have a LOF score close to 1, while outliers have a significantly larger LOF. Scikit-learn stores the negative of these values, so inliers will have values close to -1.
n_neighbors_	integer. The actual number of neighbors used for k-neighbors queries.
offset_	float. Offset used to define the binary labels from the raw scores. decision_function = score_samples - offset_.

Implementation Example (Outlier Detection):

import numpy as np
from sklearn.neighbors import LocalOutlierFactor

# Generate sample data: a cluster of normal points and a few outliers
np.random.seed(42)
X_inliers = 0.5 * np.random.randn(100, 2) + [2, 2]
X_outliers = np.random.uniform(low=-4, high=8, size=(20, 2))
X = np.vstack([X_inliers, X_outliers])

# Initialize and fit LOF model for outlier detection
# contamination='auto' implies 0.1. Adjust as needed.
lof = LocalOutlierFactor(n_neighbors=20, contamination='auto')

# For outlier detection, use fit_predict()
# It fits the model and then predicts labels (-1 for outliers, 1 for inliers)
y_pred = lof.fit_predict(X)

# Example: Print predictions for the first 5 points and last 5 points
# (assuming outliers were appended last in X)
print(f"Predictions for first 5 (likely inliers): {y_pred[:5]}")
print(f"Predictions for last 5 (likely outliers): {y_pred[-5:]}")

# Access the negative outlier factor scores
# Scores close to -1 are more normal; more negative scores (e.g., -2, -5) are more outlying.
# X_scores = lof.negative_outlier_factor_
# print(f"Negative outlier factor for first 5 points: {X_scores[:5]}")
# print(f"Negative outlier factor for last 5 points: {X_scores[-5:]}")

Expected Output (labels might vary slightly based on exact data generation and parameters):

Predictions for first 5 (likely inliers): [1 1 1 1 1]
Predictions for last 5 (likely outliers): [-1 -1 -1 -1 -1]

Practical Tip: LOF is good at finding outliers that are in low-density regions. The choice of n_neighbors can be critical. If novelty=True, it can be used for novelty detection, where fit is called on clean data and predict on new data.

One-Class SVM (svm.OneClassSVM)

One-Class Support Vector Machine is an unsupervised algorithm that learns a decision boundary separating the region of high data density from sparser regions. It is effective in high-dimensional spaces and can capture complex shapes in the data distribution. It requires the data to be ‘clean’ for training (novelty detection setup), or it will try to find a boundary around the majority of the data if used for outlier detection.

It learns a decision function for novelty detection: it classifies new data as similar or different to the training set. Key parameters include kernel (e.g., ‘rbf’, ‘linear’) and nu. The nu parameter (an upper bound on the fraction of training errors and a lower bound of the fraction of support vectors) controls the trade-off between finding a tight boundary and including all training points.

Key Parameters

Common parameters for sklearn.svm.OneClassSVM:

Parameter	Description
kernel	str, default=‘rbf’. Specifies the kernel type to be used in the algorithm (‘linear’, ‘poly’, ‘rbf’, ‘sigmoid’, ‘precomputed’).
degree	int, default=3. Degree of the polynomial kernel function (‘poly’). Ignored by all other kernels.
gamma	{‘scale’, ‘auto’} or float, default=‘scale’. Kernel coefficient for ‘rbf’, ‘poly’ and ‘sigmoid’. If gamma='scale' (default) is passed then it uses 1 / (n_features * X.var()) as value of gamma.
nu	float, default=0.5. An upper bound on the fraction of training errors and a lower bound of the fraction of support vectors. Should be in the interval (0, 1].
tol	float, default=1e-3. Tolerance for stopping criterion.
shrinking	bool, default=True. Whether to use the shrinking heuristic.

Implementation Example:

import numpy as np
from sklearn.svm import OneClassSVM

# Generate some data - mostly clustered, with a few points further away
rng = np.random.RandomState(42)
X_train = 0.3 * rng.randn(100, 2)
X_train = np.r_[X_train + 2, X_train - 2] # Two clusters
X_test_normal = [[2.1, 2.1], [-1.9, -1.9]] # Points similar to training data
X_test_outliers = [[0,0], [5,5]] # Points different from training data

# Fit the One-Class SVM model
# nu=0.1 means we expect about 10% of training data to be outliers/on the wrong side of boundary
# Or, it controls the number of support vectors.
one_class_svm = OneClassSVM(kernel='rbf', gamma='scale', nu=0.1)
one_class_svm.fit(X_train)

# Predict on new data
pred_normal = one_class_svm.predict(X_test_normal)
pred_outliers = one_class_svm.predict(X_test_outliers)

print(f"Predictions for normal-like test points: {pred_normal}")
print(f"Predictions for outlier-like test points: {pred_outliers}")

# Get raw scores (signed distance to the separating hyperplane)
# scores_normal = one_class_svm.score_samples(X_test_normal)
# print(f"Scores for normal-like test points: {scores_normal}")

Expected Output:

Predictions for normal-like test points: [ 1  1]
Predictions for outlier-like test points: [-1 -1]

For more detailed information on each algorithm and further examples, please refer to the official Scikit-Learn documentation on outlier detection.

Choosing the right anomaly detection algorithm and its parameters often depends on the characteristics of your data and the specific definition of an ‘anomaly’ in your context. Experimentation and domain knowledge are key.