Supervised Learning: Classification

AI with Python – Supervised Learning: Classification

This chapter focuses on supervised learning, specifically classification tasks. Classification is a type of supervised machine learning where the goal is to predict a categorical class label for a given input.

A classification model learns from observed data (features) and their corresponding true labels to make predictions on new, unseen data. The output is a discrete category, such as “Spam” or “Not Spam,” “Cat” or “Dog,” or medical diagnoses like “Malignant” or “Benign.” To build a classification model, a labeled training dataset is essential. Common applications include image recognition, medical diagnosis, sentiment analysis, and fraud detection.

Steps for Building a Classifier in Python with Scikit-learn

We will use Python 3 and the Scikit-learn library, a powerful and popular toolkit for machine learning in Python. Here’s a general workflow:

Step 1: Install Scikit-learn

If you haven’t already, install Scikit-learn. The most common way is using pip:

pip install scikit-learn

If you use Anaconda, you can install it via conda:

conda install scikit-learn

Step 2: Load a Dataset

Scikit-learn comes with several built-in datasets. For this example, we’ll use the Breast Cancer Wisconsin Diagnostic Database. This dataset contains information about breast cancer tumors, with labels classifying them as malignant or benign.

The dataset has 569 instances (tumors) and 30 features (attributes like radius, texture, smoothness, area).

from sklearn.datasets import load_breast_cancer

# Load the dataset
cancer_data = load_breast_cancer()

Scikit-learn datasets are often Bunch objects, which are dictionary-like. Key attributes include:

• data: The feature matrix (NumPy array).
• target: The array of labels (NumPy array).
• feature_names: Names of the features.
• target_names: Names of the target classes.

Let’s assign these to variables for clarity:

features = cancer_data.data
labels = cancer_data.target
feature_names = cancer_data.feature_names
target_names = cancer_data.target_names

Let’s inspect some of this data:

print("Target names:", target_names)
print("First 5 labels:", labels[:5]) # 0 for malignant, 1 for benign
print("First feature name:", feature_names[0])
print("First instance's features:\n", features[0])

Example output:

Target names: ['malignant' 'benign']
First 5 labels: [0 0 0 0 0]
First feature name: mean radius
First instance's features:
 [1.799e+01 1.038e+01 1.228e+02 1.001e+03 1.184e-01 2.776e-01
  3.001e-01 1.471e-01 2.419e-01 7.871e-02 1.095e+00 9.053e-01
  8.589e+00 1.534e+02 6.399e-03 4.904e-02 5.373e-02 1.587e-02
  3.003e-02 6.193e-03 2.538e+01 1.733e+01 1.846e+02 2.019e+03
  1.622e-01 6.656e-01 7.119e-01 2.654e-01 4.601e-01 1.189e-01]

This shows the first tumor is labeled as ‘malignant’ (0) and its ‘mean radius’ is approximately 17.99.

Step 3: Split Data into Training and Testing Sets

It’s crucial to evaluate your model on data it hasn’t seen during training. We split the dataset into a training set (for model learning) and a test set (for model evaluation). Scikit-learn’s train_test_split function is used for this.

from sklearn.model_selection import train_test_split

# Split data: 70% for training, 30% for testing.
# random_state ensures reproducibility.
X_train, X_test, y_train, y_test = train_test_split(
    features, labels, test_size=0.30, random_state=42
)

print(f"Training set size: {X_train.shape[0]} samples")
print(f"Test set size: {X_test.shape[0]} samples")

A common split is 70-80% for training and 20-30% for testing.

Step 4: Choose and Train a Model

We’ll use the Gaussian Naïve Bayes (GNB) algorithm as our first classifier. It’s a simple yet often effective algorithm based on Bayes’ theorem with an assumption of feature independence.

from sklearn.naive_bayes import GaussianNB

# Initialize the Gaussian Naïve Bayes classifier
gnb_model = GaussianNB()

# Train the model using the training data
gnb_model.fit(X_train, y_train)
print("Model training complete.")

Step 5: Evaluate the Model

After training, we use the test set to evaluate the model’s performance. The predict() method makes predictions on the test features.

# Make predictions on the test set
y_pred = gnb_model.predict(X_test)

print("First 10 predictions:", y_pred[:10])
print("First 10 actual labels:", y_test[:10])

To quantify performance, we use evaluation metrics. Accuracy is a common one:

from sklearn.metrics import accuracy_score

accuracy = accuracy_score(y_test, y_pred)
print(f"Model Accuracy: {accuracy * 100:.2f}%")

An accuracy of around 90-95% is typical for this dataset with GNB. For instance, an output might be Model Accuracy: 94.15%.

This structured approach—load data, preprocess (if needed), split, train, evaluate—is fundamental to building machine learning models.

Common Classification Algorithms

Let’s explore a few more widely used classification algorithms available in Scikit-learn.

Naïve Bayes Classifier (Revisited)

Naïve Bayes classifiers are a family of simple probabilistic classifiers based on applying Bayes’ theorem with strong (naïve) independence assumptions between the features. Despite their simplicity, they can perform well, especially with high-dimensional data like text.

Scikit-learn offers several types: GaussianNB (for continuous features assuming Gaussian distribution, as used above), MultinomialNB (for discrete counts, e.g., word counts in text), and BernoulliNB (for binary/boolean features).

The previous example demonstrated GaussianNB on the Breast Cancer dataset. The steps remain the same: initialize the model, fit it with training data (X_train, y_train), predict on test data (X_test), and evaluate.

Support Vector Machines (SVM)

Support Vector Machines (SVMs) are powerful and versatile supervised learning models used for classification, regression, and outlier detection. For classification, an SVM aims to find a hyperplane in an N-dimensional space (where N is the number of features) that best separates the data points of different classes.

The ‘support vectors’ are the data points closest to the hyperplane, which, if removed, would alter the position of the hyperplane. SVMs can handle linear and non-linear separation using various kernels (e.g., linear, polynomial, RBF, sigmoid).

Let’s build an SVM classifier using the Iris dataset, which has 3 classes of iris plants and 4 features (sepal length/width, petal length/width). For visualization ease, we’ll use only the first two features.

import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm, datasets

# Load Iris dataset
iris = datasets.load_iris()
X = iris.data[:, :2]  # We only take the first two features for 2D visualization
y = iris.target

# Create an SVM Classifier with a linear kernel
# C is the regularization parameter
C = 1.0
svc_model = svm.SVC(kernel='linear', C=C, decision_function_shape='ovr').fit(X, y)

# Create a mesh to plot the decision boundaries
x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
h = 0.02  # step size in the mesh
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))

# Plot decision boundary. For that, we will assign a color to each point in the mesh [x_min, x_max]x[y_min, y_max].
Z = svc_model.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

plt.figure(figsize=(8, 6))
plt.contourf(xx, yy, Z, cmap=plt.cm.coolwarm, alpha=0.8)

# Plot also the training points
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.coolwarm, edgecolors='k')
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.xticks(())
plt.yticks(())
plt.title('SVC with linear kernel on Iris dataset (first two features)')
plt.show()

This code will display a scatter plot of the Iris data (sepal length vs. sepal width), with different classes shown in different colors. The regions separated by lines (decision boundaries) indicate the classification zones predicted by the SVM model.

Kernels like ‘rbf’ (Radial Basis Function) can be used for non-linearly separable data: svm.SVC(kernel='rbf', C=C, gamma='auto').

Logistic Regression

Despite its name, Logistic Regression is a widely used algorithm for binary and multi-class classification. It models the probability that a given input point belongs to a particular class. This probability is then transformed using a logistic (or sigmoid) function to output a value between 0 and 1.

The dependent variable is the target class we are predicting, and independent variables are the features. For example, predicting if a customer will click an ad (binary: yes/no) based on features like age, browsing history, etc.

Example with Logistic Regression:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

# Generate synthetic data for classification
X_synthetic, y_synthetic = make_classification(
    n_samples=100, n_features=2, n_redundant=0,
    n_informative=2, random_state=1, n_clusters_per_class=1
)

# Create and train the Logistic Regression classifier
# 'solver' and 'C' are important hyperparameters
# 'liblinear' is good for small datasets, 'lbfgs' is a common default
log_reg_model = LogisticRegression(solver='liblinear', C=1.0, random_state=42)
log_reg_model.fit(X_synthetic, y_synthetic)

# Function to visualize decision boundaries (similar to SVM example)
def plot_decision_boundary(model, X, y, title):
    x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
    y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
    h = .02  # step size in the mesh
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))
    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    plt.figure(figsize=(8, 6))
    plt.contourf(xx, yy, Z, cmap=plt.cm.RdYlBu, alpha=0.8)
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.RdYlBu, edgecolors='k')
    plt.title(title)
    plt.xlabel('Feature 1')
    plt.ylabel('Feature 2')
    plt.show()

plot_decision_boundary(log_reg_model, X_synthetic, y_synthetic, 'Logistic Regression Decision Boundary')

The plot will show the data points and the decision boundary learned by the Logistic Regression model, separating the two classes in the synthetic dataset.

Decision Tree Classifier

A Decision Tree is a non-parametric supervised learning method used for classification and regression. It models decisions as a tree structure. Each internal node represents a ‘test’ on an attribute (e.g., is color red?), each branch represents the outcome of the test, and each leaf node represents a class label (decision taken after computing all attributes).

Example: Predicting gender based on height and hair length (a toy dataset).

For visualizing the tree structure itself (not just boundaries), graphviz can be used. Ensure it’s installed (pip install graphviz pydotplus). You also need to install the Graphviz software (graphviz.org).

from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.model_selection import train_test_split
import numpy as np

# Toy dataset: [height (cm), hair length (cm)]
X_gender = np.array([
    [165, 19], [175, 32], [136, 35], [174, 65], [141, 28],
    [176, 15], [131, 32], [166,  6], [128, 32], [179, 10],
    [136, 34], [186,  2], [126, 25], [176, 28], [112, 38],
    [169,  9], [171, 36], [116, 25], [196, 25]
])
# Labels: 0 for 'Man', 1 for 'Woman'
Y_gender = np.array([0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0])
data_feature_names = ['height', 'length of hair']
class_names_gender = ['Man', 'Woman']

# Split data (optional for this small dataset, but good practice)
X_train_g, X_test_g, y_train_g, y_test_g = train_test_split(
    X_gender, Y_gender, test_size=0.30, random_state=5
)

# Create and fit the Decision Tree model
dt_model = DecisionTreeClassifier(random_state=42, max_depth=3) # max_depth limits tree size
dt_model.fit(X_train_g, y_train_g)

# Make a prediction
sample_pred = dt_model.predict([[133, 37]])
print(f"Prediction for [133cm, 37cm hair]: {class_names_gender[sample_pred[0]]}")

# Display the tree structure as text
tree_rules = export_text(dt_model, feature_names=data_feature_names, class_names=class_names_gender)
print("\nDecision Tree Rules:\n", tree_rules)

# Evaluate accuracy
accuracy_dt = dt_model.score(X_test_g, y_test_g)
print(f"Decision Tree Accuracy: {accuracy_dt*100:.2f}%")

The output will include a prediction (e.g., ‘Woman’) and a text-based representation of the decision tree rules, showing how features are used to make splits. For example:

Prediction for [133cm, 37cm hair]: Woman

Decision Tree Rules:
|--- length of hair <= 26.50
|   |--- height <= 170.00
|   |   |--- class: Man
|   |--- height > 170.00
|   |   |--- class: Man
|--- length of hair > 26.50
|   |--- height <= 140.00
|   |   |--- class: Woman
|   |--- height > 140.00
|   |   |--- class: Woman
Decision Tree Accuracy: 83.33%

If you have graphviz and pydotplus installed, you can visualize it as an image: from sklearn.tree import export_graphviz; import pydotplus; dot_data = export_graphviz(...); graph = pydotplus.graph_from_dot_data(dot_data); graph.write_png('decision_tree.png').

Random Forest Classifier

Random Forest is an ensemble learning method that operates by constructing multiple decision trees at training time. For classification, the output is the class selected by most trees. It’s generally more robust and accurate than a single decision tree, reducing overfitting by averaging results.

We’ll use the Breast Cancer dataset again.

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer
import matplotlib.pyplot as plt
import numpy as np

# Load data
cancer = load_breast_cancer()
X_rf, y_rf = cancer.data, cancer.target

# Split data
X_train_rf, X_test_rf, y_train_rf, y_test_rf = train_test_split(X_rf, y_rf, random_state=0)

# Create and train Random Forest model
# n_estimators is the number of trees in the forest
rf_model = RandomForestClassifier(n_estimators=100, random_state=0, max_depth=5)
rf_model.fit(X_train_rf, y_train_rf)

# Evaluate accuracy
acc_train_rf = rf_model.score(X_train_rf, y_train_rf)
acc_test_rf = rf_model.score(X_test_rf, y_test_rf)
print(f'Random Forest - Training Accuracy: {acc_train_rf*100:.2f}%')
print(f'Random Forest - Test Accuracy: {acc_test_rf*100:.2f}%')

# Feature importances
importances = rf_model.feature_importances_
indices = np.argsort(importances)[::-1] # Sort features by importance

print("\nTop 10 Feature Importances:")
for i in range(10):
    print(f"{i+1}. Feature '{cancer.feature_names[indices[i]]}': {importances[indices[i]]:.4f}")

# Plotting feature importances (textual description)
plt.figure(figsize=(10,8))
plt.title('Feature Importances in Random Forest')
plt.barh(range(10), importances[indices][:10][::-1], align='center') # Plot top 10, reversed for plot order
plt.yticks(range(10), cancer.feature_names[indices][:10][::-1])
plt.xlabel('Relative Importance')
plt.tight_layout()
plt.show()

The output will show training and test accuracies (e.g., Training: ~99-100%, Test: ~96-97%). It will also list the most important features identified by the model. The plot would visualize these feature importances as a horizontal bar chart, with longer bars indicating more important features such as ‘worst concave points’, ‘worst perimeter’, ‘worst radius’.

Performance Metrics for Classifiers

After training a classifier, we need to evaluate its effectiveness. Several metrics are used, often derived from a Confusion Matrix.

Confusion Matrix

A confusion matrix is a table used to describe the performance of a classification model on a set of test data for which the true values are known. It compares the actual target values with those predicted by the machine learning model.

For a binary classification problem (e.g., Positive/Negative or 1/0):

Predicted: Positive (1) Negative (0)

Actual Positive (1): TP (True Positive) FN (False Negative)

Actual Negative (0): FP (False Positive) TN (True Negative)

• True Positives (TP): Correctly predicted positive instances.
• True Negatives (TN): Correctly predicted negative instances.
• False Positives (FP): Incorrectly predicted positive instances (Type I error).
• False Negatives (FN): Incorrectly predicted negative instances (Type II error).

from sklearn.metrics import confusion_matrix, classification_report

# Using the GaussianNB model from the first example
# y_test, y_pred were calculated earlier

cm = confusion_matrix(y_test, y_pred)
print("Confusion Matrix (GNB Model):")
print(cm)

# For more detailed metrics:
print("\nClassification Report (GNB Model):")
print(classification_report(y_test, y_pred, target_names=target_names))

Example Confusion Matrix output for GNB on breast cancer data:

Confusion Matrix (GNB Model):
[[ 59   5]
 [  5 102]]
This means: TP=102 (benign correctly id'd), TN=59 (malignant correctly id'd), FP=5 (malignant misclassified as benign), FN=5 (benign misclassified as malignant). (Assuming benign is positive class)

The Classification Report provides precision, recall, F1-score, and support for each class.

Accuracy

The proportion of correct predictions among the total number of cases evaluated.

$$Accuracy = \frac{TP+TN}{TP+FP+FN+TN}$$

Precision

Of all instances predicted as positive, what proportion was actually positive? High precision relates to a low false positive rate.

$$Precision = \frac{TP}{TP+FP}$$

Recall (Sensitivity or True Positive Rate)

Of all actual positive instances, what proportion did the model correctly identify? High recall relates to a low false negative rate.

$$Recall = \frac{TP}{TP+FN}$$

Specificity (True Negative Rate)

Of all actual negative instances, what proportion did the model correctly identify?

$$Specificity = \frac{TN}{TN+FP}$$

F1-Score

The harmonic mean of Precision and Recall. It’s useful when you need a balance between Precision and Recall, especially if there’s an uneven class distribution.

$$F1\ Score = 2 * \frac{Precision * Recall}{Precision + Recall}$$

Handling Class Imbalance

Class imbalance occurs when the number of observations in one class is significantly lower (minority class) or higher (majority class) than in other classes. This is common in real-world problems like fraud detection, rare disease identification, or defect detection in manufacturing.

Standard classifiers trained on imbalanced data often perform poorly on the minority class because they are biased towards the majority class (as this maximizes overall accuracy).

Example of Imbalanced Classes

Dataset: Credit Card Fraud Detection
Total transactions = 100,000
Fraudulent transactions (minority class) = 100 (0.1%)
Non-fraudulent transactions (majority class) = 99,900 (99.9%)
Event Rate (fraud) = 0.1%

A model that always predicts “non-fraudulent” would achieve 99.9% accuracy but would be useless for detecting fraud.

Techniques to Handle Class Imbalance

1. Resampling Techniques

Resampling aims to balance the class distribution in the training dataset.

• Random Under-Sampling: Randomly remove instances from the majority class. Reduces dataset size, potentially losing useful information.
• Random Over-Sampling: Randomly duplicate instances from the minority class. Increases dataset size, can lead to overfitting on the minority class.
• Synthetic Minority Over-sampling Technique (SMOTE): Creates synthetic samples for the minority class by interpolating between existing minority instances. More sophisticated than random oversampling.
• ADASYN (Adaptive Synthetic Sampling): Similar to SMOTE, but generates more synthetic data for minority class samples that are harder to learn.

The imbalanced-learn library (pip install imbalanced-learn) provides implementations for these techniques (e.g., RandomUnderSampler, RandomOverSampler, SMOTE).

2. Algorithmic Approaches (Cost-Sensitive Learning)

Modify existing algorithms to account for class imbalance. This often involves assigning higher misclassification costs to the minority class. Many Scikit-learn classifiers (e.g., SVC, LogisticRegression, RandomForestClassifier) have a class_weight parameter that can be set to 'balanced' or a custom dictionary of weights.

# Example with RandomForestClassifier
rf_balanced_model = RandomForestClassifier(n_estimators=100, class_weight='balanced', random_state=0)
# Or custom weights: class_weight={0:1, 1:10} if class 1 is minority
# Then train and evaluate as usual.

3. Ensemble Techniques

Ensemble methods like Random Forest or Gradient Boosting can be adapted for imbalanced data. Techniques like Balanced Random Forest or EasyEnsemble/BalanceCascade specifically address imbalance.

4. Use Appropriate Evaluation Metrics

Accuracy is misleading for imbalanced datasets. Focus on metrics like Precision, Recall, F1-Score (especially for the minority class), Area Under the ROC Curve (AUC-ROC), and Area Under the Precision-Recall Curve (AUC-PR).

Addressing class imbalance is often crucial for building effective real-world classifiers.