Introduction

Classification - Introduction

What is Classification?

Classification is a fundamental task in supervised machine learning. It involves predicting a specific category or class label for a given input data point based on its features. Think of it like sorting objects into predefined bins. The output is a discrete category, such as ‘Spam’ or ‘Not Spam’ for emails, ‘Cat’ or ‘Dog’ for images, or ‘Approved’ or ‘Rejected’ for loan applications.

Mathematically, classification aims to learn a mapping function (f) that takes input variables (X) and predicts discrete output variables (Y). Since it’s supervised learning, the algorithm learns from a dataset where both the input features (X) and the correct class labels (Y) are provided during training.

A common example is email spam detection. The algorithm learns from emails labeled as ‘spam’ or ‘not spam’. The goal is to build a model that can accurately classify new, unseen emails. When there are only two possible categories, like in this case, it’s called binary classification. If there are more than two categories (e.g., classifying news articles into ‘Sports’, ‘Politics’, ‘Technology’), it’s called multi-class classification.

To build a classifier, we first train it on a labeled dataset (the training data). The model learns patterns correlating features with specific classes. After training, we evaluate its performance on a separate dataset (the testing data) that it hasn’t seen before to see how well it generalizes.

Types of Learners in Classification

Classification algorithms can be broadly categorized based on how they learn:

Lazy Learners

Lazy learners, as the name suggests, don’t build a general model during the training phase. They simply store the entire training dataset. When a new data point needs classification, they compare it to the stored data points to make a prediction. This means training is fast, but prediction can be slow, especially with large datasets. Examples include K-Nearest Neighbors (KNN) and Case-Based Reasoning.

Eager Learners

Eager learners construct a classification model based on the training data before receiving any test data. They build a generalized representation of the patterns found. Training takes longer, but predicting new instances is typically much faster. Most common classification algorithms fall into this category, such as Decision Trees, Naïve Bayes, Support Vector Machines (SVM), Logistic Regression, and Artificial Neural Networks (ANN).

Building a Classifier in Python with Scikit-learn

Scikit-learn is the go-to Python library for machine learning. Let’s walk through the steps to build a simple classifier.

Step 1: Import Necessary Libraries

We need libraries for data handling (Pandas), numerical operations (NumPy), data splitting, the classifier itself, and evaluation metrics (Scikit-learn).

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler

Step 2: Load and Prepare the Dataset

We’ll use the Breast Cancer Wisconsin Diagnostic Database, a classic binary classification dataset available in Scikit-learn. It’s good practice to load data into a Pandas DataFrame for easier manipulation.

# Load the dataset
cancer = load_breast_cancer()

# Create a Pandas DataFrame
# Features are in cancer.data, target is in cancer.target
df = pd.DataFrame(data=np.c_[cancer['data'], cancer['target']],
                  columns=np.append(cancer['feature_names'], ['target']))

# Display basic info and first few rows
print('Dataset Shape:', df.shape)
print('\nFirst 5 rows:\n', df.head())

# Separate features (X) and target (y)
X = df[cancer.feature_names]
y = df['target']

# Print class labels (0: malignant, 1: benign)
print('\nTarget names:', cancer.target_names)
print('Class distribution:\n', y.value_counts())

Output Explanation: The code loads the dataset, creates a table (DataFrame) where each row is a sample and columns represent measurements (features) and the final diagnosis (target). It shows the size of the table, the first few examples, the meaning of the target values (0 and 1), and how many samples belong to each class.

Step 3: Split Data into Training and Testing Sets

We need to split our data to train the model on one part and test its performance on another unseen part. This helps estimate how well the model will perform on new data. We use train_test_split. Setting random_state ensures that the split is the same every time we run the code, making results reproducible. Stratifying by y ensures that the proportion of classes is the same in both train and test sets.

# Split data: 80% for training, 20% for testing
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42, stratify=y)

print('\nTraining set shape:', X_train.shape)
print('Testing set shape:', X_test.shape)

Step 3.5: Feature Scaling (Optional but Recommended)

Many algorithms perform better when features are on a similar scale. While Naive Bayes isn’t strictly sensitive to scaling, it’s good practice, especially for algorithms like SVM or Logistic Regression. We’ll scale the data using StandardScaler, which standardizes features by removing the mean and scaling to unit variance.

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Note: We fit the scaler ONLY on the training data and then transform both train and test data.
# This prevents data leakage from the test set into the training process.

Step 4: Train the Classification Model

We’ll use the Gaussian Naïve Bayes (GaussianNB) classifier, a simple and often effective algorithm based on Bayes’ theorem with an assumption of normality (Gaussian distribution) for features.

# Initialize the Gaussian Naive Bayes classifier
gnb = GaussianNB()

# Train the model using the scaled training data
model = gnb.fit(X_train_scaled, y_train)

print('\nModel training complete.')

Step 5: Make Predictions

Now we use the trained model to predict the class labels for the scaled test data.

# Make predictions on the scaled test set
y_pred = model.predict(X_test_scaled)

# Display the first few predictions compared to actual labels
print('\nFirst 10 Predictions:', y_pred[:10])
print('First 10 Actual Labels:', y_test.values[:10])

Step 6: Evaluate the Model

How well did our model perform? We use various metrics to evaluate its predictions against the true labels (y_test).

# Calculate accuracy
accuracy = accuracy_score(y_test, y_pred)
print(f'\nModel Accuracy: {accuracy:.4f}')

# Generate a confusion matrix
print('\nConfusion Matrix:')
cm = confusion_matrix(y_test, y_pred)
print(cm)
# Note: In Scikit-learn's confusion matrix for binary classification:
# Top-left: True Negatives (TN)
# Top-right: False Positives (FP)
# Bottom-left: False Negatives (FN)
# Bottom-right: True Positives (TP)
# Assuming class 1 ('benign') is the positive class

# Generate a detailed classification report
print('\nClassification Report:')
report = classification_report(y_test, y_pred, target_names=cancer.target_names)
print(report)

The accuracy tells us the overall percentage of correct predictions. The confusion matrix gives a detailed breakdown of correct and incorrect predictions for each class. The classification report provides precision, recall, and F1-score for each class, which are often more informative than accuracy, especially for imbalanced datasets.

For more robust evaluation, especially with smaller datasets, consider using Cross-Validation. See Scikit-learn docs: https://scikit-learn.org/stable/modules/cross_validation.html

Understanding Classification Evaluation Metrics

Choosing the right metric is crucial. Simply building a model isn’t enough; we need to know how effective it is. The choice depends on the problem and the consequences of different types of errors.

Confusion Matrix

A confusion matrix provides a table summarizing the performance of a classification model. For a binary classifier, it looks like this:

Predicted Negative | Predicted Positive

-----------------|------------------

Actual Negative | TN (True Negative) | FP (False Positive - Type I Error)

Actual Positive | FN (False Negative - Type II Error) | TP (True Positive)

• True Positives (TP): Correctly predicted positive instances (e.g., actual benign correctly identified as benign).
• True Negatives (TN): Correctly predicted negative instances (e.g., actual malignant correctly identified as malignant).
• False Positives (FP): Incorrectly predicted positive instances (e.g., actual malignant incorrectly identified as benign - a ‘false alarm’).
• False Negatives (FN): Incorrectly predicted negative instances (e.g., actual benign incorrectly identified as malignant - a ‘miss’).

Refer to the Scikit-learn confusion_matrix output in the code example above to see the calculated values.

Accuracy

The proportion of total predictions that were correct. Formula: Accuracy = (TP + TN) / (TP + TN + FP + FN). While intuitive, accuracy can be misleading on imbalanced datasets (where one class is much more frequent than others).

Calculated using accuracy_score(y_test, y_pred).

Precision

Of all instances predicted as positive, what proportion actually is positive? High precision means fewer false positives. Formula: Precision = TP / (TP + FP). Useful when the cost of a false positive is high (e.g., marking a non-spam email as spam).

See the ‘precision’ column in the classification_report output.

Recall (Sensitivity or True Positive Rate)

Of all actual positive instances, what proportion did the model correctly identify? High recall means fewer false negatives. Formula: Recall = TP / (TP + FN). Crucial when the cost of a false negative is high (e.g., failing to detect a malignant tumor).

See the ‘recall’ column in the classification_report output.

F1-Score

The harmonic mean of Precision and Recall. It provides a single score that balances both concerns. Formula: F1 = 2 * (Precision * Recall) / (Precision + Recall). It’s often a better measure than accuracy on imbalanced datasets.

See the ‘f1-score’ column in the classification_report output.

Specificity (True Negative Rate)

Of all actual negative instances, what proportion did the model correctly identify? Formula: Specificity = TN / (TN + FP). It’s the counterpart to recall for the negative class. While not always shown in the default classification_report, it can be calculated from the confusion matrix. High specificity means the model is good at identifying true negatives.

Calculated as TN / (TN + FP) from the confusion_matrix values.

Common Classification Algorithms

Besides Naïve Bayes, other popular and powerful classification algorithms include:

• Logistic Regression: A linear model adapted for binary classification tasks.
• Support Vector Machine (SVM): Finds the optimal hyperplane to separate classes, effective in high-dimensional spaces.
• Decision Tree: A tree-like structure where internal nodes test features and leaf nodes represent class labels.
• Random Forest: An ensemble method using multiple decision trees to improve robustness and accuracy.
• K-Nearest Neighbors (KNN): A lazy learner that classifies based on the majority class of its nearest neighbors.
• Gradient Boosting Machines (e.g., XGBoost, LightGBM): Powerful ensemble methods that build models sequentially, correcting errors of previous models.

We will explore some of these in more detail in later chapters.

Applications of Classification

Classification models are used in countless real-world applications:

• Medical Diagnosis: Identifying diseases based on patient symptoms and test results (e.g., cancer detection).
• Spam Filtering: Classifying emails as spam or not spam.
• Sentiment Analysis: Determining the sentiment (positive, negative, neutral) expressed in text.
• Image Recognition: Categorizing images (e.g., identifying objects, recognizing faces).
• Fraud Detection: Identifying fraudulent transactions or activities.
• Customer Churn Prediction: Predicting whether a customer is likely to stop using a service.
• Document Classification: Assigning documents to predefined categories (e.g., news articles by topic).