Naïve Bayes

Classification Algorithms - Naïve Bayes

Introduction to Naïve Bayes Algorithm

Naïve Bayes is a probabilistic classification algorithm based on Bayes’ Theorem. It’s called ‘naïve’ because it makes a strong (and often unrealistic) assumption: that all predictor features are independent of each other given the class label.

Despite this simplification, Naïve Bayes classifiers perform surprisingly well in many real-world scenarios, particularly in text classification (like spam filtering) and medical diagnosis.

The core idea is to use Bayes’ Theorem to calculate the probability of a data point belonging to a particular class, given its features:

P(Class | Features) = [P(Features | Class) * P(Class)] / P(Features)

Where:

P(Class | Features): Posterior Probability - The probability of the class given the observed features (what we want to find).
P(Features | Class): Likelihood - The probability of observing the features given the class. This is where the ‘naïve’ independence assumption simplifies calculation: P(feature1, feature2, ... | Class) ≈ P(feature1 | Class) * P(feature2 | Class) * ...
P(Class): Prior Probability - The overall probability of the class (based on its frequency in the training data).
P(Features): Evidence - The overall probability of observing the features. Since this is constant for all classes for a given data point, it’s often ignored during classification (we just compare the numerators).

The algorithm calculates the posterior probability for each class and assigns the data point to the class with the highest probability.

Types of Naïve Bayes Classifiers

Scikit-learn provides several Naïve Bayes implementations, differing mainly in their assumptions about the distribution of P(Features | Class):

Gaussian Naïve Bayes (GaussianNB): Assumes that features (given the class) follow a Gaussian (normal) distribution. Suitable for continuous numerical features.
Multinomial Naïve Bayes (MultinomialNB): Assumes features represent counts or frequencies (e.g., word counts in text classification). Suitable for discrete features.
Bernoulli Naïve Bayes (BernoulliNB): Assumes features are binary (0 or 1, representing presence or absence). Often used for text classification with binary term occurrence features.
Complement Naïve Bayes (ComplementNB): An adaptation of MultinomialNB particularly suited for imbalanced datasets.
Categorical Naïve Bayes (CategoricalNB): Designed for categorically distributed features.

The choice of classifier depends on the nature of your features.

Building a Model using Gaussian Naïve Bayes in Python

Let’s implement a Gaussian Naïve Bayes classifier using Scikit-learn. We’ll generate some synthetic data for demonstration.

Example

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

# 1. Generate synthetic data (blobs with Gaussian distribution)
# Using 2 features for easy visualization
X, y = make_blobs(n_samples=300, centers=2, n_features=2,
                  random_state=42, cluster_std=1.5)

# 2. Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# 3. Initialize and train the GaussianNB model
model_gnb = GaussianNB()
model_gnb.fit(X_train, y_train)

# 4. Make predictions on the test set
y_pred = model_gnb.predict(X_test)

# 5. Evaluate the model
accuracy = accuracy_score(y_test, y_pred)
conf_matrix = confusion_matrix(y_test, y_pred)
class_report = classification_report(y_test, y_pred)

print(f"Accuracy: {accuracy:.4f}\n")
print("Confusion Matrix:")
print(conf_matrix)
print("\nClassification Report:")
print(class_report)

# 6. Optional: Visualize the decision boundary (for 2D data)
plt.figure(figsize=(10, 6))

# Plot training points
sns.scatterplot(x=X_train[:, 0], y=X_train[:, 1], hue=y_train, palette='viridis', marker='o', label='Train Data')

# Create a mesh grid for plotting the decision boundary
ax = plt.gca()
xlim = ax.get_xlim()
ylim = ax.get_ylim()
xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], 100),
                   np.linspace(ylim[0], ylim[1], 100))
Z = model_gnb.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 1]
Z = Z.reshape(xx.shape)

# Plot decision boundary and margins
plt.contourf(xx, yy, Z, levels=[0, 0.5, 1], cmap='viridis', alpha=0.3)
plt.contour(xx, yy, Z, levels=[0.5], colors='black', linestyles='--')

plt.title('Gaussian Naive Bayes Decision Boundary')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.legend(loc='upper right')
plt.grid(True)
plt.show()

# 7. Optional: Show posterior probabilities for some test points
probabilities = model_gnb.predict_proba(X_test)
print("\nPosterior Probabilities for first 5 test points (Class 0, Class 1):")
print(np.round(probabilities[:5], 3))

Output (Illustrative)

Accuracy: 1.0000

Confusion Matrix:
[[45  0]
 [ 0 45]]

Classification Report:
              precision    recall  f1-score   support

           0       1.00      1.00      1.00        45
           1       1.00      1.00      1.00        45

    accuracy                           1.00        90
   macro avg       1.00      1.00      1.00        90
weighted avg       1.00      1.00      1.00        90


Posterior Probabilities for first 5 test points (Class 0, Class 1):
[[1.    0.   ]
 [0.    1.   ]
 [1.    0.   ]
 [0.    1.   ]
 [0.    1.   ]]

The output shows perfect accuracy on this simple, well-separated synthetic dataset. The confusion matrix confirms no misclassifications. The classification report provides precision, recall, and F1-score per class. The plot visualizes the data points and the linear decision boundary found by the Gaussian Naïve Bayes classifier. The posterior probabilities show the model’s confidence in its predictions for the first few test samples.

Pros & Cons

Pros

Fast and Efficient: Very quick to train and predict, even on large datasets.
Requires Less Data: Performs reasonably well even with small amounts of training data.
Scalable: Scales linearly with the number of features and data points.
Handles High Dimensions: Works well with a large number of features (e.g., text classification).
Probabilistic: Naturally provides probability estimates for predictions.
Handles Different Feature Types: Different variants handle continuous, discrete, and binary features.
Good for Multi-class: Works inherently for multi-class classification problems.

Cons

‘Naïve’ Independence Assumption: The core assumption of feature independence is often violated in real-world data, which can sometimes limit performance.
‘Zero Frequency’ Problem: If a specific feature value or category doesn’t appear in the training data for a given class, the model might assign it zero probability, potentially dominating the overall calculation. Techniques like Laplace smoothing (additive smoothing) are used to mitigate this, and are often implemented by default in libraries.
Sensitivity to Feature Distribution: Performance depends on how well the assumed feature distribution (Gaussian, Multinomial, etc.) matches the actual data distribution.

Applications of Naïve Bayes Classification

Despite its simplicity, Naïve Bayes is effective in various applications:

Text Classification: Spam filtering (identifying spam emails), sentiment analysis (classifying text as positive/negative/neutral), document categorization.
Medical Diagnosis: Predicting the probability of a disease based on symptoms (features).
Real-time Prediction: Its speed makes it suitable for applications requiring fast predictions.
Recommendation Systems: Can be used as part of a hybrid recommendation system.

Further Resources

Scikit-learn Naïve Bayes Documentation: https://scikit-learn.org/stable/modules/naive_bayes.html
StatQuest: Naïve Bayes Explained: https://statquest.org/video-index/ (Search for Naive Bayes)