Classification with Naïve Bayes

Scikit-learn - Classification with Naïve Bayes

Naïve Bayes classifiers are a family of simple probabilistic classifiers based on applying Bayes’ theorem with a strong (naïve) independence assumption between the features. This means that the presence (or absence) of a particular feature of a class is assumed to be unrelated to the presence (or absence) of any other feature, given the class variable.

Bayes’ theorem provides a way of calculating posterior probability P(Y|features):

P(Y|features) = [P(features|Y) * P(Y)] / P(features)

Where:

• P(Y|features) is the posterior probability of class Y given the observed features.
• P(Y) is the prior probability of class Y.
• P(features|Y) is the likelihood, which is the probability of observing the features given class Y.
• P(features) is the prior probability of the features (evidence).

Due to the independence assumption, the likelihood P(features|Y) can be decomposed into a product of probabilities for each feature: P(feature_1|Y) * P(feature_2|Y) * … * P(feature_n|Y). The classifier then assigns the class label Y that maximizes this posterior probability.

Scikit-learn implements several Naïve Bayes algorithms, differing mainly by the assumptions they make regarding the distribution of P(features|Y).

Types of Naïve Bayes Classifiers in Scikit-learn

Model	Description & Use Case
GaussianNB (sklearn.naive_bayes.GaussianNB)	Assumes that features follow a Gaussian (normal) distribution. Suitable for continuous numerical features that are approximately normally distributed.
MultinomialNB (sklearn.naive_bayes.MultinomialNB)	Assumes features are drawn from a multinomial distribution. Typically used for discrete count data, such as word counts in text classification (e.g., bag-of-words model).
BernoulliNB (sklearn.naive_bayes.BernoulliNB)	Assumes features are binary (0s and 1s, i.e., Boolean variables). Also used for text classification, particularly with binary term occurrence features (word present or absent).
ComplementNB (sklearn.naive_bayes.ComplementNB)	An adaptation of MultinomialNB specifically designed to perform better on imbalanced datasets. It estimates parameters using data from all classes except the one being considered.
CategoricalNB (sklearn.naive_bayes.CategoricalNB)	Assumes features are categorically distributed. Suitable for features that have discrete, unordered categories. Requires features to be encoded numerically (e.g., 0, 1, 2 for different categories).

Building a Naïve Bayes Classifier: GaussianNB Example

Let’s apply a Gaussian Naïve Bayes classifier to the breast_cancer dataset from Scikit-learn. This dataset has continuous features, making GaussianNB a suitable choice if we assume these features are normally distributed within each class.

Example

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

# Load the breast cancer dataset
data = load_breast_cancer()
X, y = data.data, data.target
feature_names = data.feature_names
target_names = data.target_names

print(f"Target names: {target_names}")
print(f"First target label: {y[0]} (corresponds to '{target_names[y[0]]}')")
print(f"First feature name: {feature_names[0]}")
print(f"Values for first sample's first feature: {X[0,0]}")

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.30, random_state=42, stratify=y
)

# Initialize and train the Gaussian Naive Bayes classifier
gnb_clf = GaussianNB()
model = gnb_clf.fit(X_train, y_train)

# Make predictions on the test set
y_pred = gnb_clf.predict(X_test)

# Evaluate the model
accuracy = accuracy_score(y_test, y_pred)
print(f"\nAccuracy: {accuracy:.4f}")

print("\nConfusion Matrix:")
print(confusion_matrix(y_test, y_pred))

print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=target_names))

Output

Target names: ['malignant' 'benign']
First target label: 0 (corresponds to 'malignant')
First feature name: mean radius
Values for first sample's first feature: 17.99

Accuracy: 0.9415

Confusion Matrix:
[[ 59   4]
 [  6 102]]

Classification Report:
              precision    recall  f1-score   support

   malignant       0.91      0.94      0.92        63
      benign       0.96      0.94      0.95       108

    accuracy                           0.94       171
   macro avg       0.93      0.94      0.94       171
weighted avg       0.94      0.94      0.94       171

The output shows the predicted classes for the test samples. accuracy_score gives the overall correctness. The confusion matrix and classification report provide more detailed performance insights per class.

Advantages of Naïve Bayes:

• Computationally fast and easy to implement.
• Requires a small amount of training data to estimate parameters.
• Performs well in multi-class prediction.
• Often works surprisingly well even if the independence assumption is violated, especially for text classification.

Disadvantages of Naïve Bayes:

• The strong independence assumption is often unrealistic in real-world data.
• If a categorical feature has a category in the test data that was not observed in the training data, the model will assign it a zero probability (Laplace smoothing, often applied by default in MultinomialNB and BernoulliNB via the alpha parameter, helps mitigate this for those variants).
• For numerical features, GaussianNB assumes a normal distribution, which might not hold true.

Despite its simplicity and naïve assumptions, Naïve Bayes classifiers are effective for many applications, particularly document classification and spam filtering. For more details and other Naive Bayes variants, consult the Scikit-learn documentation: https://scikit-learn.org/stable/modules/naive_bayes.html