Naive Bayes Classifier: Gaussian, Multinomial & Bernoulli

The Naive Bayes Classifier is a family of probabilistic supervised algorithms rooted in Bayes' Theorem. Despite its simplicity and an assumption that is almost always technically violated in the real world, Naive Bayes is blisteringly fast, scales to billions of records, and remains a top baseline for NLP text classification and spam detection.

Why is it Called "Naive"? The Independence Assumption

Bayes' Theorem requires calculating the joint likelihood of observing all features together given class C: P(x₁, x₂, ..., xᵥ | C). In real data, features interact and depend on each other (e.g., words like "Credit" and "Card" frequently co-occur).

The algorithm makes the "Naive" assumption that all features are strictly conditionally independent given the class label. This allows us to decompose the joint probability into a simple product of individual probabilities:

P(x₁, x₂, ..., xᵥ | C) = P(x₁ | C) · P(x₂ | C) · ... · P(xᵥ | C) = ∏ P(xᵢ | C)

The Three Main Naive Bayes Variants

Variant	Feature Data Type	Probability Distribution Used	Primary Use Case
Gaussian Naive Bayes	Continuous numerical features (e.g., Blood pressure, Height)	Assumes features follow a Gaussian bell curve `N(μ, σ²)`.	Medical diagnosis, sensor telemetry.
Multinomial Naive Bayes	Discrete counts and frequencies (e.g., Word occurrences)	Multinomial distribution over word count vectors.	Text categorization, sentiment analysis, spam filters.
Bernoulli Naive Bayes	Binary binary indicators (0 or 1: word present or absent)	Multivariate Bernoulli distribution.	Short text categorization (e.g., Twitter tweet analysis).

The Zero-Frequency Problem & Laplace Smoothing

If a word (e.g., "cryptocurrency") appears in a test email but was never seen in the training spam dataset, its conditional probability P("cryptocurrency" | Spam) = 0. Because probabilities are multiplied together, a single zero wipes out all other evidence, resulting in an overall probability of 0%!

To fix this, we apply Laplace Smoothing (Add-1 Smoothing) by adding a small smoothing constant α = 1 to the numerator, guaranteeing no probability is ever zero:

P(word | C) = (Count(word, C) + α) / (Count(total_words, C) + α·|V|)

Python Implementation: Building a Text Spam Classifier

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline

# Training Corpus
documents = [
    "Exclusive prize! Claim your free cash reward now",
    "URGENT: Your account has been compromised, reset password",
    "Hey, are we still meeting for lunch tomorrow at noon?",
    "Project updates and minutes from this morning's team call",
    "Win a brand new car today, text WIN to 55555"
]
labels = ["spam", "spam", "ham", "ham", "spam"]

# Build pipeline: Bag-of-Words Vectorizer + Multinomial Naive Bayes
spam_detector = make_pipeline(CountVectorizer(), MultinomialNB(alpha=1.0))
spam_detector.fit(documents, labels)

# Test on novel unseen messages
queries = [
    "Hello team, can you review the attached project document?",
    "Claim your cash prize today"
]
predictions = spam_detector.predict(queries)
print("Query Predictions:", predictions)
# Output: ['ham', 'spam']

Frequently Asked Questions (FAQ) - Student & Interview Review

Q1. Why does Naive Bayes work so well despite its unrealistic independence assumption?

Classification only requires ranking the highest probability, not exact probability calibration. Even if feature correlations skew the exact probability, the relative order of classes usually remains correct.

Q2. How does Naive Bayes handle missing features at test time?

Naive Bayes handles missing features gracefully by simply omitting that feature from the product calculation, without requiring imputation.

Q3. What is the computational complexity of training Naive Bayes?

Training complexity is O(N * d), which is purely linear. It only requires a single pass over the dataset to compute means, variances, or count histograms.