Q12Machine Learning

Naive Bayes is a highly aggressive, probabilistic classification architecture strictly founded upon Bayes' Theorem. Unlike Logistic Regression, which attempts to find a mathematical boundary between classes, Naive Bayes is a Generative Model. It mathematically calculates the absolute probability of a data point belonging to a specific class, and violently assigns the point to the class with the highest probability (Maximum A Posteriori - MAP).

The Mathematical Theorem

Bayes' Theorem is mathematically defined as: $$P(Y|X)=\frac{P(X|Y)\cdot P(Y)}{P(X)}$$

• $P(Y|X)$: The Posterior Probability (The probability that an email is Spam $Y$, given that it contains the word "Viagra" $X$).
• $P(X|Y)$: The Likelihood (The probability of seeing the word "Viagra" given that the email is already known to be Spam).
• $P(Y)$: The Prior Probability (The absolute overall percentage of emails that are Spam in the dataset).

The "Naive" Assumption

If the input $X$ contains 100 features (words), calculating the exact likelihood $P(x_1,x_2...x_{100}|Y)$ is mathematically impossible due to exponential combinations. The algorithm makes a "Naive" (and physically incorrect) assumption: it violently assumes every single feature is completely mathematically independent of every other feature.

$$P(X|Y)\approx P(x_1|Y)\times P(x_2|Y)\times ...\times P(x_n|Y)$$

Applications

Despite this horrific logical flaw (words in a sentence are obviously highly dependent), the mathematical multiplication works phenomenally well for classification. It is the absolute industry standard for Natural Language Processing (NLP), specifically Spam Filtering, Sentiment Analysis, and Document Categorization, because it executes blisteringly fast on massive datasets and requires very little training data.