NLP Text Preprocessing: TF-IDF, N-grams & Word Embeddings

Natural Language Processing (NLP) bridges human linguistics and computational machine learning. Because computers cannot perform mathematical matrix operations on raw strings of text, NLP transforms unstructured prose into structured numerical vectors through Text Preprocessing and Vectorization.

Figure 35.1: The Text Vectorization Pipeline — From Raw Text to Dense Semantic Vector Spaces.

The 4-Stage Text Preprocessing Pipeline

Normalization & Cleaning: Lowercasing text, stripping HTML tags, removing punctuation, and expanding contractions.
Tokenization: Breaking continuous strings into discrete linguistic units (words, subwords, or punctuation tokens).
Stopword Removal: Removing high-frequency, low-information grammatical words (e.g., "the", "is", "at", "which").
Stemming vs. Lemmatization:
- Stemming (Porter / Snowball): Crude heuristic rule-based chopping of word suffixes (e.g., "running", "runs" ➞ "run"; but "caring" ➞ "car" [Error!]).
- Lemmatization (WordNet): Morphological dictionary lookup that uses parts of speech to return the valid base lemma (e.g., "better" ➞ "good", "caring" ➞ "care").

TF-IDF: Term Frequency - Inverse Document Frequency

The Bag-of-Words (BoW) model simply counts word occurrences, but common words dominate the counts. TF-IDF solves this by scoring words based on how informative they are across an entire corpus:

TF-IDF Mathematical Formula

TF-IDF(t, d, D) = TF(t, d) · IDF(t, D)

TF(t, d) [Term Frequency]: Count(t in d) / Total_words(d). Measures how often word t appears in document d.
IDF(t, D) [Inverse Document Frequency]: log( Total_docs(D) / (1 + Count_docs_containing(t)) ). Punishes words that appear in every document (like "the"), while heavily weighting words that only appear in specific topics (like "quantum" or "leukemia").

From Sparse BoW to Dense Word Embeddings (Word2Vec)

TF-IDF vectors are high-dimensional (20,000+ columns) and sparse (99% zeros). Furthermore, they completely ignore word meanings: to TF-IDF, "king" and "queen" have zero similarity! Dense Embeddings (Word2Vec, GloVe) map words into continuous low-dimensional geometric vector spaces (typically 100 to 300 dimensions) where semantic relationships are captured by spatial proximity: Vector("King") - Vector("Man") + Vector("Woman") ≈ Vector("Queen")!

Frequently Asked Questions (FAQ) - Student & Interview Review

Q1. What is an N-Gram and why use them?

An N-Gram is a contiguous sequence of n tokens. Unigrams are single words ('not', 'good'); Bigrams combine pairs ('not good'). Bigrams capture negations that completely reverse sentiment in Bag-of-Words models.

Q2. What is Byte-Pair Encoding (BPE)?

BPE is the modern subword tokenization algorithm used by GPT and modern LLMs. It iteratively merges the most frequent pairs of characters into subwords, eliminating Out-of-Vocabulary (OOV) errors.

Q3. What is the difference between Word2Vec and modern Transformer embeddings?

Word2Vec produces static embeddings (the word 'bank' has the exact same vector in 'river bank' and 'bank deposit'). Transformers produce contextual embeddings that change based on surrounding sentence context.