Natural Language Processing: Text Preprocessing and Subword Tokenization

Computers cannot understand human language directly; they operate purely on numbers and matrix transformations. Natural Language Processing (NLP) is the branch of AI that enables machines to ingest, parse, understand, and generate human languages. The foundational bridge converting continuous unstructured human text into discrete numeric tokens is Tokenization.

The Evolution of Text Preprocessing

Historically, classical NLP utilized rule-based linguistic pipelines:

Sentence Segmentation: Dividing raw prose into distinct sentences using punctuation heuristics.
Word Tokenization: Splitting sentences on whitespace characters. Flaw: Fails on contractions ("don't"), compound words ("state-of-the-art"), and languages without spaces (Chinese, Japanese).
Stop Word Removal: Discarding high-frequency words with low semantic density ("the", "is", "at"). (Rarely used in modern deep learning because grammar and syntax rely heavily on stop words).
Stemming vs Lemmatization:
- Stemming (Porter / Snowball): Crude, heuristic rule-based chopping of word suffixes (e.g., "studying", "studies", "studied" all reduce to the stem studi). Fast, but frequently outputs non-words.
- Lemmatization: Uses lexical morphological databases (like WordNet) and part-of-speech context to return the true dictionary base form (lemma) (e.g., "better" reduces to good).

The Out-Of-Vocabulary (OOV) Bottleneck

Word-level tokenization suffers from a fatal dilemma: if your vocabulary contains 50,000 words, any rare word, misspelling, slang, or technical jargon (e.g., "bioinformatics", "neurodegenerative") is mapped to an unknown token: <UNK>. Conversely, character-level tokenization eliminates OOV words but produces sequence lengths 5x longer, blowing up the ( O(N^2) ) memory cost of Transformers!

Modern Subword Tokenization Algorithms

Modern foundation models solve this through Subword Tokenization. Common words remain intact as single tokens, while rare words are automatically decomposed into meaningful morphological prefixes, roots, and suffixes (e.g., "unbreakable" becomes ["un", "break", "able"]):

Algorithm	Key Mechanics & Splitting Rule	Used In Modern AI
Byte Pair Encoding (BPE) (Sennrich 2016)	Starts with individual characters. Iteratively counts the most frequent adjacent pair of symbols across the corpus and merges them into a new token until vocabulary reaches target size ( V ).	GPT-2, GPT-3.5, GPT-4, Llama 2/3, Mistral (uses Byte-level BPE on raw UTF-8 bytes).
WordPiece (Schuster & Nakajima)	Similar to BPE, but selects the pair to merge that maximizes the likelihood of the language model training data. Uses `##` to denote subword continuation.	BERT, DistilBERT, Electra
Unigram / SentencePiece (Kudo 2018)	Starts with an enormous vocabulary and iteratively prunes the tokens that contribute least to overall corpus likelihood. Treats whitespace as an explicit symbol (`_`), language-agnostic.	T5, ALBERT, Llama (early), Gemma

Byte-Level BPE (tiktoken)

Modern LLMs (like OpenAI's tiktoken) operate directly on raw UTF-8 bytes rather than Unicode characters. Because the byte space is strictly bounded between 0 and 255, any text in any human language, programming language, emoji, or mathematical symbol can be encoded without ever encountering an Out-Of-Vocabulary (<UNK>) error!

Python Implementation: Inspecting Subword Tokenization with tiktoken & Hugging Face

The following script demonstrates how modern LLM tokenizers break sentences into discrete token IDs:

import tiktoken

# 1. Load the GPT-4o / GPT-4 tokenizer (o200k_base / cl100k_base)
tokenizer = tiktoken.get_encoding("cl100k_base")

text = "Artificial Intelligence is transforming bioinformatics & quantum computing!"

# 2. Encode text into discrete token IDs
token_ids = tokenizer.encode(text)
print(f"Original Text: '{text}'")
print(f"Total Tokens : {len(token_ids)}")
print(f"Token IDs    : {token_ids}
")

# 3. Decode each token ID back to its subword piece
print("Subword Token Breakdown:")
for token_id in token_ids:
    piece = tokenizer.decode([token_id])
    print(f"  Token ID {token_id:6d} -> Subword: '{piece}'")

Frequently Asked Questions (FAQ) - Student & Interview Review

Q1. How many words are in a typical LLM token?

In the English language, 1 token corresponds to approximately 0.75 words (or roughly 4 characters). Therefore, a 1,000-word essay requires approximately 1,333 tokens. In languages with non-Latin scripts (like Hindi, Arabic, or Chinese), words may decompose into multiple byte tokens, consuming more tokens per word.

Q2. Why did modern LLMs eliminate Stop Word Removal?

In early search engines and keyword indexing, stop words ('the', 'and', 'which') were removed to save database space. In modern deep learning with attention mechanisms, stop words provide critical grammatical syntax, pronoun reference, and logical negation ('not', 'never'). Discarding them destroys contextual meaning.

Q3. What is the difference between Stemming and Lemmatization?

Stemming is a crude heuristic algorithm (like Porter Stemmer) that chops off common word endings without understanding grammatical context, often producing non-words (e.g., 'arguing' -> 'argu'). Lemmatization uses a morphological dictionary and part-of-speech tags to return the true grammatical root word (e.g., 'ran' -> 'run', 'mice' -> 'mouse').

Q4. Why do LLMs struggle with basic character counting tasks?

LLMs do not see individual letters; they see discrete subword tokens. For example, the word 'strawberry' is ingested as two tokens: ['straw', 'berry']. Because the model never sees the raw individual letters during self-attention, it cannot easily count how many 'r's are in 'strawberry' without step-by-step spelling prompts.