Text as Data: Tokenization and the Messiness of Language

Hook #

Every machine-learning model you've built so far took numbers as input — pixels, feature vectors, measurements. But language arrives as text: a stream of characters with no inherent numeric structure, riddled with ambiguity, irregularity, and edge cases that a model can't touch until you turn it into something numeric. Natural Language Processing is the field that makes computers work with human language, and it starts with a deceptively hard problem: how do you even chop text into pieces to feed a model? That's tokenization — splitting text into units (words, subwords, characters) — and it sounds trivial until you actually try it. Where does one "word" end? Is "don't" one token or two ("do" + "n't")? Is "New York" one unit or two? What about "🙂", a URL, a hashtag, code, or a language like Chinese with no spaces between words? Beneath tokenization sits the deeper truth of NLP: human language is messy in ways that formal systems (code, math, protocols) are not. It's ambiguous (one sentence, many meanings), irregular (rules with endless exceptions), and context-dependent (the same word means different things in different places). This lesson is the groundwork for the whole field — the preprocessing pipeline that turns raw text into model-ready input: tokenization (split into units), normalization (lowercase, strip punctuation, handle accents), stemming/lemmatization (reduce "running"/"ran"/"runs" to a common root), and — the version that matters for modern models — subword tokenization (BPE/WordPiece: split into pieces that balance vocabulary size against coverage). You've used the output of tokenization already (the transformer consumed "tokens"); this lesson is where those tokens come from, and why getting them right is harder than it looks.

What you'll be able to do by the end of this lesson #

• Explain why text must be turned into discrete units (tokens) before any model can process it, and why tokenization is genuinely hard — word boundaries, contractions, multi-word units, punctuation, emoji, URLs, and languages without spaces all resist a simple "split on spaces" rule.
• Distinguish the classical preprocessing steps — normalization (lowercasing, accent/punctuation handling), stemming (crude rule-based suffix chopping: "running" → "run"), and lemmatization (dictionary-based reduction to a real base form: "better" → "good") — and when each is appropriate.
• Explain subword tokenization (BPE, WordPiece, SentencePiece) — splitting text into frequent sub-word pieces rather than whole words — and why it's what modern transformers use: it handles any word (including never-seen ones) with a fixed, manageable vocabulary.
• Recognize the deeper theme of NLP — that language is ambiguous, irregular, and context-dependent — and why that messiness (not the algorithms) is what makes the field hard.

A quick try before we start #

Take one sentence: "Dr. Smith didn't think the U.S.-based startup's AI-powered app was worth <span class="katex-inline" data-formula="2.5M — but she wasn't sure." Now try to split it into "words." Immediately, problems: Is "Dr." a word or an abbreviation (and does the period end the sentence — it doesn't)? Is "didn't" one token or two ("did" + "n't")? What about "U.S.-based" — one token, or "U.S." + "based", or "U" + "S" + "based"? Is "">2.5M" a word? Does "AI-powered" split on the hyphen? Is "she" the same entity as "Dr. Smith" (that's a later problem, but note it's lurking)? There is no single correct answer — it depends on what you're doing downstream. This is the first lesson of NLP: language does not come pre-chopped into clean units. A naïve "split on whitespace" gives you "didn't", "U.S.-based", "$2.5M" as single messy tokens; a naïve "split on every non-letter" shatters "U.S." into "U" and "S" and loses the meaning. Classical NLP handled this with hand-built rules (regex-heavy tokenizers, one per language, full of special cases). Modern NLP mostly sidesteps the word-boundary problem with subword tokenization: don't try to find "words" at all — learn a vocabulary of frequent character sequences ("token", "##ization", "Ġthe") from data, and split any text into those pieces. A rare word like "tokenization" becomes "token" + "ization"; a never-seen word still splits into known pieces; and the vocabulary stays a fixed size (say 50,000 pieces) no matter how much text you see. Hold this: before a model can touch language, you must turn text into discrete units — and because language is messy, how you do that (whole words? subwords? characters?) is a real engineering decision with real consequences.

Why this matters here #

This lesson matters because it's the entry point to an entire field — and because the preprocessing it covers is the unglamorous but load-bearing first step of every NLP system, classical or neural. You spent the Deep Learning courses learning architectures (RNNs, transformers) that consume "tokens" and produce predictions — but those courses assumed the tokens already existed. This lesson is where tokens come from, and it turns out that decision shapes everything downstream: the vocabulary size (how many distinct tokens the model must handle), how the model deals with rare or unseen words, how long sequences get (subword splitting makes text longer in tokens), even how fairly the model treats different languages (subword tokenizers trained mostly on English fragment other languages into far more tokens — a real equity issue in who pays more per API call and whose text fits in the context window). Getting tokenization wrong quietly poisons everything after it.

The deeper significance is that this lesson introduces the central difficulty of NLP: language is fundamentally messy in a way the rest of computer science is not. Everything you've built — compilers, protocols, databases, algorithms — operates on formal languages: precise grammars, unambiguous syntax, deterministic rules. A parser rejects a malformed program; a protocol rejects a malformed packet. Human language has none of that rigor. It's ambiguous ("I saw her duck" — did she lower her head, or do you see her waterfowl?), irregular (English plurals: cat→cats, but child→children, mouse→mice, sheep→sheep), context-dependent ("bank" means something different by a river than in a sentence about money), and endlessly creative (people coin new words, misspell, use slang, mix languages, and break every rule you'd write down). This is why NLP resisted the rule-based approaches that worked elsewhere in CS: you cannot write down all the rules of a natural language, because there is no finite, consistent rulebook — the "rules" have exceptions, the exceptions have exceptions, and usage shifts constantly. This messiness is the reason NLP eventually turned to statistical and then neural methods — instead of encoding the rules of language by hand, learn the patterns from vast amounts of real text. This lesson's preprocessing (tokenization, normalization, stemming) is partly the last gasp of the rule-based era (hand-built rules to tame the messiness) and partly the foundation the statistical/neural era still stands on (you still have to tokenize). Understanding why language is hard — and why that hardness pushed the field from rules to learning — is understanding the shape of NLP's whole history, which is the arc this course traces.

The engineer's lens #

The first lens is tokenization: turning a character stream into discrete units, and why "split on spaces" fails. A model — any model — needs discrete units it can map to vectors and process. The obvious approach, split on whitespace, breaks immediately on real text: punctuation clings to words ("word." vs. "word"), contractions fuse two units into one ("can't" = "can" + "not"), multi-word expressions act as one unit ("New York", "kick the bucket"), and many languages (Chinese, Japanese, Thai) don't put spaces between words at all. Classical tokenizers handled this with hand-written rules — regex patterns to peel off punctuation, expand contractions, protect abbreviations and URLs and numbers — a fiddly, language-specific, exception-riddled process (Jurafsky & Martin Ch. 2 walks through the linguistic reasoning). This is the same lesson you learned in compilers — lexing turns a character stream into tokens — except a programming language has a precise, finite set of token rules, while a natural language has fuzzy, exception-laden ones that no rulebook fully captures. For the engineer, the takeaway is that tokenization is a modeling decision, not a given: the units you choose (words? subwords? characters?) determine your vocabulary size, your handling of rare words, and how the model sees the text — and there's no universally right choice, only a right choice for a task and a language.

The second lens is normalization, stemming, and lemmatization: collapsing surface variation to expose meaning. Once you have tokens, the same meaning often appears in many surface forms, and classical NLP tried to collapse that variation so the model sees fewer, more meaningful units. Normalization handles surface noise: lowercasing ("The" = "the"), stripping or standardizing punctuation, folding accents ("café" = "cafe"), normalizing whitespace and unicode. Stemming crudely chops suffixes to a common root using rule-based algorithms (the Porter stemmer): "running", "runs", "ran"→"run"; it's fast and rough — it often produces non-words ("argument"→"argu", "happiness"→"happi") and over- or under-collapses, but it's cheap and good enough for search/retrieval. Lemmatization does it properly — using a dictionary and part-of-speech to reduce a word to its real base form (its lemma): "better"→"good", "was"→"be", "mice"→"mouse" — more accurate but slower (it needs linguistic knowledge). The classical motivation was sparsity: if "run", "runs", "running", "ran" are four separate tokens, a model must learn each independently from limited data; collapse them to one and the model learns once and generalizes. This is the same sparsity problem you met in classical ML (too many rare features, not enough data per feature). The twist worth noting: modern neural NLP often skips stemming and lemmatization entirely — because subword tokenization + learned embeddings let the model discover that "run" and "running" are related from data, so the hand-built collapsing becomes unnecessary (and sometimes harmful — it destroys information the model could use). Knowing when these classical steps help (search, classical pipelines, low-data settings) versus when to skip them (modern transformer pipelines) is exactly the engineer's judgment this course builds.

The third lens is subword tokenization: the modern answer, and why every LLM uses it. The classical "split into words" approach has a fatal problem for neural models: the vocabulary is unbounded. Natural language has an open vocabulary — new words, names, typos, and rare forms appear constantly — so a word-level vocabulary either grows impossibly large or hits out-of-vocabulary words it can't represent at all ("I've never seen 'tokenization' — I have no vector for it"). Subword tokenization solves this elegantly: don't split into words, split into frequent sub-word pieces learned from data. Byte-Pair Encoding (BPE) starts from individual characters and repeatedly merges the most frequent adjacent pair into a new token, until it reaches a target vocabulary size — so common words become single tokens ("the", "ing") while rare words split into pieces ("tokenization" → "token" + "ization"). WordPiece (BERT) and SentencePiece (used by many multilingual models) are variants of the same idea. The payoff is threefold: (1) fixed vocabulary size (say 30k–100k pieces) no matter how much text you see; (2) no out-of-vocabulary problem — any string splits into known pieces (worst case, individual characters/bytes); and (3) a natural handling of morphology — related words share pieces ("run", "running" both contain "run"), so the model gets some of stemming's benefit for free. This is the tokenization of the modern era — every transformer you met in the DL course (BERT, GPT) consumes subword tokens, which is why a word like "unbelievable" might be 3 tokens and why you're billed per token (not per word) by LLM APIs, and why token counts, not word counts, determine whether text fits in a context window. For the engineer, subword tokenization is the bridge between "language is messy and open-ended" and "models need a fixed, finite input vocabulary" — it's the pragmatic compromise the whole field settled on, and understanding it demystifies a lot of practical LLM behavior (weird token splits, per-token pricing, context limits, the multilingual fairness gap).

What to focus on in the resources #

• Jurafsky & Martin, Ch. 2 (free) — primary. The canonical, rigorous treatment: tokenization, normalization, stemming/lemmatization, and edit distance, with the linguistic reasoning behind each. Read Ch. 2 to understand why the messiness of language forces each preprocessing choice. Bookmark the whole book — it's the reference for the course.
• Hugging Face NLP Course, 'Tokenizers' (free) — primary. The best hands-on explanation of modern subword tokenization (BPE/WordPiece/SentencePiece) — how real transformers actually split text — with runnable code. Do it to connect classical preprocessing to how models see text today.
• spaCy free course (free). Do the preprocessing in working code on real text — tokenization, lemmatization, normalization — so the abstractions become concrete. spaCy is the production NLP library; using it builds practical fluency.
• Skip on first pass: the exact Porter stemmer rules, edit-distance dynamic programming (revisit from the algorithms course if needed), the fine differences between BPE/WordPiece/Unigram, and language-specific tokenizer quirks. Get: tokenization (turn text into discrete units; "split on spaces" fails on real language); normalization/stemming/lemmatization (collapse surface variation to fight sparsity — but often skipped in modern neural pipelines); subword tokenization (BPE/WordPiece: split into frequent pieces → fixed vocabulary, no out-of-vocabulary, morphology for free — what every LLM uses); and the deep theme — language is ambiguous, irregular, and context-dependent, which is why NLP moved from rules to learning.

Explain it back #

Explain to a colleague why text has to be tokenized before a model can use it, why tokenization is harder than "split on spaces," and what subword tokenization is. A strong answer: models need numbers, but language arrives as raw text, so the first step is tokenization — chopping text into discrete units (tokens) that can be mapped to vectors. "Split on spaces" fails immediately: punctuation sticks to words, contractions fuse two units ("can't" = "can"+"not"), multi-word expressions act as one ("New York"), and many languages have no spaces at all — so classical tokenizers were fiddly, hand-built, language-specific rule systems. Once you have tokens, the same meaning shows up in many surface forms ("run"/"runs"/"running"), so classical NLP used normalization (lowercase, strip punctuation), stemming (crude suffix-chopping → "run", often producing non-words), and lemmatization (dictionary-based reduction to a real base form → "good" from "better") to collapse that variation and fight sparsity. But modern neural NLP mostly skips stemming/lemmatization and instead uses subword tokenization — BPE/WordPiece/SentencePiece — which splits text into frequent sub-word pieces learned from data ("tokenization" → "token"+"ization"). That gives three wins: a fixed vocabulary size, no out-of-vocabulary problem (any string splits into known pieces), and morphology for free (related words share pieces). It's what every transformer (BERT, GPT) consumes — which is why you're billed per token, why context limits are in tokens not words, and why rare words cost more. Underneath all of it is the theme of NLP: language is ambiguous, irregular, and context-dependent — messy in a way formal languages (code, protocols) never are — which is exactly why the field moved from hand-written rules to learning from data.

Where this connects #

Backward: Tokenization is lexing from the compilers course — turn a character stream into tokens — but for a natural language with fuzzy, exception-laden boundaries instead of a programming language's precise ones. The sparsity motivation for stemming/normalization is the same sparse-feature problem from classical ML (too many rare features, too little data each). And the tokens this lesson produces are exactly the tokens the transformer (Deep Learning course) consumed — this lesson is where they come from. Subword embeddings connect to the word embeddings of the next lesson (each token gets a learned vector).

Forward: Every remaining lesson in this course starts from tokenized text. The next lesson (word embeddings) takes these tokens and asks how to represent their meaning as vectors. Classical NLP tasks (language modeling, POS tagging, NER) and text classification all consume tokenized, sometimes normalized, text. And subword tokenization is the direct setup for the LLM course — BPE/tiktoken tokenization, per-token pricing, and context windows are all here in embryo. This lesson is the field's foundation: before you can process language, you must turn its messy, open-ended, ambiguous stream into discrete units a model can hold.