Words and Tokens

Computational Linguistics

Course 1 · Chapter 2 · Words and Tokens

Chapter 1 turned text into characters and bytes. The next question sounds trivial: where do the words begin and end? Almost every program that handles language starts by cutting text into pieces, and the way it cuts decides everything it counts afterwards.

The idea in a box

A token is a piece the program chooses to treat as a unit. It is often a word, but "word" has no single definition, so a tokenizer is always a set of decisions, and those decisions can be right or wrong for the job.

What Is a Word?

Wikipedia's Word article says "there is no consensus among linguists on its definition", and that different criteria apply at different levels of analysis without converging. It names three of them:

Kind	Defined by	The article's wording
Orthographic word	writing	"a segment indicated by blank spaces in writing or print"
Phonological word	sound	"a unit larger or equal to a syllable", found through segmental or prosodic features or phonological rules
Grammatical word	morphology and inflection	distinct from word-forms

Lexeme and word form. A lexeme is the dictionary-listed unit and covers all its inflected forms, so teapot and teapots belong to one lexeme. Chapter 4 returns to this.
No spaces. Chinese, Japanese, Thai and Lao often do not separate words with spaces. The article says Mandarin is analytic, so delimiting words orthographically is "unnecessary", and that Japanese uses kanji/kana switching as a soft cue.
Compounds and contractions. Compound words such as cowboys and typewriter have more than one root; contractions like can't are described as words formed from multiple words.

Tokenization

The Lexical analysis article is mostly about programming languages, but it has a passage on natural language. It says a machine does not implicitly segment text on spaces the way a human reader does, so delimiters must be defined explicitly, and defining a "word" is hard. Its edge cases are contractions, hyphenated words, emoticons and URIs; its example is New York-based, where a naive tokenizer may split at the space when the hyphen is the better break. It adds that tokenization is especially difficult for languages written in scriptio continua, naming Ancient Greek, Chinese and Thai, and that agglutinative languages such as Korean also complicate it. Suggested remedies are more complex heuristics, lookup tables of special cases, and language models that identify collocations in a later step.

Three tokenizers, one sentence

The sentence and code below are my own. I ran each and copied the output.

import re

s = "Don't call the New York-based e-mail team before 5 p.m., said Dr. Smith."
print(s.split())
print(re.findall(r"\w+", s))
print(re.findall(r"\w+(?:[-']\w+)*", s))

["Don't", 'call', 'the', 'New', 'York-based', 'e-mail', 'team', 'before', '5', 'p.m.,', 'said', 'Dr.', 'Smith.'] ['Don', 't', 'call', 'the', 'New', 'York', 'based', 'e', 'mail', 'team', 'before', '5', 'p', 'm', 'said', 'Dr', 'Smith'] ["Don't", 'call', 'the', 'New', 'York-based', 'e-mail', 'team', 'before', '5', 'p', 'm', 'said', 'Dr', 'Smith']

Tokenizer	What it gets right	What it gets wrong
Split on spaces	keeps Don't, York-based, e-mail whole	leaves punctuation stuck on p.m., and Smith.
Letters and digits only	removes the punctuation	tears Don't into Don + t, and breaks e-mail and p.m.
Letters, digits, inner hyphens and apostrophes	keeps Don't, York-based, e-mail	still breaks p.m. into p and m, and loses the full stop in Dr.

None of the three is correct everywhere. Whether New York-based should be one token, two or three depends on what you want to do next, which is the Lexical analysis article's point.

When there are no spaces

zh = "我喜欢咖啡"
print(zh.split())
print(len(zh))

['我喜欢咖啡'] 5

Splitting on spaces returns the whole string as a single "word", because nothing in it is a space. Cutting it into words needs knowledge of the language, not just a rule about spaces. I have given the string only to show that failure; I do not segment it here, and I do not claim a segmentation, because I have not checked one against a source.

Types and Tokens

The Type–token distinction article credits Charles Sanders Peirce, who drew the distinction in 1906. Types are "abstract descriptive concepts"; tokens are "objects that instantiate concepts", the particular occurrences of a type. Because each type can have many tokens, there are generally more tokens than types. Its example is "A rose is a rose is a rose": three word types, with three tokens of a, two of is and three of rose.

from collections import Counter

text = "A rose is a rose is a rose."
words = text.lower().replace(".", "").split()
print(len(words), len(set(words)))
print(Counter(words))

8 3 Counter({'a': 3, 'rose': 3, 'is': 2})

Eight tokens, three types, matching the article's counts. The article does not mention the type-token ratio, so this course does not rely on it here.

Word Frequencies and Zipf's Law

The Zipf's law article states that in a natural-language text, a word's frequency is roughly inversely proportional to its rank: "the most common word occurs approximately twice as often as the next common one" (and about three times as often as the third). Its Brown Corpus example: the is the most frequent word at nearly 7% of occurrences (69,971 of slightly over 1 million), of is second at slightly over 3.5% (36,411) and and follows with 28,852.

Rank	Word	Count (article)	Zipf prediction from rank 1 (my arithmetic)
1	the	69,971	69,971
2	of	36,411	34,986
3	and	28,852	23,324

The prediction column is 69,971 divided by the rank, which I calculated: the real rank-2 count is close to half of rank 1 (ratio 1.92), but rank 3 is about 2.43 times smaller, not 3. That is the article's own caveat: the law holds only approximately.

History. Named after George Kingsley Zipf, who observed the word-frequency relation in 1932 but "never claimed to have originated it"; earlier discoveries are credited to Jean-Baptiste Estoup (1916), Godfrey Dewey (1923) and Edward Condon (1928).
Form. The Zipf–Mandelbrot version is frequency ∝ 1 / (rank + b)^a, with a ≈ 1 and b ≈ 2.7 as fitted parameters.
Where it bends. Real plots deviate at the high- and low-frequency ends, depending on language, topic, author, translation and spelling. Across a large set of English texts only about 15% fit well, rising to nearly 50% with small changes to the definition. Some languages deviate more, including Romance languages' most frequent words and East Asian languages where each morpheme is one syllable.
Why. The article says the reason is still poorly understood; proposals include random-text models, the principle of least effort, and preferential attachment.
Uses. It applies to most natural languages and some constructed ones, such as Esperanto and Toki Pona, and is used in authorship attribution and in analyzing the Voynich manuscript.

Why a small text will not show it

t = ("the cat sat on the mat and the dog sat on the log "
     "while the bird sang and the cat slept")
c = Counter(t.split())
print(len(t.split()), len(c))
for rank, (w, n) in enumerate(c.most_common(5), 1):
    print(rank, w, n)

21 12 1 the 6 2 cat 2 3 sat 2 4 on 2 5 and 2

Twenty-one tokens and twelve types: the has six, then four words tie on two. A tiny text gives ties and jumps, not the smooth curve. The law is a statement about large collections, so this chapter does not claim my toy sentence demonstrates it.

Watch out: a count is not a meaning

The is the commonest word because it is a grammatical word, not because it is the most important. Frequency lists answer "how often", not "how much it matters". Chapter 3 shows what a corpus can and cannot tell you.

Note on sources

This chapter uses Wikipedia's Word, Type–token distinction and Lexical analysis (all read in full) and Zipf's law (read to 100,000 of 121,078 characters, so its last part was not read). The Word page does not cover clitics, the Type–token page does not mention type-token ratio or corpus linguistics, and the Zipf page gives no percentage for and. The sentences, code, outputs and the prediction column are mine; the Chinese string is shown only to display a failure, not segmented. Reading Chapter 1's "tokenization" as the same step is my own link.

Try It Yourself

Exercise 1

Run the three tokenizers on "It's a well-known fact, isn't it?" and compare the outputs. For each of It's, well-known and isn't, say which tokenizer keeps it whole.

📄 View solution

Exercise 2

Count the tokens and types in "to be or not to be", then say which is the type and which are the tokens for the word be. What happens to the type count if the first word is written To and you do not lowercase?

📄 View solution

Exercise 3

Using the article's Brown Corpus counts, calculate what Zipf's law predicts for ranks 2 and 3 from rank 1, compare with the real counts, and say what that shows about calling it a "law".

📄 View solution

Chapter 2 Quick Reference

No agreed definition of "word": orthographic, phonological, grammatical
Tokenization is a set of choices; edge cases include contractions, hyphens, abbreviations and scripts without spaces
Type = distinct form; token = each occurrence (8 tokens, 3 types in "A rose is a rose is a rose")
Zipf's law: frequency roughly proportional to 1 / rank, approximate and bending at the ends

Coming next

Computational Linguistics 3 looks at corpora: what a body of text can show, and what it cannot.