Counting Words

Words by the Numbers 🏆

Chapter 1 · Counting Words

Before computers could read, people counted. Medieval friars indexed the Bible by hand, and later scholars tallied the words of poets by the thousand. Counting sounds dull, but it is the first move in almost everything this course does: spotting a language, naming an author, scoring how hard a page is to read, or finding the words that matter in a document. This first chapter treats a piece of writing as data, cuts it into words, counts them, and builds the small counter that later chapters keep reusing. You need arithmetic only; the maths stays light, and every idea is something you can run.

What was run, and what to trust

Every Python snippet in this chapter and in the three exercise solutions was executed, and the output in the comments is what Python printed. Background facts come from a web search that returned summaries of encyclopedia and reference pages (the type–token distinction, the history of concordances); such fetched pages are often truncated or model-summarised, so treat the wording as a paraphrase, not a quotation. The sample sentences are either the public-domain opening of an 1813 novel (check the exact wording against an edition) or short texts invented for the exercises. Every count below describes those few lines only, so it is illustrative, not a finding about English. Statements marked (check) were not confirmed against a source.

Text as Data

A computer sees a passage as a long string of characters. To count anything meaningful we have to decide what the units are. Word counting needs two distinctions that look pedantic and turn out to matter all the way through the course. A word token is one occurrence of a word in the text; a word type is a distinct word, however many times it occurs. The terms come from the philosopher C. S. Peirce, who used “the” as an illustration: one word in the language, many physical copies of it on a page (this is the reference accounts’ summary, not a quotation). In the sentence “the cat sat on the mat” there are six tokens and five types, because “the” is one type with two tokens.

Term	Counts	“the cat sat on the mat”
Token	every occurrence	6
Type	each distinct word once	5
Frequency	how many tokens a type has	the = 2, every other type = 1
Vocabulary	the set of types	the, cat, sat, on, mat

Our sample for the whole chapter is the opening of Jane Austen’s Pride and Prejudice (1813, in the public domain): two sentences, short enough to check by eye. The first attempt at cutting it into words is the one most beginners try, splitting on spaces.

PASSAGE = (
    "It is a truth universally acknowledged, that a single man in possession "
    "of a good fortune, must be in want of a wife. However little known the "
    "feelings or views of such a man may be on his first entering a "
    "neighbourhood, this truth is so well fixed in the minds of the "
    "surrounding families, that he is considered the rightful property of "
    "some one or other of their daughters."
)

# First attempt: split on whitespace and count.
words = PASSAGE.split()
print("pieces:", len(words))
print(words[:8])
print([w for w in words if not w.isalpha()])
# pieces: 70
# ['It', 'is', 'a', 'truth', 'universally', 'acknowledged,', 'that', 'a']
# ['acknowledged,', 'fortune,', 'wife.', 'neighbourhood,', 'families,', 'daughters.']
    

Seventy pieces, but six of them carry punctuation: acknowledged, is not the same string as acknowledged, so a program would count it as a different word. This is the heart of tokenising: deciding where one word ends and the next begins. There is no single correct answer, only a choice that suits the question. A second attempt lower-cases everything and keeps runs of letters and apostrophes.

import re
from collections import Counter
PASSAGE = (
    "It is a truth universally acknowledged, that a single man in possession "
    "of a good fortune, must be in want of a wife. However little known the "
    "feelings or views of such a man may be on his first entering a "
    "neighbourhood, this truth is so well fixed in the minds of the "
    "surrounding families, that he is considered the rightful property of "
    "some one or other of their daughters."
)

# Second attempt: lower-case, then keep runs of letters and apostrophes.
tokens = re.findall(r"[a-z']+", PASSAGE.lower())
counts = Counter(tokens)
print("tokens:", len(tokens), "| types:", len(counts))
for word, n in counts.most_common(8):
    print(f"{word:>8} {n}")
# tokens: 70 | types: 48
#        a 6
#       of 6
#      the 4
#       is 3
#       in 3
#    truth 2
#     that 2
#      man 2
    

Now the passage has 70 tokens and 48 types. The top of the list holds small grammatical words (a, of, the, is, in), not the interesting words, and that pattern is general. a and of tie at six, and Counter.most_common lists ties in the order they first appeared (an implementation detail worth not relying on). The next chapter, Words by the Numbers 2, asks how steeply these counts fall as you go down the list.

Choices in Tokenising

Lower-casing merges “The” at the start of a sentence with “the” in the middle, which is usually what we want, but it also merges “Polish” with “polish”. Keeping apostrophes keeps it’s as one token but makes don’t and dont different. Every rule has a price.

Choice	Helps with	Costs
Split on spaces	simplicity	punctuation sticks to words
Lower-case	merging sentence-start words	merges names and common words
Letters only	clean words	splits `e-mail`, drops `3.50`
Keep apostrophes	`don’t`, `it’s` stay whole	quote marks can be mistaken for apostrophes

Measuring the Vocabulary

With tokens and types in hand, a few simple numbers describe a text. The type–token ratio is types divided by tokens: 48 ÷ 70 gives 0.686. A word seen only once is called a hapax legomenon, from Greek for “said once” (plural: hapax legomena).

import re
from collections import Counter
PASSAGE = (
    "It is a truth universally acknowledged, that a single man in possession "
    "of a good fortune, must be in want of a wife. However little known the "
    "feelings or views of such a man may be on his first entering a "
    "neighbourhood, this truth is so well fixed in the minds of the "
    "surrounding families, that he is considered the rightful property of "
    "some one or other of their daughters."
)

tokens = re.findall(r"[a-z']+", PASSAGE.lower())
counts = Counter(tokens)
total = len(tokens)
print("type-token ratio:", round(len(counts) / total, 3))
hapaxes = [w for w, n in counts.items() if n == 1]
print("words seen once:", len(hapaxes), f"({len(hapaxes) / len(counts):.0%} of types)")
print("share of the top 5 types:", f"{sum(n for _, n in counts.most_common(5)) / total:.1%}")
# type-token ratio: 0.686
# words seen once: 38 (79% of types)
# share of the top 5 types: 31.4%
    

Of the 48 types, 38 (79%) occur once, and the five commonest types account for 31.4% of all the tokens. Remember how small the sample is: with 70 tokens almost any word is a hapax, and the type–token ratio tends to fall as a text gets longer, because common words keep repeating. So the ratio is not comparable between texts of different lengths. Words by the Numbers 6 returns to this when it compares authors.

Counting Letters

The same method works one level down. Letters are tokens too, and in the passage the letter e leads comfortably.

from collections import Counter
PASSAGE = (
    "It is a truth universally acknowledged, that a single man in possession "
    "of a good fortune, must be in want of a wife. However little known the "
    "feelings or views of such a man may be on his first entering a "
    "neighbourhood, this truth is so well fixed in the minds of the "
    "surrounding families, that he is considered the rightful property of "
    "some one or other of their daughters."
)

letters = [ch for ch in PASSAGE.lower() if ch.isalpha()]
counts = Counter(letters)
total = len(letters)
print("letters:", total)
for ch, n in counts.most_common(5):
    print(f"{ch}: {n:3d}  {n / total:.1%}  " + "#" * n)
vowels = sum(counts[v] for v in "aeiou")
print(f"vowels: {vowels / total:.1%}")
# letters: 300
# e:  34  11.3%  ##################################
# i:  27  9.0%  ###########################
# o:  27  9.0%  ###########################
# t:  26  8.7%  ##########################
# s:  23  7.7%  #######################
# vowels: 38.3%
    

Of 300 letters, e has 34 (11.3%), and vowels make up 38.3%. That one sample is far too small to quote as the share of e in English; a larger reference count would be needed (check). Letter profiles are the subject of Words by the Numbers 3, where they tell languages apart.

A Small Reusable Counter

The pieces so far fit in two functions that later chapters build on. Each snippet in the course imports what it uses, so you can paste any block into a fresh file and run it.

import re
from collections import Counter

def tokenise(text):
    # Lower-case, then keep runs of letters and apostrophes.
    return re.findall(r"[a-z']+", text.lower())

def word_report(text, top=5):
    tokens = tokenise(text)
    counts = Counter(tokens)
    return {
        "tokens": len(tokens),
        "types": len(counts),
        "top": counts.most_common(top),
    }

sample = "the cat sat on the mat and the dog sat on the cat"
print(word_report(sample, top=3))
# {'tokens': 13, 'types': 7, 'top': [('the', 4), ('cat', 2), ('sat', 2)]}
    

Counting is much older than code. The Dominican friar Hugh of Saint-Cher is generally credited with the first verbal concordance of the Bible, an index of where each word occurs, produced in Paris in the 13th century with many helpers. Sources disagree on the date: one puts completion at 1230 and another gives 1230 to 1239 (conflict; check). It is the same job our counter does, with parchment instead of a Counter.

Common mistakes

Counting word and word, as different things because punctuation was left in; forgetting to lower-case, so The and the are separate types; mixing up tokens and types (“how many words?” has two honest answers); comparing the type–token ratio of a short text with that of a long one; and quoting results from a tiny sample, such as this chapter’s 70-word passage, as if they were facts about a language.

Exercise 1

Take a short invented passage of your own (the solution uses 28 words). Report the number of tokens and types, the five commonest words, and the list of words that occur exactly once.

📄 View solution

Exercise 2

Cut the sentence “Don’t pay $3.50 for a well-known e-mail tool; it’s cheaper in Sao Paulo.” into words three ways: splitting on spaces, keeping letters and apostrophes, and a pattern that also keeps hyphens, decimal points and digits. Compare the counts. Which result would you trust for a word count, and which for a price search?

📄 View solution

Exercise 3

Compare the letter profiles of the Austen sentence and a short invented passage: the commonest letter and its share, the share of vowels, and how many different letters appear. Do the two profiles look alike, and why is it risky to conclude anything from them?

📄 View solution

Quick Reference

Token: one occurrence of a word; type: a distinct word; a text has at least as many tokens as types
Tokenising: choosing where words start and end; every rule (case, punctuation, apostrophes, hyphens) has a cost
Type–token ratio: types ÷ tokens; it tends to fall as texts get longer, so compare equal lengths only
Hapax: a type seen once; in short texts most types are hapaxes
Python habit: re.findall to tokenise, Counter to tally, most_common(n) for the top of the list
Check before trusting a count: which tokeniser, how long the text is, and whether the sample is representative

Coming next

Words by the Numbers 2 is about Zipf’s law: rank and frequency, the long tail of rare words, log-log curves and testing the law on different texts.