Zipf's Law

Words by the Numbers

Chapter 2 · Zipf’s Law

Words by the Numbers 1 ended with a hint: in a tiny passage the commonest words were small grammatical ones, and most other words appeared once. This chapter asks how steep that drop is. The surprise, noticed by the American linguist George Kingsley Zipf, is that the drop follows a simple pattern across very different texts, and a few lines of arithmetic and Python are enough to see it, measure it and, importantly, to see where it stops working. The maths is light: division, a logarithm, and a line of best fit that Python computes for you.

What was run, and what to trust

Every Python snippet in this chapter and in the three exercise solutions was executed (Python 3.14), and the output in the comments is what Python printed. The real text used for counting is the reference documentation that ships inside Python itself (the pydoc_data.topics module): about 75,000 tokens of technical English, available on any machine, but not typical English prose, and its counts change a little between Python versions. Background facts come from web searches that returned summaries of encyclopedia and paper pages; such fetched pages are often truncated or model-summarised, so treat the wording as a paraphrase. The sources agree on the basic statement and the Brown Corpus figure but differ on exact numbers and framing (conflict; check). The vocabulary size and text length in the first snippet are assumed, illustrative numbers. Statements marked (check) were not confirmed against a source.

Rank and Frequency

Sort the types of a text by how often they occur: the commonest gets rank 1, the next rank 2, and so on. Zipf’s law, as the sources summarise it, says that the frequency of a word is roughly inversely proportional to its rank. The word of rank 2 should occur about half as often as the word of rank 1, rank 3 about a third as often, rank 10 about a tenth. Written as a formula, with f the frequency and r the rank, f ≈ C ÷ r, where C is a constant for the text. Another way to say it: rank × frequency stays roughly constant. Zipf set out the idea at length in his 1949 book on the “principle of least effort”, which argued that speakers and listeners both try to save effort (this is the reference accounts’ summary; check against the book).

Here is what a perfect 1/rank text would look like. The snippet assumes a vocabulary of 1,000 types and 100,000 tokens; both numbers are illustrative, chosen only to make the arithmetic easy to follow.

# Ideal Zipf: the word of rank r has frequency proportional to 1/r.
VOCAB = 1000      # assumed vocabulary size (illustrative)
TOKENS = 100000   # assumed text length (illustrative)

norm = sum(1 / r for r in range(1, VOCAB + 1))
print("normaliser (harmonic sum):", round(norm, 3))
print("rank  expected count  rank*count")
for r in (1, 2, 3, 5, 10, 100, 1000):
    expected = TOKENS * (1 / r) / norm
    print(f"{r:4d}  {expected:14.1f}  {r * expected:10.1f}")
print("share of rank 1:", f"{(1 / norm):.1%}")
# normaliser (harmonic sum): 7.485
# rank  expected count  rank*count
#    1         13359.2     13359.2
#    2          6679.6     13359.2
#    3          4453.1     13359.2
#    5          2671.8     13359.2
#   10          1335.9     13359.2
#  100           133.6     13359.2
# 1000            13.4     13359.2
# share of rank 1: 13.4%
    

The last column never changes: that is the whole law. The constant 13,359 is simply the number of tokens divided by the harmonic sum 1 + 1/2 + 1/3 + … + 1/1000 (about 7.485), so rank 1 takes 13.4% of the text. Real texts only approximately follow this, as the next section shows.

Testing It on Real Text

Now the same table for real text. The snippet tokenises the whole Python reference documentation with the tokeniser from Words by the Numbers 1 and prints the top ten ranks with their rank × count products.

import re
from collections import Counter
from pydoc_data.topics import topics

def tokenise(text):
    return re.findall(r"[a-z']+", text.lower())

text = " ".join(topics.values())
tokens = tokenise(text)
counts = Counter(tokens)
print("tokens:", len(tokens), "| types:", len(counts))
print("rank word        count  rank*count")
for rank, (word, n) in enumerate(counts.most_common(10), start=1):
    print(f"{rank:4d} {word:<10} {n:7d} {rank * n:11d}")
once = sum(1 for n in counts.values() if n == 1)
print("types seen once:", once, f"({once / len(counts):.0%} of types)")
# tokens: 75022 | types: 3445
# rank word        count  rank*count
#    1 the           5201        5201
#    2 a             2188        4376
#    3 is            2019        6057
#    4 of            1561        6244
#    5 in            1557        7785
#    6 to            1445        8670
#    7 and           1183        8281
#    8 for            945        7560
#    9 if             804        7236
#   10 are            760        7600
# types seen once: 918 (27% of types)
    

The top word, the, takes 5,201 of 75,022 tokens, which is 6.9%. For comparison, the sources report that in the Brown Corpus, a one-million-word sample of American English from the 1960s, the is also the commonest word and accounts for nearly 7% of all occurrences (69,971 out of slightly over a million; second-hand figures, check). The rank × count column wobbles between about 4,400 and 8,700 in the top ten, so the product is not constant, but it stays within a factor of two, over a range of counts from 5,201 down to 760. That is why people say “Zipf’s law” for something this loose. Note also the bottom line of the output: 918 of the 3,445 types (27%) occur exactly once. That crowd of rare words is the long tail.

Part of the curve	Ranks (this text)	What lives there
Head	1 to about 10	grammar words: `the`, `a`, `is`, `of`
Body	about 10 to several hundred	common content words and technical terms
Long tail	hundreds to 3,445	rare words; 918 appear once

Log-Log Plots and the Slope

If f = C ÷ r, take logarithms: log f = log C − log r. So on a plot with log rank across and log frequency up, a Zipf text is a straight line with slope −1. A log-log plot turns a curve that plunges and then crawls along the floor into something the eye can judge. Python can fit the line without a plotting library. The snippet below uses the standard least-squares formula, then draws a crude text-mode picture in which each bar’s length is ten times the base-10 logarithm of the count.

import math
import re
from collections import Counter
from pydoc_data.topics import topics

def tokenise(text):
    return re.findall(r"[a-z']+", text.lower())

def zipf_slope(tokens, max_rank=1000):
    # Least-squares line through (log rank, log count); returns the slope.
    freqs = sorted(Counter(tokens).values(), reverse=True)[:max_rank]
    xs = [math.log(r) for r in range(1, len(freqs) + 1)]
    ys = [math.log(f) for f in freqs]
    mx, my = sum(xs) / len(xs), sum(ys) / len(ys)
    num = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
    den = sum((x - mx) ** 2 for x in xs)
    return num / den

tokens = tokenise(" ".join(topics.values()))
print("slope, top 1000 ranks:", round(zipf_slope(tokens), 2))
print("slope, top 100 ranks: ", round(zipf_slope(tokens, 100), 2))

# A crude log-log picture: bar length = log10(count) * 10.
counts = sorted(Counter(tokens).values(), reverse=True)
for rank in (1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024):
    print(f"{rank:5d} {counts[rank - 1]:6d} " + "#" * round(math.log10(counts[rank - 1]) * 10))
# slope, top 1000 ranks: -1.04
# slope, top 100 ranks:  -0.81
#     1   5201 #####################################
#     2   2188 #################################
#     4   1561 ################################
#     8    945 ##############################
#    16    639 ############################
#    32    313 #########################
#    64    177 ######################
#   128     95 ####################
#   256     52 #################
#   512     22 #############
#  1024      8 #########
    

The slope over the top 1,000 ranks is −1.04, close to the textbook −1, while over only the top 100 ranks it is −0.81. The same text gives different slopes depending on which ranks you fit, so a slope is always quoted together with the rank range. In the picture, each doubling of rank takes the count down by a factor that is not quite two at the left (5,201, 2,188, 1,561) and somewhat steeper later (95, 52 at ranks 128 and 256), which is a straight-ish line on log axes with a gentle bend. The usual fix in the literature is the Zipf–Mandelbrot form, which has frequency proportional to 1/(rank + b)^a, with two adjustable numbers instead of one; the sources attribute it to Benoit Mandelbrot (check the details).

Form	Frequency at rank r	Straight line on log-log?
Zipf (simple)	C ÷ r	yes, slope −1
Generalised	C ÷ r^a	yes, slope −a
Zipf–Mandelbrot	C ÷ (r + b)^a	bends at low ranks, straight later

Does Text Length Matter, and Is It Special to Language?

Two honest tests. First, fit the same text at different lengths, and add a second, quite different text: the doc-strings of Python’s built-in names (a few thousand tokens).

import builtins
import inspect
import math
import re
from collections import Counter
from pydoc_data.topics import topics

def tokenise(text):
    return re.findall(r"[a-z']+", text.lower())

def zipf_slope(tokens, max_rank=1000):
    freqs = sorted(Counter(tokens).values(), reverse=True)[:max_rank]
    xs = [math.log(r) for r in range(1, len(freqs) + 1)]
    ys = [math.log(f) for f in freqs]
    mx, my = sum(xs) / len(xs), sum(ys) / len(ys)
    num = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
    den = sum((x - mx) ** 2 for x in xs)
    return num / den

docs = tokenise(" ".join(topics.values()))
# Slope against text length: the same text, first N tokens.
for n in (1000, 10000, 40000, len(docs)):
    print(f"first {n:7d} tokens: slope {zipf_slope(docs[:n], 1000):.2f}  types {len(set(docs[:n]))}")

# A second, different text: the docstrings of the built-in names.
doc_text = " ".join(inspect.getdoc(obj) or "" for obj in vars(builtins).values())
small = tokenise(doc_text)
print("builtin docstrings:", len(small), "tokens, slope", round(zipf_slope(small, 300), 2),
      "| top word:", Counter(small).most_common(1)[0])
# first    1000 tokens: slope -0.90  types 286
# first   10000 tokens: slope -1.15  types 1284
# first   40000 tokens: slope -1.06  types 2553
# first   75022 tokens: slope -1.04  types 3445
# builtin docstrings: 4908 tokens, slope -0.93 | top word: ('the', 342)
    

With only the first 1,000 tokens the slope is −0.90; at 10,000 it is −1.15; at 40,000, −1.06; and for the full text, −1.04. Short samples are noisy, so do not judge the law from a paragraph (the 70-word passage from Words by the Numbers 1 has far too few repeats). The small doc-string collection gives −0.93 over its top 300 ranks, and its top word is again the. These are two samples of one narrow genre; they say nothing about novels or speech.

The second test is a famous deflation. The psychologist George Miller pointed out in 1957 that text typed by a monkey hitting random keys, including the space bar, also yields a power law; Wentian Li gave a formal argument in 1992 (the sources agree on this core story, but differ on how to credit Miller, Mandelbrot and Li; conflict; check). The snippet below does the experiment with a fixed random seed: one million random characters from 26 letters and the space, all equally likely.

import math
import random
import re
from collections import Counter
from pydoc_data.topics import topics

def tokenise(text):
    return re.findall(r"[a-z']+", text.lower())

def zipf_slope(tokens, max_rank=1000):
    freqs = sorted(Counter(tokens).values(), reverse=True)[:max_rank]
    xs = [math.log(r) for r in range(1, len(freqs) + 1)]
    ys = [math.log(f) for f in freqs]
    mx, my = sum(xs) / len(xs), sum(ys) / len(ys)
    num = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
    den = sum((x - mx) ** 2 for x in xs)
    return num / den

# Monkey typing: 26 letters and a space, all equally likely, fixed seed.
random.seed(1)
typed = "".join(random.choice("abcdefghijklmnopqrstuvwxyz ") for _ in range(1_000_000))
monkey = typed.split()
real = tokenise(" ".join(topics.values()))
for name, toks in (("monkey", monkey), ("real text", real)):
    print(f"{name:10} tokens {len(toks):7d}  types {len(set(toks)):6d}  slope {zipf_slope(toks):.2f}")
print("most common monkey 'words':", Counter(monkey).most_common(4))
# monkey     tokens   35637  types  33591  slope -0.73
# real text  tokens   75022  types   3445  slope -1.04
# most common monkey 'words': [('l', 64), ('a', 61), ('m', 61), ('f', 58)]
    

The monkey text has 35,637 “words” of which 33,591 are different, and the fit over the top 1,000 ranks gives −0.73, not −1. Its curve is a staircase, because all words of the same length are equally likely, so many types tie on the same count. Short words are common simply because there are fewer of them to type. The lesson is not that the law is meaningless but that a straight line on log-log axes is a weak test: judge a claim by its slope, its rank range and how much data lies behind it, and ask what a random-text control gives.

Text	Tokens	Top-1000 slope
Python documentation	75,022	−1.04
Monkey typing (seed 1)	35,637	−0.73
Built-in doc-strings	4,908	−0.93 (top 300)

Common mistakes

Calling a rank–frequency pattern “proof” of the law from a very short text; quoting a slope without saying which ranks were fitted; plotting frequency against rank on ordinary axes, where the tail is invisible; forgetting that ties at the tail make steps; treating the two samples here as typical English; and reading the “constant” rank × frequency as exact when it only holds within a factor of about two.

Exercise 1

Build a rank–frequency table for a short invented passage and print rank × count for the top eight ranks. How many types share the lowest count, and what does that do to the shape of the tail?

📄 View solution

Exercise 2

Predict the counts at ranks 1, 2, 5, 10, 50, 100, 500 and 1,000 with the ideal 1/rank formula, using the real text’s own token and type totals, and print the ratio of actual to predicted. Where does the simple law over-predict, and where under-predict?

📄 View solution

Exercise 3

Fit the log-log slope separately over the rank bands 1–10, 10–100, 100–1,000 and 1,000–2,000, and count the types with frequency 1. In which band does a single straight line stop being a good description, and why?

📄 View solution

Quick Reference

Rank: position in the list sorted by frequency, commonest = 1
Zipf’s law: frequency ≈ C ÷ rank, so rank × frequency is roughly constant
Log-log plot: log rank against log frequency; a straight line of slope −1 means Zipf
Long tail: many rare words; in a 75,022-token text, 27% of the types occurred once
Zipf–Mandelbrot: adds an offset b to the rank so the head can bend
Random-text control: monkey typing also gives a rough power law, so check slope, range and data size
Python habit: sorted(Counter(tokens).values(), reverse=True) gives the frequencies by rank

Coming next

Words by the Numbers 3 is about letters and secrets: letter frequency in languages, early frequency analysis and telling languages apart by their letter profiles.