Style and Authorship

Words by the Numbers

Chapter 6 · Style and Authorship

Words by the Numbers 5 generated text from counts. This chapter runs the other way: given a text, can counts tell you who wrote it? Every writer has habits they do not notice, such as how often they reach for upon, by or would, and those habits show up as steady rates across hundreds of pages. The chapter uses a famous test case, the Federalist Papers, where historians disagreed for a century and a half about who wrote twelve of the essays. The maths is light: rates per thousand words, a z-score, and an average of differences. All the Python is runnable.

What was run, and what to trust

Every Python snippet in this chapter and in the three exercise solutions was executed (Python 3.14), and the output in the comments is what Python printed. The text is the plain-text Federalist Papers file from Project Gutenberg (eBook 1404), which the code downloads once and saves as federalist.txt; you need an internet connection the first time. The essays are cut at the start of the greeting and at the closing signature, so the headings and the file's footnotes are not counted. The author labels come from that file, except that the twelve disputed essays (numbers 49 to 58, 62 and 63) and the three joint ones (18 to 20) are set aside by the code. Counts depend on exactly how words are cut, so a different tokeniser gives slightly different numbers. The two-line passages in Exercise 1 are invented, illustrative examples. Background facts come from a web search that returned summaries of web pages; such fetched pages are often truncated or model-summarised, so treat the wording as a paraphrase. The search supported the essay counts, the disputed essay numbers, and the 1963-64 Mosteller and Wallace study with its conclusion that Madison wrote all twelve. It also showed a source conflict, noted below. Statements marked (check) were not confirmed against a source.

A Puzzle from 1787

The Federalist Papers are 85 newspaper essays that argued for adopting the new United States Constitution. They appeared from October 1787 (the file's own dates) over the single pen name Publius. The writers were Alexander Hamilton, James Madison and John Jay. Lists of who wrote what were drawn up soon after, but the lists disagreed, and twelve essays (numbers 49 to 58, 62 and 63) were claimed by both Hamilton and Madison. Historians argued from the content and the political context without settling it (general background, check). In 1963-64 the statisticians Frederick Mosteller and David Wallace tried a different kind of evidence: counting words. According to the search summaries, they used Bayes' theorem to weigh the evidence from many words, noticed that the two men used the same words at different rates, and concluded that Madison wrote all twelve. Some later work questioned this for Federalist 55, per one source; I could not confirm the details (check).

One source conflict is worth knowing before counting. Some sources say Madison wrote 14 essays, Hamilton 51, Jay 5, with 3 joint and 12 disputed (this adds up to 85). Others say Madison wrote 29, which is the same 14 plus the 12 disputed plus the 3 joint ones once the disputes are settled in his favour. Both are consistent; they count the disputed essays differently. The code below uses the first scheme, because the disputed ones are exactly what you want to keep out of the training data.

Function Words: Habits You Cannot See

A function word is a small grammatical word (the, of, by, to, upon) as opposed to a content word that carries the topic (treaty, army). Function words suit authorship work for two reasons. They are frequent, so even one essay gives usable counts; and writers choose them without thinking, so they are hard to imitate and are less tied to the subject. The usual unit is a rate: uses per 1,000 words, which makes a long essay and a short one comparable. The first snippet loads the essays, prints how much text each author has, and compares rates.

import pathlib, re, urllib.request

URL = "https://www.gutenberg.org/cache/epub/1404/pg1404.txt"
DISPUTED = set(range(49, 59)) | {62, 63}   # the twelve disputed essays
JOINT = {18, 19, 20}                       # joint work, left out of the comparisons

def load(path="federalist.txt"):
    p = pathlib.Path(path)
    if not p.exists():                     # download once, then reuse the saved copy
        req = urllib.request.Request(URL, headers={"User-Agent": "Mozilla/5.0"})
        p.write_bytes(urllib.request.urlopen(req).read())
    raw = p.read_text(encoding="utf-8-sig").replace("\r", "")
    parts = re.split(r"^FEDERALIST No\. (\d+)\s*$", raw, flags=re.M)
    papers = {}                            # essay number -> (author, list of words)
    for k in range(1, len(parts), 2):
        n, body = int(parts[k]), parts[k + 1]
        m = re.search(r"^(HAMILTON|MADISON|JAY)$", body, re.M)
        author = m.group(1) if m else "?"
        if n in DISPUTED: author = "DISPUTED"
        if n in JOINT: author = "JOINT"
        start = body.find("To the People")
        end = body.rfind("\nPUBLIUS")
        text = body[start:end if end > start else None]
        papers[n] = (author, re.findall(r"[a-z]+(?:'[a-z]+)?", text.lower()))
    return papers

from collections import Counter

papers = load()
essays, words, pool = Counter(), Counter(), {}
for n, (author, w) in papers.items():
    essays[author] += 1
    words[author] += len(w)
    pool.setdefault(author, Counter()).update(w)

for a in ("HAMILTON", "MADISON", "JAY", "JOINT", "DISPUTED"):
    print(f"{a:9} essays {essays[a]:2d}   words {words[a]:7,d}")

print()
print(f"{'per 1000 words':14} {'Hamilton':>9} {'Madison':>9} {'Jay':>9} {'disputed':>9}")
for word in ("upon", "whilst", "while", "by", "to", "on", "would", "there"):
    rates = [1000 * pool[a][word] / words[a] for a in ("HAMILTON", "MADISON", "JAY", "DISPUTED")]
    print(f"{word:14}", *[f"{r:9.2f}" for r in rates])

for a in ("HAMILTON", "MADISON"):
    own = [w for author, w in papers.values() if author == a]
    print(a, "essays with no 'upon' at all:", sum(1 for w in own if "upon" not in w), "of", len(own))
# HAMILTON  essays 51   words 111,764
# MADISON   essays 14   words  38,890
# JAY       essays  5   words   8,426
# JOINT     essays  3   words   5,653
# DISPUTED  essays 12   words  24,165
#
# per 1000 words  Hamilton   Madison       Jay  disputed
# upon                3.33      0.18      0.12      0.12
# whilst              0.01      0.31      0.00      0.37
# while               0.32      0.00      0.24      0.00
# by                  7.56     11.62      9.73     11.50
# to                 40.72     32.42     34.77     31.74
# on                  3.36      7.59      5.22      7.82
# would               8.25      4.09      8.07      4.55
# there               3.39      0.85      1.19      1.32
# HAMILTON essays with no 'upon' at all: 0 of 51
# MADISON essays with no 'upon' at all: 11 of 14
    

Hamilton uses upon 3.33 times per 1,000 words and Madison 0.18, about eighteen times less often. Eleven of Madison's 14 essays do not contain upon at all, while all 51 of Hamilton's do. Madison prefers whilst where Hamilton says while, and Madison writes by and on more, Hamilton to, would and there. Now look at the last column. The twelve disputed essays use upon at 0.12 per 1,000 words, whilst at 0.37 and by at 11.50. Those figures sit with Madison's, not Hamilton's. Jay, with only five essays, looks different again (for example a low the, which would need a closer look; check).

Per 1,000 words	Hamilton	Madison	Disputed essays
upon	3.33	0.18	0.12
whilst	0.01	0.31	0.37
by	7.56	11.62	11.50
to	40.72	32.42	31.74
would	8.25	4.09	4.55

One word is a clue, not a proof. The disputed essays are about the structure of the new government (the code does not read them, so that is background; check), and topic can move even function-word rates a little. That is why serious studies use many words at once, as the third snippet does.

Vocabulary Richness

A second idea is vocabulary richness: does one author use a wider range of words? The simplest measure is the type-token ratio, the number of distinct words divided by the total number of words (compare the vocabulary counts in Words by the Numbers 1). It has a well-known flaw, which the next snippet measures: the longer the text, the more words repeat, so the ratio falls. To compare writers fairly, cut every essay to the same length first. The snippet also counts the share of the 1,000 words that occur only once in the sample.

import pathlib, re, urllib.request

URL = "https://www.gutenberg.org/cache/epub/1404/pg1404.txt"
DISPUTED = set(range(49, 59)) | {62, 63}   # the twelve disputed essays
JOINT = {18, 19, 20}                       # joint work, left out of the comparisons

def load(path="federalist.txt"):
    p = pathlib.Path(path)
    if not p.exists():                     # download once, then reuse the saved copy
        req = urllib.request.Request(URL, headers={"User-Agent": "Mozilla/5.0"})
        p.write_bytes(urllib.request.urlopen(req).read())
    raw = p.read_text(encoding="utf-8-sig").replace("\r", "")
    parts = re.split(r"^FEDERALIST No\. (\d+)\s*$", raw, flags=re.M)
    papers = {}                            # essay number -> (author, list of words)
    for k in range(1, len(parts), 2):
        n, body = int(parts[k]), parts[k + 1]
        m = re.search(r"^(HAMILTON|MADISON|JAY)$", body, re.M)
        author = m.group(1) if m else "?"
        if n in DISPUTED: author = "DISPUTED"
        if n in JOINT: author = "JOINT"
        start = body.find("To the People")
        end = body.rfind("\nPUBLIUS")
        text = body[start:end if end > start else None]
        papers[n] = (author, re.findall(r"[a-z]+(?:'[a-z]+)?", text.lower()))
    return papers

import statistics
from collections import Counter

papers = load()
rows = [(len(w), len(set(w)) / len(w)) for a, w in papers.values() if a in ("HAMILTON", "MADISON", "JAY")]
r = statistics.correlation([n for n, _ in rows], [t for _, t in rows])
print(f"{len(rows)} essays, lengths {min(n for n, _ in rows)} to {max(n for n, _ in rows)} words")
print(f"correlation between essay length and whole-essay type-token ratio: {r:.2f}")

print()
print("first 1,000 words of each essay that is at least 1,000 words long")
for a in ("HAMILTON", "MADISON", "JAY", "DISPUTED"):
    ws = [w[:1000] for author, w in papers.values() if author == a and len(w) >= 1000]
    ttr = [len(set(w)) / 1000 for w in ws]
    once = [sum(1 for c in Counter(w).values() if c == 1) / 1000 for w in ws]
    print(f"{a:9} essays {len(ws):2d}  distinct {statistics.mean(ttr):.3f} (spread {statistics.stdev(ttr):.3f})  used once {statistics.mean(once):.3f}")
# 70 essays, lengths 969 to 5725 words
# correlation between essay length and whole-essay type-token ratio: -0.76
#
# first 1,000 words of each essay that is at least 1,000 words long
# HAMILTON  essays 50  distinct 0.405 (spread 0.029)  used once 0.289
# MADISON   essays 14  distinct 0.390 (spread 0.037)  used once 0.269
# JAY       essays  5  distinct 0.407 (spread 0.028)  used once 0.282
# DISPUTED  essays 12  distinct 0.377 (spread 0.024)  used once 0.249
    

The correlation of -0.76 says that longer essays have lower ratios, so comparing whole essays would mostly compare their lengths. With every essay cut to 1,000 words the ratios are close: Hamilton 0.405, Jay 0.407 and Madison 0.390. The spread between essays by the same author (0.03 to 0.04) is about as big as the gap between the authors. So vocabulary richness does not separate these writers, and it is a weaker tool here than function-word rates. The disputed essays score 0.377, a little under Madison's average but within the spread; I would not lean on that. A measure that cannot beat its own noise is a result worth reporting, not hiding.

Comparing Authors: A Simple Delta

To use many words at once, put each essay on a common scale. For each of the k commonest words, take its rate in the essay and convert it to a z-score: how many standard deviations it is above or below the average essay (z = (rate − mean) / sd). Do the same for an author's pooled writing to get that author's profile. The distance between an essay and a profile is the average of the absolute differences between their z-scores. This is a simple version of the measure called Burrows' Delta (the name and the idea are from literary statistics; the details here are my simplified version). The smaller the Delta, the closer the essay is to that author.

Before trusting it on the disputed essays, test it where the answer is known. The snippet below leaves out one essay at a time, builds the profiles from the others, and guesses the author of the one left out. Only then does it score the twelve disputed essays, using all the known essays and the 30 commonest words.

import pathlib, re, urllib.request

URL = "https://www.gutenberg.org/cache/epub/1404/pg1404.txt"
DISPUTED = set(range(49, 59)) | {62, 63}   # the twelve disputed essays
JOINT = {18, 19, 20}                       # joint work, left out of the comparisons

def load(path="federalist.txt"):
    p = pathlib.Path(path)
    if not p.exists():                     # download once, then reuse the saved copy
        req = urllib.request.Request(URL, headers={"User-Agent": "Mozilla/5.0"})
        p.write_bytes(urllib.request.urlopen(req).read())
    raw = p.read_text(encoding="utf-8-sig").replace("\r", "")
    parts = re.split(r"^FEDERALIST No\. (\d+)\s*$", raw, flags=re.M)
    papers = {}                            # essay number -> (author, list of words)
    for k in range(1, len(parts), 2):
        n, body = int(parts[k]), parts[k + 1]
        m = re.search(r"^(HAMILTON|MADISON|JAY)$", body, re.M)
        author = m.group(1) if m else "?"
        if n in DISPUTED: author = "DISPUTED"
        if n in JOINT: author = "JOINT"
        start = body.find("To the People")
        end = body.rfind("\nPUBLIUS")
        text = body[start:end if end > start else None]
        papers[n] = (author, re.findall(r"[a-z]+(?:'[a-z]+)?", text.lower()))
    return papers

import statistics
from collections import Counter

papers = load()
known = {n: a for n, (a, w) in papers.items() if a in ("HAMILTON", "MADISON")}
counts = {n: Counter(w) for n, (a, w) in papers.items()}
size = {n: len(w) for n, (a, w) in papers.items()}

def make_model(train, k):
    pooled = Counter()
    for n in train:
        pooled.update(counts[n])
    vocab = [w for w, _ in pooled.most_common(k)]          # the k commonest words
    rates = [[counts[n][v] / size[n] for v in vocab] for n in train]
    mean = [statistics.mean(col) for col in zip(*rates)]
    sd = [statistics.pstdev(col) for col in zip(*rates)]
    profiles = {}
    for author in ("HAMILTON", "MADISON"):
        own = Counter()
        for n in train:
            if known[n] == author:
                own.update(counts[n])
        total = sum(own.values())
        profiles[author] = [(own[v] / total - mean[i]) / sd[i] for i, v in enumerate(vocab)]
    return vocab, mean, sd, profiles

def delta(n, model):
    vocab, mean, sd, profiles = model
    z = [(counts[n][v] / size[n] - mean[i]) / sd[i] for i, v in enumerate(vocab)]
    return {a: statistics.mean(abs(z[i] - p[i]) for i in range(len(vocab))) for a, p in profiles.items()}

K = 30
right, seen = Counter(), Counter()
for n, a in known.items():                                  # leave one essay out, then guess it
    model = make_model([m for m in known if m != n], K)
    d = delta(n, model)
    seen[a] += 1
    right[a] += min(d, key=d.get) == a
print("test on essays of known authorship:", dict(right), "right out of", dict(seen))

model = make_model(list(known), K)
print("the", K, "words used:", " ".join(model[0]))
tally = Counter()
for n in sorted(papers):
    if papers[n][0] == "DISPUTED":
        d = delta(n, model)
        best = min(d, key=d.get)
        tally[best] += 1
        print(f"essay {n}: Hamilton {d['HAMILTON']:.2f}  Madison {d['MADISON']:.2f}  -> {best}")
print(dict(tally))
# test on essays of known authorship: {'HAMILTON': 49, 'MADISON': 12} right out of {'HAMILTON': 51, 'MADISON': 14}
# the 30 words used: the of to and in a be that it is which as by this would have or for not will with from their an are they states been on state
# essay 49: Hamilton 0.96  Madison 0.87  -> MADISON
# essay 50: Hamilton 1.13  Madison 0.99  -> MADISON
# essay 51: Hamilton 1.15  Madison 1.01  -> MADISON
# essay 52: Hamilton 0.76  Madison 0.67  -> MADISON
# essay 53: Hamilton 0.85  Madison 0.66  -> MADISON
# essay 54: Hamilton 1.05  Madison 0.85  -> MADISON
# essay 55: Hamilton 0.76  Madison 0.85  -> HAMILTON
# essay 56: Hamilton 1.12  Madison 1.06  -> MADISON
# essay 57: Hamilton 1.04  Madison 0.80  -> MADISON
# essay 58: Hamilton 0.79  Madison 0.60  -> MADISON
# essay 62: Hamilton 0.81  Madison 0.64  -> MADISON
# essay 63: Hamilton 0.64  Madison 0.52  -> MADISON
# {'MADISON': 11, 'HAMILTON': 1}
    

On essays whose authors are known, the method names 49 of Hamilton's 51 and 12 of Madison's 14 correctly, 61 out of 65 (94%). That is the honest test, and it comes before looking at the disputed essays. For the twelve disputed essays, eleven are closer to Madison's profile and one, essay 55, is closer to Hamilton's, by a margin of 0.09 (0.76 against 0.85). That essay is also the one the search summary flagged as doubtful, which is interesting but does not prove anything: with 94% accuracy on known essays, about one wrong call in twelve is about what that accuracy leads you to expect, and the code cannot tell you why it was wrong. Note also what is missing: this method gives no probability, only a nearer or farther.

Words used (commonest k)	Known essays right (of 65)	Disputed essays closer to Madison
10	55	8 of 12
30	61	11 of 12
100	61	11 of 12
200	61	12 of 12

The table comes from Exercise 3. With 10 words the test accuracy is lower (55 of 65) and the disputed verdicts wobble; from 30 words up, the test accuracy is steady and the verdicts mostly agree. A result that survives changing the settings is more believable than one that depends on a lucky choice. This does not reproduce Mosteller and Wallace's work, which used selected marker words and Bayesian probabilities; it only shows that a much simpler method points the same way (their method as summarised in the search; check).

Common mistakes

Training on the essays you are testing, which makes the method look perfect; leaving the disputed essays in the training data; comparing whole texts of different lengths with a length-sensitive measure such as the type-token ratio; using topic words (department, confederation) as if they were style, since they change with the subject; relying on one marker word; reading a small margin as a verdict; having very few essays from one author (Jay has five), so his profile is unreliable; and forgetting that the numbers measure similarity to a profile, not who actually sat at the desk. Authorship statistics give evidence, not certainty.

Exercise 1

Write two short invented passages by two imaginary authors and one mystery passage. Choose ten function words, compute each passage's rate per 100 words, add up the absolute differences between the mystery rates and each author's, and say which author is closer. Then say why such short samples prove very little.

📄 View solution

Exercise 2

On the Federalist text, find the words (used at least 40 times in total by Hamilton and Madison together) whose rates differ most between the two. Use the base-2 logarithm of the ratio of rates, print the top eight each way, and decide which of them look like style and which like topic.

📄 View solution

Exercise 3

Repeat the leave-one-out test with the 10, 30, 100 and 200 commonest words. For each, report how many of the 65 known essays are guessed correctly and how many of the twelve disputed essays come out closer to Madison. Is the answer stable?

📄 View solution

Quick Reference

Function words: small grammatical words; frequent, unconscious, less tied to topic
Rate: 1000 * count / total_words, so essays of different lengths compare
Type-token ratio: len(set(words)) / len(words); falls as the text grows, so cut to equal length (here, r = -0.76 with length)
z-score: (rate - mean) / sd for each word
Simple Delta: the mean of abs(z_essay - z_author) over the k commonest words; smaller is closer
Honest test: leave one known essay out, rebuild the profiles, guess it (61 of 65 here)
Check robustness: change k and see whether the verdicts move
Source conflict: Madison's essay count is 14 or 29 depending on whether the disputed and joint essays are included

Coming next

Words by the Numbers 7 is about readability: sentence and word length, Flesch-style formulas, and what the scores miss.