Letters and Secrets

Words by the Numbers

Chapter 3 · Letters and Secrets

Words by the Numbers 2 counted whole words and found a steep, lopsided pattern. This chapter drops down a level and counts letters. Letters are far fewer than words (26 of them, in the English alphabet) so even a short text gives a usable count, and the counts turn out to be stubbornly regular. That regularity has two famous uses: breaking secret messages, which is where this story starts, and recognising which language a text is written in without understanding a word of it. The maths is light: percentages, a remainder after division, and a distance that is just a sum of differences.

What was run, and what to trust

Every Python snippet in this chapter and in the three exercise solutions was executed (Python 3.14), and the output in the comments is what Python printed. The English text counted in the first snippet is the reference documentation that ships inside Python (the pydoc_data.topics module), about 363,000 letters of technical English; it is not typical prose, and its counts change a little between Python versions. The short passages used for the language-recognition snippet were written for this course (the same small story in five languages, with my own translations, not checked by native speakers), so treat that experiment as a demonstration on tiny, parallel, same-topic samples and not as a measurement of the languages. Background facts come from web searches that returned summaries of web pages; such fetched pages are often truncated or model-summarised, so treat the wording as a paraphrase. The sources agree on Al-Kindi’s treatise and its method but disagree on the date of Babbage’s work on the Vigenère cipher (conflict, below). Messages and shift keys in the cipher snippets are invented, illustrative examples. Statements marked (check) were not confirmed against a source.

Letters in Real Text

Start by counting. The snippet below keeps only the 26 unaccented letters, lowercased, from the whole Python reference documentation and prints the eight commonest. If every letter were equally likely each would take 1 ÷ 26, about 3.8%. Real English is nothing like that.

import re
from collections import Counter
from pydoc_data.topics import topics

def letters_only(text):
    return re.findall(r"[a-z]", text.lower())

letters = letters_only(" ".join(topics.values()))
counts = Counter(letters)
total = len(letters)
print("letters:", total, "| an even spread would give each", f"{1 / 26:.1%}")
print("letter   count  share")
for letter, n in counts.most_common(8):
    print(f"{letter:>6} {n:7d} {n / total:6.1%}")
rarest = sorted(counts.items(), key=lambda pair: pair[1])[:4]
print("rarest:", [(letter, f"{n / total:.2%}") for letter, n in rarest])
# letters: 363372 | an even spread would give each 3.8%
# letter   count  share
#      e   48945  13.5%
#      t   37536  10.3%
#      a   28415   7.8%
#      i   26992   7.4%
#      n   26949   7.4%
#      s   26622   7.3%
#      o   23879   6.6%
#      r   21733   6.0%
# rarest: [('z', '0.09%'), ('q', '0.19%'), ('j', '0.32%'), ('k', '0.38%')]
    

The letter e takes 13.5% of all letters, more than three times the even-spread share, and the top four letters (e, t, a, i) together make up about 39%. At the other end, z accounts for under a tenth of one percent. The familiar “e, t, a…” order is the standard story about English; here the order is e, t, a, i, n, s, o, r, because this is documentation text with plenty of i from words such as is, if and in. Different texts give slightly different shares, but a text of any decent length reorders only the middle of the list. One source in my search also notes that th is the most likely pair of letters in English; pairs are the subject of Words by the Numbers 4, so take that as a preview, not a result.

Unit counted	Types to count	Text needed for a stable picture
Letters	26	a few dozen letters gave usable results in the experiment below
Words	thousands, with a long tail (Words by the Numbers 2)	tens of thousands of tokens for a steady slope

Al-Kindi and the Shift Cipher

Regular letter counts are a gift to anyone trying to read a message not meant for them. According to the sources I found, the earliest known description of breaking a cipher by counting letters is a treatise by the Arab scholar Al-Kindi, A Manuscript on Deciphering Cryptographic Messages, usually dated to about 850 CE in Baghdad, with a copy rediscovered in an Ottoman archive in Istanbul in 1987 (a Simon Singh article and a history-of-information page; second-hand, check). The method is simple: count how often each symbol appears in the ciphertext, then match the commonest symbols to the commonest letters of the language. One summary says he began from the observation that in a given language the letters settle at stable frequencies. Another source cautions that the manuscript’s examples are easiest to explain with the English alphabet, but that Al-Kindi worked on Arabic.

The simplest cipher to attack this way is the shift cipher, usually called the Caesar cipher after the tradition that Julius Caesar used a shift of three (my recollection of the Suetonius account; not confirmed in my search; check). Each letter moves a fixed number of places along the alphabet, wrapping round from z to a. Wrapping is where a little arithmetic comes in: the new position is (old position + k) remainder 26, which Python writes (i + k) % 26. There are only 26 possible shifts, so you could try them all, but frequency gives a shortcut: assume the commonest ciphertext letter stands for e, and the shift is the distance between them.

import string
from collections import Counter

ALPHA = string.ascii_lowercase

def shift(text, k):
    # Move every letter k places along the alphabet; leave other characters alone.
    return "".join(
        ALPHA[(ALPHA.index(ch) + k) % 26] if ch in ALPHA else ch for ch in text
    )

plain = ("meet me by the old bridge when the evening bells are heard and "
         "bring the letter that the secretary left on the table")
secret = shift(plain, 3)
print("ciphertext:", secret)

cipher_counts = Counter(ch for ch in secret if ch in ALPHA)
commonest = cipher_counts.most_common(1)[0][0]
guess = (ALPHA.index(commonest) - ALPHA.index("e")) % 26
print("commonest cipher letter:", commonest, "| assumed to be e, so shift =", guess)
print("recovered:", shift(secret, -guess))
# ciphertext: phhw ph eb wkh rog eulgjh zkhq wkh hyhqlqj ehoov duh khdug dqg eulqj wkh ohwwhu wkdw wkh vhfuhwdub ohiw rq wkh wdeoh
# commonest cipher letter: h | assumed to be e, so shift = 3
# recovered: meet me by the old bridge when the evening bells are heard and bring the letter that the secretary left on the table
    

The commonest cipher letter is h, which is three places after e, so the shift is 3 and the message comes back intact. The message here is 116 characters long, invented for the example, and plenty of es made the guess easy. The next section shows what happens when it is not.

When the “e” Guess Fails

Frequency analysis is a statistical argument, and statistics need data. A very short message may have no e at all. Here are two ways to attack the same ciphertexts: the “commonest letter is e” rule, and a more careful best-fit rule that tries all 26 shifts and keeps the one whose letter shares are closest to the English reference profile (using the sum of absolute differences, so the distance is 0 for identical profiles). The reference profile comes from the documentation text counted earlier. All four messages are enciphered with the same invented key, 7.

import string
from collections import Counter
from pydoc_data.topics import topics

ALPHA = string.ascii_lowercase

def shift(text, k):
    return "".join(
        ALPHA[(ALPHA.index(ch) + k) % 26] if ch in ALPHA else ch for ch in text
    )

def shares(text):
    letters = [ch for ch in text.lower() if ch in ALPHA]
    counts = Counter(letters)
    return [counts[ch] / len(letters) for ch in ALPHA]

english = shares(" ".join(topics.values()))   # reference profile

def crack_by_e(cipher):
    commonest = Counter(ch for ch in cipher if ch in ALPHA).most_common(1)[0][0]
    return (ALPHA.index(commonest) - ALPHA.index("e")) % 26

def crack_by_fit(cipher):
    # Try all 26 shifts; keep the one whose letter shares are closest to English.
    def gap(k):
        trial = shares(shift(cipher, -k))
        return sum(abs(a - b) for a, b in zip(trial, english))
    return min(range(26), key=gap)

messages = [
    "the secret meeting is moved to the usual place after the evening bells",
    "attack at dawn",
    "buy milk and a loaf of bread",
    "send help to the old mill",
]
KEY = 7
for plain in messages:
    cipher = shift(plain, KEY)
    by_e, by_fit = crack_by_e(cipher), crack_by_fit(cipher)
    print(f"{len(plain):3d} chars | e-guess {'ok  ' if by_e == KEY else 'FAIL'}"
          f" | best-fit {'ok  ' if by_fit == KEY else 'FAIL'} | {plain[:28]}")
#  70 chars | e-guess ok   | best-fit ok   | the secret meeting is moved
#  14 chars | e-guess FAIL | best-fit FAIL | attack at dawn
#  28 chars | e-guess FAIL | best-fit FAIL | buy milk and a loaf of bread
#  25 chars | e-guess FAIL | best-fit ok   | send help to the old mill
    

On the 70-character message both rules succeed. On attack at dawn (14 characters) and buy milk and a loaf of bread (28 characters) both fail; on send help to the old mill (25 characters) only best-fit succeeds. Four messages prove nothing statistical, but they show the shape of the problem: with a few dozen letters the counts are noisy, and a smarter rule, or a human who notices that the output is gibberish, has to take over. Note, too, that the reference profile here comes from technical documentation, not from the kind of message a spy would write, so a profile that matches the sender’s style would do better.

Cipher	Keys	Letter counts of the ciphertext
Shift (Caesar)	26 shifts, including the do-nothing shift of 0	same shape as the plaintext, only moved along the alphabet
Simple substitution	26! = 403,291,461,126,605,635,584,000,000 (about 4.0 × 10²⁶, computed by Python)	same set of counts, each tied to one symbol: Al-Kindi’s method still applies
Vigenère	a keyword; each letter is shifted by the matching keyword letter in turn	flattened, because one plaintext letter becomes several different symbols

That last row is why the Vigenère cipher was thought unbreakable for a long time. A search on the Kasiski examination reports that Friedrich Kasiski published a method in 1863, based on spotting repeated strings in the ciphertext that reveal the keyword length, and that Charles Babbage had independently found a way into the cipher earlier without ever explaining it. The sources conflict on when: one gives 1846, another 1854 (conflict; check). Once the length is known, each position is a separate shift cipher and the e-guess works again.

Letter Profiles: Telling Languages Apart

Turn the idea around. Instead of using a known language to read a hidden message, use the letter counts to identify the language. A profile is a list of 26 numbers: the share of each letter. Two texts in the same language should have close profiles, and two different languages should differ. To compare profiles use the distance from the last section: add up the absolute differences, letter by letter. The smaller the sum, the more alike the texts.

The experiment below uses five short training passages (the same invented harbour story in English, French, German, Spanish and Italian), builds a profile from each, and then tries to say which language each of five different test sentences is in. Accents are stripped first (é becomes e, ö becomes o), which keeps the profile to 26 letters. To see how much text is needed, it classifies the first 20, 40 and 80 letters of each test sentence, and then all of it. The passages are long because the data is in the snippet, so every line can be run as it stands.

TRAIN = {
    "English": (
        "The old harbour was quiet in the early morning, and the fishermen were "
        "already mending their nets on the stone quay. A cold wind came in from "
        "the sea, carrying the smell of salt and tar, while the children of the "
        "village walked slowly to school with their books under their arms. "
        "Nobody spoke much, for the week had been long and the weather had been "
        "hard on everyone who worked on the water."
    ),
    "French": (
        "Le vieux port était calme au petit matin, et les pêcheurs réparaient "
        "déjà leurs filets sur le quai de pierre. Un vent froid venait de la mer, "
        "apportant l'odeur du sel et du goudron, pendant que les enfants du "
        "village marchaient lentement vers l'école, leurs livres sous le bras. "
        "Personne ne parlait beaucoup, car la semaine avait été longue et le "
        "temps avait été dur pour tous ceux qui travaillaient sur l'eau."
    ),
    "German": (
        "Der alte Hafen war am frühen Morgen still, und die Fischer flickten "
        "schon ihre Netze am steinernen Kai. Ein kalter Wind kam vom Meer und "
        "brachte den Geruch von Salz und Teer mit, während die Kinder des Dorfes "
        "langsam zur Schule gingen, ihre Bücher unter dem Arm. Niemand sprach "
        "viel, denn die Woche war lang gewesen und das Wetter hatte allen "
        "zugesetzt, die auf dem Wasser arbeiteten."
    ),
    "Spanish": (
        "El viejo puerto estaba tranquilo a primera hora de la mañana, y los "
        "pescadores ya remendaban sus redes en el muelle de piedra. Un viento "
        "frío llegaba del mar, trayendo el olor a sal y alquitrán, mientras los "
        "niños del pueblo caminaban despacio hacia la escuela con los libros "
        "bajo el brazo. Nadie hablaba mucho, porque la semana había sido larga "
        "y el tiempo había sido duro para todos los que trabajaban en el agua."
    ),
    "Italian": (
        "Il vecchio porto era tranquillo di primo mattino, e i pescatori "
        "rammendavano già le loro reti sulla banchina di pietra. Un vento freddo "
        "veniva dal mare, portando l'odore di sale e di catrame, mentre i "
        "bambini del villaggio camminavano lentamente verso la scuola con i "
        "libri sotto il braccio. Nessuno parlava molto, perché la settimana era "
        "stata lunga e il tempo era stato duro per tutti quelli che lavoravano "
        "sull'acqua."
    ),
}
TEST = {
    "English": (
        "She opened the window and listened to the rain falling on the roof of "
        "the little house, thinking about the letter that had arrived the day "
        "before and what she should write in answer to it."
    ),
    "French": (
        "Elle ouvrit la fenêtre et écouta la pluie qui tombait sur le toit de la "
        "petite maison, en pensant à la lettre qui était arrivée la veille et à "
        "ce qu'elle devait écrire en réponse."
    ),
    "German": (
        "Sie öffnete das Fenster und hörte dem Regen zu, der auf das Dach des "
        "kleinen Hauses fiel, und dachte an den Brief, der am Tag zuvor "
        "angekommen war, und was sie darauf antworten sollte."
    ),
    "Spanish": (
        "Ella abrió la ventana y escuchó la lluvia que caía sobre el tejado de "
        "la casita, pensando en la carta que había llegado el día anterior y en "
        "lo que debía escribir como respuesta."
    ),
    "Italian": (
        "Lei aprì la finestra e ascoltò la pioggia che cadeva sul tetto della "
        "piccola casa, pensando alla lettera che era arrivata il giorno prima e "
        "a cosa dovesse scrivere in risposta."
    ),
}

import string
import unicodedata
from collections import Counter

ALPHA = string.ascii_lowercase

def base_letters(text):
    # Strip accents (e-acute becomes e, o-umlaut becomes o) and keep a-z only.
    decomposed = unicodedata.normalize("NFD", text.lower())
    return [ch for ch in decomposed if ch in ALPHA]

def profile(text):
    letters = base_letters(text)
    counts = Counter(letters)
    return [counts[ch] / len(letters) for ch in ALPHA]

def distance(p, q):
    # Sum of the absolute differences between two letter-share lists.
    return sum(abs(a - b) for a, b in zip(p, q))

profiles = {lang: profile(text) for lang, text in TRAIN.items()}

def classify(text):
    p = profile(text)
    return min(profiles, key=lambda lang: distance(p, profiles[lang]))

for lang, p in profiles.items():
    top = sorted(zip(ALPHA, p), key=lambda pair: -pair[1])[:3]
    print(f"{lang:8}", ", ".join(f"{letter} {share:.1%}" for letter, share in top))

for n in (20, 40, 80, None):
    wrong = []
    for lang, text in TEST.items():
        sample = "".join(base_letters(text)[:n])
        if classify(sample) != lang:
            wrong.append((lang, classify(sample)))
    label = "all" if n is None else n
    print(f"first {label} letters: {5 - len(wrong)}/5 right", wrong)
# English  e 14.7%, h 8.3%, o 8.0%
# French   e 17.4%, a 9.8%, t 8.8%
# German   e 17.4%, n 10.0%, r 8.1%
# Spanish  a 16.5%, e 11.4%, o 8.7%
# Italian  a 13.2%, e 9.9%, o 9.4%
# first 20 letters: 4/5 right [('Italian', 'French')]
# first 40 letters: 5/5 right []
# first 80 letters: 5/5 right []
# first all letters: 5/5 right []
    

The top-three lists already look different: German and French put e at 17.4%, while Spanish and Italian lead with a. English has h as its second letter in this small sample, from words like the. The classifier gets all five right given 40 letters or more, and four of five at 20 letters, where an Italian sentence is taken for French. That is a nice result, but it is a very small, friendly test: the passages are translations of each other, there are five sentences, and published frequency tables, built from far larger collections, would differ from these small-sample shares (the 14.7% for e in English here, for instance, is well above the figure I recall for large English corpora; check). Real language identifiers use more than single letters, typically groups of two or three, which is where the next chapter goes.

Sample of the test sentence	Correct of 5	Mistake
first 20 letters	4	Italian taken for French
first 40 letters	5	none
first 80 letters	5	none
whole sentence	5	none

Common mistakes

Assuming the commonest ciphertext letter is always e (short messages, or messages with unusual vocabulary, break this); building a profile from a text in a different genre and expecting a perfect match; forgetting to strip accents or to lowercase, so that E and é count as different letters; treating a result on five sentences as a measured accuracy; and forgetting that the shift formula needs % 26 to wrap around the end of the alphabet.

Exercise 1

Count the letters in a short invented passage of your own (two sentences will do). Print the top five letters with their percentage shares, the share of e compared with the even-spread 3.8%, and how many of the 26 letters never appear.

📄 View solution

Exercise 2

Encrypt five short messages with different shift keys. Break each with two rules: (A) the commonest letter is e; (B) try each of the three commonest letters as e and keep the shift that makes the words “the”, “and”, “to” appear most often. How many does each rule get right?

📄 View solution

Exercise 3

Using the five training passages from this chapter, print the three closest pairs of languages by profile distance and the furthest pair. Then add the Python documentation as a second kind of English and see whether it sits closer to the story English or to the other languages.

📄 View solution

Quick Reference

Letter profile: a list of 26 shares, one per letter, summing to 100%
Even spread: 1 ÷ 26, about 3.8% each; real English is far from it (e was 13.5% in the documentation text)
Frequency analysis: match the commonest ciphertext symbols to the commonest letters of the language (Al-Kindi, about 850 CE per the sources)
Shift cipher: ALPHA[(i + k) % 26]; to undo it, shift by -k
Distance between profiles: sum(abs(a - b) for a, b in zip(p, q)), smaller means more alike
Limits: short messages, unusual vocabulary and genre differences all weaken the argument
Strip accents: unicodedata.normalize("NFD", text), then keep only a to z

Coming next

Words by the Numbers 4 is about neighbouring words: bigrams and trigrams, collocations, and which word pairs are more than chance.