Text Detective

A stranger hands you a page of text with no title and no name. Can you say what language it is in, who probably wrote it, and what it is about? This last chapter puts the course to work on exactly that. Each question gets one tool you already own: letter counts from Words by the Numbers 3, the author comparison from Words by the Numbers 6, and the keyword ranking from Words by the Numbers 8. The new skill is not another formula. It is knowing how far to trust each verdict, which is why the chapter spends as much time on the failures as on the successes.

Case 1: What Language Is This?

Language identification is the easiest case, and it is old. A 1994 paper by William Cavnar and John Trenkle, "N-Gram-Based Text Categorization" (presented at the SDAIR-94 symposium in Las Vegas, pages 161 to 175, per the search), built a profile of the most frequent short letter sequences for each language and gave a new text to the language whose profile it matched best. Secondary summaries report accuracy of about 99.8% on postings of at least 300 bytes, using the 400 most frequent sequences, and 99.3% on shorter ones; I did not check those figures in the paper, and the search did not say which sequence lengths they used (check). The idea is the one from Words by the Numbers 3, that every language has a letter profile, plus the idea from Words by the Numbers 4, that pairs of neighbours say more than single items.

The first snippet trains one profile per language on one book, then tests passages of 20 to 300 characters taken at random from different books. It does this twice: once with single letters as the features and once with letter pairs. The score is cosine similarity, which measures how closely two lists of frequencies point the same way (1 means identical shape, 0 means nothing in common). Each result below is out of 500 passages, 100 per language.

Three things stand out. Pairs beat single letters at every length, because a pair like th or qu carries more of a language's habits than a letter count does. Both improve with more text, as Words by the Numbers 1 would lead you to expect: more data, less luck. And the letter table at the end of the output shows why a single-letter profile works at all, and where it is weak. The letters w and k are absent from the French, Spanish and Italian profiles, z is six times commoner in German than in English (1.03% against 0.17%), and ñ and ç identify Spanish and French almost alone. But e and n are everywhere, and the pairs add the rest. The training books are modest (the German one is only 179 thousand characters), and the test is five languages that all use the Latin alphabet; languages written in other scripts would be easier to tell apart, and closely related languages harder.

Case 2: Who Wrote This?

Authorship is harder, because every author writes the same language. Words by the Numbers 6 used the habit that is hardest to fake: how often a writer uses small function words such as the, of and but. Here is the measure in its general form. Standardise each word's rate against a reference collection (subtract the mean, divide by the standard deviation) so that all words count equally, then take the average absolute difference between the passage's standardised rates and each author's. This is Burrows' Delta, from John Burrows, "'Delta': a Measure of Stylistic Difference and a Guide to Likely Authorship", Literary and Linguistic Computing, volume 17, number 3, 2002, pages 267 to 287, per the search. The search also reported a reviewer's note that it works least well on texts of a genre unlike the rest of an author's work, and a 2004 test of the measure by David Hoover in the same journal (volume 19, pages 453 to 475). My version standardises against 2,000-word chunks of the five training books, which is a simplification of Burrows' reference set.

The second snippet trains on one book by each of five authors and tests on a different book by each. It varies two knobs: how many of the commonest words are used, and how long the passage is. It ends with a deliberate trap, a sixth author the system has never heard of, to see whether Delta can recognise a stranger.

Chance is 20%, so even ten words and 200 words of text do far better than guessing, and both more words and more text help. With 300 words and a 2,000-word passage the score is 99%. Each cell is only 250 trials with one seed, so differences of a few points between cells are not reliable. The last two lines are the sobering part. For passages by the five known authors the best Delta has a median of 0.93 and a 95th percentile of 1.21. For passages by the stranger the median is 0.95, practically the same, and only 2 of 50 pass the 1.21 cut-off, about the 5% you would expect from known authors by construction. The stranger was blamed on Dickens 20 times, Doyle 16, Wells 12 and Austen 2. Delta always names someone. It measures which candidate is nearest, and nearest is not the same as right.

Case 3: What Is It About?

For topic, Words by the Numbers 8 gave the tool: TF-IDF, which scores a word highly when it is frequent in the passage but rare across the collection. The third snippet is the full detective. It takes six mystery passages of 1,000 words, from books the profiles were not trained on (one is by an author outside the candidate list). For each it works out the language from the first 300 characters, names the nearest author if the language is English (flagging a Delta above the 1.21 cut-off from the previous snippet), and lists the five top TF-IDF keywords, measured against 500-word pieces of the books on the shelf for that language. The true answers are printed last.

Every language verdict is right, and so are the three author verdicts on passages by candidate authors; the stranger is the lesson. Passage F is by an author who is not on the list, and the detective named Doyle with a Delta of 0.83, which is below the median of 0.93 for passages by the real candidates, so no cut-off could have caught it. Yet the keywords for F (rochester, governess, eyre) are the strongest clue in the whole report, since a character name is a fingerprint of one book. A good detective uses the tools against each other. A keyword list full of proper names says the topic evidence is really about the story, not the style; a Delta that is merely adequate says to hold the verdict loosely; and language, the most reliable of the three, comes first because the other two depend on it. Notice also why the keywords are mostly names: with a shelf this small, a character who appears in one book is rare everywhere else, so TF-IDF loves characters. A bigger shelf with many books on the same subject would favour subject words instead (my expectation, not tested here).

The last table lines the three tools up so you can choose between them. The reliability column summarises what this chapter measured, not a general rule.