Measuring Success

Computational Linguistics

Course 1 · Chapter 9 · Measuring Success

Chapters 7 and 8 quoted accuracy figures such as "over 95%" for taggers and "51.4%" for always choosing the most frequent word sense. A number like that means little until you know what was counted, what the easy alternative scores, and whether human judges agree on the right answer. This chapter covers those three questions.

The idea in a box

A score answers a question only when you can say what it counts, what a lazy system would get and how far people agree on the answer key.

Counting Right and Wrong

Wikipedia's Precision and recall article starts from four counts for a yes/no decision: true positives (TP, a positive prediction that matches an actual positive), false positives (FP, a false alarm), false negatives (FN, a miss) and true negatives (TN, a correct rejection). From these:

Measure	Formula	In words (the article's wording)
Precision	TP / (TP + FP)	"the fraction of relevant instances among the retrieved instances"
Recall	TP / (TP + FN)	"the fraction of relevant instances that were retrieved"
F1	2 · P · R / (P + R)	"the harmonic mean of precision and recall"
Accuracy	(TP + TN) / all	the share of all decisions that were right

The article says precision and recall often pull in opposite directions: perfect recall is trivial (retrieve everything) and perfect precision is easy (select only a few very likely items). It notes accuracy "can be a limited/misleading metric" when classes are unbalanced: with 95 negatives and 5 positives, predicting everything negative gives 0.95. Its worked example is a photo with 10 cats and 12 dogs, where a program picks out 8 dogs of which 5 are right: TP 5, FP 3, FN 7, TN 7, so precision 0.625 and recall about 0.417. Wikipedia's F-score article adds the general F-beta, where recall counts β² times as much as precision (β = 2 favours recall, β = 1/2 favours precision), and the form F1 = 2TP / (2TP + FP + FN). It also lists criticisms: F-measures "do not take true negatives into account", F1 gives precision and recall equal weight, and the score depends on the ratio of positive to negative cases. It says the F-score is common in NLP, for example for named entity recognition and word segmentation.

An evaluation you can run

The 20 sentences, their labels and the keyword detector below are mine. The task: decide whether a sentence is about a lighthouse (5 of the 20 are). I wrote the labels by hand, which is also a weakness; see agreement, below. Before trusting my functions, I fed them the article's dog counts (the last line of the output) and got its 0.625 and 0.417.

data = [
 ("the keeper lit the lamp", 1), ("the light turned all night", 1),
 ("the tower stood on the rocks", 1), ("the keeper climbed the stairs", 1),
 ("ships saw the beam from far out", 1), ("she carried a light bag", 0),
 ("the market sold fresh bread", 0), ("it was a light rain", 0),
 ("we walked to the station", 0), ("the train was late again", 0),
 ("he read a book on the sofa", 0), ("the cat slept by the stove", 0),
 ("the bakery opens at six", 0), ("the lamp on my desk broke", 0),
 ("rain fell on the town", 0), ("they ate soup for lunch", 0),
 ("the bus stopped near the park", 0), ("a child drew a red kite", 0),
 ("the teacher marked the essays", 0), ("the river froze in january", 0),
]
def detect(s):
    return int(any(w in s.split() for w in ("lamp", "light", "keeper", "lighthouse")))
def counts(gold, pred):
    tp = sum(g == 1 and p == 1 for g, p in zip(gold, pred))
    fp = sum(g == 0 and p == 1 for g, p in zip(gold, pred))
    fn = sum(g == 1 and p == 0 for g, p in zip(gold, pred))
    tn = sum(g == 0 and p == 0 for g, p in zip(gold, pred))
    return tp, fp, fn, tn
def scores(tp, fp, fn, tn):
    p = tp / (tp + fp) if tp + fp else 0.0
    r = tp / (tp + fn) if tp + fn else 0.0
    f1 = 2 * p * r / (p + r) if p + r else 0.0
    return p, r, f1, (tp + tn) / (tp + fp + fn + tn)
gold = [g for _, g in data]
pred = [detect(s) for s, _ in data]
print("tp fp fn tn:", counts(gold, pred))
print("precision, recall, F1, accuracy:", [round(x, 3) for x in scores(*counts(gold, pred))])
for (s, g), p in zip(data, pred):
    if g != p:
        print("wrong:", s, "gold", g, "pred", p)
none = [0] * len(gold)
print("always-no:", counts(gold, none), [round(x, 3) for x in scores(*counts(gold, none))])
print("dog check:", [round(x, 3) for x in scores(5, 3, 7, 7)])

tp fp fn tn: (3, 3, 2, 12) precision, recall, F1, accuracy: [0.5, 0.6, 0.545, 0.75] wrong: the tower stood on the rocks gold 1 pred 0 wrong: ships saw the beam from far out gold 1 pred 0 wrong: she carried a light bag gold 0 pred 1 wrong: it was a light rain gold 0 pred 1 wrong: the lamp on my desk broke gold 0 pred 1 always-no: (0, 0, 5, 15) [0.0, 0.0, 0.0, 0.75] dog check: [0.625, 0.417, 0.5, 0.545]

The detector's accuracy is 0.75, which sounds respectable. But the lazy system that says "no" to every sentence also scores 0.75, because 15 of the 20 sentences are negatives, so the detector has gained nothing in accuracy. Only precision (0.5) and recall (0.6) reveal what it is doing: three of its six alarms are wrong (light bag, light rain, lamp on my desk, which are the word sense problem of Chapter 8) and it misses two lighthouse sentences that contain none of its words.

Baselines

A baseline is the simple system any new method must beat. The always-no rule above is one. Earlier chapters met others on real data, all from the articles I read: giving each word its most common tag and unknown words the proper-noun tag "will approach 90% accuracy" (Part-of-speech tagging, citing Charniak, 1997), so a tagger at 93-95% has to be read against that; and always choosing the most frequent sense scored 51.4% (SemEval-2007) and 57% (Senseval-2) in Word-sense disambiguation. This is why Chapter 7 treated its own 24/24 with suspicion. The term "baseline" as a general concept comes from those pages' usage and my own explanation; I did not read a separate article on it.

Watch out: do not tune on your test set

A detector built after looking at the errors will look better than it is. In Exercise 1 I change the keyword list after seeing the five mistakes above, and the score rises; that rise says little about new text, because the test sentences shaped the fix. Keep a test set apart from the data you tune on.

Do the Judges Agree?

The answer key (gold) was written by one person. Wikipedia's Inter-rater reliability article defines the field as "the degree of agreement among independent observers who rate, code, or assess the same phenomenon". Its simplest measure, joint probability of agreement, is the percentage of cases where raters agree; the weakness is that it ignores chance, so with few categories two raters can agree "by pure chance" often. The kappa family corrects for that: Cohen's kappa for two raters, Fleiss' kappa for any fixed number. The article gives no numeric cut-offs.

Wikipedia's Cohen's kappa article gives κ = (p_o − p_e) / (1 − p_e), where p_o is the observed proportion of agreement and p_e the agreement expected by chance from each rater's own label proportions. It ranges from −1 to 1, with 0 meaning agreement no better than chance. Its worked example (50 grant applicants, Yes/No; 20 both-Yes, 5, 10, 15 both-No) gives p_o = 0.7, p_e = 0.5 and κ = 0.4; my function reproduces that (first output line). The article also lists the Landis and Koch bands (for example 0.61-0.80 "substantial") and Fleiss's, but calls such guidelines arbitrary, based on opinion rather than evidence. Its limitations include that kappa can mislead with rare categories.

def kappa(a, b):
    n = len(a)
    po = sum(x == y for x, y in zip(a, b)) / n
    pe = sum((a.count(c) / n) * (b.count(c) / n) for c in set(a) | set(b))
    return po, pe, (po - pe) / (1 - pe)

# the Cohen's kappa article's 50-applicant table
A = ["Y"] * 20 + ["Y"] * 5 + ["N"] * 10 + ["N"] * 15
B = ["Y"] * 20 + ["N"] * 5 + ["Y"] * 10 + ["N"] * 15
print("article check:", [round(x, 3) for x in kappa(A, B)])

# my 20 sentences: annotator B disagrees with my labels on two
A = gold
B = list(A); B[2] = 0; B[13] = 1
print("agree", sum(x == y for x, y in zip(A, B)), "of", len(A))
print("po, pe, kappa:", [round(x, 3) for x in kappa(A, B)])

# annotator C says "yes" to just two sentences
C = [0] * 20; C[0] = 1; C[5] = 1
print("annotator C:", sum(x == y for x, y in zip(A, C)), "of 20;", [round(x, 3) for x in kappa(A, C)])

article check: [0.7, 0.5, 0.4] agree 18 of 20 po, pe, kappa: [0.9, 0.625, 0.733] annotator C: 15 of 20; [0.75, 0.7, 0.167]

B agrees on 18 of 20 (90%) and gets κ ≈ 0.73. Annotator C agrees on 15 of 20 (75%) but κ is only about 0.17: because 75% of the labels are "no", chance alone would produce 70% agreement, so only a little of C's agreement is real. The 75% agreement figure and the 0.75 accuracy of always-no are the same arithmetic seen from two sides. The invented annotators B and C are only to show the maths; no real annotators were used.

Why this matters for the field: the Word-sense disambiguation article says human annotators in Senseval-2 "agreed in only 85% of word occurrences" with fine-grained senses and treats human performance as an upper bound, so a system cannot sensibly be asked to beat the people who made the key.

Note on sources

This chapter uses Wikipedia's Precision and recall, F-score and Inter-rater reliability (each read in full) and Cohen's kappa (read to 100,000 of 105,852 characters; the unread tail was navigation per the fetch). The fetch tool flagged one sentence in Precision and recall (about an all-negative classifier and "100% recall") as inconsistent; I did not use it. The baseline figures are from the Part-of-speech tagging and Word-sense disambiguation pages as read in Chapters 7 and 8. The 20 sentences, labels, detector, annotators B and C, and every output are mine and were run with py -X utf8. I did not read an article on baselines, nor on micro/macro averaging beyond the F-score page's brief mention.

Try It Yourself

Exercise 1

Replace the keyword list with keeper, tower, beam and lighthouse, and recompute the counts and scores. Why is the improvement not good evidence that the new detector is better?

📄 View solution

Exercise 2

Using the article's F-beta formula, compute F1, F2 and F0.5 for the original detector. For a tool that flags sentences for a human to read, which would you report, and why?

📄 View solution

Exercise 3

Compute raw agreement and kappa between the answer key and an annotator who labels every sentence "no". Explain the result.

📄 View solution

Chapter 9 Quick Reference

Precision = TP / (TP + FP); recall = TP / (TP + FN); F1 = harmonic mean of the two
Accuracy can mislead when one class is common: compare with the always-majority baseline
Baseline: the simple method a new system must beat (most frequent tag, most frequent sense)
Kappa = (p_o − p_e) / (1 − p_e): agreement corrected for chance
Do not tune on the test set

Coming next

Computational Linguistics 10 looks at what the field cannot yet do, who it affects, and where to go from here, including the NLP and LLMs courses.