Computational Linguistics 9, Exercise 1: a new keyword list

Code (run with py -X utf8, after the chapter's script):

def detect2(s):
    return int(any(w in s.split() for w in ("keeper", "tower", "beam", "lighthouse")))
p2 = [detect2(s) for s, _ in data]
print(counts(gold, p2), [round(x, 3) for x in scores(*counts(gold, p2))])
print([s for (s, g), p in zip(data, p2) if g != p])

Output from the run:

(4, 0, 1, 15) [1.0, 0.8, 0.889, 0.95]
['the light turned all night']

Precision rose from 0.5 to 1.0, recall from 0.6 to 0.8, F1 from 0.545 to
0.889, accuracy from 0.75 to 0.95. The only remaining error is a lighthouse
sentence that now has no keyword.

Why this is weak evidence: I chose "tower" and "beam" and dropped "lamp" and
"light" after seeing which sentences the first detector got wrong. The test
sentences shaped the fix, so the same 20 sentences cannot honestly measure it.
It also has no test on text it has not seen: "the keeper of the shop" would be
a false alarm, which I did not run. A fair test needs new, separately labelled
sentences, kept out of the tuning.