Computational Linguistics 9, Exercise 3: the annotator who always says "no"

Code (run with py -X utf8, after the chapter's scripts):

D = [0] * 20
print([round(x, 3) for x in kappa(gold, D)])

Output from the run:

[0.75, 0.75, 0.0]   (observed agreement, chance agreement, kappa)

Raw agreement is 75% because 15 of the 20 answers in the key are "no", so D
is right on all of them. Chance agreement is also 0.75: the key says "yes"
25% of the time and D never does, so by the chance formula
0.25 * 0 + 0.75 * 1 = 0.75. Kappa = (0.75 - 0.75) / (1 - 0.75) = 0: no
agreement beyond chance. D has noticed nothing about the sentences.

This is the same arithmetic as the chapter's always-no detector (accuracy
0.75, recall 0): a high raw figure on unbalanced data can hide that nothing
has been learned. The Cohen's kappa article says 0 means agreement no better
than chance. It also warns that kappa can mislead with rare categories; I
have not tested other class balances.