Computational Linguistics 7, Exercise 1: unseen words

Code (run with py -X utf8, after the chapter's script):

s = "the keeper climbed the stairs".split()
print([w for w in s if w not in best])
print(list(zip(s, baseline(s))))
print(list(zip(s, viterbi(s))))
def suffix_rule(words, tags):
    return ["VERB" if w.endswith("ed") and t == "NOUN" and w not in best else t
            for w, t in zip(words, tags)]
print(list(zip(s, suffix_rule(s, baseline(s)))))

Output from the run:

unseen words: climbed, stairs
baseline: the DET, keeper NOUN, climbed NOUN, the DET, stairs NOUN
viterbi:  the DET, keeper NOUN, climbed VERB, the DET, stairs NOUN
+ suffix rule: the DET, keeper NOUN, climbed VERB, the DET, stairs NOUN

Reading:
- The baseline has no entry for "climbed" and "stairs", so both become NOUN.
  "climbed" is wrong.
- The Viterbi tagger got "climbed" as VERB, which I read as being because the tag sequence
  NOUN -> VERB -> DET is well supported by the counts (and an unseen word costs
  the same smoothing amount under every tag), so context decides.
- My suffix rule also gets it, using spelling instead of context. It would be
  wrong for an unseen noun ending in -ed (for example "hundred"), which I have
  not run.
- The Brill tagger article says unknown words get NOUN by default and that
  lexical rules may use capitalization and prefix or suffix strings; my rule is
  of that kind but is my own, not Brill's.