Computational Linguistics 7, Exercise 3: why 24 of 24 is weak evidence

Scores from the run (5 hand-tagged test sentences, 24 tokens):
baseline 22 of 24; baseline plus the rule 24 of 24; Viterbi 24 of 24.

Three reasons the score is weak:
1. Size. 24 tokens: one mistake changes the score by about 4 percentage
   points, and a lucky sample can hide whole classes of error.
2. Overlap. The test sentences reuse the training words and sentence patterns
   ("the keeper will light the lamp"), so there are almost no unseen words.
   The Exercise 1 sentence, which has two unseen words, is much harder.
3. Tuning on the errors. I wrote the rule after seeing the baseline get
   "we light the lamp" wrong, so the rule is shaped to this data. Likewise I
   wrote both the training and the test text myself.

A better test: tag a larger set of sentences written or collected
independently of the training set, never used to write rules or choose the
smoothing constant, tagged by someone else, and report accuracy separately for
known and unseen words. Chapter 9 of the course covers measurement.

For scale, the Part-of-speech tagging article's figures (about 90% for the
most-common-tag baseline, over 95% for the HMM methods) come from large
corpora; I have not run anything of that size, so mine cannot be compared.