Computational Linguistics 10, Exercise 2: fixing the word list

Code (run with py -X utf8, after the chapter's script):

known |= {"dinnae", "ken", "guid", "nicht"}
for ref, said in pairs:
    out = toy_recognise(said)
    print(repr(said), "->", repr(out), round(wer(ref, out), 2))

Output from the run:

'i do not know' -> 'i do not know' 0.0
'i dinnae ken' -> 'i dinnae ken' 0.75
'it was a guid nicht' -> 'it was a guid nicht' 0.4

The <unk> markers are gone, but the WER is exactly the same: 0.75 and 0.4.

Why: the reference for each pair is the standard English wording ("i do not
know", "it was a good night"). The recogniser now transcribes the Scots
speaker's words correctly, but the score compares them with different words,
so each Scots word counts as a substitution (and "i dinnae ken" is also one
word shorter than "i do not know", which adds a deletion).

What the remaining score measures: not recognition errors but the distance
between the speaker's variety and the standard wording in the key. The
transcript is right; the key penalises it.

Change to the key: let the reference follow what the speaker said (a
transcription of "i dinnae ken"), or accept a set of acceptable variants.
If the task is to get a standard-English meaning from the speech, that is a
separate, explicit translation step, and it should be scored as such. This
is my suggestion; I have not tested it on real data, and the toy recogniser
does no real recognition.

It links to Sociolinguistics: Language, Dialect and Society 8 (Standards and
Attitudes): which form counts as "correct" is a social choice built into the
data. That link is my reading, not a claim from the Wikipedia articles.