Computational Linguistics 10, Exercise 3: scores by group

Code (run with py -X utf8, after the chapter's wer function, with the
original word list):

results = [("A", "i do not know", "i do not know"),
           ("A", "it was a good night", "it was a good night"),
           ("B", "i do not know", "i dinnae ken"),
           ("B", "it was a good night", "it was a guid nicht")]
by = {}
for g, ref, said in results:
    by.setdefault(g, []).append(wer(ref, toy_recognise(said)))
print({g: round(sum(v) / len(v), 3) for g, v in by.items()})

Output from the run:

{'A': 0.0, 'B': 0.575}

Group A scores 0.0 and group B 0.575. The overall average over all four
items is (0 + 0 + 0.75 + 0.4) / 4 = 0.2875 (computed by hand from the chapter's
numbers, not printed by the script). That figure looks like "about 29%
errors", but it describes neither group: half the users get no errors and
half get more than half.

An overall average hides the gap because it mixes the groups. To find the
problem you have to break the score down by speaker group, as the Algorithmic
bias article's examples (accent differences in speech recognition) imply.
The groups, sentences and word list here are invented, and four items are far
too few to conclude anything; they show the method, not a finding. Chapter 9's
points about small test sets and agreement on the answer key apply here too.