Computational Linguistics 5, Exercise 3: generating text with a bigram model

Code (run with py -X utf8, after the chapter's script):

nxt = {}
for (w1, w2), c in bi.items():
    nxt.setdefault(w1, []).append((c, w2))
w = "the"
out = [w]
for _ in range(9):
    w = max(nxt[w], key=lambda t: t[0])[1] if w in nxt else None
    if w is None:
        break
    out.append(w)
print(" ".join(out))

Output from the run:

the keeper climbed the keeper climbed the keeper climbed the

What it looks like: a loop. The model always picks the most frequent follower
of the current word ("the" -> "keeper" -> "climbed" because ties go to the first
one counted, "climbed" -> "the"), then repeats.

What it says: a bigram model remembers only the previous word, so once it
returns to a word it has no memory of having been there. Always choosing the
most likely word also removes variety. Sampling by probability instead would
vary the text, but it would still have only one word of memory. The loop is
produced by my rule (always the maximum, first on ties) and my 66-word text;
the articles do not discuss generation or the limits of n-gram models in the
parts I read, so the general lesson is my reading.