Computational Linguistics 5, Exercise 2: changing alpha

Code (run with py -X utf8, after the chapter's script):

for a in (1, 0.5, 0.1, 0.01):
    p = 1.0
    for w1, w2 in zip(test, test[1:]):
        p *= (bi[(w1, w2)] + a) / (uni[w1] + a * V)
    print(a, round(p, 8), round(p ** (-1 / 4), 1))

Output from the run:

1 5.7e-06 20.5
0.5 1.448e-05 16.2
0.1 7.308e-05 10.8
0.01 6.373e-05 11.2

Lowest perplexity: alpha = 0.1 (10.8), slightly better than 0.01 (11.2).

The problem: I chose alpha by looking at the very sentence I then score. That
makes the result look better than it would on new text, because the setting is
fitted to the test. The Perplexity article defines perplexity on a separate
test sample, and the Additive smoothing article says the choice of alpha is
partly subjective and that alpha = 1 or 1/2 are used when there is no prior
knowledge. The standard remedy (my statement, not from the articles) is to
choose alpha on one held-out set and report the score on another. Here the
whole text is 66 words, so neither is large enough to mean anything.