Computational Linguistics 3, Exercise 1: concordance lines for "lamp"

Code (run with py -X utf8, after the chapter's script):

kwic(tokens, "lamp", 2)

Output from the run:

               lit the [lamp] strong tea
              grey the [lamp] was bright
               but the [lamp] never failed

What to notice: the first line reads "lit the [lamp] strong tea". In the
original text the sentence ends after "lamp." and a new sentence begins with
"Strong tea". The script discards punctuation, so the window crosses the
sentence boundary and the right-hand context belongs to a different sentence.
The same happens in "grey the [lamp]": "grey" ends the previous sentence
("The sea was grey.").

What it tells you: a tokenizer that drops punctuation (Chapter 2) loses
information that concordance lines need. A fix would be to keep sentence
boundaries as tokens or to stop the window at them. That is a design
decision, not something the sources specify.