Computational Linguistics 2, Exercise 2: types and tokens

Code (run with py -X utf8):

from collections import Counter
w = "to be or not to be".split()
print(len(w), len(set(w)), Counter(w))
w2 = "To be or not to be".split()
print(len(w2), len(set(w2)), len(set(x.lower() for x in w2)))

Output from the run:

6 4 Counter({'to': 2, 'be': 2, 'or': 1, 'not': 1})
6 5 4

Answer: 6 tokens and 4 types. The word "be" is one type with two tokens (its
two occurrences); the type is the abstract form, the tokens are the
occurrences, as in the Type-token distinction article's "A rose is a rose is
a rose" example.

If the first word is written "To" and you do not lowercase, "To" and "to" count
as two different types, so the type count rises from 4 to 5 while the token
count stays 6. Lowercasing first brings it back to 4. Whether to lowercase is
a tokenization decision (Chapter 2): it merges forms a human reads as the same
word, at the cost of also merging cases where capitals matter, such as a name.
That last point is my own comment.