Computational Linguistics 8, Exercise 2: why lamp and keeper look alike

Code (run with py -X utf8, after the chapter's script; the vectors are rebuilt
from the original twelve sentences plus the four new ones):

print(dict(vec["keeper"]), dict(vec["lamp"]))

Before adding sentences, the run gave (keeper vs lamp cosine 0.943):
keeper: lit 2, climbed 1, watched 2
lamp:   lit 1, watched 1

Both words appear next to "lit" and "watched" and little else. The cosine
compares direction: lamp's vector is nearly in the direction of keeper's
because both are dominated by the same two context words. The "climbed"
context, which only keeper has, is a small part of keeper's vector.

After adding "the keeper slept" x2 and "the lamp glowed" x2:

keeper: lit 2, climbed 1, watched 2, slept 2
lamp:   lit 1, watched 1, glowed 2
lantern: lit 1, watched 1

lamp keeper   0.453   (was 0.943)
lamp lantern  0.577   (was 1.0)
keeper sailor 0.462   (was 0.556)

What changed: lamp and keeper moved apart because each gained a context the
other lacks (slept, glowed). Lamp and lantern also moved apart, because lamp
alone gained "glowed" and the new data says nothing about lantern. So adding
evidence about one word lowers its similarity to a word with no such
evidence. Keeper and sailor also dropped, which is not what I would want for
two people, because sailor got no new sentence either.

Lesson: with 12-16 sentences the scores swing a lot, so they are not
trustworthy. The Distributional semantics article mentions data sparsity as a
limit; my numbers illustrate that but do not test it on a large corpus.