Meaning

So far the course has dealt with forms: characters, tokens, stems, tags and trees. A program that only handles forms can say that lamp is a noun, but not that a lamp is more like a lantern than like a harbour. This chapter shows two ways a machine can represent meaning: by writing it down as a network of relations, and by measuring the company a word keeps.

Meaning Written Down: WordNet

Wikipedia's WordNet article calls it "a lexical database of semantic relations between words", a combination of a dictionary and a thesaurus, created in 1985 at Princeton University's Cognitive Science Laboratory under the psychologist George Armitage Miller, and later directed by Christiane Fellbaum. Roughly synonymous words of the same part of speech are grouped into synsets, each with a short gloss and usage examples; a word with several senses appears in several synsets. The main relations:

The article lists uses in word-sense disambiguation (where it calls WordNet the most commonly used computational lexicon of English), information retrieval, text classification, summarization and machine translation, and says word similarity can be measured by distance in the graph. It also notes that other resources build on it: ImageNet is organised by WordNet's noun hierarchy.

A hypernym tree you can run

I have not used WordNet's data in this course (Python's nltk is not installed here, and I have not read its interface). Instead, the tree below is a tiny made-up hypernym network of my own, and the distance is the number of steps up to the nearest shared ancestor, added for both words. The "steps through the graph" idea is the article's; this particular measure is just the simplest one I could write. I ran it and copied the output.

Closer in the tree means more alike: a lantern is one step from a lamp, three from a candle, six from a keeper. Everything this tells us was put there by whoever drew the tree, which is both the strength (it is deliberate and inspectable) and the cost (someone must build it, word by word).

Choosing a Sense: Word-Sense Disambiguation

Wikipedia's Word-sense disambiguation article defines it as "the process of identifying which sense of a word is meant in a sentence or other segment of context", usually subconscious in people. Its examples include pen (writing instrument or enclosure) and bank, which a translator might render as French banque or rive. It names Lesk (1986) as "the seminal dictionary-based method": find "the pair of dictionary senses with the greatest word overlap" in their definitions. It groups approaches into knowledge-based (dictionaries and thesauri, "without using any corpus evidence"), supervised (classifiers trained on sense-tagged corpora, held back by a "knowledge acquisition bottleneck"), semi-supervised and unsupervised (clustering occurrences, "the greatest challenge"). Its figures:

• Coarse-grained, homograph-level English WSD "routinely above 90%" (as of 2009); fine-grained top results "from 59.1% to 69.0%" in SemEval-2007 and Senseval-2.
• In Senseval-2, with fine-grained senses, human annotators "agreed in only 85% of word occurrences", so human performance is treated as an upper bound.
• Always choosing the most frequent sense scored 51.4% (SemEval-2007) and 57% (Senseval-2), showing how much comes from frequency alone.
• History: Warren Weaver raised it as a computational problem in 1949; the 1990s statistical revolution made it a supervised-learning problem.

I read the article to 100,000 of 113,521 characters. Below is a very simplified overlap method in the spirit of Lesk's idea, with two glosses I wrote for light. It is my sketch, not Lesk's algorithm as published, which I have not read.

The first two work because their context words overlap a gloss. In the third there is no overlap at all, so max simply returns the first sense: the program has no evidence but still answers. A real system must be able to say "I can't tell", or fall back on a baseline such as the most frequent sense.

Meaning Measured: Distributional Semantics

The Distributional semantics article describes it as a research area that measures and classifies semantic similarity between linguistic items using their distributional properties in large corpora, built on the distributional hypothesis: "linguistic items with similar distributions have similar meanings." It attributes the phrase "a word is characterized by the company it keeps" to Firth, who popularised it in the 1950s, and cites Harris (1954) for the hypothesis. It gives no longer Firth quotation and no Harris wording, so I quote nothing more. Its other points:

• Co-occurrence information is collected in high-dimensional vectors, and similarity is defined as vector similarity; when the context is other words the models are called "word space" or "vector space" models.
• It lists cosine similarity and Minkowski distance as similarity measures, and dimension reduction (random indexing, singular value decomposition) as one parameter; it explains neither method.
• Models differ in context type, window size, frequency weighting, dimension reduction and similarity measure; latent semantic analysis, Hyperspace Analogue to Language and topic models are named.
• It says distributional semantics serves as an input to neurally inspired deep-learning models, and lists word embedding and Word2vec only in "See also". NLP 5 (Word Embeddings and Word2Vec) covers that route, and I have not re-read it.
• Limits it admits: it calls it an open question whether similar meaning really tracks similar distribution; it mentions data sparsity; and it says word-level meaning is only part of a sentence's meaning, with Tigers love rabbits as its example.

Wikipedia's Cosine similarity article gives the measure: "the dot product of the vectors divided by the product of their lengths", from −1 to +1, and 0 to 1 when no component can be negative (as with counts). It says that for documents it works as a way of "normalizing document length". Below is my own tiny corpus (12 invented sentences) and a window of two words either side, ignoring the.

Lamp and lantern score 1.0 and boat and ship score 1.0, which is the pattern the hypothesis predicts: I wrote them to appear in identical contexts, so this is partly built in. The surprise is the last line: lamp and keeper score 0.943, higher than keeper and sailor (0.556), though a keeper is a person and a lamp is not. The reason is in the counts: both are only ever seen next to lit and watched in my twelve sentences. The measure is faithfully reporting a corpus that is too small and too repetitive, which echoes the article's data sparsity point. I did not test how the score changes with a larger corpus.

The table's two columns are my summary of the two sections, not a comparison made by the articles.

Try It Yourself