Corpora

Computational Linguistics

Course 1 · Chapter 3 · Corpora

Chapter 2 counted words in a single sentence. A real study counts them in millions. The collection a linguist or a program learns from is called a corpus, and what it contains decides what any result built on it can possibly say.

The idea in a box

A corpus is evidence, and every body of evidence has a shape. Before trusting a count, ask what text went in, and who or what chose it.

What a Corpus Is

Definition. Wikipedia's Corpus linguistics article calls the field "an empirical method for the study of language by text corpus". Corpora are balanced, stratified collections of real-world speech or writing, and today they are generally machine-readable.
Text corpus. The Text corpus article defines one as "a dataset, consisting of natively digital and older, digitalized, language resources", which may be annotated or unannotated. It calls corpora "the main knowledge base in corpus linguistics" and says annotated ones support statistical hypothesis testing and checking how linguistic rules apply. Other uses it lists: language technology, machine translation, language teaching and philology.

Two famous corpora

Corpus	What the Corpus linguistics article says
Brown Corpus	a balanced corpus of about one million words of American English from 1961, analyzed by Kučera and Francis in 1967; "the first computerized corpus designed for linguistic research"
British National Corpus	"a 100 million word collection of a range of spoken and written texts", created in the 1990s by a consortium that included Oxford, Lancaster and the British Library

The article adds that Quirk's corpus "was the first modern corpus to be built with the purpose of representing the whole language", and that corpora fed dictionaries and grammars such as the American Heritage Dictionary (1969) and Quirk et al.'s 1985 grammar.

Annotation

A raw corpus is just text. Annotation adds information. The Corpus linguistics article says it may include "structural markup, part-of-speech tagging, parsing, and numerous other representations", and that most lexical corpora are part-of-speech tagged. It also reports a disagreement: John Sinclair preferred minimal annotation, the Survey of English Usage team supported more. Wallis and Nelson's 3A framework describes the work as annotation (applying a scheme), abstraction (mapping the scheme onto a theoretical model) and analysis ("statistically probing, manipulating and generalising from the dataset").

Kind of annotation	What the Text corpus article says
Part-of-speech tagging	adds a tag for each word's part of speech, such as verb, noun or adjective
Lemmatization	marks each word's base form
Interlinear glossing	makes annotation bilingual when the corpus language is not the researchers' working language
Treebanks	fully parsed corpora, usually smaller, "containing around one to three million words"; consistent annotation of a whole corpus is difficult

The two sources differ a little on size: Chapter 1's Computational linguistics article gives the Penn Treebank as "over 4.5 million words" of American English, above the Text corpus article's "around one to three million". I report both and do not try to reconcile them.

Looking Inside: Concordance Lines

The Key Word in Context article calls KWIC "the most common format for concordance lines": each line places a keyword in a fixed, central position with the words before and after it. The term was coined by Hans Peter Luhn; the underlying "keyword in titles" idea was first proposed for Manchester libraries in 1864 by Andrea Crestadoro.

The text, code and outputs below are mine: an invented 66-word paragraph, run through a small script.

import re
from collections import Counter

text = ("The old lighthouse stood on the cliff. The keeper climbed the stairs every night "
        "and lit the lamp. Strong tea warmed the keeper after the climb. "
        "On stormy nights the keeper drank strong tea and watched the sea. "
        "The sea was grey. The lamp was bright. The keeper made a decision: "
        "he would stay another winter. Strong winds shook the lighthouse, "
        "but the lamp never failed.")
tokens = re.findall(r"[a-z]+", text.lower())
print(len(tokens), len(set(tokens)))

def kwic(tokens, word, width=3):
    for i, t in enumerate(tokens):
        if t == word:
            left = " ".join(tokens[max(0, i - width):i]).rjust(22)
            right = " ".join(tokens[i + 1:i + 1 + width])
            print(left, "[" + t + "]", right)

kwic(tokens, "keeper")

66 40 the cliff the [keeper] climbed the stairs tea warmed the [keeper] after the climb stormy nights the [keeper] drank strong tea was bright the [keeper] made a decision

Four lines, one keyword in the middle. A reader sees at once what tends to follow keeper: a verb. A caution about my own script: it throws away punctuation, so a window can cross a sentence boundary. The line "was bright the [keeper]" runs from one sentence into the next.

Collocations

The Collocation article defines a collocation, in corpus linguistics, as "a series of words or terms that co-occur more often than would be expected by chance". Its examples are crystal clear, middle management, nuclear family and cosmetic surgery, plus make + decision as a verb–object pair. A collocation's meaning can be understood from its words; for an idiom "the meaning of the whole cannot be inferred from its parts". Statistically, association measures test whether co-occurrence is due to chance, and because most collocations prove significant, the scores mainly rank results. The page names mutual information, t-scores and log-likelihood, and notes KWIC analysis as a way to find them.

The idea in miniature, in my own code: compare how often two words appear next to each other with how often they would if the words were scattered at random. The "expected" figure here is my simplified calculation, not one of the measures the article names.

n = len(tokens)
c = Counter(tokens)
pairs = Counter(zip(tokens, tokens[1:]))
for pair in [("strong", "tea"), ("the", "keeper"), ("the", "lamp")]:
    obs = pairs[pair]
    exp = c[pair[0]] * c[pair[1]] / n
    print(pair, obs, round(exp, 2))

('strong', 'tea') 2 0.09 ('the', 'keeper') 4 0.85 ('the', 'lamp') 3 0.64

Strong tea occurs twice where chance predicts under 0.1; the keeper four times where chance predicts under 1. The ratio ranks strong tea as the stronger pairing. But the keeper also beats chance, which fits the article's remark that most collocations prove significant and scores mostly rank. A 66-word text is far too small for any of this to count as evidence about English; the point is to see the arithmetic.

What a Corpus Cannot Tell You

It shows the language that went in. The Brown Corpus is American English from 1961; the BNC dates from the 1990s. Neither tells you how people write today.
Balance is a design choice. The Corpus linguistics article describes corpora as "balanced, often stratified collections of authentic" texts meant to represent a variety. Someone decided what counted as balanced, and the page does not describe how.
Absence is not impossibility. A construction missing from a million words may be rare, or may belong to a kind of text the corpus left out.
Annotation has errors and theories in it. A tag is a decision; the Text corpus article says consistent annotation of a whole corpus is difficult.

Watch out: a count from one corpus is not a fact about the language

"The word occurs 4 times per million" is a fact about that corpus. Whether it generalises depends on how the texts were chosen. The sources I read give corpus sizes and dates, but the Corpus linguistics page says nothing about how representativeness is checked, and the Text corpus page does not cover sampling, so I make no claim about it beyond those cautions.

Note on sources

This chapter uses Wikipedia's Corpus linguistics, Text corpus, Collocation and Key Word in Context articles (all read in full). The Corpus linguistics page does not discuss concordance, collocation or frequency methods, corpus-based versus corpus-driven work, or criticism, and does not mention Chomsky. The Text corpus page does not mention the Brown Corpus, gives no BNC size, and does not cover sampling. The Collocation page has no Firth quotation, no "strong tea" example and no mention of pointwise mutual information, so "strong tea" is my example. The paragraph, script, outputs and the "expected" calculation are mine. Chapter 1's Penn Treebank figure comes from the Computational linguistics article.

Try It Yourself

Exercise 1

Using the chapter's tokens and kwic, print concordance lines for lamp with a window of 2. What do you notice about the first line, and what does it tell you about the script?

📄 View solution

Exercise 2

Compute observed and expected counts for the sea, the lighthouse and strong winds. Which ratio is largest, and why should you hesitate to call it a collocation?

📄 View solution

Exercise 3

You want to study how people talk about money today. Explain why the Brown Corpus and the British National Corpus, as the sources describe them, would each be a poor choice on its own, and say what kind of corpus you would want instead.

📄 View solution

Chapter 3 Quick Reference

Corpus: a balanced, stratified collection of real language, usually machine-readable
Annotation: markup, part-of-speech tags, lemmas, parse trees (treebanks)
KWIC: keyword centred with context on both sides
Collocation: co-occurrence more often than chance; scores mostly rank
A corpus shows what went in: date, variety, selection

Coming next

Computational Linguistics 4 looks at words from the inside: stems, lemmas and patterns of endings.