Limits and Society

The methods in this course work on text that someone has already typed, spelt in a standard way, in a language with plenty of data. This last chapter looks at the places where those conditions fail: spoken language, translation between languages, and speakers whose way of talking is under-represented. It ends with a recap of the course and pointers to where to go next.

Speech: From Sound to Text

Wikipedia's Speech recognition article calls it a sub-field of computational linguistics that develops methods to "translate spoken language into text or other interpretable forms". Its history, briefly: Bell Labs' Audrey recognised single-speaker digits in 1952; hidden Markov models and n-gram language models (Chapters 5 and 7) took over in the 1980s; deep neural networks and related methods drove large error reductions in the 2000s and 2010s; and in 2017 Microsoft reported human parity on the Switchboard task. The classic design has an acoustic model (audio turned into spectral features about every 10 ms) and a language model that constrains plausible word sequences; the article says end-to-end neural models can learn these parts together.

It defines the standard score, word error rate, as (s + d + i) / n: substitutions, deletions and insertions over the number of words in the reference. It gives vocabulary-size figures (recognising the ten digits "essentially perfectly", while vocabularies of 200, 5,000 and 100,000 words may have error rates of 3%, 7% and 45%) and lists accent and dialect among the factors that affect accuracy. I read the article to 100,000 of 226,473 characters, so it may say more on accents and demographics later; in the part I read it gives no figures for gender or age.

Below is my own implementation of word error rate (edit distance over words), checked on a few cases, followed by a deliberately simple "recogniser" that only knows a short list of words. The word list and the Scots-speaking example are mine; this shows the mechanism, not how any real system performs.

The first four lines match the formula by hand: one substitution in five words is 0.2, two deletions is 0.4, one insertion is 0.2. In the second block, the same meaning, said by a speaker who uses Scots words (dinnae for do not, ken for know, guid for good, nicht for night; the glosses are from my own knowledge and not from the articles), gets a much worse score. The toy recogniser has not failed through any defect in the speaker; its word list was drawn from one variety. Exercise 2 shows that a second problem hides in the answer key.

Translation Between Languages

Wikipedia's Machine translation article defines it as "the use of computational techniques to translate text or speech from one language to another" and says differences in language, grammar, tone and culture make it "inadequate to replace real translators fully". The milestones it gives: Warren Weaver's 1949 memorandum, "perhaps the single most influential publication in the earliest days of machine translation"; the 1954 Georgetown-IBM demonstration; the 1966 ALPAC report, which found a decade of research had not met expectations and led to greatly reduced funding; a shift to statistical models from the late 1980s; and, in the 2020s, neural machine translation and large language models superseding earlier methods. It describes three approaches:

Among its known problems it lists word-sense and syntactic disambiguation (raised by Bar-Hillel in the 1950s), and says machine translation handles slang, dialect and casual speech less accurately than standard language. It names BLEU, NIST, METEOR and LEPOR as automatic metrics and says human judges are the most reliable but slow; Chapter 9's agreement question applies to those judges. It also says human post-editing is usually still needed, and that medical and legal use carries risks, with some courts prohibiting it in formal proceedings. I read this article to 100,000 of 145,834 characters, and the part I read does not discuss gender bias in translation.

Who Gets Left Out

Wikipedia's Algorithmic bias article describes it as "systematic and repeatable errors that create unfair outcomes", and separates pre-existing bias (from underlying social and institutional ideologies), technical bias (from limits of a program or its design) and emergent bias (from use in new contexts). Its language-related claims, which the page carries a maintenance notice warning may include text produced by a large language model, so they should be checked against the sources it cites (I have not done that):

• English-trained models often "present the Anglo-American views as truth", and models may be biased against people based on their dialect.
• A 2024 study is described as finding that models show more negative stereotypes about speakers of African American English than recorded human biases; some hate-speech detection tools were reported 2.2 times more likely to flag content written in that dialect.
• Speech recognition accuracy can vary by accent, possibly due to "a lack of training data for speakers of that accent".
• Machine translation shows "a strong tendency towards male defaults", especially in STEM fields.
• Large language models may link nurses or secretaries with women and engineers or CEOs with men.

Each of these is the article's report. Whether it is true in a particular case needs the measures from Chapter 9, applied separately to each group of speakers: an overall score can be high while one group's is poor. Exercise 3 does this on a tiny invented example.

Links to the sociolinguistics course

Sociolinguistics: Language, Dialect and Society 2 (Dialect and Accent) treats the line between language and dialect as partly social. Sociolinguistics: Language, Dialect and Society 8 (Standards and Attitudes) covers how standard languages are made and how attitudes to accents arise. In computational terms, a "standard" is whatever the training data and the answer key contain. I am drawing that link myself; neither Wikipedia article makes it.

Course Complete: What You Can Now Do

Across the course, the same pattern recurred: a simple method works on neat data, a limit appears, and a measurement tells you how bad it is. Everything was written in short standard-library Python on small invented data, so none of the numbers say how well any real system performs.

Where to go next: NLP picks up the same tasks with libraries and machine learning (its Chapter 1 covers the text-preprocessing pipeline, Chapter 5 word embeddings, Chapter 7 named-entity recognition and part-of-speech tagging; I have only read the chapter titles). LLMs covers how large language models are built; for example, LLMs 2 is on subword tokenisation (compare Chapter 2) and LLMs 10 on context windows and limitations. Those are the headline titles from the courses' banners, not a detailed check of their contents.

Try It Yourself