Language as Data

Computational Linguistics

Course 1 · Chapter 1 · Language as Data

A computer does not see words. It sees numbers. Before a program can count a word, parse a sentence or translate a paragraph, someone has to decide how language becomes numbers, and that decision already contains linguistics. This chapter says what the field is, where it came from, and then looks at the very first step: how text is stored.

The idea in a box

Wikipedia describes computational linguistics as an interdisciplinary field "concerned with the computational modelling of natural language". In this course, the question behind every chapter is the same: what exactly is the program being asked to do with language, and how would we know it did it?

What the Field Is

Definition. The Computational linguistics article calls it an interdisciplinary area concerned with the computational modelling of natural language and with finding suitable computational approaches to linguistic questions. It is closely related to mathematical linguistics.
Origins. It overlapped with AI from the 1950s US efforts to use computers for machine translation, especially of Russian scientific journals into English. Researchers expected rule-based methods could also capture lexicon, morphology, syntax and semantics. After those approaches failed, David Hays coined the term to separate the field from AI, and helped found the ACL and ICCL in the 1970s and 1980s.
Neighbours. The page lists linguistics, computer science, AI, mathematics, logic, philosophy, cognitive science, psycholinguistics, anthropology and neuroscience. It says the translation effort grew into the wider field of natural language processing (NLP).

Computational linguistics and NLP

The two pages do not give a definitional distinction. The NLP article defines NLP as "the processing of natural language information by a computer" and a subfield of computer science closely associated with AI, and says it "is also related to" computational linguistics. The Computational linguistics article implies NLP is the broader field that grew out of its translation origins. In practice this course uses the split described in its outline: NLP 1 teaches the machine-learning pipeline; this course keeps the linguist's questions in view. That working split is mine, not the sources'.

A short timeline (from the NLP article)

Date	Event or period
1950	Alan Turing published "Computing Machinery and Intelligence", proposing what is now the Turing test
1954	The Georgetown experiment demonstrated automatic translation of more than sixty Russian sentences into English; its authors predicted translation would be solved within a few years, but progress was much slower
1950s to early 1990s	Symbolic NLP: hand-written rules; the 1980s and early 1990s are called its heyday
1990s onward	Statistical NLP: machine learning replaced hand-written rules starting in the late 1980s; in 2003 a neural probabilistic language model outperformed the word n-gram model
From 2015	Neural methods increasingly replace traditional statistical approaches

The Tasks, Mapped to This Course

The NLP article lists common tasks. The grouping below is the article's; which chapter of this course covers each group is my own mapping.

Level	Tasks the article lists	Chapter here
Text and speech	tokenization, optical character recognition, speech recognition, text-to-speech	1 and 2 (text); 10 (speech)
Morphology	lemmatization, morphological segmentation, part-of-speech tagging, stemming	4, 7
Syntax	parsing, sentence breaking, grammar induction	6
Semantics	named entity recognition, word-sense disambiguation, sentiment analysis, semantic role labelling	8
Applications	machine translation, question answering, summarization	10

Text Is Numbers: Unicode

The Unicode article calls Unicode "a character encoding standard maintained by the Unicode Consortium" that aims to support text in the world's writing systems that can be digitized. It states that version 18.0 defines 172,808 characters and 175 scripts (a figure I have not checked against the standard itself).

Code points. Unicode gives each character a number, a code point, in a codespace from 0 to 1114111, written U+0000 to U+10FFFF: "U+" plus hexadecimal, with at least four digits (the article's example is U+00F7, DIVISION SIGN).
Bytes. A code point is an abstract number. Text is stored by an encoding that maps code points to sequences of bytes.
UTF-8. The most widely used encoding by a large margin, partly because of its backwards-compatibility with ASCII. In UTF-8, code points in the Basic Multilingual Plane take one to three bytes and supplementary-plane code points take four. UTF-16 uses a single code unit for the Basic Multilingual Plane and surrogate pairs beyond it.
Combining characters. A combining mark can be added after a base character, so "é" can be one precomposed code point or "e" plus a combining acute accent; the article says canonical equivalence makes these interchangeable in practice.
Before Unicode. Many encodings were built for single contexts and were often incompatible, so text in one could appear as garbage when read with another (the article calls one such case mojibake).

Seeing it in Python

The examples below are my own; I ran each and copied its output. Hungarian is used because it has accented letters beyond the ASCII range.

for ch in ["A", "é", "ő", "語"]:
    print(ch, "U+%04X" % ord(ch), len(ch.encode("utf-8")), "bytes in UTF-8")

A U+0041 1 bytes in UTF-8 é U+00E9 2 bytes in UTF-8 ő U+0151 2 bytes in UTF-8 語 U+8A9E 3 bytes in UTF-8

So one character is not one byte. Counting "length" depends on what you count:

s = "Szia, világ!"
print(len(s), len(s.encode("utf-8")))

12 13

Twelve characters, thirteen bytes: the "á" takes two. And the same visible word can be two different strings:

import unicodedata
a = "café"      # "é" as one code point
b = "café"     # "e" plus a combining acute accent
print(a, b, a == b, len(a), len(b))
print(unicodedata.normalize("NFC", b) == a)

café café False 4 5 True

The two strings look the same on screen but are not equal until they are normalised to a single form (NFC joins the pieces). A program that counts words without normalising would count "café" twice.

Finally, the garbage the article describes. Here a UTF-8 string is read back with the wrong encoding:

print("café".encode("utf-8").decode("latin-1"))

cafÃ©

Watch out: this is not the article's mojibake example

The Unicode article describes mojibake through C1 control code points misdecoded as Windows-1252. My example above is a different mix-up (UTF-8 bytes read as Latin-1) that produces the same kind of damage. I give it as my own illustration, not as the article's case.

Note on sources

This chapter uses Wikipedia's Computational linguistics (read in full), Natural language processing (read to 100,000 of 142,965 characters; it ends partway through the references) and Unicode (read to 100,000 of 257,769 characters; it ends partway through "Standardized subsets", so later sections were not read). The Computational linguistics page does not mention the Cold War, and the Unicode page gives no UTF-32 detail and does not use U+0041 as an example, so neither appears as sourced. All Python examples and their outputs are mine, run on this machine with py -X utf8; the Unicode figure for version 18.0 is the article's own and unchecked. The task-to-chapter mapping is mine.

Try It Yourself

Exercise 1

In Python, print the code point (in U+XXXX form) and the number of UTF-8 bytes for "n", "ñ", "€" and "😀". What pattern do you see between the size of the code point and the number of bytes?

📄 View solution

Exercise 2

The word "naïve" has 5 characters. Use unicodedata.normalize("NFD", ...) to split the "ï" into pieces. How many characters does the result have, and what are the pieces called?

📄 View solution

Exercise 3

Sort these tasks by level using the NLP article's grouping: stemming, parsing, tokenization, word-sense disambiguation, machine translation. Then say in two sentences why this course treats the history of the Georgetown experiment as a warning and not a success story.

📄 View solution

Chapter 1 Quick Reference

Computational linguistics: computational modelling of natural language; NLP grew from its machine-translation origins
Eras per the NLP article: symbolic (hand-written rules), statistical, neural (increasingly from 2015)
Unicode: code points U+0000 to U+10FFFF; encodings (UTF-8 most used) turn them into bytes
Characters are not bytes, and identical-looking strings may differ until normalised

Coming next

Computational Linguistics 2 asks the question that sounds easy: what counts as a word?