Legacy Encodings

Text Encoding and Unicode

Course 1 · Chapter 3 · Legacy Encodings

UTF-8 did not arrive on an empty field. For decades, text meant one byte per character, and every region and vendor filled the spare half of the byte differently. Those older encodings are still around in old files, databases, email and web pages, and they are the source of most garbled text you will meet. This chapter explains the main ones: ASCII, Latin-1, and Windows-1252.

What was run, and what was not
Every program in this chapter was run on Python 3.14 and the output shown is the real output. Facts about ASCII, ISO-8859-1 and Windows-1252 come from their Wikipedia articles, each read only in part (see "Not Yet Verified" at the end). Details not checked against a source are marked "(check)".

ASCII: Seven Bits

ASCII was first published in 1963 as ASA X3.4-1963 (Wikipedia). It uses 7 bits and defines 128 codes, numbered 0 to 127: 33 control codes (the first 32 plus DEL at 127) and 95 printable characters. Space is 32 (0x20), the digit 0 is 48 (0x30), A is 65 (0x41) and a is 97 (0x61).

The designers placed each lowercase letter exactly 0x20 above its capital, so the two differ in a single bit (bit 5). That lets you change case with one XOR. ASCII also cannot represent anything outside basic English text, and Python refuses rather than guesses:

print(sum(1 for c in range(128) if chr(c).isprintable()), "printable of 128") print(ord("A"), ord("a"), ord("a") - ord("A"), 0x20) print(chr(ord("g") ^ 0x20), chr(ord("G") ^ 0x20)) print(f"{ord('A'):08b}") print(f"{ord('a'):08b}") try: "caf\u00e9".encode("ascii") except UnicodeEncodeError as e: print(e) print("caf\u00e9".encode("ascii", errors="replace")) print("caf\u00e9".encode("ascii", errors="backslashreplace")) print("caf\u00e9".encode("ascii", errors="xmlcharrefreplace"))
95 printable of 128 65 97 32 32 G g 01000001 01100001 'ascii' codec can't encode character '\xe9' in position 3: ordinal not in range(128) b'caf?' b'caf\\xe9' b'café'

The last three lines show Python's error handlers: replace silently swaps in a question mark (data is lost), backslashreplace keeps a readable escape, and xmlcharrefreplace writes an HTML numeric entity. The default, strict, raises an error, which is usually what you want. (Output is plain bytes, so the entity é shown above is literally the five characters é in the byte string.)

The Eighth Bit: Everyone Filled It Differently

A byte has 8 bits, so ASCII leaves 128 spare values (128 to 255). Each region and vendor filled them with what its users needed, producing many incompatible code pages. The same byte therefore meant different things depending on which table the reader used.

ISO-8859-1, called Latin-1, covers Western European languages. Its printable upper half runs from 0xA0 to 0xFF. Its special property: every byte value is equal to the Unicode code point with the same number (Unicode's first 256 code points were chosen to match), which Python confirms below. Here is the single byte 0xE9 and the byte 0xA4 under six code pages:

same = all(bytes([b]).decode("latin-1") == chr(b) for b in range(256)) print("latin-1 byte value == code point for all 256:", same) for name in ["latin-1", "cp1252", "iso8859-15", "iso8859-7", "cp437", "koi8-r"]: print(f"{name:11} 0xE9 -> {bytes([0xE9]).decode(name)} 0xA4 -> {bytes([0xA4]).decode(name)!r}")
latin-1 byte value == code point for all 256: True latin-1 0xE9 -> é 0xA4 -> '¤' cp1252 0xE9 -> é 0xA4 -> '¤' iso8859-15 0xE9 -> é 0xA4 -> '€' iso8859-7 0xE9 -> ι 0xA4 -> '€' cp437 0xE9 -> Θ 0xA4 -> 'ñ' koi8-r 0xE9 -> И 0xA4 -> '╓'

One byte, six tables, four different letters for 0xE9: é, ι (Greek iota), Θ and И. Nothing in the byte tells you which table was meant; that information must travel with the data (a later chapter covers how files and web pages declare it) or be guessed. The name cp437 is the original IBM PC character set (check), koi8-r is a Russian encoding, and iso8859-15 is a revision of Latin-1 that includes the euro sign. That iso8859-7 (Greek) shows a euro at 0xA4 reflects a later revision of that standard in Python's table (check).

Windows-1252: The Latin-1 Lookalike

ISO-8859-1 reserves 0x80 to 0x9F for C1 control codes, which are almost never used. It also has no euro sign (the standard dates from 1987, before the euro was presented in 1996, Wikipedia), no curly quotes and no dashes. Microsoft's Windows-1252 matches Latin-1 everywhere except in that block, where it puts printable characters: euro sign, curly quotes, dashes, the trademark sign. It is so common that HTML5 browsers treat the label "iso-8859-1" as Windows-1252 (Wikipedia). Python shows the difference exactly:

diff = [b for b in range(256) if bytes([b]).decode("latin-1") != bytes([b]).decode("cp1252", errors="replace")] print(len(diff), "bytes differ; range", hex(min(diff)), "to", hex(max(diff))) undefined = [] for b in range(0x80, 0xA0): try: bytes([b]).decode("cp1252") except UnicodeDecodeError: undefined.append(hex(b)) print("undefined in Python's cp1252:", undefined) print(bytes([0x80]).decode("cp1252"), bytes([0x92]).decode("cp1252"), repr(bytes([0x92]).decode("latin-1"))) print("\u20ac".encode("cp1252"), "\u20ac".encode("iso8859-15")) try: "\u20ac".encode("latin-1") except UnicodeEncodeError as e: print(e)
32 bytes differ; range 0x80 to 0x9f undefined in Python's cp1252: ['0x81', '0x8d', '0x8f', '0x90', '0x9d'] € ’ '\x92' b'\x80' b'\xa4' 'latin-1' codec can't encode character '\u20ac' in position 0: ordinal not in range(256)

All 32 bytes of the C1 block differ, and only there. Five of them (0x81, 0x8D, 0x8F, 0x90, 0x9D) are undefined in Windows-1252: Wikipedia says the same five are unused, and notes that some Windows APIs map them to the C1 controls anyway, a behaviour Python's strict codec does not copy. The euro sign is byte 0x80 in Windows-1252 but 0xA4 in ISO-8859-15. This is why a text pasted from a word processor, full of curly quotes, shows odd symbols if read as the wrong one of the two.

Why Mojibake Has a Shape

Last chapter ended with UTF-8 bytes read as Latin-1. The most common modern accident is the same mistake with Windows-1252 as the wrong reader. UTF-8 bytes for accented letters and curly punctuation start with 0xC3, 0xE2 and similar, which the old tables display as "Ã" and "â", followed by symbols. Recognise that shape and you know the cause: UTF-8 text decoded as a legacy encoding.

original = "It\u2019s caf\u00e9" raw = original.encode("utf-8") garbled = raw.decode("cp1252") print(garbled) print(garbled.encode("cp1252").decode("utf-8") == original) quote = "\u201d".encode("utf-8") # right double quote print(quote.hex(" ")) try: quote.decode("cp1252") except UnicodeDecodeError as e: print(e) print(quote.decode("latin-1").encode("latin-1").decode("utf-8"))
It’s café True e2 80 9d 'charmap' codec can't decode byte 0x9d in position 2: character maps to <undefined> ”

The first half is a clean repair: encode the garbled text back with the same wrong encoding to recover the original bytes, then decode correctly. The second half shows where it breaks: the right double quote is the bytes E2 80 9D, and 9D is one of Windows-1252's undefined bytes, so the strict decode fails. Latin-1, which defines all 256 bytes, never fails and can always be reversed. Whether a damaged copy can be fixed therefore depends on which wrong decoder was used, and on whether anything was replaced by ? or U+FFFD along the way (then the data is gone). Libraries exist that automate such repairs (check).

Common mistake
Treating "it decoded without an error" as proof that the encoding is right. Latin-1 decodes any byte string without an error, so a successful decode proves nothing. Exercise 3 shows this.

Exercises

Exercise 1

Write flip_case(s) that swaps the case of ASCII letters using XOR with 0x20 and leaves everything else alone. Check it against str.swapcase for all 128 ASCII characters, then try it on "café" and say why the é is untouched.

📄 View solution
Exercise 2

Write compare(b) that decodes one byte as Latin-1 and as Windows-1252, returning "(undefined)" when the second fails. Print it for 0x80, 0x85, 0x92, 0x93, 0x99, 0x9D, 0xA3 and 0xE9, then count how many of the 256 byte values differ and confirm they all lie in 0x80 to 0x9F.

📄 View solution
Exercise 3

Write which_work(raw, names) returning the encodings in a list that decode the bytes without error. Try it on plain English, UTF-8 text with an accent, Latin-1 text with an accent, and a euro sign in Windows-1252. What does the result tell you about guessing an encoding from "no error"?

📄 View solution

Chapter 3 Quick Reference

  • ASCII: 7 bits, 128 codes (33 control, 95 printable); A is 65, a is 97, and case differs by one bit (0x20)
  • Bytes 128 to 255 were assigned differently by each region and vendor: the same byte, different letters
  • Latin-1 (ISO-8859-1): byte value equals Unicode code point for all 256 values; 0x80-0x9F are controls; no euro sign
  • Windows-1252: Latin-1 except 0x80-0x9F, which hold printable characters (euro at 0x80, curly quotes); five bytes undefined
  • HTML5 treats the "iso-8859-1" label as Windows-1252
  • Mojibake like "é" or "’" usually means UTF-8 bytes read by a legacy decoder; reverse it by encoding with the wrong codec and decoding as UTF-8
  • Latin-1 never raises a decode error, so "no error" does not prove an encoding is right
  • Encode error handlers: strict (default), replace, backslashreplace, xmlcharrefreplace
Not Yet Verified
Checked against Wikipedia's ASCII, ISO/IEC 8859-1 and Windows-1252 articles: the 1963 publication, 7 bits and 128 codes, the 33/95 split, the one-bit case difference, Latin-1's relationship to Unicode, the missing euro sign and the reason, the HTML5 label behaviour, the euro at 0x80 and the five undefined Windows-1252 bytes. The ASCII and ISO-8859-1 articles were read only in part (first 100,000 of about 170,000 and 126,000 characters); the Windows-1252 article was summarised by a fetch tool. What the names cp437, koi8-r and iso8859-15 refer to, the Greek euro revision, how Python's strict codec compares with Windows APIs beyond the Wikipedia remark, and the existence of automatic repair libraries are from general knowledge (check). Sources: en.wikipedia.org/wiki/ASCII, en.wikipedia.org/wiki/ISO/IEC_8859-1, en.wikipedia.org/wiki/Windows-1252.
Coming next
Text Encoding and Unicode 4 covers UTF-16 and UTF-32: why a second Unicode encoding exists, what surrogate pairs are, and what byte order marks are for.