Characters and Bytes
Text Encoding and Unicode 🏆
Course 1 · Chapter 1 · Characters and Bytes
Every programmer meets the same bug sooner or later: a name like "café" turns into "café", a string is "four characters long" in one tool and "five" in another, or a database column rejects an emoji. All of these come from one misunderstanding: text is not bytes. This chapter separates the layers (characters, numbers, bytes) and shows that "how long is this string?" has at least three different answers. Later chapters build on this map.
Computers Only Store Numbers
A computer stores numbers, so to store text we agree a table that gives each character a number. An early and famous table is ASCII, which numbers 128 characters (0 to 127): unaccented English letters, digits, punctuation and some control codes. The capital letter A is 65. That works for English and nothing else, so many incompatible tables were invented for other languages. Unicode was created to replace them with one table for all writing systems. Wikipedia describes it as assigning "a unique number called a code point to each character".
A code point is just that number, written in hexadecimal with a U+ prefix and at least four digits: A is U+0041, the euro sign is U+20AC.
Python gives you the number with ord() and goes back with chr():
Notice what is not in the output of the first line: no bytes. The code point 8364 for the euro sign is an abstract number. How many bytes it takes on disk depends on a second decision, the encoding.
Code Space: How Many Numbers Are There?
Unicode's code space runs from U+0000 to U+10FFFF, which is 1,114,112 numbers, split into 17 planes of 65,536. Python reports the top of the range:
Not every number is a valid character: Wikipedia says 1,112,064 of them are valid code points (the rest are surrogates, reserved for a different encoding, UTF-16, which comes later in the course). Most of the valid ones are still unassigned. Wikipedia gives the number of assigned characters in the newest version as 172,808, counting graphic and format characters only (check: this figure changes with each Unicode release).
Encoding: From Code Points to Bytes
An encoding is a rule for turning code points into bytes and back. The one that won on the web is UTF-8, designed by Ken Thompson and Rob Pike in September 1992. It uses one to four bytes per code point:
| Code point range | UTF-8 bytes | Typical content |
|---|---|---|
| U+0000 to U+007F | 1 | ASCII: English letters, digits, punctuation |
| U+0080 to U+07FF | 2 | Accented Latin letters, Greek, Cyrillic, Hebrew, Arabic (check the exact scripts) |
| U+0800 to U+FFFF | 3 | Most CJK characters, the euro sign |
| U+10000 to U+10FFFF | 4 | Emoji and rarer historic scripts |
The ranges match the byte counts you saw in the first program: A takes 1, é takes 2, € and 語 take 3. Because the first 128 values encode exactly like ASCII, a pure-ASCII file is already valid UTF-8, which is a large part of why UTF-8 spread so easily. Wikipedia reports that about 99.1% of surveyed websites use it (W3Techs, as quoted there).
Three Ways to Count One String
Now the trap. A reader sees "characters", but a program can count three different things:
| Unit | What it is | Python |
|---|---|---|
| Bytes | What is stored or sent, after an encoding is chosen | len(s.encode("utf-8")) |
| Code points | The Unicode numbers in the string | len(s) |
| Grapheme clusters | What a reader perceives as one character | Not built in (needs a library or the Unicode rules) |
The letter "é" can be written two ways. One is a single code point, U+00E9. The other is the letter e followed by U+0301, a combining acute accent that attaches to the letter before it. They look identical and are different strings:
Two strings that look the same compare unequal, and one is longer. (Fixing this is called normalization, a later chapter.) The same effect is bigger with emoji, which are often several code points joined together:
A flag is two "regional indicator" code points that a font draws as one picture. A family emoji is three person emoji glued by two invisible U+200D (zero-width joiner) code points: 5 code points, 18 bytes, one picture on screen (how it is drawn depends on your font, check). Counting "characters" for a username limit or a text box is therefore a design decision, not a function call.
len() as "the number of characters a person sees". In Python 3 it is the number of code points. In some other languages the same call counts bytes or UTF-16 units instead (check per language; a later chapter covers this).
When Decoding Goes Wrong: Mojibake
Bytes mean nothing without knowing the encoding. If you write text as UTF-8 and read it back as Latin-1 (an old encoding that maps each byte to one character), each multi-byte character falls apart into several odd ones:
The two bytes C3 A9 are read as the two characters "Ã" and "©". This garbling is called mojibake, and recognising its shape ("Ã" followed by a symbol) is a useful debugging skill. The cure is never to guess: know which encoding produced the bytes, and decode with it. You can ask Python what a code point is:
Mn is the category "Mark, nonspacing": a code point that draws on top of the previous one instead of taking its own space.
Exercises
Predict, from the table above, how many UTF-8 bytes each of these needs: A, ß, Ω, €, 語 and 🎉. Then write a loop that prints each character's code point in U+XXXX form and its byte count, and compare with your predictions.
Write describe(s), which prints a string's code-point count, UTF-8 byte count and each code point in hex. Use it on "cafe", "café" (one code point for é), "café" (e plus U+0301) and the Japan flag (two regional indicators, U+1F1EF and U+1F1F5). Predict the numbers first.
Encode "naïve café" as UTF-8, decode the bytes wrongly as Latin-1 to make mojibake, then reverse the damage and check you recovered the original. Also find the exact error Python raises when you decode those bytes as ASCII.
📄 View solutionChapter 1 Quick Reference
- Text is stored as numbers: a code point (
U+0041) is Unicode's number for a character; an encoding turns code points into bytes - Code space is
U+0000toU+10FFFF(1,114,112 numbers, 17 planes) - UTF-8 uses 1 byte for U+0000-007F (same as ASCII), 2 for up to U+07FF, 3 for up to U+FFFF, 4 above
- A string has three sizes: bytes, code points, and what the reader sees (grapheme clusters); Python's
len()gives code points - "é" can be one code point or two, so visually equal strings can compare unequal
- Wrong decoding produces mojibake like "café"; know the encoding, do not guess
ord(),chr(),str.encode(),bytes.decode()andunicodedata.name()are your inspection tools
len() in other languages. Rendering of the flag and family emoji depends on your font. Sources: en.wikipedia.org/wiki/Unicode, en.wikipedia.org/wiki/UTF-8.