Inside UTF-8

Text Encoding and Unicode

Course 1 · Chapter 2 · Inside UTF-8

Text Encoding and Unicode 1 showed that UTF-8 turns each code point into one to four bytes. This chapter opens the box. The rule is small enough to fit in one table, and once you see it you can write an encoder and a decoder in about thirty lines, understand why UTF-8 survives being cut in the wrong place, and see why decoders must refuse certain byte sequences.

What was run, and what was not

Every program in this chapter was run on Python 3.14 and the output shown is the real output. The bit patterns and validity rules were checked against RFC 3629 and the Wikipedia article on UTF-8 (the Wikipedia article was read only in part: see "Not Yet Verified" at the end). Anything else is marked "(check)".

The Whole Rule in One Table

UTF-8 puts the code point's bits into a fixed pattern. The first byte says how long the character is; every following byte is marked as a continuation. This is the table from RFC 3629, section 3:

Code point range	Byte pattern (x = code point bits)	Free bits
U+0000 to U+007F	`0xxxxxxx`	7
U+0080 to U+07FF	`110xxxxx 10xxxxxx`	11
U+0800 to U+FFFF	`1110xxxx 10xxxxxx 10xxxxxx`	16
U+10000 to U+10FFFF	`11110xxx 10xxxxxx 10xxxxxx 10xxxxxx`	21

Read the leading bits of a byte and you know its role:

0... is a complete one-byte (ASCII) character.
10... is a continuation byte, never the start of a character.
110..., 1110..., 11110... are lead bytes for sequences of 2, 3 and 4 bytes. The number of leading 1s is the length.

Worked Example: the Euro Sign

U+20AC is in the three-byte range. Written in 16 bits it is 0010 0000 1010 1100. Split it 4 + 6 + 6 and drop the pieces into the pattern:

0010 000010 101100          the 16 bits of U+20AC, split 4 + 6 + 6
1110|0010  10|000010  10|101100
11100010 10000010 10101100  =  E2 82 AC

Python agrees, and shows the same layout for one character of each length:

for ch in "A\u00e9\u20ac\U0001F600":
    bits = " ".join(f"{b:08b}" for b in ch.encode("utf-8"))
    print(f"U+{ord(ch):04X}  {bits}")

U+0041 01000001 U+00E9 11000011 10101001 U+20AC 11100010 10000010 10101100 U+1F600 11110000 10011111 10011000 10000000

Writing an Encoder

The table translates directly into shifts and masks. cp >> 6 drops the low six bits; cp & 0x3F keeps only them; 0x80 | sets the continuation marker. The function also refuses numbers that are not valid characters (surrogates and values above U+10FFFF), a rule covered below.

def encode_cp(cp):
    if cp < 0 or cp > 0x10FFFF or 0xD800 <= cp <= 0xDFFF:
        raise ValueError(f"not a valid scalar value: {cp:#x}")
    if cp < 0x80:
        return bytes([cp])
    if cp < 0x800:
        return bytes([0xC0 | cp >> 6, 0x80 | cp & 0x3F])
    if cp < 0x10000:
        return bytes([0xE0 | cp >> 12, 0x80 | cp >> 6 & 0x3F, 0x80 | cp & 0x3F])
    return bytes([0xF0 | cp >> 18, 0x80 | cp >> 12 & 0x3F,
                  0x80 | cp >> 6 & 0x3F, 0x80 | cp & 0x3F])

def encode(s):
    return b"".join(encode_cp(ord(c)) for c in s)

text = "Hi \u00e9 \u20ac \u8a9e \U0001F600"
mine = encode(text)
print(mine == text.encode("utf-8"), len(mine))
bad = 0
for cp in range(0x110000):
    if 0xD800 <= cp <= 0xDFFF:
        continue
    if encode_cp(cp) != chr(cp).encode("utf-8"):
        bad += 1
print("mismatches over every valid code point:", bad)

True 18 mismatches over every valid code point: 0

The second half is the strongest test available: it compares the hand-built encoder with Python's for every one of the 1,112,064 valid code points, and finds no difference. (Operator precedence matters in the shifts: in Python >> binds tighter than &, which binds tighter than |, so the expressions need no brackets, but brackets are kinder to readers.)

Self-Synchronising: Recovering from Damage

Because continuation bytes always match 10xxxxxx and no lead byte or ASCII byte does, you can drop into the middle of a UTF-8 stream and find the next character boundary: skip forward past continuation bytes. Equally, you can find where the current character began by backing up, at most three bytes (Wikipedia). Counting the non-continuation bytes counts the code points without decoding anything:

def is_continuation(b):
    return b & 0xC0 == 0x80

def char_starts(data):
    return [i for i, b in enumerate(data) if not is_continuation(b)]

data = "a\u00e9\u20ac\U0001F600z".encode("utf-8")
print(data.hex(" "))
print("boundaries:", char_starts(data))
print("chars:", len(char_starts(data)))

cut = data[2:]                      # start in the middle of a character
print("cut at byte 2:", cut.hex(" "))
print("starts in cut:", char_starts(cut))
print(cut.decode("utf-8", errors="replace"))

61 c3 a9 e2 82 ac f0 9f 98 80 7a boundaries: [0, 1, 3, 6, 10] chars: 5 cut at byte 2: a9 e2 82 ac f0 9f 98 80 7a starts in cut: [1, 4, 8] �€😀z

Starting in the middle of "é" damages only that one character (it becomes the replacement character U+FFFD); the decoder resynchronises at the very next lead byte and the rest is intact. Older multi-byte encodings were not so forgiving (check: this comparison is from general knowledge). The same property is why a practical text editor can jump to an arbitrary byte offset in a big file.

What a Decoder Must Refuse

The table has a subtle gap: the character U+003F could in principle be written in two bytes (C0 BF) by filling the two-byte pattern with leading zeros. That is an overlong encoding. The standard allows exactly one way to write each character, and Wikipedia explains the risk: overlong forms can "bypass other security validations", for example a filter that looks for the byte for "/" would miss a two-byte spelling of it. RFC 3629 requires implementations to protect against decoding such sequences; its example is a naive decoder turning C0 80 into U+0000.

Three more things are invalid:

Surrogates U+D800 to U+DFFF, reserved for UTF-16 (a later chapter). Their would-be encoding starts ED A0 to ED BF.
Values above U+10FFFF, such as the sequence F4 90 80 80. Earlier standards allowed up to six bytes and 31 bits; RFC 3629 (2003) cut UTF-8 back to four bytes to match the range UTF-16 can reach.
Malformed structure: a continuation byte with no lead byte, or a lead byte not followed by enough continuation bytes.

That also explains the byte values that never appear in valid UTF-8: C0 and C1 could only start overlong forms, and F5 to FF could only lead to values past U+10FFFF. Python's own decoder rejects all of these:

tests = {
    "overlong slash": bytes([0xC0, 0xAF]),
    "overlong NUL": bytes([0xC0, 0x80]),
    "surrogate": bytes([0xED, 0xA0, 0x80]),
    "too big": bytes([0xF4, 0x90, 0x80, 0x80]),
    "lone continuation": bytes([0x80]),
    "truncated": bytes([0xE2, 0x82]),
}
for name, raw in tests.items():
    try:
        raw.decode("utf-8")
        print(f"{name:18} decoded?!")
    except UnicodeDecodeError as e:
        print(f"{name:18} {e.reason}")

overlong slash invalid start byte overlong NUL invalid start byte surrogate invalid continuation byte too big invalid continuation byte lone continuation invalid start byte truncated unexpected end of data

The wording is Python's own; other decoders word these differently. Notice that the surrogate case is reported as an "invalid continuation byte" rather than "surrogate": Python rejects ED A0 because RFC 3629 limits the second byte after ED to 80-9F. The RFC's other restricted lead bytes work the same way: E0 needs A0-BF, F0 needs 90-BF, F4 needs 80-8F.

Common mistake

Writing a decoder that only checks the lead byte, or a validator that runs after decoding. Check on the bytes, and decode strictly (no overlongs, no surrogates), before any security check such as looking for a forbidden path or tag. Wikipedia notes that the surrogate rule is often ignored in practice, for example in Windows file names (check the details before relying on this).

Exercises

Exercise 1

Encode U+20AC by hand using the table: write its 16 bits, split them 4 + 6 + 6, add the pattern bits, and give the hex bytes. Then reproduce it in Python with shifts and masks only (no .encode() until the final comparison).

📄 View solution

Exercise 2

Write a strict decode(data) that returns a string and raises ValueError for an invalid start byte, a truncated sequence, a bad continuation byte, an overlong form, a surrogate, and a value above U+10FFFF. Check it agrees with Python on valid text and rejects the five bad inputs from this chapter.

📄 View solution

Exercise 3

A database column holds at most N bytes. Write safe_truncate(data, limit) that cuts UTF-8 bytes to at most limit bytes without ever splitting a character, using the continuation-byte test. Compare it with a naive slice on a string containing a two-byte, a three-byte and a four-byte character.

📄 View solution

Chapter 2 Quick Reference

Patterns: 0xxxxxxx, 110xxxxx 10xxxxxx, 1110xxxx 10xxxxxx 10xxxxxx, 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
The count of leading 1s in the first byte is the length; 10xxxxxx is always a continuation byte
Encode with shifts and masks: 0x80 | (cp >> n) & 0x3F for each continuation byte
Self-synchronising: skip or back up over bytes where b & 0xC0 == 0x80 to find a boundary (at most three bytes back)
Invalid: overlong forms, surrogates U+D800-DFFF, values above U+10FFFF, stray or missing continuation bytes
Never valid as bytes: C0, C1, F5 to FF
RFC 3629 limits to four bytes; second-byte limits after E0, ED, F0, F4 enforce the rules above

Not Yet Verified

Checked against RFC 3629 (the octet table, the second-byte restrictions after E0, ED, F0 and F4, the surrogate ban, the overlong warning and the C0 80 example) and the Wikipedia UTF-8 article (overlong risk, back up at most three bytes, the six-byte limit before RFC 3629, the Windows file name remark). The Wikipedia article was read only in part (its first 100,000 of about 137,000 characters). The RFC page text was summarised by a fetch tool rather than read line by line. The remarks on older multi-byte encodings and the exact error wording of other decoders are from general knowledge (check). Sources: rfc-editor.org/rfc/rfc3629, en.wikipedia.org/wiki/UTF-8.

Coming next

Text Encoding and Unicode 3 looks at the encodings UTF-8 replaced: ASCII, Latin-1 and the Windows code pages, and why they leave mojibake behind.