UTF-16 and Friends

Text Encoding and Unicode

Course 1 · Chapter 4 · UTF-16 and Friends

UTF-8 is not the only way to turn code points into bytes. Two other encodings of the same Unicode table are in daily use inside operating systems, programming languages and file formats: UTF-16 and UTF-32. You will meet them whenever you read a file saved by certain Windows tools, work with strings in some languages, or see a mysterious pair of bytes at the start of a file. This chapter explains how they work and what those bytes are.

What was run, and what was not

Every program in this chapter was run on Python 3.14 and the output shown is the real output. Facts about UTF-16, UTF-32 and byte order marks come from their Wikipedia articles (the UTF-16 article was read only in part: see "Not Yet Verified" at the end). Details not checked against a source are marked "(check)".

Three Encodings of One Table

All three encodings can represent every code point from U+0000 to U+10FFFF. They differ in the size of the unit they use:

Encoding	Unit	Bytes per code point	Strength
UTF-8	8 bits	1 to 4	Compact for ASCII-heavy text; no byte-order problem
UTF-16	16 bits	2 or 4	Compact for many non-Latin scripts (check: depends on the text)
UTF-32	32 bits	always 4	The Nth code point is at a fixed position (Wikipedia: code points are "directly indexed")

UTF-32 wastes space: Wikipedia notes that 11 of its 32 bits are always zero, making it close to twice the size of UTF-16 and up to four times the size of UTF-8, depending on how many ASCII characters the text has. Here is the same four-character string in each, and the cost per character:

s = "A\u00e9\u20ac\U00010437"
for enc in ["utf-8", "utf-16-le", "utf-32-le"]:
    print(f"{enc:10} {len(s.encode(enc)):2} bytes")
for ch in s:
    print(f"U+{ord(ch):05X}  utf-8 {len(ch.encode('utf-8'))}  utf-16 {len(ch.encode('utf-16-le'))}  utf-32 {len(ch.encode('utf-32-le'))}")

utf-8 10 bytes utf-16-le 10 bytes utf-32-le 16 bytes U+00041 utf-8 1 utf-16 2 utf-32 4 U+000E9 utf-8 2 utf-16 2 utf-32 4 U+020AC utf-8 3 utf-16 2 utf-32 4 U+10437 utf-8 4 utf-16 4 utf-32 4

The -le suffix means little-endian byte order, explained below. U+10437 is a letter of the Deseret alphabet, chosen because it sits above U+FFFF (check: the name of the script is from general knowledge) and is also the worked example in Wikipedia's UTF-16 article.

Surrogate Pairs

A 16-bit unit can hold only 65,536 values: the range U+0000 to U+FFFF, called the Basic Multilingual Plane (check the term; it is plane 0 from the previous chapter's 17 planes). The early 16-bit encoding, UCS-2, stopped there, and Wikipedia calls it obsolete. To reach the other 16 planes UTF-16 uses surrogate pairs: two 16-bit units that together stand for one code point. For a code point from U+10000 up:

Subtract 0x10000, leaving a 20-bit number.
Add the top 10 bits to 0xD800: this is the high surrogate (range D800 to DBFF).
Add the bottom 10 bits to 0xDC00: this is the low surrogate (range DC00 to DFFF).

This is why U+D800 to U+DFFF are not assignable characters, and why UTF-8 (previous chapter) must reject their encodings. In Python:

def to_surrogates(cp):
    v = cp - 0x10000
    return 0xD800 + (v >> 10), 0xDC00 + (v & 0x3FF)

def from_surrogates(hi, lo):
    return 0x10000 + ((hi - 0xD800) << 10) + (lo - 0xDC00)

hi, lo = to_surrogates(0x10437)
print(f"{hi:04X} {lo:04X}", f"{from_surrogates(hi, lo):X}")
print("\U00010437".encode("utf-16-be").hex(" "))
print("\U00010437".encode("utf-16-le").hex(" "))

bad = 0
for cp in range(0x10000, 0x110000):
    raw = chr(cp).encode("utf-16-be")
    if (int.from_bytes(raw[:2], "big"), int.from_bytes(raw[2:], "big")) != to_surrogates(cp):
        bad += 1
print("mismatches against Python for all 1,048,576 supplementary code points:", bad)

D801 DC37 10437 d8 01 dc 37 01 d8 37 dc mismatches against Python for all 1,048,576 supplementary code points: 0

D801 DC37 matches the worked example in Wikipedia. The two 16-bit units appear as bytes d8 01 dc 37 in big-endian order and 01 d8 37 dc in little-endian order, which brings us to byte order. The final loop checks the formula against Python's own encoder for every one of the 1,048,576 code points above U+FFFF.

The Length Trap

A system that stores text as UTF-16 units counts those units, not code points. Windows APIs and Java use UTF-16 internally, and JavaScript may use UCS-2 or UTF-16 (Wikipedia), so the "length" of a string with a character above U+FFFF is 2 there, not 1 (check this per language and version). Python 3 counts code points, so you can compute the UTF-16 count yourself. The same code shows what happens to a lone surrogate:

lone = "\ud800"
print(len(lone), hex(ord(lone)))
try:
    lone.encode("utf-8")
except UnicodeEncodeError as e:
    print(e.reason)
print(lone.encode("utf-8", errors="surrogatepass").hex(" "))
try:
    b"\x00\xd8".decode("utf-16-le")
except UnicodeDecodeError as e:
    print(e.reason)
for s in ["a", "\u00e9", "\U0001F600", "\U00010437"]:
    print(f"code points={len(s)}  utf-16 units={len(s.encode('utf-16-le')) // 2}")

1 0xd800 surrogates not allowed ed a0 80 unexpected end of data code points=1 utf-16 units=1 code points=1 utf-16 units=1 code points=1 utf-16 units=2 code points=1 utf-16 units=2

Python lets a string hold a lone surrogate (it is one "code point"), but refuses to encode it as UTF-8, because UTF-8 forbids them. The surrogatepass handler forces it through, producing the invalid bytes ED A0 80 that last chapter's decoder rejects. In UTF-16 a high surrogate with nothing after it is just as broken, here reported as "unexpected end of data". A string cut between two halves of a pair is a real bug in languages that index by UTF-16 unit.

Byte Order and the Byte Order Mark

A 16-bit unit is stored in two bytes, and there are two orders: big-endian (most significant byte first) and little-endian (least significant first). UTF-8 has no such problem, because its unit is one byte. UTF-16 and UTF-32 come in both orders. A file may begin with a byte order mark (BOM): the code point U+FEFF, which Wikipedia says is formally ZERO WIDTH NO-BREAK SPACE, written in the file's own byte order. That gives these signatures:

Encoding	BOM bytes
UTF-8	`EF BB BF` (optional)
UTF-16 big-endian	`FE FF`
UTF-16 little-endian	`FF FE`
UTF-32 big-endian	`00 00 FE FF`
UTF-32 little-endian	`FF FE 00 00`

If a reader uses the wrong order, the BOM turns into U+FFFE, a "noncharacter" that should never appear in text, which is the clue that the order is wrong. The UTF-8 BOM carries no order information; Wikipedia says the Unicode Standard neither requires nor recommends it, and that some Microsoft tools treat it as required. Python names the three behaviours separately: utf-16 and utf-32 write a BOM and read it to choose the order, utf-16-le and similar have a fixed order and no BOM, and utf-8-sig writes or strips the UTF-8 BOM.

import codecs
text = "Hi"
print(text.encode("utf-16").hex(" "))
print(text.encode("utf-16-le").hex(" "), "|", text.encode("utf-16-be").hex(" "))
print(text.encode("utf-32").hex(" "))
print(text.encode("utf-8-sig").hex(" "))
raw = text.encode("utf-16-le")
print(raw.decode("utf-16-be"))
print(repr(codecs.BOM_UTF16_LE.decode("utf-16-be")))
print(repr(b"\xef\xbb\xbfHi".decode("utf-8")), repr(b"\xef\xbb\xbfHi".decode("utf-8-sig")))

ff fe 48 00 69 00 48 00 69 00 | 00 48 00 69 ff fe 00 00 48 00 00 00 69 00 00 00 ef bb bf 48 69 䠀椀 '\ufffe' '\ufeffHi' 'Hi'

The plain utf-16 output begins ff fe, little-endian, because this machine is little-endian (Python reports it in sys.byteorder; on a big-endian machine it would differ, check). Reading little-endian bytes as big-endian gives the unrelated CJK-looking characters on line six; the wrong-order BOM becomes U+FFFE. The last line shows a classic bug: a UTF-8 file with a BOM, opened as plain utf-8, begins with an invisible U+FEFF that breaks string comparisons, headers and parsers.

Common mistake

Assuming a file's encoding from the BOM alone, or ignoring it. A missing BOM proves nothing (most UTF-8 files have none). A present BOM is strong evidence, but note that FF FE 00 00 could also be a UTF-16 little-endian BOM followed by a NUL character, so check UTF-32 signatures first (Exercise 3).

Exercises

Exercise 1

By hand, then in Python, work out the surrogate pairs for U+1F600 and U+24B62. Show the value after subtracting 0x10000 in binary, split it into its top and bottom 10 bits, add the bases, and compare with chr(cp).encode("utf-16-be").

📄 View solution

Exercise 2

Write utf16_units(s) that returns how many 16-bit units a string takes in UTF-16 without encoding it (2 for code points above U+FFFF, otherwise 1). Check it against the encoder on six strings, including a family emoji built from several code points, and explain why it differs from len.

📄 View solution

Exercise 3

Write sniff(data) that looks for a BOM and returns the encoding name and the remaining bytes (or None and the data unchanged). Test it on text encoded as utf-8-sig, utf-16, utf-16-be and utf-32. Why must the UTF-32 little-endian signature be tested before the UTF-16 one?

📄 View solution

Chapter 4 Quick Reference

UTF-8 uses 1-4 bytes, UTF-16 uses 2 or 4, UTF-32 always 4
Surrogate pair: subtract 0x10000; high = 0xD800 + top 10 bits, low = 0xDC00 + bottom 10 bits
High surrogates D800-DBFF, low surrogates DC00-DFFF; they are not characters and are illegal in UTF-8
UCS-2 is the obsolete 16-bit-only predecessor of UTF-16
Systems that index by UTF-16 unit report 2 for a character above U+FFFF; Python reports 1
BOM is U+FEFF: FE FF / FF FE (UTF-16 BE/LE), 00 00 FE FF / FF FE 00 00 (UTF-32), EF BB BF (UTF-8, optional)
Wrong byte order turns the BOM into U+FFFE; a UTF-8 BOM read as plain UTF-8 leaves an invisible U+FEFF (use utf-8-sig)
Python codecs: utf-16 and utf-32 add and read a BOM; -le/-be variants do not

Not Yet Verified

Checked against Wikipedia's UTF-16, Byte order mark and UTF-32 articles: the surrogate formula and ranges, the D801 DC37 worked example, UCS-2 being obsolete, which systems use UTF-16, the BOM byte sequences, U+FEFF's formal name, the wrong-order U+FFFE effect, the UTF-8 BOM remarks, and UTF-32's fixed width, space cost and direct indexing. The UTF-16 article was read only in part (first 100,000 of about 113,000 characters); the other two were summarised by a fetch tool. The UTF-16 article did not itself spell out the BOM bytes, so the byte values come from the BOM article and from Python's own output. The "Basic Multilingual Plane" name, the script name for U+10437, how each language counts length, and the compactness claims are from general knowledge (check). Sources: en.wikipedia.org/wiki/UTF-16, en.wikipedia.org/wiki/Byte_order_mark, en.wikipedia.org/wiki/UTF-32.

Coming next

Text Encoding and Unicode 5 returns to the two-spellings problem from chapter 1 and fixes it: normalization forms, and how to compare and search text that looks the same.