Normalization

Text Encoding and Unicode

Course 1 · Chapter 5 · Normalization

In Text Encoding and Unicode 1 we met two strings that look identical but compare unequal: "é" as one code point, and "e" followed by a combining accent. That was a preview of a real problem: Unicode often gives several ways to write the same text. A login that accepts "Zoë" on one device and rejects it on another, a search that misses a word, a duplicate row in a database: these usually come from this. The cure is normalization, and this chapter explains what it does and when to use which form.

What was run, and what was not

Every program in this chapter was run on Python 3.14 and the output shown is the real output. Facts about normalization come from the Wikipedia article "Unicode equivalence" and from Unicode Standard Annex #15 (UAX #15), both read through a summarising tool (see "Not Yet Verified" at the end). Details not checked against a source are marked "(check)".

Two Kinds of "Same"

Unicode defines two levels of equivalence between sequences of code points:

Kind	Meaning	Example
Canonical equivalence	Assumed to have the same appearance and meaning; may be substituted for each other	`U+00E9` and `e + U+0301`; the angstrom sign `U+212B` and `U+00C5`
Compatibility equivalence	May look different but share a meaning in some contexts; treating them alike is sometimes right and sometimes wrong	The ligature `U+FB03` and "ffi"; the superscript `U+2075` and "5"; fullwidth Ａ and A

Canonical equivalence implies compatibility equivalence, but not the other way round (Wikipedia). The first kind is about the same text spelled two ways; the second is about different characters that are close enough for some purposes, such as searching.

The Four Normalization Forms

A normalization form converts a string into one fixed representative of its equivalence class, so that equivalent strings become identical. There are four, built from two operations: decomposition (split into base letter plus marks) and composition (join back where a single code point exists):

Form	What it does
NFD	Canonical decomposition, with combining marks in a set order
NFC	Canonical decomposition, then canonical composition: the shortest, "precomposed" form
NFKD	Compatibility decomposition, with combining marks in a set order
NFKC	Compatibility decomposition, then canonical composition

The "K" stands for compatibility. Python exposes all four through unicodedata.normalize. Start with the problem from chapter 1:

import unicodedata as ud

a = "\u00e9"
b = "e\u0301"
print(a == b, len(a), len(b))
for form in ["NFC", "NFD"]:
    na, nb = ud.normalize(form, a), ud.normalize(form, b)
    print(form, na == nb, len(na), len(nb))

def same_text(x, y):
    return ud.normalize("NFC", x) == ud.normalize("NFC", y)

print(same_text(a, b))

False 1 2 NFC True 1 1 NFD True 2 2 True

Either form makes the two strings equal; NFC makes both one code point long and NFD makes both two. UAX #15 states that equivalent strings have "precisely the same normalized form", which is what allows plain binary comparison afterwards. It also states that each form is idempotent: normalizing twice changes nothing.

What Each Form Does to Awkward Characters

Run all four forms over six characters and the differences between the canonical pair (NFC, NFD) and the compatibility pair (NFKC, NFKD) become clear:

import unicodedata as ud

forms = ["NFC", "NFD", "NFKC", "NFKD"]
samples = {
    "angstrom sign": "\u212b",
    "A with ring": "\u00c5",
    "ffi ligature": "\ufb03",
    "superscript 5": "\u2075",
    "roman numeral IV": "\u2163",
    "fullwidth A": "\uff21",
}
for label, s in samples.items():
    row = []
    for f in forms:
        n = ud.normalize(f, s)
        row.append(" ".join(f"{ord(c):04X}" for c in n))
    print(f"{label:17} U+{ord(s):04X} | " + " | ".join(f"{f}: {r}" for f, r in zip(forms, row)))

Reading the table:

The angstrom sign and the Swedish letter Å are canonically equivalent, so all forms fold the sign into U+00C5 (NFC) or A plus U+030A, a combining ring (NFD).
The ligature, the superscript, the Roman numeral and the fullwidth letter are only compatibility equivalents: NFC and NFD leave them alone, while NFKC and NFKD replace them with plain letters and digits. A search for "f" matches the ligature under NFKC but not under NFC (Wikipedia).

Combining Marks Have an Order

When a letter carries several marks, the order they are typed in can differ without changing the look. Each mark has a combining class number, and normalization sorts adjacent marks of non-zero class by that number (UAX #15), keeping the original order for marks of equal class, since those interact visually. Python reports the class with unicodedata.combining:

import unicodedata as ud

s = "e\u0323\u0302"          # e + dot below + circumflex
t = "e\u0302\u0323"          # e + circumflex + dot below
print(s == t, [ud.combining(c) for c in s], [ud.combining(c) for c in t])
print(ud.normalize("NFD", s) == ud.normalize("NFD", t))
print(ud.normalize("NFC", s) == ud.normalize("NFC", t), [f"{ord(c):04X}" for c in ud.normalize("NFC", s)])

u = "\u1ebf"                # e with circumflex and acute
d = ud.normalize("NFD", u)
print([f"{ord(c):04X}" for c in d], [ud.combining(c) for c in d])
print(ud.normalize("NFC", "e\u0302\u0301") == u, ud.normalize("NFC", "e\u0301\u0302") == u)
print(ud.normalize("NFC", ud.normalize("NFC", s)) == ud.normalize("NFC", s))

False [0, 220, 230] [0, 230, 220] True True ['1EC7'] ['0065', '0302', '0301'] [0, 230, 230] True False True

The dot below has class 220 and the circumflex class 230, so both orders sort to the same sequence, and under NFC both become the single code point U+1EC7. In the second part the acute and circumflex share class 230, so their typed order is kept: "e, circumflex, acute" composes to U+1EBF, but "e, acute, circumflex" does not. The last line demonstrates idempotence.

Which Form, When?

NFC is the usual choice for storing and exchanging text and for comparing strings. It is compact, and it keeps compatibility characters as they were. (I believe the W3C recommends NFC for web content: check, neither source I read said so.)
NFD is useful as a working form when you want to take apart letters and marks, for example to strip accents (Exercise 3).
NFKC / NFKD are for matching, such as search or identifier comparison, where folding ligatures and fullwidth letters is wanted. UAX #15 warns they "must not be blindly applied to arbitrary text", because they discard distinctions.

The loss is real. Here a user-name key is built with NFKC followed by casefold() (a stronger lowercase), and the same call is shown ruining a sentence:

import unicodedata as ud

names = ["Zo\u00eb", "Zoe\u0308", "ZO\u00cb"]
print(len(set(names)), len(set(ud.normalize("NFC", n) for n in names)))
print(len(set(ud.normalize("NFC", n).casefold() for n in names)))

text = "The \ufb01nal \uff21\uff22\uff23 costs 2\u00b2 \u2163"
print(text)
print(ud.normalize("NFC", text))
print(ud.normalize("NFKC", text))
print("final" in ud.normalize("NFC", text), "final" in ud.normalize("NFKC", text))

x = ud.normalize("NFC", "e")
y = ud.normalize("NFC", "\u0301")
print(len(ud.normalize("NFC", x + y)), len(x + y))

3 2 1 The ﬁnal ＡＢＣ costs 2² Ⅳ The ﬁnal ＡＢＣ costs 2² Ⅳ The final ABC costs 22 IV False True 1 2

Three lessons from the output. First, three spellings of one name collapse to two strings after NFC (the third differs only in case) and to one after casefold. Second, NFKC makes "final" findable (the ligature is gone), but it also turned "2²" into "22", a meaning change, which is why NFKC suits matching and not stored text. Third, normalized strings are not closed under concatenation (UAX #15): "e" and a lone accent are each already in NFC, but joined they are not, because they now compose into one code point (length 1, not 2). Normalize after you join, not before.

Common mistake

Normalizing only one side of a comparison, or normalizing in one place and not another. Pick a form, apply it where text enters your system (input, import, API), and compare only normalized strings. Also remember that normalization does not fix case, accents you think of as "the same letter" (it will not turn "ø" into "o"), or a mix-up of encodings (that is chapter 3's problem).

Exercises

Exercise 1

Write canon_eq(a, b) and compat_eq(a, b) using NFD and NFKD. Test six pairs: é and e+acute, Å and the angstrom sign, the ffi ligature and "ffi", two mark orders that share a combining class, fullwidth A and A, and a and A. Predict each answer first.

📄 View solution

Exercise 2

Write key(name) (NFKC then casefold()) and use it to group six user names that differ by composition, case, fullwidth letters, a missing accent and a plural. How many distinct names are there raw, and how many after keying?

📄 View solution

Exercise 3

Write strip_accents(s): decompose with NFD, drop characters with a non-zero combining class, and recompose with NFC. Try it on "Zoë", "crème brûlée", "Ångström", "øre", "straße" and "Łódź". Which letters does it not change, and why?

📄 View solution

Chapter 5 Quick Reference

Canonical equivalence: same text spelled differently (é vs e+accent); compatibility equivalence: similar characters (ligatures, superscripts, fullwidth)
NFD = canonical decompose; NFC = decompose then compose; NFKD / NFKC = the same with compatibility mappings
Equivalent strings share one normalized form; every form is idempotent
Combining marks are sorted by combining class; marks of the same class keep their order
Use NFC to store and compare; NFD to take letters apart; NFKC/NFKD only for matching, since they lose information ("2²" became "22")
Normalization is not closed under concatenation: normalize after joining
Python: unicodedata.normalize(form, s), unicodedata.combining(c), str.casefold()

Not Yet Verified

Checked, through a summarising fetch tool and not by reading line by line, against Wikipedia's "Unicode equivalence" article and UAX #15: the two kinds of equivalence, the definitions of the four forms, the angstrom, ligature, superscript and fullwidth examples, canonical ordering by combining class, idempotence, the NFKC warning, and non-closure under concatenation. Neither source, as summarised, stated which form the W3C recommends, so that remark is general knowledge (check). The specific outputs for U+2163, U+00B2 and U+1EC7 come from Python's own tables, not from the sources. The remark that casefold is "a stronger lowercase" is from general knowledge (check). Sources: en.wikipedia.org/wiki/Unicode_equivalence, unicode.org/reports/tr15/.

Coming next

Text Encoding and Unicode 6 asks what a "character" is to a reader: grapheme clusters, how emoji sequences are built, and why counting characters correctly needs more than normalization.