Grapheme Clusters

Text Encoding and Unicode

Course 1 · Chapter 6 · Grapheme Clusters

Text Encoding and Unicode 1 showed three ways to measure a string: bytes, code points, and what a reader sees. Normalization (the previous chapter) cannot give you the third one. A family emoji, a flag, or a letter with two accents is still several code points after any normalization, yet a reader sees one thing. This chapter defines that unit, the grapheme cluster, shows how emoji and Hangul are built from pieces, and uses it to count, cut and reverse text without breaking it.

What was run, and what was not

Every program in this chapter was run on Python 3.14 and the output shown is the real output. The standard library has no grapheme function (check), so the examples use the third-party regex module (install with pip install regex), whose \X pattern matches one extended grapheme cluster. Facts come from Unicode Standard Annex #29 (UAX #29) and the Wikipedia article on emoji, both read through a summarising tool (see "Not Yet Verified" at the end). Details not checked against a source are marked "(check)".

What Is a Character, to a Reader?

UAX #29 calls what a person thinks of as one character a user-perceived character, and approximates it with the grapheme cluster. Its own example is the letter G with a grave accent: two code points, one perceived character. Grapheme clusters are the units for moving a cursor, selecting, deleting and counting "characters", and a word, line or sentence break should never fall inside one. The standard defines extended grapheme clusters, the recommended default, and older legacy ones kept for compatibility.

Many things that count as a single cluster are built from several code points, each with a job:

Piece	Code points	Effect
Combining mark	e.g. `U+0301`	Attaches to the previous letter
Skin tone modifier	`U+1F3FB` to `U+1F3FF`	Follows a human emoji and sets its tone (added in Unicode 8.0, Wikipedia)
Zero-width joiner (ZWJ)	`U+200D`	Glues emoji into one picture, e.g. a family; unsupported systems show the parts side by side
Regional indicators	`U+1F1E6` to `U+1F1FF`	26 letters; a pair spells a flag
Variation selector 16	`U+FE0F`	Asks for the colourful emoji look (VS15, `U+FE0E`, asks for plain text look)
Keycap sequence	digit or # or *, `U+FE0F`, `U+20E3`	Makes a keycap picture (the exact recipe is from general knowledge: check)

One Cluster, Many Code Points

import regex

samples = {
    "e + acute": "e\u0301",
    "thumbs up + tone": "\U0001F44D\U0001F3FD",
    "UK flag": "\U0001F1EC\U0001F1E7",
    "family": "\U0001F468\u200D\U0001F469\u200D\U0001F467",
    "red heart": "\u2764\uFE0F",
    "keycap 1": "1\uFE0F\u20E3",
    "CR LF": "\r\n",
}
for label, s in samples.items():
    clusters = regex.findall(r"\X", s)
    print(f"{label:17} code points={len(s)}  utf-8 bytes={len(s.encode('utf-8')):2}  clusters={len(clusters)}")

e + acute code points=2 utf-8 bytes= 3 clusters=1 thumbs up + tone code points=2 utf-8 bytes= 8 clusters=1 UK flag code points=2 utf-8 bytes= 8 clusters=1 family code points=5 utf-8 bytes=18 clusters=1 red heart code points=2 utf-8 bytes= 6 clusters=1 keycap 1 code points=3 utf-8 bytes= 7 clusters=1 CR LF code points=2 utf-8 bytes= 2 clusters=1

Every sample is one thing to a reader and one cluster to the library, while the code point and byte counts range up to 5 and 18. Even CR LF (a Windows-style line break) is one cluster, because rule GB3 forbids a break between them. Zoom in on emoji parts:

import regex

def count(s):
    return len(regex.findall(r"\X", s))

def report(s):
    cps = " ".join(f"U+{ord(c):04X}" for c in s)
    print(f"{len(s)} code points, {count(s)} clusters: {cps}")

report("\U0001F44D")
report("\U0001F44D\U0001F3FD")
report("\U0001F469\U0001F3FB\u200D\U0001F4BB")
report("\U0001F1E9\U0001F1EA")
report("\u2764")
report("\u2764\uFE0F")

1 code points, 1 clusters: U+1F44D 2 code points, 1 clusters: U+1F44D U+1F3FD 4 code points, 1 clusters: U+1F469 U+1F3FB U+200D U+1F4BB 2 code points, 1 clusters: U+1F1E9 U+1F1EA 1 code points, 1 clusters: U+2764 2 code points, 1 clusters: U+2764 U+FE0F

The third line is a tone-modified person joined by ZWJ to another emoji: four code points, one picture. The heart shows that U+2764 alone and followed by U+FE0F are both one cluster, differing in how they should be drawn.

The Rules Behind the Clusters

UAX #29 gives every code point a Grapheme_Cluster_Break property (CR, LF, Control, Extend, ZWJ, Regional_Indicator, Prepend, SpacingMark, and the Hangul types L, V, T, LV and LVT) and then a list of rules. In plain terms:

GB3: do not break between CR and LF. GB4, GB5: otherwise always break around controls.
GB6 to GB8: keep Hangul jamo (and syllable) sequences together.
GB9: do not break before Extend or ZWJ characters (this keeps a letter with its marks). GB9a: do not break before a SpacingMark. GB9b: do not break after a Prepend character.
GB11: do not break inside an emoji ZWJ sequence.
GB12, GB13: regional indicators pair up into flags; an odd one out is left alone.
GB999: anywhere else, break.

Hangul shows why composition does not matter to the rules: a syllable written as one code point and the same syllable as three conjoining jamo are both single clusters. Flags show the odd-one-out rule:

import regex
import unicodedata as ud

for form in ["NFC", "NFD"]:
    s = ud.normalize(form, "\uac01")
    print(form, [f"{ord(c):04X}" for c in s], len(regex.findall(r"\X", s)))

two_flags = "\U0001F1EC\U0001F1E7\U0001F1EB\U0001F1F7"
three_ri = two_flags + "\U0001F1E9"
print(len(regex.findall(r"\X", two_flags)), len(regex.findall(r"\X", three_ri)))

NFC ['AC01'] 1 NFD ['1100', '1161', '11A8'] 1 2 3

Four regional indicators make two flags; a fifth makes a third cluster that is not a valid flag by itself, but is still a cluster. The Hangul syllable 각 (U+AC01) is the example UAX #29 itself uses.

A Small Segmenter

You do not need to write one in practice, but building a simplified one makes the rules concrete. This version treats marks, joiners, variation selectors and skin tones as "extend", glues whatever follows a ZWJ, pairs regional indicators, and keeps CR LF together. It leaves out Hangul, Prepend and SpacingMark, so it is a teaching sketch and not UAX #29:

import unicodedata as ud

def is_extend(c):
    cp = ord(c)
    return (ud.category(c) in ("Mn", "Me")      # nonspacing and enclosing marks
            or cp == 0x200D                      # zero width joiner (handled separately)
            or 0xFE00 <= cp <= 0xFE0F            # variation selectors
            or 0x1F3FB <= cp <= 0x1F3FF)         # emoji skin tone modifiers

def is_ri(c):
    return 0x1F1E6 <= ord(c) <= 0x1F1FF

def clusters(s):
    out = []
    i = 0
    while i < len(s):
        j = i + 1
        if s[i:i + 2] == "\r\n":
            j = i + 2
        elif is_ri(s[i]) and j < len(s) and is_ri(s[j]):
            j = i + 2
        while j < len(s) and (is_extend(s[j]) or s[j - 1] == "\u200D"):
            j += 1
        out.append(s[i:j])
        i = j
    return out

text = "e\u0301 \U0001F44D\U0001F3FD \U0001F1EC\U0001F1E7 \U0001F468\u200D\U0001F469\u200D\U0001F467 \u2764\uFE0F 1\uFE0F\u20E3 \r\n"
mine = clusters(text)
print(len(text), len(mine))
import regex
print(mine == regex.findall(r"\X", text))

24 13 True

On this sample, 24 code points make 13 clusters (seven things plus six spaces), and the sketch agrees with the library. It would not on Hangul jamo or Indic scripts, and the rule list above is the reason: the real rules have more cases than a page of code. Use a maintained library.

Using Clusters: Counting, Cutting, Reversing

Cutting a string at a code point index can split a cluster, leaving a stray accent, half a family, or a dangling joiner. Cut at cluster boundaries instead:

import regex

def graphemes(s):
    return regex.findall(r"\X", s)

def truncate(s, n):
    return "".join(graphemes(s)[:n])

s = "cafe\u0301 \U0001F468\u200D\U0001F469\u200D\U0001F467!"
print(len(s), len(graphemes(s)))
print(ascii(s[:4]), ascii(truncate(s, 4)))
print(ascii(s[:7]))
print(ascii(truncate(s, 6)))
rev = graphemes(s)[::-1]
print(ascii(s[::-1][:3]), ascii(rev[:2]))

12 7 'cafe' 'cafe\u0301' 'cafe\u0301 \U0001f468' 'cafe\u0301 \U0001f468\u200d\U0001f469\u200d\U0001f467' '!\U0001f467\u200d' ['!', '\U0001f468\u200d\U0001f469\u200d\U0001f467']

(ascii() shows the escapes so the invisible parts can be seen.) A four-code-point slice lost the accent from "café", and a seven-code-point slice kept only the man of the family. Reversing by code points left the string starting with a stray girl and a joiner, while reversing the list of clusters kept each cluster intact. The same logic applies to character limits in forms: count clusters if you mean what a reader sees, but remember that storage limits are still in bytes or code points (a database may count differently: next chapters).

Common mistake

Assuming the cluster rules are fixed. New emoji sequences appear with each Unicode release, so a library built on an older Unicode version may split a newer sequence (check). Python's own unicodedata reports its Unicode version in unicodedata.unidata_version, and a library's data may be older or newer than that. Also, a cluster is a good default for "one character", not a perfect one: what a reader sees depends on the font and language.

Exercises

Exercise 1

Write report(s) printing the number of code points, the number of clusters and each code point in hex. Use it on a thumbs-up, the same thumbs-up with a skin tone, a woman with a tone joined to a laptop (use ZWJ), a flag, a plain heart and a heart with U+FE0F. Predict the cluster counts first.

📄 View solution

Exercise 2

Show that composition does not change cluster counts. Take three Hangul syllables, count code points and clusters under NFC and under NFD. Then show that "a", CR LF, "b" is 3 clusters but 4 code points, and compare a letter followed by two combining marks with a combining mark followed by a letter.

📄 View solution

Exercise 3

Write truncate(s, limit) that shortens text to at most limit clusters, ending with an ellipsis when it cuts, and never splitting a cluster. Test it on plain text, text with a combining accent, and text containing a family emoji. Compare with a naive slice.

📄 View solution

Chapter 6 Quick Reference

A grapheme cluster approximates a user-perceived character; use it for counting, cursor movement, selection, truncation and reversal
Emoji parts: skin tone modifiers (U+1F3FB-1F3FF), ZWJ (U+200D), regional indicators (U+1F1E6-1F1FF, pairs make flags), variation selector U+FE0F
Rules (UAX #29): GB3 keeps CR LF together; GB9 keeps marks and ZWJ with their base; GB11 keeps emoji ZWJ sequences; GB12/13 pair regional indicators
Hangul syllables and conjoining jamo are single clusters either way, so normalization does not change cluster counts
Python's standard library has no grapheme function (check); the regex module's \X matches one extended grapheme cluster
Never slice or reverse by code point when the output is for people; cut and reverse lists of clusters
Segmentation rules grow with each Unicode version; keep libraries current

Not Yet Verified

Checked, through a summarising fetch tool and not by reading line by line, against UAX #29 (user-perceived characters, extended versus legacy clusters, the property values, rules GB3 to GB13 in plain terms, the Hangul and flag examples) and Wikipedia's emoji article (skin tone range and Unicode 8.0, ZWJ, regional indicator range, VS15 and VS16). The emoji article was read only in part (first 100,000 of about 228,000 characters). The article did not explain keycap sequences or the flag country-code mapping, so the keycap recipe is from general knowledge (check). That the standard library has no grapheme segmentation, how third-party libraries track Unicode versions, and the behaviour of the sample outputs on other library versions are also from general knowledge or from this one run (check). The simplified segmenter is my own sketch and deliberately incomplete. Sources: unicode.org/reports/tr29/, en.wikipedia.org/wiki/Emoji.

Coming next

Text Encoding and Unicode 7 follows text out of the program: files, network protocols and web pages, and how each of them says (or fails to say) which encoding the bytes use.