Security

Everything that makes Unicode useful also makes it a place to hide things. Characters from different alphabets can look identical. Some characters are invisible. Others change the direction in which text is displayed, so that what you read on screen is not the order in which a program reads it. This chapter shows the main tricks, with code to detect them, and the rule that ties the course together: validate text only after you have normalized it. The programs here are defensive checks, written to be run on suspicious text, not to produce it.

Look-Alikes: Homoglyphs

Different scripts contain letters that look almost the same. The Cyrillic small letter a (U+0430) and the Latin small letter a (U+0061) are different code points, so a string using one is a different string. UTS #39 gives the classic example of "paypal" with the Cyrillic a, and calls such pairs confusables. Normalization does not help, because these characters are not canonically or compatibly equivalent: they are simply different letters that look alike.

On screen the two words in line six look identical (in most fonts, check), yet they compare unequal and the second is not ASCII. Both have six characters, so a length check sees nothing. Two defences are common. The simplest is to restrict identifiers such as user names to ASCII, or to a single script. UTS #39 describes graduated restriction levels (from ASCII-only and single-script up to unrestricted), and a detection method: reduce each string to a skeleton by mapping every character to a prototype, and treat strings with equal skeletons as confusable. The standard stresses that skeletons are for internal comparison and should not be shown or stored in place of the original. A small mixed-script check, a rough version of the first idea, uses the Unicode character name to find the script:

The mixed-script word is caught; an all-Cyrillic word is not flagged, because it is a legitimate word in one script (a fully Cyrillic string made only of look-alike letters would still pass: another reason to prefer a real library implementing UTS #39 for anything important, check what is available for your language). My first-word-of-the-name trick is a crude stand-in: UTS #39 uses the Unicode Script_Extensions property and treats Common and Inherited characters as matching any script, which this sketch ignores.

Invisible Characters

Unicode has characters that take no space on screen. Python classifies them as category Cf (format). Some are essential in some languages (the zero-width joiner of Text Encoding and Unicode 6 builds emoji sequences), but in a user name or a file name they are a way to make two different strings look the same:

"admin" with a trailing zero-width space prints as "admin" but is a different, longer string. You have already met two of these characters: U+FEFF, the byte order mark (Text Encoding and Unicode 4), and the joiner. UTS #39 removes Default_Ignorable characters when building skeletons for exactly this reason. Always display suspicious strings through ascii() or a similar escaping view when debugging.

Direction Changers: Bidirectional Controls

Scripts such as Arabic and Hebrew run right to left. The Unicode Bidirectional Algorithm (UAX #9) therefore defines invisible control characters that set the direction of a stretch of text. Per UAX #9 these are: embeddings (LRE U+202A, RLE U+202B), overrides (LRO U+202D, RLO U+202E), the terminator PDF (U+202C), and the isolates LRI U+2066, RLI U+2067, FSI U+2068 with their terminator PDI U+2069. The standard itself advises avoiding overrides where possible because of security concerns.

The security problem, named Trojan Source by researchers Nicholas Boucher and Ross Anderson (University of Cambridge; USENIX Security 2023), is that compilers and interpreters "adhere to the logical ordering of source code, not the visual order", so bidirectional controls hidden in comments or strings can make code look different in a review from what runs. Their page lists CVE-2021-42574 for the bidirectional attack and CVE-2021-42694 for a related homoglyph attack, and recommends that tools warn or error on unterminated bidi controls and on confusable identifiers, and that editors and repository front ends make such characters visible. A detector is short:

The names and positions are the printed proof. Without the detector, the line would simply display with its pieces reordered. Legitimate right-to-left text uses these controls too, so a detector should flag them for human review in source code, file names and identifiers, not delete them everywhere.

Order of Operations: Normalize First, Then Validate

Several earlier parts of this course have pointed at one rule. If you check text for forbidden content and then normalize it, the normalization can create what the check was meant to prevent. Compatibility normalization turns fullwidth angle brackets into ordinary ones, so a filter that looks for ASCII markup misses the fullwidth version, and the later step produces the dangerous text:

The first False is the bug: the filter passed text that becomes <script> after NFKC. Checking the normalized form (last line) catches it. The same logic applies to the overlong UTF-8 forms of Text Encoding and Unicode 2: decode strictly first, then check the decoded text, never the raw bytes. (A blocklist is a weak defence on its own; the point here is only the ordering. Better defences allow known-good input and escape output for its context, check.) Combining all the checks of this chapter into one reporting function:

Note the policy choices inside the function: it tolerates the joiner and non-joiner (needed by some scripts and by emoji, check which your users need), accepts a fully Cyrillic word, and reports rather than rejects. A real policy depends on what the field is: a password field should accept almost anything, a user name should accept very little, and a file name sits somewhere between.

Exercises