Debugging Encodings

Text Encoding and Unicode

Course 1 · Chapter 10 · Debugging Encodings

You now know what text is (Text Encoding and Unicode 1), how UTF-8 and its rivals are built (Text Encoding and Unicode 2, 3 and 4), why equal-looking strings differ (Text Encoding and Unicode 5 and 6), and where encodings are declared or lost (Text Encoding and Unicode 7 and 8), and the security angle (Text Encoding and Unicode 9). This final chapter turns that knowledge into a repeatable debugging method, builds a small toolkit, and then solves a realistic damaged string end to end.

What was run, and what was not

Every program in this chapter was run on Python 3.14 and the output shown is the real output. This chapter introduces no new reference material: the facts it uses were established, with sources, in the earlier parts of the course, and the new material (the workflow and the toolkit) is my own design, so it is a worked method and not a standard. Details not checked against a source are marked "(check)". The toolkit uses heuristics and can be wrong on unusual text.

The Six-Step Method

Look at the bytes and code points, not the rendered text. Print ascii(s) and the UTF-8 hex. A screen hides invisible characters and may itself be using the wrong font or encoding.
Recognise the pattern. Most damage has a recognisable shape (table below).
Decide: reversible or lost? If the original bytes are still all present (just misread) you can reverse it. If a step replaced characters with ? or U+FFFD, the information is gone and you must re-fetch the source.
Find the boundary. Text passes through boundaries (file, network, database, program). Test the text after each one and find the first where it stops matching.
Fix at the source. Declare or set the right encoding at the faulty boundary. Do not scatter repairs through the program.
Prevent. Always pass an encoding, normalize on input, and keep test data with non-ASCII characters, emoji and a decomposed accent in it.

Step 1: Look at the Real Data

def show(s):
    print("text :", s)
    print("repr :", ascii(s))
    print("utf-8:", s.encode("utf-8", errors="surrogatepass").hex(" "))

show("caf\u00e9 \u2014 \u201cok\u201d")

text : café — “ok” repr : 'caf\xe9 \u2014 \u201cok\u201d' utf-8: 63 61 66 c3 a9 20 e2 80 94 20 e2 80 9c 6f 6b e2 80 9d

The repr line shows exactly which code points are present (including any invisible ones, Text Encoding and Unicode 9), and the hex line shows the bytes, which is what you compare with what you expect from a file or a network capture.

Step 2: Recognise the Pattern

Take one clean string and damage it in the common ways, to learn what each looks like:

original = "caf\u00e9 \u2014 \u201cok\u201d"
raw = original.encode("utf-8")
for wrong in ["cp1252", "cp437"]:
    print(f"{wrong:8} {raw.decode(wrong, errors='replace')}")
print("ascii    ", raw.decode("ascii", errors="replace"))
print("lossy    ", original.encode("ascii", errors="replace").decode("ascii"))

cp1252 cafÃ© â€” â€œokâ€� cp437 caf├⌐ ΓÇö ΓÇ£okΓÇ¥ ascii caf�� ok�� lossy caf? ? ?ok?

The Latin-1 version of this damage looks much the same but with invisible control characters in place of the Windows-1252 punctuation. Use this as a field guide:

You see	Likely cause	Reversible?
Ã© Ã¨ Ã¶ and â€™ â€œ	UTF-8 bytes read as Windows-1252 or Latin-1 (Text Encoding and Unicode 3)	Usually yes, if no byte was replaced; the last quote above ended in U+FFFD (the replacement character), because byte 9D is undefined in Windows-1252, so that one is lost
Ã¯Â»Â¿ at the start	A UTF-8 byte order mark read as text, often after two layers (Text Encoding and Unicode 4)	Yes: decode with `utf-8-sig`
Ã followed by more Ã, Â and ‚	Several layers of the same mistake	Yes, one layer at a time
� (U+FFFD)	A decoder replaced bytes it could not read	No: re-read the original source
? where a letter should be	An encoder replaced characters it could not represent	No
é or `\u00e9` shown literally	Escaped text that was never unescaped (Text Encoding and Unicode 7)	Yes: unescape at the right layer
Letters separated by invisible NULs	16-bit text read as 8-bit (Text Encoding and Unicode 4)	Yes: decode as UTF-16
Empty boxes only	Characters are right but the font lacks glyphs	Nothing to repair: change font

Steps 3 to 5: A Toolkit and a Capstone

For the reversible case, "UTF-8 read as Windows-1252", the repair is to encode the garbled text back to bytes with the wrong codec, then decode those bytes as UTF-8. Real data complicates this in two ways: some bytes are undefined in Windows-1252 (so many tools turn them into the matching control character, which we copy), and damage may have happened more than once. The program below also contains a guard: it only accepts a repair if it reduces the number of tell-tale marker sequences, so clean text, and legitimate text such as the Portuguese city name in capitals, is left alone.

MARKERS = ("\u00c3", "\u00c2", "\u00e2\u20ac", "\u00ef\u00bb\u00bf")

def suspicion(s):
    return sum(s.count(m) for m in MARKERS)

def lenient_decode(raw):
    """Windows-1252, but the five undefined bytes become the matching control characters."""
    out = []
    for b in raw:
        try:
            out.append(bytes([b]).decode("cp1252"))
        except UnicodeDecodeError:
            out.append(chr(b))
    return "".join(out)

def lenient_encode(s):
    out = bytearray()
    for c in s:
        try:
            out += c.encode("cp1252")
        except UnicodeEncodeError:
            if ord(c) > 0xFF:
                raise
            out.append(ord(c))
    return bytes(out)

def undo_once(s):
    """Try to reverse one layer of 'UTF-8 bytes read as Windows-1252'."""
    try:
        fixed = lenient_encode(s).decode("utf-8")
    except (UnicodeEncodeError, UnicodeDecodeError):
        return None
    return fixed if suspicion(fixed) < suspicion(s) else None

def peel(s, max_layers=5):
    layers = 0
    while layers < max_layers:
        fixed = undo_once(s)
        if fixed is None:
            break
        s = fixed
        layers += 1
    return s, layers

once = lenient_decode("caf\u00e9 \u2014 \u201cok".encode("utf-8"))
print(once)
print(peel(once))
for fine in ["already fine \u00e9", "S\u00c3O PAULO"]:
    print(peel(fine))

# capstone: a BOM, a curly apostrophe, accents, a euro sign, a dash and a quote, damaged twice
original = "\ufeffZo\u00eb\u2019s caf\u00e9 costs 5 \u20ac \u2014 \u201copen"
layer1 = lenient_decode(original.encode("utf-8"))
layer2 = lenient_decode(layer1.encode("utf-8"))
print(ascii(original))
print(layer1)
print(layer2.replace(chr(0x9d), "<9D>"))
fixed, layers = peel(layer2)
print(layers, fixed == original)
print(ascii(fixed[:6]))
print(fixed.lstrip("\ufeff"))

cafÃ© â€” â€œok ('café — “ok', 1) ('already fine é', 0) ('SÃO PAULO', 0) '\ufeffZo\xeb\u2019s caf\xe9 costs 5 \u20ac \u2014 \u201copen' ï»¿ZoÃ«â€™s cafÃ© costs 5 â‚¬ â€” â€œopen Ã¯Â»Â¿ZoÃƒÂ«Ã¢â‚¬â„¢s cafÃƒÂ© costs 5 Ã¢â€šÂ¬ Ã¢â‚¬â€<9D> Ã¢â‚¬Å“open 2 True '\ufeffZo\xeb\u2019s' Zoë’s café costs 5 € — “open

The capstone follows the method. Step 1: the repr of the original shows an invisible U+FEFF (a BOM) at the front. Step 2: the first damaged copy shows "ï»¿" and "Ã«", the second shows the same marks twice over ("Ã¯Â»Â¿"): two layers. Step 3: every byte survived, so the damage is reversible. Step 4: peel reports 2 layers and recovers the original exactly (True), which means the fault happened twice, that is, at two boundaries. Step 5 would be to find those two boundaries and set the encoding correctly there; step 6, to add a test containing a BOM, an accent and an euro sign. The final line strips the BOM, which should be removed once its job is done. As a check on the guard, "SÃO PAULO" was left untouched because reversing it would not produce valid UTF-8.

Finding the Boundary

Step 4 is a search: run the text through the stages in order and report the first stage whose output differs from the original. Here each stage is a real round trip (file, JSON, database, export), and in each run we inject a fault at one stage:

import json, sqlite3, tempfile, pathlib

original = "Zo\u00eb\u2019s caf\u00e9 \u20ac5"
tmp = pathlib.Path(tempfile.mkdtemp())

def lenient_decode(raw):
    out = []
    for b in raw:
        try:
            out.append(bytes([b]).decode("cp1252"))
        except UnicodeDecodeError:
            out.append(chr(b))
    return "".join(out)

def file_ok(s):
    p = tmp / "a.txt"
    p.write_text(s, encoding="utf-8", newline="\n")
    return p.read_text(encoding="utf-8")

def json_ok(s):
    return json.loads(json.dumps({"v": s}))["v"]

def db_ok(s):
    con = sqlite3.connect(":memory:")
    con.execute("CREATE TABLE t (v TEXT)")
    con.execute("INSERT INTO t VALUES (?)", (s,))
    return con.execute("SELECT v FROM t").fetchone()[0]

def wrong_read(s):
    return lenient_decode(s.encode("utf-8"))

def to_ascii(s):
    return s.encode("ascii", errors="replace").decode("ascii")

def first_bad_stage(text, stages):
    current = text
    for name, fn in stages:
        current = fn(current)
        if current != text:
            return name, current
    return None, current

names = ["file", "json", "database", "export"]
good = {"file": file_ok, "json": json_ok, "database": db_ok, "export": json_ok}
for fault, bad in [(None, None), ("file", wrong_read), ("database", wrong_read), ("export", to_ascii)]:
    stages = [(n, bad if n == fault else good[n]) for n in names]
    name, result = first_bad_stage(original, stages)
    print(f"fault in {fault!s:9} -> first bad stage: {name!s:9} {result}")

fault in None -> first bad stage: None Zoë’s café €5 fault in file -> first bad stage: file ZoÃ«â€™s cafÃ© â‚¬5 fault in database -> first bad stage: database ZoÃ«â€™s cafÃ© â‚¬5 fault in export -> first bad stage: export Zo??s caf? ?5

Each injected fault is found at its own stage. Without a fault, no stage is reported. The last row is the irreversible case, with question marks: it can be found, but not repaired. The method scales: in a real system the "stages" are captures and log lines taken at each boundary, and the comparison is done by eye against the hex dump from step 1.

Common mistake

Repairing garbled text with a fix-up function at the point where you noticed it. That hides the real fault, which will return with the next record. Use a repair function like peel only to rescue existing damaged data, then fix the boundary that caused it. Also never repair blindly: keep a copy of the original, and review a sample before bulk-fixing.

What the Course Gave You

Part	The idea to keep
Text Encoding and Unicode 1	Bytes, code points and what a reader sees are three different counts
Text Encoding and Unicode 2	UTF-8 is a small, strict bit pattern; decoders must refuse overlong forms and surrogates
Text Encoding and Unicode 3	Legacy encodings assign the upper half of a byte differently; "no error" proves nothing
Text Encoding and Unicode 4	UTF-16 uses surrogate pairs; byte order marks tell the order but are not always present
Text Encoding and Unicode 5	Normalize (NFC) before comparing; use NFKC only for matching
Text Encoding and Unicode 6	Count, cut and reverse by grapheme cluster when the audience is a person
Text Encoding and Unicode 7	Say the encoding at every boundary: files, headers, pages
Text Encoding and Unicode 8	Databases store what they are given; check character set, connection and collation
Text Encoding and Unicode 9	Decode strictly, normalize, then validate; make invisible characters visible

Exercises

Exercise 1

Write report(s) that checks a string for the symptoms in the field guide (replacement characters, a question mark between letters, BOM bytes shown as text, mojibake markers, numeric entities, literal backslash-u escapes, NUL characters) and returns a list of findings. Try it on eight samples, including a clean one.

📄 View solution

Exercise 2

Using the toolkit from this chapter, damage a clean string by zero, one, two and three layers of "UTF-8 read as Windows-1252" and check that peel recovers the original each time. Then show two irreversible cases (ASCII replacement with question marks, and UTF-8 decoded as ASCII with replacement) and what peel does with them.

📄 View solution

Exercise 3

Extend the boundary search: build a four-stage pipeline (file, JSON, database, export) with correct round-trip stages, then inject a fault at each stage in turn (a wrong-encoding read at the file, the database, and a lossy ASCII export) and confirm first_bad_stage names the right stage each time. What does the output of the lossy case tell you about repair?

📄 View solution

Chapter 10 Quick Reference

Method: look at bytes, recognise the pattern, decide reversible or lost, find the boundary, fix at the source, prevent
ascii(s) and s.encode("utf-8").hex(" ") show what is really there
Reverse one layer: s.encode(wrong).decode("utf-8"), copying the tolerant handling of the undefined bytes; guard against false repairs
A faulty boundary is found by comparing the text after each stage with the original
Repair functions rescue existing data; the real fix is to declare the right encoding where the fault happens
Test with non-ASCII data: accents, emoji, a decomposed accent, a BOM

Not Yet Verified

This chapter's facts are the ones established with sources in the earlier parts of the course (the Windows-1252 undefined bytes, the BOM bytes, the behaviour of the replacement handlers); it cites no new external source. The field guide, the six-step method, the marker list and the guard heuristic are my own design from this course, not an established standard (check against your own data). The claim that many tools map the five undefined bytes to control characters is from the Windows-1252 discussion in Text Encoding and Unicode 3 (which cited one source for the existence of such mappings), and that the lenient decoder reproduces any particular tool's behaviour is not verified (check). The Portuguese city-name example shows a false positive being avoided, not a general guarantee. The font remark in the table (boxes mean missing glyphs) is general knowledge (check).

Course complete

That is the end of Text Encoding and Unicode. You can now say what a string is made of at three levels, read and write the main encodings by hand, compare and search text safely, send it across boundaries without damage, defend against the main tricks, and track a garbled string back to its cause. Keep the field guide table and the six steps; they will outlast any particular library.