prize18-2  Exercise 2: A Strict Decoder
========================================

Plan: read the lead byte to learn the length; collect the payload bits from the lead byte and each continuation byte; then reject
overlong forms (value below the minimum for that length), surrogates and values above U+10FFFF. Lead bytes 0xC0 and 0xC1 can only
begin overlong forms, and 0xF5-0xFF could only give values above U+10FFFF, so they are rejected as start bytes.

ex2.py:

  def decode(data):
      out = []
      i = 0
      while i < len(data):
          b = data[i]
          if b < 0x80:
              n, cp, low = 1, b, 0
          elif 0xC2 <= b <= 0xDF:
              n, cp, low = 2, b & 0x1F, 0x80
          elif 0xE0 <= b <= 0xEF:
              n, cp, low = 3, b & 0x0F, 0x800
          elif 0xF0 <= b <= 0xF4:
              n, cp, low = 4, b & 0x07, 0x10000
          else:
              raise ValueError(f"invalid start byte {b:#04x} at {i}")
          if i + n > len(data):
              raise ValueError("truncated sequence")
          for k in range(1, n):
              c = data[i + k]
              if c & 0xC0 != 0x80:
                  raise ValueError(f"bad continuation byte at {i + k}")
              cp = cp << 6 | c & 0x3F
          if cp < low:
              raise ValueError(f"overlong form at {i}")
          if cp > 0x10FFFF or 0xD800 <= cp <= 0xDFFF:
              raise ValueError(f"invalid code point {cp:#x} at {i}")
          out.append(chr(cp))
          i += n
      return "".join(out)

  good = "Hi \u00e9 \u20ac \u8a9e \U0001F600".encode("utf-8")
  print(decode(good) == good.decode("utf-8"))
  for raw in [bytes([0xC0, 0xAF]), bytes([0xED, 0xA0, 0x80]),
              bytes([0xF4, 0x90, 0x80, 0x80]), bytes([0xE2, 0x82]), bytes([0x80])]:
      try:
          decode(raw)
      except ValueError as e:
          print(raw.hex(" "), "->", e)

Real output (Python 3.14):

  True
  c0 af -> invalid start byte 0xc0 at 0
  ed a0 80 -> invalid code point 0xd800 at 0
  f4 90 80 80 -> invalid code point 0x110000 at 0
  e2 82 -> truncated sequence
  80 -> invalid start byte 0x80 at 0

Note: the overlong form of "/" (C0 AF) is caught at the start byte here. An overlong form that starts with a legal lead byte, such as E0 80 AF,
is caught by the "cp < low" check instead. Python's own decoder gives different wording for the same failures; the point is that all are refused.