prize18-7  Exercise 3: Decoding an HTTP Body
============================================

Plan: use the header's charset when there is one (and let a wrong one raise an error, since silently ignoring a declared charset would hide a server bug);
with no charset, try strict UTF-8 first and fall back to Windows-1252.

ex3.py:

  from email.message import Message

  def http_charset(content_type):
      m = Message()
      m["Content-Type"] = content_type
      return m.get_content_charset()

  def decode_response(content_type, body):
      cs = http_charset(content_type)
      if cs:
          return body.decode(cs), f"header says {cs}"
      try:
          return body.decode("utf-8"), "no charset, strict utf-8 worked"
      except UnicodeDecodeError:
          return body.decode("cp1252", errors="replace"), "no charset, utf-8 failed, used cp1252"

  body_utf8 = "caf\u00e9".encode("utf-8")
  body_1252 = "caf\u00e9".encode("cp1252")
  cases = [
      ("text/html; charset=utf-8", body_utf8),
      ('text/plain; charset="cp1252"', body_1252),
      ("text/html", body_utf8),
      ("text/html", body_1252),
      ("text/html; charset=ascii", body_utf8),
  ]
  for ct, body in cases:
      try:
          text, how = decode_response(ct, body)
          print(f"{ct:32} -> {text!r:8} {how}")
      except UnicodeDecodeError as e:
          print(f"{ct:32} -> error: {e.reason}")

Real output (Python 3.14):

  text/html; charset=utf-8         -> 'café'   header says utf-8
  text/plain; charset="cp1252"     -> 'café'   header says cp1252
  text/html                        -> 'café'   no charset, strict utf-8 worked
  text/html                        -> 'café'   no charset, utf-8 failed, used cp1252
  text/html; charset=ascii         -> error: ordinal not in range(128)

The last case fails: the header claims ASCII but the body holds UTF-8 bytes above 127. A wrong header that happens to name a single-byte encoding such as
Latin-1 would not fail at all; it would produce mojibake (the header example in this chapter). So a header is a claim you can check only by looking at the bytes,
and "no error" is not proof, as Text Encoding and Unicode 3 showed.