Files and the Web

Inside a program, text is a sequence of code points. The moment it is saved to a file, sent over a network or put in a web page, it becomes bytes, and the reader at the other end needs to know which encoding to use to get the text back. This chapter walks through the main places where that information lives, or fails to live: file defaults, newline conventions, HTTP headers, HTML declarations, JSON and CSV. Most real-world mojibake starts at one of these boundaries.

Files: Always Name the Encoding

A plain text file has no field saying how it is encoded (unless it starts with a BOM, see Text Encoding and Unicode 4). When Python opens a file in text mode without an encoding argument, it picks a default that depends on the platform. PEP 686 describes this default as platform-dependent, and says that UTF-8 mode, which makes it consistent, becomes the default starting with Python 3.15 (check the final release status). On this Windows machine the difference is visible when the same program runs with and without UTF-8 mode:

Each line shows the UTF-8 mode flag, the machine's locale encoding, and the encoding a text-mode open() would use. This output is specific to this machine (a Western European Windows install): by default open() uses Windows-1252, so a UTF-8 file written on Linux and read here without an encoding argument is silently misread as in Text Encoding and Unicode 3. With UTF-8 mode on, the locale is unchanged but open() uses UTF-8. PEP 686 notes this default mainly affects Windows users, because most Unix systems already use a UTF-8 locale. The fix needs no flag: always pass encoding.

The file holds ten bytes: "caf" is three, é is the two bytes c3 a9, then a space, then € as the three bytes e2 82 ac, then the line feed. Read as UTF-8 it round-trips; read as Windows-1252 it turns into mojibake; read as ASCII it fails, or with errors="replace" every non-ASCII byte becomes its own replacement character (five here for two characters: another reminder that bytes and characters differ).

Newlines Are Part of the Encoding Problem

Line endings differ too: LF (\n) on Unix-like systems and CR LF (\r\n) on Windows. Python's text mode translates between \n in your program and the platform's convention. The newline argument controls it, and it matters for byte-exact files:

With newline="" on reading, nothing is translated and the CR LF survives; by default, reading converts CR LF to \n. Python's csv module asks you to open files with newline="" so it can handle line endings itself (check the wording in the csv documentation).

HTTP: The Header Carries the Charset

An HTTP response describes its body with a Content-Type header, whose charset parameter names the encoding. MDN gives the example text/html; charset=utf-8 and says the value is case-insensitive but lowercase is preferred. MDN's page does not say what applies when the charset is missing; for web pages the HTML rules below fill the gap. Python can parse the parameter with the standard email package (the header syntax is shared; this is a convenience, not a full HTTP client):

The parser lowercases the value, strips quotes, and returns None when absent. The last line shows what a wrong header does: the same bytes, decoded as the header's Latin-1, give mojibake. The server's header is the claim; the bytes are the truth; mismatch is a bug on the server side.

JSON: Settled by the Standard

JSON removed the guesswork. RFC 8259 says JSON exchanged between systems outside a closed ecosystem "MUST be encoded using UTF-8", that implementations must not add a BOM to network JSON, and that parsers may ignore one. Inside a JSON string, a character outside the BMP can be written as an escaped surrogate pair (the RFC's example is the G clef, \uD834\uDD1E), which is why the surrogates of Text Encoding and Unicode 4 still matter. Python's json escapes non-ASCII by default and can be told not to:

The first line is pure ASCII, safe in any pipeline, with the emoji as a surrogate pair; the second keeps the characters and relies on the receiver reading UTF-8. The third line shows a parser joining the pair back into one character. The last line shows a CSV written with the utf-8-sig codec, which starts with the BOM bytes EF BB BF. That BOM is a workaround: some spreadsheet programs only recognise UTF-8 when it is present (check: this is common experience, not something I verified in a source). It conflicts with the JSON rule above, so decide per format.

HTML: Declarations and a 1024-Byte Window

A browser must decide the encoding of a page from several clues. The WHATWG HTML Standard lists them in order of priority:

1. a byte order mark;
2. an explicit choice by the user;
3. the transport layer, such as the HTTP charset;
4. a prescan of the first bytes for a <meta> declaration (the standard encourages limiting it to 1024 bytes, and says authoring rules restrict declarations to that window);
5. the parent document, a previous visit, autodetection (discouraged for network resources);
6. an implementation default, which depends on the user's locale; for all locales not listed, windows-1252.

The standard recommends UTF-8 where the author can choose, and describes the locale-based defaults as a compatibility measure for legacy content. Here is a deliberately simplified version of the first, third, fourth and last of those steps (it omits the user override and the other tentative clues, and a real prescan is stricter than this regular expression):

Read the output line by line. A page with a meta tag is read as UTF-8; an HTTP header overrides the meta tag (priority order); a BOM beats the header; a meta tag that starts after 1100 bytes of comment is never seen, so the page falls back to the default and breaks; and a page with no declaration at all falls back too. So the practical advice is to send the HTTP charset, and put <meta charset="utf-8"> as early as possible in the <head>, and save the file as UTF-8.

Exercises