Other Formats

Working with JSON

Chapter 10 ยท JSON and Its Neighbours

JSON is rarely the only choice. Configuration files are often YAML or TOML, data feeds are sometimes XML or CSV, services between companies use Protocol Buffers, and editors accept "JSON with comments". Each format fixes something JSON lacks and gives something up for it. This chapter does not argue from reputation: it feeds the same awkward inputs to real parsers and reports what came out, converts one document through each format and counts what was lost, and measures size and speed on 100,000 records. The most useful result is a list of places where a format quietly changes your data.

Every example here was run, except where marked
Python 3.14.0 with PyYAML 6.0.3 (with libyaml), ruamel.yaml 0.19.1, the standard-library tomllib with tomli_w 1.2.0, json5 0.17.3, xmltodict 1.0.4, the standard csv module and protobuf 7.36.2 (the upb backend) with grpcio-tools 1.84.0; Node 24.18.0 with yaml 2.9.1, js-yaml 5.4.3, smol-toml 1.9.1, json5 2.2.3, jsonc-parser 3.3.1 and fast-xml-parser 5.11.2. Windows 11, all in a scratch folder. Not run: the yq command-line tool, Excel's handling of CSV, TOML 1.1, YAML anchors and merge keys, YAML's unsafe loaders, XML namespaces, schemas and XPath, Protocol Buffers schema evolution, and other languages' parsers. A different version of a library may behave differently, especially for YAML, where the behaviour depends on which YAML version a library implements.

At a Glance

FormatCommentsnullTyped valuesBuilt for
JSONnoyesstring, number, boolean, null, array, objectdata exchange between programs
YAMLyesyes (null, ~, empty)depends on the loader's rules (see below)human-edited configuration
TOMLyesnostrings, integers, floats, booleans, dates and times, arrays, tableshuman-edited configuration
JSON5 / JSONCyesyesas JSON (JSON5 adds hex, Infinity, NaN)hand-edited JSON files
XMLyesno standard waynone: all textdocuments, and long-lived enterprise interfaces
CSVnononone: all textflat tables, spreadsheets
Protocol Buffersin the schema file onlyno (absent field)strict, from a schemacompact, fast, schema-checked messages

YAML: The Friendly Format with Sharp Edges

YAML uses indentation instead of braces, allows comments and is easy to read. The catch is that a YAML value with no quotes is guessed: the loader decides whether NO is a string, a boolean or something else. The guess depends on which version of the YAML specification the library follows (1.1 or 1.2) and on the library. I wrote one small YAML file full of unquoted values and loaded it with four loaders. A selection:

Unquoted valuePyYAML 6.0.3ruamel.yaml 0.19.1Node yaml 2.9.1yaml with version: "1.1"
norway: NOboolean falsestringstringboolean false
on_word: on, yes_word: yesboolean truestringstringboolean true
quoted_no: "NO"stringstringstringstring
version: 1.10float 1.1float 1.1number 1.1number 1.1
zip: 01234integer 668 (octal)integer 1234number 1234number 668
octal_old: 014integer 12integer 14number 14number 12
octal_new: 0o14stringinteger 12number 12string
sexagesimal: 22:22integer 1342stringstringnumber 1342
thousands: 1_000integer 1000integer 1000stringnumber 1000
exp: 1e3stringfloat 1000.0number 1000number 1000
date: 2026-10-12a date objecta date objectstringa Date object
tilde: ~, empty:nullnullnullnull

The first row is the famous Norway problem: a list of country codes containing NO turns Norway into false in a YAML 1.1 loader (PyYAML, and Node's yaml when told to use 1.1). The other rows are the same disease: 1.10 becomes 1.1 in every loader I ran, so a version number must be quoted; 01234 is a zip code in one loader and the number 668 in another; 22:22 is a time of day or the integer 1342. I did not find a case where quoting a value ("NO", "1.10") was read as anything but a string. When PyYAML writes a document it does the quoting for you: yaml.safe_dump wrote the string "1.10" as '1.10' and the string "2026-10-12" as '2026-10-12', and both came back as strings.

"YAML is a superset of JSON": Only Sometimes

The YAML 1.2 specification is designed so that a JSON document is also a YAML document. I fed JSON texts to the same loaders to see how true that is in practice:

JSON inputjson.loads / JSON.parsePyYAMLruamel.yamlNode yaml / js-yaml
ordinary object, escaped slash, u00e9 escapefinefinefinefine
a tab between tokensfineerror (ScannerError)finefine
a surrogate-pair escape for an emojione emojitwo lone surrogatestwo lone surrogatesone emoji
duplicate key {"a":1,"a":2}last wins: 2last wins: 2, silentlyerrorerror (both)
1E400infinitythe string "1E400"infinityyaml: infinity; js-yaml: the string "1E400"
1e31000.0the string "1e3"1000.01000

A YAML 1.2 loader reads ordinary JSON, but PyYAML (a 1.1 implementation) changed the meaning of 1e3 and broke a valid JSON number into a string, and mangled an emoji. So "just load my JSON with a YAML parser" is only safe with a loader that targets 1.2, and even then the duplicate-key behaviour differs from a JSON parser. (The Node JSON.parse line for 1E400 printed null because I displayed it through JSON.stringify, which prints infinity as null.) One more difference from JSON: a plain value containing a colon and a space, colon_str: a: b, is an error in the loaders I tried (PyYAML: "mapping values are not allowed here"; Node yaml: "Nested mappings are not allowed in compact mappings"), so it must be quoted.

Rules that follow from the results
Quote every string that is not obviously a word. Use a YAML 1.2 loader if you have a choice, and know which one you have. Prefer safe_load-style functions (I ran only the safe loaders; the unsafe ones are not covered here). Never use unquoted version numbers, zip codes, country codes, phone numbers or times.

TOML: Configuration Without Guessing

TOML is aimed at configuration files. Values have explicit types (a quoted string is a string, 2026-10-12 is a date) so there is much less guessing. I parsed a small TOML file in Python and Node:

# a comment title = "demo" when = 2026-10-12T08:30:00Z day = 2026-10-12 mixed = [1, "two", 3.0] hex = 0xFF big = 1_000 [owner] name = "Ada"
  • Types: when came back as a Python datetime with a UTC offset, day as a date, hex as 255 and big as 1000. In Node, smol-toml returned TomlDate objects (a subclass of Date) for both.
  • Dates are not JSON: json.dumps on the parsed data raised "Object of type datetime is not JSON serializable". The usual fix, default=str, wrote "2026-10-12 08:30:00+00:00" with a space instead of a T, which is not the ISO form most JSON consumers expect. Convert dates with .isoformat() deliberately.
  • No null: n = null is an error in both libraries, and tomli_w refused to write a Python None ("Object of type 'NoneType' is not TOML serializable"). A JSON document with null values cannot go to TOML without deciding what an absent value means.
  • Mistakes are caught: a duplicate key (a = 1 then a = 2) and redefining a as a table both raised TOMLDecodeError.
  • nan is allowed: x = nan parsed to a NaN float in Python, so a TOML file can contain a value JSON cannot represent.

JSON5 and JSONC: JSON You Can Comment

JSON5 is a superset of JSON that allows comments, trailing commas, single-quoted and unquoted keys, hex numbers, +1, .5, Infinity and NaN. JSONC ("JSON with Comments") is just JSON plus comments and, depending on the tool, trailing commas. I ran thirteen inputs through JSON.parse, JSON5 2.2.3 and Microsoft's jsonc-parser 3.3.1:

InputJSON.parseJSON5 2.2.3jsonc-parser
line comment, block commenterrorokok
trailing commaerrorokerror by default, ok with allowTrailingComma
single quotes, unquoted key, hex 0x1F, +1, .5, Infinity, NaN, multi-line stringerrorok (hex gives 31)errors
undefinederrorerrorerrors
ordinary JSONokokok

The Python json5 package agreed on the cases I tried (comments, trailing comma, unquoted key, Infinity). Two cautions. Comments do not survive: parsing a commented file and writing it back with JSON5.stringify produced a file with the comment gone, and with unquoted keys and single quotes ({ port: 8080, tags: [ 'a', ], }), which is not valid JSON. And a JSON5 or JSONC file is not JSON: JSON.parse, jq and most servers reject it, so strip or convert it before handing it to anything that expects JSON. (I did not test which tools or editors accept comments in which file names; check the documentation of the tool that reads your file.)

XML: Everything Is Text

XML has attributes, mixed content, namespaces and schemas, and it is still the format of many long-lived interfaces. For data, its limit is that there are no number, boolean or list types: everything is text, and a list is only "an element that repeats". Converting a document to XML and back with xmltodict lost types and one list:

FieldBeforeAfter JSON, XML, JSON
debugboolean falsethe string "false"
port, ratio, each server's portnumbersstrings
tags["web"] (a list of one)the string "web": a one-item list is indistinguishable from a single value
owner.emailnullnull again here, but an empty string and null both became None: they cannot be told apart

That is 6 differences in a 9-field document. The Node library fast-xml-parser showed the same list ambiguity (two <s> elements gave an array, one gave a plain string) and, by default, turned the text 1.10 into the number 1.1 and 01234 into 1234, the YAML problem again. A node with both an attribute and text came out as {"@id": "7", "#text": "hi"} in xmltodict; the names @id and #text are that library's convention, not a standard. Any JSON-to-XML mapping is a set of choices, so two tools will give two different answers.

CSV: Flat and Untyped

CSV is a table of text. It is the right answer for flat data that goes to a spreadsheet. I wrote one record with awkward values using Python's csv module and read it back:

FieldWrittenRead back
id 1, price 19.99numbersstrings '1' and '19.99'
flag truebooleanthe string 'True' (Python's spelling, not JSON's true)
tags a lista listthe string "['a', 'b']" (Python's repr, not JSON)
n null, empty ""differentboth the empty string
zip "01234", a note with a comma and quotes, a value with a newlinestringsidentical (the module quoted them)

The module handled commas, quotes and embedded newlines correctly by quoting, and kept the leading zero, because it never converts anything. The damage is in types, nulls and nesting: you must write the schema yourself. In my size test below the nested tags list went into one cell as compact JSON, and a reader had to convert each column back. (What a spreadsheet does to 01234, long numbers or dates is a separate problem and I did not test it.)

Protocol Buffers: A Schema and Bytes

Protocol Buffers ("protobuf") describes messages in a .proto schema file and encodes them as compact binary; the schema is needed to read the bytes. (Protobuf in a real service is the subject of the course gRPC.) I used this schema:

syntax = "proto3"; message Record { int64 id = 1; string name = 2; string status = 3; double amount = 4; repeated string tags = 5; string ts = 6; }

For the first record, the JSON line is 100 bytes and the protobuf message is 52 bytes. Three things I observed that do not show in a size table:

  • Default values vanish. The first record has id 0. In proto3 a zero is the default and is not stored: it is absent from the bytes, and the library's own JSON conversion (MessageToJson) left "id" out entirely. A reader cannot tell "zero" from "not provided".
  • int64 is a string in protobuf's JSON mapping. MessageToJson wrote the id 253+1 as "9007199254740993" (a string), so JavaScript does not lose the digits (Chapter 1) but your JSON schema must expect a string.
  • The bytes are unreadable without the schema, and you must keep schema versions in step between sender and receiver. I did not test schema evolution.

Round Trip: One Document Through Each Format

I took one 9-field configuration document (including a string "1.10", a boolean, a number, a one-item list, a nested object, a list of objects and a null), wrote it with each format's own library, read it back and compared it with the original:

Format (library)Result
YAML (PyYAML dump, then load)identical (the dumper quoted '1.10' and the date string)
TOML (tomli_w, tomllib)failed on the null; identical once the null field was removed
XML (xmltodict)6 differences: booleans and numbers became strings, a one-item list became a string

The lesson is the same in each row: a format round-trips what it can represent. If your data has null, TOML cannot hold it; if it has numbers and lists, XML cannot tell you which is which unless something else (a schema) does.

Size and Speed on 100,000 Records

The first 100,000 records of the Chapter 9 data set (about 10.7 MB as JSON) were written and read back in each format, with each result checked against the original (the same records came back for every row except plain XML, where all values are strings). Single run, one machine:

FormatBytesAfter gzip -6WriteRead
JSON array10,690,6401,883,1720.11 s0.12 s
JSON Lines10,690,6391,883,1690.28 s0.25 s
YAML (PyYAML, pure Python)10,840,5491,901,69914.43 s28.04 s
YAML (PyYAML with libyaml)10,840,5491,901,6994.19 s13.95 s
TOML12,690,1751,937,6661.04 s2.28 s
CSV (tags as a JSON cell)6,140,4431,684,1280.28 s0.31 s
XML14,664,8732,001,2541.55 s1.27 s with xmltodict; 0.18 s just building the element tree with ElementTree
Protocol Buffers (length-prefixed)6,041,8041,859,1220.16 s0.21 s

What the numbers say. YAML was about 230 times slower to read than JSON in the pure-Python loader and about 115 times slower with libyaml, for almost the same number of bytes: fine for a config file, wrong for a data feed. After gzip, every format landed between 1.68 MB (CSV) and 2.00 MB (XML), within about 11% of JSON's 1.88 MB, so size is rarely a reason to leave JSON if the transfer is compressed. CSV and Protobuf are smaller raw because they do not repeat the field names on every record. The speed ranking is partly a statement about these libraries (the Python JSON parser is written in C; TOML's is pure Python), not only the formats, and the CSV and Protobuf rows include my own conversion code. The XML "Read" column mixes two different jobs, as labelled.

Choosing and When JSON Is the Wrong Choice

You needReach for
Data between programs, web APIs, browser codeJSON
A configuration file people edit by hand, with commentsTOML (flat, typed) or YAML (nested, but quote every string and know your loader); JSONC if your tooling already reads it
Hand-edited JSON in a tool that supports itJSONC or JSON5, converted to JSON before use elsewhere
A table for a spreadsheetCSV, with the types written down somewhere
A document with mixed text and markup, or an existing XML interfaceXML
High volume, strict schema, many languages, small messagesProtocol Buffers (see gRPC), or MessagePack/CBOR (Chapter 9) if you have no schema
Exact money, 64-bit ids, dates, binary dataJSON works if you represent them as strings (Chapters 1 and 2); a typed format helps if you cannot

JSON is the wrong choice when you need comments or a human-edited file with no ceremony, when data is naturally a flat table, when you need typed binary data at high volume, or when the other end only speaks something else. It is not the wrong choice just because it is plain: after gzip it was within about 11% of the others on size, and it was the fastest to read in these tests.

Mistakes while preparing this chapter
1. I wrote a YAML test value, colon_str: a: b, and "fixed" it before running it. I had assumed it would be a parse error and quoted it without checking. Running it afterwards confirmed the error in both loaders (it is now a row of evidence in the chapter). How to avoid it: when you think a line is invalid, run it first; a fix applied to an unverified guess is still a guess.

2. My first Node YAML test crashed at import. I wrote import jsyaml from "js-yaml", but version 5 has no default export. How to avoid it: after a major-version install, check how the package exports its functions before writing a test around it.

3. A column of my YAML table said null for 1E400, and it was wrong. The Node yaml loader returned infinity; my display went through JSON.stringify, which prints infinity as null. How to avoid it: do not judge a value through a function that changes it; print the type and the raw value.

4. I expected the first record's id to appear in protobuf's JSON output. The value was 0, so it was omitted as a default. How to avoid it: when you compare formats, include values that are zero, empty or false; they are where the formats disagree most.

Exercises

Exercise 1

Write a YAML file of at least twenty unquoted values (country codes, yes/on, 1.10, 01234, octal and hex forms, 22:22, 1_000, 1e3, dates, ~, an empty value, .inf and .nan). Load it with PyYAML, ruamel.yaml, the Node yaml package (default and version: "1.1") and js-yaml, and print a table of the type and value each gives. Then feed ten JSON texts (including a tab between tokens, a surrogate-pair escape, a duplicate key, 1E400 and 1e3) to json.loads/JSON.parse and the YAML loaders, and report where "YAML is a superset of JSON" fails.

๐Ÿ“„ View solution
Exercise 2

Parse a TOML file with a datetime, a date, a mixed-type array, hex and underscored numbers in Python and Node, and show what json.dumps does with them (with and without default=str). Try a null, a duplicate key and nan in TOML. Run thirteen inputs through JSON.parse, JSON5 and jsonc-parser (with and without allowTrailingComma) and show what a JSON5 round trip does to comments. Show the CSV and XML losses described in the chapter (a one-item list, the empty string versus null, 1.10 and 01234 in fast-xml-parser).

๐Ÿ“„ View solution
Exercise 3

Take a nine-field configuration document with a null, a one-item list and a string "1.10"; write and re-read it as YAML, TOML and XML and list every difference. Then write and read 100,000 records as JSON, JSON Lines, YAML (pure Python and libyaml), TOML, CSV, XML and Protocol Buffers (write the .proto file, compile it with grpc_tools.protoc), checking each round trip, and report bytes, gzip size, write time and read time. Show the first record as a JSON line and as protobuf bytes in hex, and what protobuf's JSON mapping does with a zero id and an id of 253+1.

๐Ÿ“„ View solution

Chapter 10 Quick Reference

  • YAML guesses types: NO is false in a 1.1 loader (PyYAML), 1.10 became 1.1 in every loader, 01234 was 668 or 1234, 22:22 was 1342 or a string; quote strings and know whether your loader is YAML 1.1 or 1.2
  • "YAML is a JSON superset" held for 1.2 loaders only on my cases: PyYAML errored on a tab, mangled a surrogate pair, and read 1e3 as a string; duplicate keys error in ruamel, yaml and js-yaml but not PyYAML or json.loads; a: b as a plain value is an error
  • TOML: comments, typed dates, no null, duplicate keys rejected, nan allowed; parsed dates are not JSON-serialisable (default=str gives a space, not T)
  • JSON5 (comments, trailing commas, hex, NaN) and JSONC (comments) are not JSON; JSON5.stringify drops comments and writes unquoted keys
  • XML and CSV have no types: booleans, numbers and one-item lists were lost (6 differences in a 9-field document); empty string and null merge; fast-xml-parser turned 1.10 into 1.1 and 01234 into 1234
  • Protobuf: schema needed; 52 bytes against 100 for the same record; zero values are omitted; int64 is a string in its JSON mapping
  • 100,000 records: YAML read 28 s (pure Python) or 14 s (libyaml) against 0.12 s for JSON; after gzip -6 every format was within about 11% of JSON's 1.88 MB (CSV 1.68, XML 2.00)
Coming next
Chapter 11 is JSON security and robustness: untrusted input, depth and size limits, duplicate keys and how different parsers disagree, prototype pollution in JavaScript, eval and other unsafe loading, injection when JSON is built by string concatenation, JSON in HTML, and what a safe API boundary looks like. Start with Working with JSON 11.