Lines and Scale

A JSON document is one value. To read the first record of a file that is one huge array, an ordinary parser must read, and build in memory, the whole file, because it cannot know the array is valid until it reaches the final ]. That is fine for a 10 KB API response and a real problem for a database export. This chapter measures how bad it gets, then shows the fixes in the order you should reach for them: JSON Lines (one record per line), streaming parsers for arrays you cannot change, gzip, the binary cousins (MessagePack, CBOR, BSON), and, on the API side, pagination. The numbers come from one generated data set and one Windows machine; treat the ratios as the lesson and the absolute figures as examples.

The Problem, Measured

The test is the simplest realistic job: count the records whose status is "paid" and add up their amount. Each record looks like this (one line of the data set):

The same records were written two ways: as one array (data.json, one record per line inside [ ... ]) and as JSON Lines (data.jsonl). Every program printed the same answer, 250,574 paid records with a total of 62,748,185.57, so the programs did equal work. Peak memory for the "load everything" approach, at three sizes:

Memory grows with the file. At a million records Node used about 4 times the file size, Python about 6.5 times, and jq loading the same array used 1,172 MB, about 11 times (table below). If that ratio held, which I did not test, a 2 GB file would want somewhere around 9 GB in Node and 13 GB in Python. The same job reading one line at a time stayed at 50-57 MB in Node and 17 MB in Python at every size: memory depends on the size of one record, not of the file.

JSON Lines: One Value per Line

JSON Lines (also called NDJSON, newline-delimited JSON) puts one complete JSON value on each line, separated by \n. Because every line is valid JSON by itself, a program can read a line, process it, throw it away, and move on; a writer can append a line without touching the rest of the file; head, tail, wc -l and split work on records; and a damaged line costs you one record, not the file.

In practice JSON Lines and NDJSON are the same thing apart from how they treat blank lines and the final newline, and the same file satisfies both if it has no blank lines and ends with \n. The RFC 7464 form is different: the record separator makes it safe to resume reading after damage, which a plain newline cannot promise, and jq --seq speaks it. Newlines inside a string value are never a problem: JSON.stringify and json.dumps write them as \n (two characters), so a compactly written record cannot contain a raw line break. Pretty-printed JSON, written with indent, is not JSON Lines.

Reading JSON Lines in Python and Node

Both were measured with the million-record file: Node 1.3 s and 57 MB, Python 2.6 s and 17 MB. Both programs are short, but both hide decisions that real files test. I ran each case below:

The last row is the one that surprises people. JSON allows the raw character U+2028 (LINE SEPARATOR) inside a string, and some line-splitting helpers treat it as a line break, cutting a valid record in two. Python's for line in f and a split on \n are safe; splitlines() is not. In Node I measured readline splitting on it, so for data that may contain U+2028 (text typed by users, for example) split on \n yourself. I tested one version of each and one character; other separators in the Unicode line-break set were not tried.

jq and JSON Lines

jq already treats its input as a stream of values (Chapter 4), so a JSON Lines file is natural input. What matters is how you ask it to read. Measured on the million-record file, for the same "paid" job:

-s throws away the benefit of JSON Lines: it builds the array in memory exactly as if the file had been one. Use -n with inputs (or reduce/foreach over inputs) when you need totals, and plain jq -c when each line maps to an output line. --stream turns a large array into path/leaf events, so memory stays small, but at 46 seconds against 4-8 it is the slow way: my advice is to convert the array to JSON Lines once (next section) and use the fast forms. A few more jq facts I checked:

• Array to lines, and back: jq -c '.[]' and jq -s -c . (or jq -n -c '[inputs]'). Both need the whole array in memory on one side, so they suit small files.
• CRLF on Windows: jq -c wrote \r\n after each line when I redirected to a file; jq -b wrote \n. Parsers accept either, but file sizes, checksums and strict tools differ, so use -b when you write JSON Lines on Windows.
• A truncated last line: jq printed the earlier results, then reported "Unfinished JSON term at EOF at line 3, column 6" and exited with status 5.
• --seq changes input as well as output: it writes RS before and LF after each value, which is RFC 7464 output. But I fed it a plain JSON Lines file and it printed "ignoring parse error: Unfinished abandoned text" and no values, exit status 0: with --seq, input must be RS-framed too. A silent exit status 0 is the danger.
• Piping on this machine was much slower than a file argument: jq -n 'reduce inputs ...' data.jsonl took 4.4 s, while cat data.jsonl | jq ... in Git Bash took 13.9 s and gzip -dc data.jsonl.gz | jq ... took 18.4 s (the last two ran in Git Bash). I did not find the cause and did not test PowerShell, cmd or Linux; if it matters to you, time both forms on your machine.
• input_line_number reports the line a value came from, which is handy for error messages.

Streaming an Array You Cannot Change

Sometimes the file is a single array and somebody else made it. A streaming (event) parser reads tokens and hands you each array element as it completes, without building the whole document.

Two things to know. ijson returns numbers with a fractional part as Decimal by default (I printed the type: Decimal), exact but slower to add up and not mixable with floats; use_float=True gives ordinary floats, as I used. And streaming costs CPU: stream-json took 7.8 s against 1.2 s for plain JSON.parse on the same file, in exchange for 108 MB of memory instead of 461 MB. Streaming a file that fits comfortably in memory is not a win. Results on the million-record array:

Converting once is the practical answer. Streaming the array with ijson and writing one compact line per record took 6.0 s and 20 MB, and the resulting file was byte-for-byte the same (same SHA-256) as the JSON Lines file the data generator wrote directly. When you control the producer, write JSON Lines from the start.

Compression: gzip

JSON is repetitive text (the same keys on every record), and gzip is very good at that. Python's gzip module at three levels, on the 107.9 MB JSON Lines file, one run each:

Compression is cheap to undo and can be read as a stream: Node fs.createReadStream(...).pipe(zlib.createGunzip()) into readline took 1.9 s and 66 MB, and Python gzip.open(path, "rt", encoding="utf-8") took 2.9 s and 18 MB, both with the same answer as the uncompressed file. Two gzip members written one after the other decompress as a single stream (I checked), which is why .jsonl.gz files can be appended to. The price is that gzip is not seekable: to read the millionth line you decompress the first 999,999. (Formats that add an index, such as bgzip, exist; I did not run them.) For HTTP, the server and browser can compress for you with Content-Encoding (HTTP itself is covered in API Types & Design 2-4, which I did not re-check for this topic); you do not need to gzip a JSON response by hand.

Binary Cousins: MessagePack, CBOR, BSON

MessagePack, CBOR and BSON are binary formats with the same data model as JSON (plus extras such as byte strings and, in BSON, dates). I encoded the same million records with each, timed Python's encoder and decoder, compressed each with gzip level 6, and checked that every decoded result equals the original records:

Three lessons. Gzip does most of the work. Raw, MessagePack and CBOR are 21% smaller than compact JSON; after gzip, MessagePack is only 2.8% smaller than gzipped JSON, and BSON is larger than JSON both ways. Pretty-printing is almost free once compressed (164 MB raw but 20.5 MB gzipped), so do not minify for size if the transfer is compressed. Speed differs by library: the JSON Lines row includes splitting lines in my Python code, so it is not a fair judgment of the format. The binary formats are not human-readable, cannot be edited in a text editor or inspected with curl alone, and need a library on every side. BSON has no top-level array (a document is an object), which is why it is stored as one document per record. I also decoded the MessagePack and CBOR files, written by Python, in Node (@msgpack/msgpack and cbor-x) and the results equalled the JSON for 1,000 records. Not tested: integers beyond 64 bits, dates and byte strings (where these formats and JSON diverge), and encoders in languages other than Python and Node.

Pagination: The API-Side Answer

A server should not send a million records in one response either. Pagination returns a page at a time, and there are two common designs. Offset: ?limit=100&offset=200. Cursor: ?limit=100&after=<last id seen>, with the server returning a next value. I wrote a small server of 1,000 items, newest first, and walked it with a client, once with nothing changing and once with 5 new items inserted right after the client fetched the first page:

With offset paging, the insert pushed the last five items of page 1 onto page 2, so the client saw them twice. (The mirror problem, a delete making a client skip items, follows from the same arithmetic but I did not run it.) The cursor walk saw each original item once; the five new items were not returned because the cursor was already past them, which is usually what you want for an export and not what you want for a live feed. Cursor paging also avoids the server counting past a large offset, but I did not measure database cost: my server holds an array. The server's design (what next is called, whether the cursor is a bare id) is mine; real APIs choose their own and often use an opaque string.

A server can also stream JSON Lines in a single response. My /stream endpoint sent Content-Type: application/x-ndjson and wrote one line every 200 ms; a Node client reading fetch's body stream parsed the first line at 222 ms and the fifth at 1,031 ms, rather than waiting for the end. That is the main reason NDJSON turns up in log and event APIs. The client has to keep any partial line in a buffer until the next newline arrives (the leftover after the last line was 0 bytes), because a network chunk can end in the middle of a record.

Choosing

Exercises