json1-9  Exercise 1: Whole File versus Line by Line, Measured
=============================================================

Run with Node 24.18.0 (stream-json 3.7.0, @msgpack/msgpack 3.1.3, cbor-x 1.6.6), Python 3.14.0 (ijson 3.5.1 with the yajl2_c backend, msgpack 1.2.3, cbor2 6.1.5, pymongo 4.18.3 for bson, psutil 7.2.2), jq 1.8.2 and gzip 1.13 on Windows 11, installed in a scratch folder. Times are single runs on one machine; compare the ratios, not the seconds. Memory is the peak working set sampled every 5 ms (so a very short spike could be missed).
Order: py gen.py 1000000; make the smaller files (commands below); bash bench.sh. The output pasted at the end is the run whose numbers appear in the chapter.

--- gen.py (1,000,000 records, fixed seed; writes data.json as an array and data.jsonl as JSON Lines, with newline=LF) ---
import json, random, sys
N = int(sys.argv[1]) if len(sys.argv) > 1 else 1_000_000
random.seed(9)
names = ["Ada", "Grace", "Linus", "Zoë", "Søren", "Kenji", "Amara", "Noor", "Ines", "Tomás"]
status = ["new", "paid", "shipped", "returned"]
def rec(i):
    return {"id": i, "name": random.choice(names) + str(random.randint(1, 999)),
            "status": random.choice(status), "amount": round(random.uniform(1, 500), 2),
            "tags": random.sample(["a", "b", "c", "d", "e"], random.randint(0, 3)),
            "ts": "2026-%02d-%02dT%02d:%02d:00Z" % (random.randint(1, 12), random.randint(1, 28), random.randint(0, 23), random.randint(0, 59))}
with open("data.jsonl", "w", encoding="utf-8", newline="\n") as fl, open("data.json", "w", encoding="utf-8", newline="\n") as fa:
    fa.write("[")
    for i in range(N):
        s = json.dumps(rec(i), ensure_ascii=False, separators=(",", ":"))
        fl.write(s + "\n")
        fa.write(("," if i else "") + "\n" + s)
    fa.write("\n]\n")

--- making the 100,000 and 300,000 record files ---
for n in 100000 300000; do
  head -n $n data.jsonl > s$n.jsonl
  { echo "["; head -n $n data.jsonl | sed '$!s/$/,/'; echo "]"; } > s$n.json
done
(Checked: both array files parse with Python's json.load.)

--- m.py (the harness: runs a command, samples peak memory, prints time, memory, exit status and the last output line) ---
# m.py LABEL CMD... : run CMD (stdout discarded to a file), sample peak working set every 5 ms, print one line
import subprocess, sys, time, psutil, os
label, cmd = sys.argv[1], sys.argv[2:]
out = open("m_out.txt", "wb")
t0 = time.perf_counter()
p = subprocess.Popen(cmd, stdout=out, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL)
ps = psutil.Process(p.pid); peak = 0
while p.poll() is None:
    try:
        tot = 0
        for q in [ps] + ps.children(recursive=True):
            try: tot += q.memory_info().peak_wset
            except psutil.Error: pass
        peak = max(peak, tot)
    except psutil.Error: pass
    time.sleep(0.005)
t = time.perf_counter() - t0
out.close()
last = open("m_out.txt", "rb").read().decode("utf-8", "replace").strip().splitlines()
print("%-34s %6.2f s  peak %7.0f MB  rc=%d  | %s" % (label, t, peak / 1048576, p.returncode, (last[-1] if last else "")[:70]))

--- t/whole.mjs ---
import fs from "node:fs";
const a = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
let n = 0, s = 0; for (const r of a) if (r.status === "paid") { n++; s += r.amount; }
console.log("paid", n, "sum", s.toFixed(2));

--- t/lines.mjs ---
import fs from "node:fs"; import readline from "node:readline";
let n = 0, s = 0;
const rl = readline.createInterface({ input: fs.createReadStream(process.argv[2], "utf8"), crlfDelay: Infinity });
for await (const line of rl) { if (!line) continue; const r = JSON.parse(line); if (r.status === "paid") { n++; s += r.amount; } }
console.log("paid", n, "sum", s.toFixed(2));

--- t/sj.mjs (stream-json 3.x) ---
import fs from "node:fs";
import { parser } from "stream-json";
import { streamArray } from "stream-json/streamers/stream-array.js";
import chain from "stream-chain";
let n = 0, s = 0;
const pipeline = chain([fs.createReadStream(process.argv[2]), parser(), streamArray()]);
pipeline.on("data", ({ value: r }) => { if (r.status === "paid") { n++; s += r.amount; } });
pipeline.on("end", () => console.log("paid", n, "sum", s.toFixed(2)));

--- t/whole.py ---
import json, sys
with open(sys.argv[1], encoding="utf-8") as f: a = json.load(f)
n = s = 0
for r in a:
    if r["status"] == "paid": n += 1; s += r["amount"]
print("paid", n, "sum", "%.2f" % s)

--- t/lines.py ---
import json, sys
n = s = 0
with open(sys.argv[1], encoding="utf-8") as f:
    for line in f:
        if not line.strip(): continue
        r = json.loads(line)
        if r["status"] == "paid": n += 1; s += r["amount"]
print("paid", n, "sum", "%.2f" % s)

--- t/ij.py ---
import ijson, sys
n = s = 0
with open(sys.argv[1], "rb") as f:
    for r in ijson.items(f, "item", use_float=True):
        if r["status"] == "paid": n += 1; s += r["amount"]
print("paid", n, "sum", "%.2f" % s, "backend", ijson.backend)

--- bench.sh ---
P=./.venv/Scripts/python.exe; J=/c/lwdj/json/c4/bin/jq.exe
M="$P m.py"
for n in 100000 300000; do
  $M "node whole  $n" node t/whole.mjs s$n.json
  $M "node lines  $n" node t/lines.mjs s$n.jsonl
  $M "py   whole  $n" $P t/whole.py s$n.json
  $M "py   lines  $n" $P t/lines.py s$n.jsonl
done
$M "node whole  1000000" node t/whole.mjs data.json
$M "node lines  1000000" node t/lines.mjs data.jsonl
$M "node stream-json 1000000" node t/sj.mjs data.json
$M "py   whole  1000000" $P t/whole.py data.json
$M "py   lines  1000000" $P t/lines.py data.jsonl
$M "py   ijson  1000000" $P t/ij.py data.json
$M "jq whole array" $J '[.[]|select(.status=="paid")]|length' data.json
$M "jq -s on JSON Lines" $J -s '[.[]|select(.status=="paid")]|length' data.jsonl
$M "jq -n reduce inputs" $J -n 'reduce inputs as $r (0; if $r.status=="paid" then .+1 else . end)' data.jsonl
$M "jq -c select (stream out)" $J -c 'select(.status=="paid")|.id' data.jsonl
$M "jq --stream + fromstream" $J -c --stream -n 'fromstream(1|truncate_stream(inputs))|select(.status=="paid")|.id' data.json
$M "node gz lines" node t/gzlines.mjs data.jsonl.gz
$M "py gz lines" $P t/gzlines.py data.jsonl.gz
$M "gzip -dc | jq reduce" "C:/Program Files/Git/bin/bash.exe" t/jqgz.sh
$M "cat | jq reduce" "C:/Program Files/Git/bin/bash.exe" t/jqcat.sh

Output of bash bench.sh (columns: label, time, peak memory, exit status, last line of the program's output):
node whole  100000                   0.36 s  peak      81 MB  rc=0  | paid 24981 sum 6247872.69
node lines  100000                   0.34 s  peak      50 MB  rc=0  | paid 24981 sum 6247872.69
py   whole  100000                   0.56 s  peak      87 MB  rc=0  | paid 24981 sum 6247872.69
py   lines  100000                   0.58 s  peak      17 MB  rc=0  | paid 24981 sum 6247872.69
node whole  300000                   0.60 s  peak     188 MB  rc=0  | paid 74969 sum 18821723.21
node lines  300000                   0.57 s  peak      50 MB  rc=0  | paid 74969 sum 18821723.21
py   whole  300000                   0.95 s  peak     224 MB  rc=0  | paid 74969 sum 18821723.21
py   lines  300000                   1.04 s  peak      17 MB  rc=0  | paid 74969 sum 18821723.21
node whole  1000000                  1.20 s  peak     461 MB  rc=0  | paid 250574 sum 62748185.57
node lines  1000000                  1.26 s  peak      57 MB  rc=0  | paid 250574 sum 62748185.57
node stream-json 1000000             7.84 s  peak     108 MB  rc=0  | paid 250574 sum 62748185.57
py   whole  1000000                  2.74 s  peak     704 MB  rc=0  | paid 250574 sum 62748185.57
py   lines  1000000                  2.59 s  peak      17 MB  rc=0  | paid 250574 sum 62748185.57
py   ijson  1000000                  2.16 s  peak      20 MB  rc=0  | paid 250574 sum 62748185.57 backend yajl2_c
jq whole array                       6.28 s  peak    1172 MB  rc=0  | 250574
jq -s on JSON Lines                  6.47 s  peak    1171 MB  rc=0  | 250574
jq -n reduce inputs                  4.44 s  peak       6 MB  rc=0  | 250574
jq -c select (stream out)            8.22 s  peak       6 MB  rc=0  | 999999
jq --stream + fromstream            45.90 s  peak       6 MB  rc=0  | 999999
node gz lines                        1.85 s  peak      66 MB  rc=0  | paid 250574 sum 62748185.57
py gz lines                          2.91 s  peak      18 MB  rc=0  | paid 250574 sum 62748185.57
gzip -dc | jq reduce                18.42 s  peak      40 MB  rc=0  | 250574
cat | jq reduce                     13.88 s  peak      33 MB  rc=0  | 250574

WHY THIS WORKS AS AN ANSWER
---------------------------
Every program printed the same answer (paid 250574, sum 62748185.57 at one million records; 24981 / 6247872.69 and 74969 / 18821723.21 at the smaller sizes, and the jq programs print the count 250574), so the programs did equal work and the timings compare like with like. Peak memory for the whole-file approach grows with the file
(Node 81, 188, 461 MB; Python 87, 224, 704 MB) while the line-by-line approach stays flat (Node 50, 50, 57 MB; Python 17, 17, 17 MB): that is the point of the exercise. jq loading the array or slurping the lines used about 1,170 MB; jq -n with inputs used 6 MB. stream-json is
slower than JSON.parse (7.8 s against 1.2 s) and ijson is as fast as the line-by-line reader. jq --stream took 45.9 s. The harness prints the exit status because a first stream-json run 'finished' in 0.35 s with 39 MB: that was a crash (a wrong module path), and without the exit status it looked like a win.
What the harness cannot tell you: memory a process used between two samples, the cost of the disk cache, anything about other machines, and how the numbers change at gigabytes (the 2 GB figures in the chapter are extrapolated and were not run). One run each; no warm-up control.