json1-9  Exercise 3: gzip, Binary Formats, and Pagination
==========================================================

Run with Node 24.18.0 (stream-json 3.7.0, @msgpack/msgpack 3.1.3, cbor-x 1.6.6), Python 3.14.0 (ijson 3.5.1 with the yajl2_c backend, msgpack 1.2.3, cbor2 6.1.5, pymongo 4.18.3 for bson, psutil 7.2.2), jq 1.8.2 and gzip 1.13 on Windows 11, installed in a scratch folder.

--- gz.py (three gzip levels; two members back to back) ---
import gzip, time
raw = open("data.jsonl", "rb").read()
print("raw bytes", format(len(raw), ","))
for lvl in (1, 6, 9):
    t = time.perf_counter(); g = gzip.compress(raw, lvl); tc = time.perf_counter() - t
    t = time.perf_counter(); back = gzip.decompress(g); td = time.perf_counter() - t
    print("level %d: %11s bytes (%.1f%% of raw)  compress %.2f s  decompress %.2f s  identical=%s" % (lvl, format(len(g), ","), 100 * len(g) / len(raw), tc, td, back == raw))
    if lvl == 6: open("data.jsonl.gz", "wb").write(g)
cut = raw.rfind(b"\n", 0, 1000000) + 1
a, b = raw[:cut], raw[cut:]
cat = gzip.compress(a) + gzip.compress(b)
print("two gzip members back to back decompress to the original:", gzip.decompress(cat) == raw)

Output:
raw bytes 107,918,860
level 1:  31,027,868 bytes (28.8% of raw)  compress 0.26 s  decompress 0.37 s  identical=True
level 6:  18,870,741 bytes (17.5% of raw)  compress 1.08 s  decompress 0.70 s  identical=True
level 9:  17,537,608 bytes (16.3% of raw)  compress 2.32 s  decompress 0.15 s  identical=True
two gzip members back to back decompress to the original: True

--- t/gzlines.mjs ---
import fs from "node:fs"; import zlib from "node:zlib"; import readline from "node:readline";
let n = 0, s = 0;
const rl = readline.createInterface({ input: fs.createReadStream(process.argv[2]).pipe(zlib.createGunzip()), crlfDelay: Infinity });
for await (const line of rl) { const r = JSON.parse(line); if (r.status === "paid") { n++; s += r.amount; } }
console.log("paid", n, "sum", s.toFixed(2));

--- t/gzlines.py ---
import gzip, json, sys
n = s = 0
with gzip.open(sys.argv[1], "rt", encoding="utf-8") as f:
    for line in f:
        r = json.loads(line)
        if r["status"] == "paid": n += 1; s += r["amount"]
print("paid", n, "sum", "%.2f" % s)

--- t/jqgz.sh ---
gzip -dc data.jsonl.gz | /c/lwdj/json/c4/bin/jq.exe -n 'reduce inputs as $r (0; if $r.status=="paid" then .+1 else . end)'

The three compressed-stream readers (and two jq pipelines) were timed with the Exercise 1 harness; their lines are at the end of the bench.txt output in Exercise 1 (node gz lines, py gz lines, gzip -dc | jq reduce, cat | jq reduce).

--- fmt.py (six encodings, gzip each, decode and compare with the original records) ---
import json, gzip, time, io, msgpack, cbor2, bson, os
recs = [json.loads(l) for l in open("data.jsonl", encoding="utf-8")]
print("records:", len(recs))
def timed(f):
    t = time.perf_counter(); r = f(); return r, time.perf_counter() - t
rows = []
def add(name, enc, dec):
    b, te = timed(enc)
    g, tg = timed(lambda: gzip.compress(b, 6))
    back, td = timed(lambda: dec(b))
    ok = back == recs
    rows.append((name, len(b), len(g), te, td, ok))
add("JSON array, compact", lambda: json.dumps(recs, ensure_ascii=False, separators=(",", ":")).encode(), lambda b: json.loads(b))
add("JSON array, indent=2", lambda: json.dumps(recs, ensure_ascii=False, indent=2).encode(), lambda b: json.loads(b))
add("JSON Lines, compact", lambda: ("\n".join(json.dumps(r, ensure_ascii=False, separators=(",", ":")) for r in recs) + "\n").encode(), lambda b: [json.loads(l) for l in b.decode().splitlines()])
add("MessagePack (array)", lambda: msgpack.packb(recs), lambda b: msgpack.unpackb(b))
add("CBOR (array)", lambda: cbor2.dumps(recs), lambda b: cbor2.loads(b))
add("BSON (1 doc per record, concatenated)", lambda: b"".join(bson.encode(r) for r in recs), lambda b: list(bson.decode_all(b)))
print("%-40s %12s %12s %8s %8s %s" % ("format", "bytes", "gzip -6", "enc s", "dec s", "round trip equal"))
for n, a, g, te, td, ok in rows:
    print("%-40s %12s %12s %8.2f %8.2f %s" % (n, format(a, ","), format(g, ","), te, td, ok))

Output:
records: 1000000
format                                          bytes      gzip -6    enc s    dec s round trip equal
JSON array, compact                       107,918,861   18,870,743     1.61     1.65 True
JSON array, indent=2                      164,171,274   20,464,364     1.80     1.83 True
JSON Lines, compact                       107,918,860   18,870,741     4.43     4.89 True
MessagePack (array)                        84,963,287   18,335,396     0.61     1.30 True
CBOR (array)                               84,963,391   18,334,512     2.38     2.19 True
BSON (1 doc per record, concatenated)     119,598,105   21,341,959     1.48     1.43 True

--- xl.py (Python writes 1,000 records in three formats) ---
import json, msgpack, cbor2
recs = [json.loads(l) for l in open("data.jsonl", encoding="utf-8")][:1000]
open("x.msgpack", "wb").write(msgpack.packb(recs)); open("x.cbor", "wb").write(cbor2.dumps(recs))
json.dump(recs, open("x.json", "w", encoding="utf-8", newline="\n"), ensure_ascii=False)
print("python wrote 1000 records as MessagePack, CBOR and JSON")

Output:
python wrote 1000 records as MessagePack, CBOR and JSON

--- xl.mjs (Node decodes the MessagePack and CBOR files) ---
import fs from "node:fs"; import { decode as mp } from "@msgpack/msgpack"; import { decode as cb } from "cbor-x";
const j = JSON.parse(fs.readFileSync("x.json", "utf8"));
const same = (a) => JSON.stringify(a) === JSON.stringify(j);
console.log("MessagePack decoded in Node equals the JSON:", same(mp(fs.readFileSync("x.msgpack"))));
const c = cb(fs.readFileSync("x.cbor")); console.log("CBOR decoded in Node equals the JSON:", same(c));
console.log("sample:", JSON.stringify(mp(fs.readFileSync("x.msgpack"))[1]));

Output:
MessagePack decoded in Node equals the JSON: true
CBOR decoded in Node equals the JSON: true
sample: {"id":1,"name":"Tomás83","status":"shipped","amount":277.58,"tags":[],"ts":"2026-12-13T05:45:00Z"}

--- srv9.mjs (a small server made for this exercise; the paths, parameter names and the shape of next are my choices) ---
import http from "node:http";
let items = []; let nextId = 1;
const reset = () => { items = []; nextId = 1; for (let i = 0; i < 1000; i++) items.unshift({ id: nextId++, title: "item " + (nextId - 1) }); }; // newest first
reset();
http.createServer((req, res) => {
  const u = new URL(req.url, "http://x"); const q = u.searchParams;
  const send = (code, body, type = "application/json") => { res.writeHead(code, { "Content-Type": type }); res.end(typeof body === "string" ? body : JSON.stringify(body)); };
  if (u.pathname === "/reset") { reset(); return send(200, { ok: true }); }
  if (u.pathname === "/insert") { for (let i = 0; i < +(q.get("n") || 1); i++) items.unshift({ id: nextId++, title: "item " + (nextId - 1) }); return send(200, { total: items.length }); }
  if (u.pathname === "/offset") { const l = +q.get("limit") || 100, o = +q.get("offset") || 0; return send(200, { items: items.slice(o, o + l) }); }
  if (u.pathname === "/cursor") {
    const l = +q.get("limit") || 100, after = q.get("after"); let start = 0;
    if (after !== null) start = items.findIndex((x) => x.id < +after);   // newest first: "after" means a smaller id
    if (start < 0) return send(200, { items: [], next: null });
    const page = items.slice(start, start + l);
    return send(200, { items: page, next: start + l < items.length ? page[page.length - 1].id : null });
  }
  if (u.pathname === "/stream") {
    res.writeHead(200, { "Content-Type": "application/x-ndjson" });
    let i = 0; const n = +q.get("n") || 5;
    const t = setInterval(() => { res.write(JSON.stringify({ seq: i, at: Date.now() }) + "\n"); if (++i >= n) { clearInterval(t); res.end(); } }, 200);
    return;
  }
  send(404, { error: "not found" });
}).listen(8769, "127.0.0.1", () => console.log("listening"));

--- pg.mjs ---
const B = "http://127.0.0.1:8769";
const get = async (p) => (await fetch(B + p)).json();
await get("/reset").catch(() => {});
async function offsetWalk(insertAfterPage) {
  await fetch(B + "/reset"); const seen = []; let off = 0, page = 0;
  while (true) {
    const { items } = await get(`/offset?limit=100&offset=${off}`); if (!items.length) break;
    seen.push(...items.map((x) => x.id)); off += 100; page++;
    if (page === insertAfterPage) await fetch(B + "/insert?n=5");
  }
  return seen;
}
async function cursorWalk(insertAfterPage) {
  await fetch(B + "/reset"); const seen = []; let after = null, page = 0;
  while (true) {
    const r = await get(`/cursor?limit=100` + (after === null ? "" : `&after=${after}`));
    seen.push(...r.items.map((x) => x.id)); page++;
    if (page === insertAfterPage) await fetch(B + "/insert?n=5");
    if (r.next === null) break; after = r.next;
  }
  return seen;
}
const rep = (name, ids) => { const set = new Set(ids); console.log(name.padEnd(34), "fetched", ids.length, "| distinct", set.size, "| duplicates", ids.length - set.size, "| all 1000 original ids present:", [...Array(1000).keys()].every((i) => set.has(i + 1))); };
rep("offset, nothing changes", await offsetWalk(0));
rep("offset, 5 inserted after page 1", await offsetWalk(1));
rep("cursor, nothing changes", await cursorWalk(0));
rep("cursor, 5 inserted after page 1", await cursorWalk(1));
// NDJSON streamed over HTTP: when does each line arrive?
const t0 = Date.now(); const resp = await fetch(B + "/stream?n=5");
console.log("response headers arrived after", Date.now() - t0, "ms; content-type:", resp.headers.get("content-type"));
const dec = new TextDecoder(); let buf = "";
for await (const chunk of resp.body) {
  buf += dec.decode(chunk, { stream: true }); let i;
  while ((i = buf.indexOf("\n")) >= 0) { const line = buf.slice(0, i); buf = buf.slice(i + 1); console.log("  line", JSON.parse(line).seq, "parsed at +" + (Date.now() - t0) + " ms"); }
}
console.log("leftover bytes after the last newline:", buf.length);

--- pg.py ---
import json, urllib.request
B = "http://127.0.0.1:8769"
get = lambda p: json.load(urllib.request.urlopen(B + p))
get("/reset")
n, after = 0, None
with open("pages.jsonl", "w", encoding="utf-8", newline="\n") as out:
    while True:
        r = get("/cursor?limit=250" + ("" if after is None else "&after=%d" % after))
        for it in r["items"]: out.write(json.dumps(it, separators=(",", ":")) + "\n"); n += 1
        if r["next"] is None: break
        after = r["next"]
print("cursor walk wrote", n, "lines to pages.jsonl; first:", open("pages.jsonl", encoding="utf-8").readline().strip())

Output of node pg.mjs (offset and cursor walks, then the NDJSON stream):
offset, nothing changes            fetched 1000 | distinct 1000 | duplicates 0 | all 1000 original ids present: true
offset, 5 inserted after page 1    fetched 1005 | distinct 1000 | duplicates 5 | all 1000 original ids present: true
cursor, nothing changes            fetched 1000 | distinct 1000 | duplicates 0 | all 1000 original ids present: true
cursor, 5 inserted after page 1    fetched 1000 | distinct 1000 | duplicates 0 | all 1000 original ids present: true
response headers arrived after 219 ms; content-type: application/x-ndjson
  line 0 parsed at +219 ms
  line 1 parsed at +423 ms
  line 2 parsed at +627 ms
  line 3 parsed at +828 ms
  line 4 parsed at +1034 ms
leftover bytes after the last newline: 0

Output of py pg.py (cursor walk with 250-item pages, written as JSON Lines):
cursor walk wrote 1000 lines to pages.jsonl; first: {"id":1000,"title":"item 1000"}

Output of curl (the raw responses):
{"items":[{"id":1000,"title":"item 1000"},{"id":999,"title":"item 999"}],"next":999}
{"items":[{"id":998,"title":"item 998"},{"id":997,"title":"item 997"}],"next":997}
HTTP/1.1 200 OK
Content-Type: application/x-ndjson
Date: Mon, 12 Oct 2026 17:35:22 GMT
Connection: keep-alive
Keep-Alive: timeout=5
Transfer-Encoding: chunked

{"seq":0,"at":1791826523133}

WHY THIS WORKS AS AN ANSWER
---------------------------
Compression: gzip level 6 gives 17.5% of the original and level 9 gives 16.3% for about twice the time; decompression stays near a tenth of a second; the concatenated-members check prints True. Formats: every row's round-trip check prints True, so the sizes are for equal data; after gzip the spread shrinks from 21% (raw MessagePack/CBOR against JSON) to under 3%, BSON is larger, and
indented JSON costs only 1.6 MB more than compact once compressed. The JSON Lines encode/decode times include my own line splitting and joining, so they say little about the format. Pagination: with 5 items inserted after page 1, offset paging returns 1,005 items with 5 duplicates and cursor paging returns 1,000 with none; the NDJSON lines are parsed at roughly 200 ms intervals,
which shows they were delivered as sent rather than at the end. Not tested: a delete during an offset walk, database cost of large offsets, integers beyond 64 bits, dates and byte strings in the binary formats, Brotli or zstd, bgzip, and any server other than the one in this exercise.