json1-9  Exercise 2: JSON Lines Readers, jq Behaviour, and Constant-Memory Conversion
=====================================================================================

Run with Node 24.18.0 (stream-json 3.7.0, @msgpack/msgpack 3.1.3, cbor-x 1.6.6), Python 3.14.0 (ijson 3.5.1 with the yajl2_c backend, msgpack 1.2.3, cbor2 6.1.5, pymongo 4.18.3 for bson, psutil 7.2.2), jq 1.8.2 and gzip 1.13 on Windows 11, installed in a scratch folder. The two reader scripts contain the character U+2028 as an escape sequence in their source.

--- pit.py (Python: seven bad-input cases, then the U+2028 trap) ---
import json, io
def show(label, text):
    good = bad = 0; errs = []
    for no, line in enumerate(io.StringIO(text, newline=None), 1):
        if not line.strip(): continue
        try: json.loads(line); good += 1
        except json.JSONDecodeError as e: bad += 1; errs.append("line %d: %s" % (no, e.msg))
    print("%-34s good=%d bad=%d %s" % (label, good, bad, errs))
show("clean", '{"a":1}\n{"a":2}\n')
show("no final newline", '{"a":1}\n{"a":2}')
show("blank line + CRLF", '{"a":1}\r\n\r\n{"a":2}\r\n')
show("pretty-printed (invalid as lines)", '{\n  "a": 1\n}\n')
show("BOM at start", '\ufeff{"a":1}\n{"a":2}\n')
show("truncated last line", '{"a":1}\n{"a":2}\n{"a":')
show("one bad line in the middle", '{"a":1}\nnot json\n{"a":3}\n')
s = json.dumps({"t": "line1\nline2"})
print("embedded newline is escaped:", s, "| lines in it:", len(s.splitlines()))
u = '{"t":"a\u2028b"}\n{"t":"c"}\n'
print("U+2028 inside a string, 2 records:")
print("  str.splitlines() ->", len(u.splitlines()), "pieces; json.loads on each:", end=" ")
r = []
for p in u.splitlines():
    try: json.loads(p); r.append("ok")
    except json.JSONDecodeError: r.append("ERR")
print(r)
print("  iterating the file object -> ", len(list(io.StringIO(u, newline=None))), "lines")
print("  split on LF ->", len(u.split("\n")) - 1, "records")

Output:
clean                              good=2 bad=0 []
no final newline                   good=2 bad=0 []
blank line + CRLF                  good=2 bad=0 []
pretty-printed (invalid as lines)  good=0 bad=3 ['line 1: Expecting property name enclosed in double quotes', 'line 2: Extra data', 'line 3: Expecting value']
BOM at start                       good=1 bad=1 ['line 1: Unexpected UTF-8 BOM (decode using utf-8-sig)']
truncated last line                good=2 bad=1 ['line 3: Expecting value']
one bad line in the middle         good=2 bad=1 ['line 2: Expecting value']
embedded newline is escaped: {"t": "line1\nline2"} | lines in it: 1
U+2028 inside a string, 2 records:
  str.splitlines() -> 3 pieces; json.loads on each: ['ERR', 'ERR', 'ok']
  iterating the file object ->  2 lines
  split on LF -> 2 records

--- pit.mjs (Node) ---
import readline from "node:readline"; import { Readable } from "node:stream";
async function lines(text) { const out = []; const rl = readline.createInterface({ input: Readable.from([text]), crlfDelay: Infinity }); for await (const l of rl) out.push(l); return out; }
const u = '{"t":"a\u2028b"}\n{"t":"c"}\n';
const got = await lines(u);
console.log("readline lines for the U+2028 text:", got.length);
console.log("split on LF pieces:", u.split("\n").filter(Boolean).length);
const bom = await lines('\ufeff{"a":1}\n');
try { JSON.parse(bom[0]); console.log("BOM line parses"); } catch (e) { console.log("BOM line: JSON.parse ->", e.message); }
const crlf = await lines('{"a":1}\r\n{"a":2}\r\n'); console.log("CRLF lines:", crlf.length, JSON.stringify(crlf));
const lone = await lines('{"a":1}\r{"a":2}\n'); console.log("lone CR splits:", lone.length);

Output:
readline lines for the U+2028 text: 3
split on LF pieces: 2
BOM line: JSON.parse -> Unexpected token '<BOM>', "<BOM>{"a":1}" is not valid JSON
CRLF lines: 2 ["{\"a\":1}","{\"a\":2}"]
lone CR splits: 2

--- jqtests.sh (CRLF with and without -b, --seq, truncation, conversions, --stream) ---
J=/c/lwdj/json/c4/bin/jq.exe
printf '{"a":1}\n{"a":2}\n' > tiny.jsonl
echo "== 1. jq -c redirected to a file (od -c shows the bytes):"; $J -c . tiny.jsonl > o1.jsonl; od -c o1.jsonl
echo "== 2. jq -b -c:"; $J -b -c . tiny.jsonl > o2.jsonl; od -c o2.jsonl
echo "== 3. --seq output (-b so only the format shows):"; $J -n -b --seq -c '{a:1},{a:2}' | od -c
echo "== 4. plain JSON Lines fed to --seq:"; $J -c --seq .a tiny.jsonl; echo "exit status $?"
echo "== 5. truncated last line:"; printf '{"a":1}\n{"a":2}\n{"a":3' > trunc.jsonl; $J -c .a trunc.jsonl; echo "exit status $?"
echo "== 6. array to lines, lines to array, input_line_number:"
echo '[{"a":1},{"a":2}]' | $J -c '.[]'; $J -s -c . tiny.jsonl; $J -n -c '[inputs]' tiny.jsonl; $J -c '[input_line_number,.a]' tiny.jsonl
echo "== 7. --stream events for a small document:"; echo '{"a":[1,{"b":2}],"c":"x"}' | $J -c --stream .

Output:
== 1. jq -c redirected to a file (od -c shows the bytes):
0000000   {   "   a   "   :   1   }  \r  \n   {   "   a   "   :   2   }
0000020  \r  \n
0000022
== 2. jq -b -c:
0000000   {   "   a   "   :   1   }  \n   {   "   a   "   :   2   }  \n
0000020
== 3. --seq output (-b so only the format shows):
0000000 036   {   "   a   "   :   1   }  \n 036   {   "   a   "   :   2
0000020   }  \n
0000022
== 4. plain JSON Lines fed to --seq:
jq: ignoring parse error: Unfinished abandoned text at EOF at line 3, column 0
exit status 0
== 5. truncated last line:
jq: parse error: Unfinished JSON term at EOF at line 3, column 6
1
2
exit status 5
== 6. array to lines, lines to array, input_line_number:
{"a":1}
{"a":2}
[{"a":1},{"a":2}]
[{"a":1},{"a":2}]
[1,1]
[2,2]
== 7. --stream events for a small document:
[["a",0],1]
[["a",1,"b"],2]
[["a",1,"b"]]
[["a",1]]
[["c"],"x"]
[["c"]]

--- conv.py (stream the million-record array into JSON Lines) ---
import ijson, json
n = 0
with open("data.json", "rb") as f, open("conv.jsonl", "w", encoding="utf-8", newline="\n") as out:
    for r in ijson.items(f, "item", use_float=True):
        out.write(json.dumps(r, ensure_ascii=False, separators=(",", ":")) + "\n"); n += 1
print("converted", n, "records")

Measured with the harness from Exercise 1 (py m.py ...):
array -> jsonl via ijson             6.22 s  peak      20 MB  rc=0  | converted 1000000 records

--- cmpf.py (checksum of the converted file against the generated one) ---
import hashlib
h = lambda p: hashlib.sha256(open(p, "rb").read()).hexdigest()[:16]
print("sha256 data.jsonl", h("data.jsonl"), "| conv.jsonl", h("conv.jsonl"), "| same:", h("data.jsonl") == h("conv.jsonl"))

Output:
sha256 data.jsonl a02440d574dcdc84 | conv.jsonl a02440d574dcdc84 | same: True

WHY THIS WORKS AS AN ANSWER
---------------------------
The reader cases are the ones real files produce. Valid inputs (no final newline, CRLF with a blank line) parse; invalid ones fail on the right line with a reason (pretty-printed, BOM, truncated, garbage in the middle), and the good lines around a bad one still parse, so a reader can count and skip. The U+2028 case is the dangerous one:
the file holds two records, str.splitlines() makes three pieces (two parse errors), and Node's readline makes three lines, while a split on LF or iterating a Python file object gives two. The fix is to split on LF only. The jq output shows the bytes: CRLF after each line without -b and LF with it; --seq writes 0x1E before each value and LF after;
a plain JSON Lines file fed to --seq produces only a warning and exits 0; a truncated last line prints the earlier values, then a parse error, exit status 5. The conversion used 20 MB at one million records, and the checksum equality shows the streamed result is identical to the file written directly. Not tested: other Unicode line separators, other Node versions, and the same jq commands in PowerShell or cmd.