json1-5  Exercise 3: Formats Checked by Another Program, Modules and Version Differences
=======================================================================================

Run in Git Bash with jq 1.8.2 (the Chapter 4 install in C:/lwdj/json/c4/bin):  bash ex3.sh

--- items.json ---
[{"id": 1, "name": "Tea, hot", "note": "plain"}, {"id": 2, "name": "say \"hi\"", "note": "quote inside"}, {"id": 3, "name": "two\nlines", "note": "newline inside"}, {"id": 4, "name": null, "note": "null name"}, {"id": 5, "name": "café", "note": "accent"}]

--- ex3_check.py (Python 3.14.0: parses the CSV on stdin and compares with items.json) ---
import csv, io, json, sys
rows = json.load(open('items.json', encoding='utf-8'))
text = sys.stdin.read()
got = list(csv.reader(io.StringIO(text, newline='')))
header, body = got[0], got[1:]
print('header:', header)
ok = True
for r, g in zip(rows, body):
    want = [str(r['id']), '' if r['name'] is None else r['name'], r['note']]
    if g != want:
        ok = False; print('MISMATCH', want, g)
print('rows parsed:', len(body), '| all match the original data:', ok and len(body) == len(rows))

--- lib/shoplib.jq ---
# helpers for the shop data
def in_stock: select(.stock > 0);
def money: . * 100 | round / 100;
def with_vat($rate): .price * (1 + $rate) | money;

--- ex3.sh ---
# Exercise 3: output formats (checked by a second program), modules, and what changed between jq versions
export PATH="/c/lwdj/json/c4/bin:$PATH"
echo '--- 1. CSV with awkward fields, parsed back by Python csv and compared with the data'
jq -r '(["id","name","note"] | @csv), (.[] | [.id, .name, .note] | @csv)' items.json | tr -d '\r' | tee items.csv
echo '   -- Python reads that CSV back:'
py -X utf8 ex3_check.py < items.csv
echo '--- 1b. control: the same rows built by joining with commas (no quoting) must FAIL the check'
jq -r '(["id","name","note"] | join(",")), (.[] | [.id, .name, .note] | map(. // "") | map(tostring) | join(","))' items.json | tr -d '\r' | py -X utf8 ex3_check.py
echo '--- 2. @sh: quote a hostile string for bash, then run it'
s=$(jq -nr '"it'"'"'s a test; echo INJECTED" | @sh' | tr -d '\r'); echo "quoted: $s"; eval "printf '[%s]\n' $s"
echo '--- 2b. @uri and @urid, @base64 and @base64d round trips'
jq -nr '"a b&c=d/é?" | @uri' | tr -d '\r'
jq -nr '"a b&c=d/é?" | @uri | @urid' | tr -d '\r'
jq -nr '"héllo wörld" | @base64' | tr -d '\r'
jq -nr '"héllo wörld" | @base64 | @base64d' | tr -d '\r'
echo '--- 3. a module'
cat lib/shoplib.jq
jq -L lib -c 'import "shoplib" as s; [.products[] | s::in_stock | {name, vat: s::with_vat(0.2)}]' shop.json
jq -L lib -c 'include "shoplib"; [.products[] | in_stock | .sku]' shop.json
jq -c 'import "shoplib" as s; 1' shop.json 2>&1 | tr -d '\r' | head -1; echo "   exit status without -L: ${PIPESTATUS[0]}"
echo '--- 4. the jq 1.8.0 release note says this prints 8; what does 1.8.2 print?'
jq -nc 'reduce range(5) as $x (0; .+$x | select($x!=2))'
jq -nc '[foreach range(5) as $x (0; .+$x | select($x!=2); [$x,.])]'
echo '   (NEWS for 1.8.1 says: "Revert the change of reduce/foreach state variable in 1.8.0")'
echo '--- 5. features new in 1.7 and 1.8, each run'
jq -nc '{"a":1,"b":{"c":2,"d":3},"e":4} | pick(.a, .b.c, .x)'
jq -nc '"abAB" | [scan("ab"; "i")]'
jq -nc '[-5, 5, -0.5] | map(abs)'
jq -nc '5 | if . > 3 then "big" end'
jq -nc '{a:{b:[1,2]}} | .a.b.[0]'
jq -nc '"k" as $k | {$k, ($k): 1}'
jq -nc '1 | debug("value is \(.)") | . + 1' 2>&1 | tr -d '\r'
jq -nc '" hello " | [trim, ltrim, rtrim]'
jq -nc '"foobarfoo" | trimstr("foo")'
jq -nc '{"xs":[1,2,3]} | .sum = add(.xs[])'
jq -nc '[1,2,3,4,5] | [skip(2; .[])]'
jq -nc '["true","false"] | map(toboolean)'
jq -nr '"%6a%71" | @urid' | tr -d '\r'
jq -nc '"  12 " | tonumber' 2>&1 | tr -d '\r'
jq -nc '"  12 " | trim | tonumber'
jq -nc '"héllo" | index("l")'
jq -nc '"héllo" | utf8bytelength'
jq -nc '[limit(-1; 1, 2)]' 2>&1 | tr -d '\r'
jq -nc '1 | ltrimstr("a")' 2>&1 | tr -d '\r'
jq -nc '[1] | toarray' 2>&1 | tr -d '\r' | head -1

Output:
--- 1. CSV with awkward fields, parsed back by Python csv and compared with the data
"id","name","note"
1,"Tea, hot","plain"
2,"say ""hi""","quote inside"
3,"two
lines","newline inside"
4,,"null name"
5,"café","accent"
   -- Python reads that CSV back:
header: ['id', 'name', 'note']
rows parsed: 5 | all match the original data: True
--- 1b. control: the same rows built by joining with commas (no quoting) must FAIL the check
header: ['id', 'name', 'note']
MISMATCH ['1', 'Tea, hot', 'plain'] ['1', 'Tea', ' hot', 'plain']
MISMATCH ['3', 'two\nlines', 'newline inside'] ['3', 'two']
MISMATCH ['4', '', 'null name'] ['lines', 'newline inside']
MISMATCH ['5', 'café', 'accent'] ['4', '', 'null name']
rows parsed: 6 | all match the original data: False
--- 2. @sh: quote a hostile string for bash, then run it
quoted: 'it'\''s a test; echo INJECTED'
[it's a test; echo INJECTED]
--- 2b. @uri and @urid, @base64 and @base64d round trips
a%20b%26c%3Dd%2F%C3%A9%3F
a b&c=d/é?
aMOpbGxvIHfDtnJsZA==
héllo wörld
--- 3. a module
# helpers for the shop data
def in_stock: select(.stock > 0);
def money: . * 100 | round / 100;
def with_vat($rate): .price * (1 + $rate) | money;
[{"name":"Tea","vat":4.2},{"name":"Biscuits","vat":2.39},{"name":"Milk","vat":1.07}]
["A1","C3","D4"]
jq: error: module not found: shoplib
   exit status without -L: 3
--- 4. the jq 1.8.0 release note says this prints 8; what does 1.8.2 print?
7
[[0,0],[1,1],[3,3],[4,7]]
   (NEWS for 1.8.1 says: "Revert the change of reduce/foreach state variable in 1.8.0")
--- 5. features new in 1.7 and 1.8, each run
{"a":1,"b":{"c":2},"x":null}
["ab","AB"]
[5,5,0.5]
"big"
1
{"k":1}
["DEBUG:","value is 1"]
2
["hello","hello "," hello"]
"bar"
{"xs":[1,2,3],"sum":6}
[3,4,5]
[true,false]
jq
jq: error (at <unknown>): string ("  12 ") cannot be parsed as a number
12
2
6
jq: error (at <unknown>): limit doesn't support negative count
jq: error (at <unknown>): startswith() requires string inputs
jq: error: toarray/0 is not defined at <top-level>, line 1, column 7:

WHY THIS WORKS AS AN ANSWER
---------------------------
The CSV is verified by a different program (Python's csv module) that must reconstruct the original five records, including the comma, the doubled quote, the embedded newline, the null and the accented letter; the control (plain join with
commas) fails that check on four rows, so a pass is meaningful. The @sh test runs the quoted result through eval and shows the hostile text came out as data and nothing ran. The round trips for @uri/@urid and @base64/@base64d return the
original string. The module is used both ways and fails with exit status 3 when -L is missing. Section 4 is the version lesson: the jq 1.8.0 release notes say the reduce example prints 8, but 1.8.2 prints 7 because 1.8.1 reverted the change.
Section 5 runs each 1.7/1.8 feature listed in the chapter; toarray is not defined on 1.8.2. Not tested: jq 1.6, 1.7.x or 1.8.0 themselves, so what those versions did comes only from the release notes.