json1-12  Exercise 3: Patch One Record Safely, Prove It Repeats, Write the README
=================================================================================

Run with Node 24.18.0 (Ajv 8.20.0, ajv-formats 3.0.1), Python 3.14.0 (jsonschema 4.26.0, jsonpatch 1.35, psutil 7.2.2), jq 1.8.2, curl 8.12.1 and sha256sum on Windows 11. All counts are from the single snapshot taken on 2026-10-12 (feed clock 19:05:41Z) and will differ for any later download. The patch changes a record for teaching; it is not a real seismic correction.

--- patch_good.json ---
[
  { "op": "test", "path": "/id", "value": "aka2026ufzwkj" },
  { "op": "test", "path": "/status", "value": "automatic" },
  { "op": "replace", "path": "/status", "value": "reviewed" },
  { "op": "replace", "path": "/mag", "value": 1.5 }
]

--- patch_wrong_precondition.json ---
[
  { "op": "test", "path": "/status", "value": "reviewed" },
  { "op": "replace", "path": "/mag", "value": 9.9 }
]

--- patch_extra_field.json ---
[
  { "op": "test", "path": "/status", "value": "automatic" },
  { "op": "add", "path": "/note", "value": "checked by hand" }
]

--- 06_patch.py ---
# Step 6: apply a JSON Patch (RFC 6902) to ONE record of quakes.jsonl, streaming, writing a new file.
# The patch is applied to a copy (jsonpatch's default does this), then the patched record is checked against record.schema.json
# before it is accepted. If either step fails the output file is not written at all.
import json, sys, os, hashlib
import jsonpatch
from jsonschema import Draft202012Validator

src, target_id, patch_file, out = sys.argv[1:5]
schema = json.load(open("record.schema.json", encoding="utf-8"))
validator = Draft202012Validator(schema)
patch = jsonpatch.JsonPatch(json.load(open(patch_file, encoding="utf-8")))

def sha(p): return hashlib.sha256(open(p, "rb").read()).hexdigest()[:16]
tmp = out + ".tmp"
matched = 0
try:
    with open(src, encoding="utf-8") as fin, open(tmp, "w", encoding="utf-8", newline="\n") as fout:
        for line in fin:
            rec = json.loads(line)
            if rec["id"] == target_id:
                matched += 1
                new = patch.apply(rec)                     # raises JsonPatchConflict / JsonPatchTestFailed; rec itself is left alone
                errs = [e.message for e in validator.iter_errors(new)]
                if errs: raise ValueError("patched record fails the schema: " + "; ".join(errs))
                line = json.dumps(new, separators=(",", ":"), ensure_ascii=False) + "\n"
            fout.write(line)
    if matched != 1: raise ValueError("expected exactly one record with id %s, found %d" % (target_id, matched))
    os.replace(tmp, out)
    print("OK   patch %s applied to %s -> %s" % (patch_file, target_id, out))
except Exception as e:
    if os.path.exists(tmp): os.remove(tmp)
    print("FAIL patch %s: %s: %s -> no output file written" % (patch_file, type(e).__name__, str(e)[:110]))

--- 06_patch.sh ---
set -e
PY=./.venv/Scripts/python.exe
J=/c/lwdj/json/c4/bin/jq.exe
ID=aka2026ufzwkj
echo "== before"; grep "\"id\":\"$ID\"" quakes.jsonl
echo "== 1. a good patch"
$PY -X utf8 06_patch.py quakes.jsonl $ID patch_good.json quakes_patched.jsonl
echo "== 2. a patch whose test precondition is false"
$PY -X utf8 06_patch.py quakes.jsonl $ID patch_wrong_precondition.json should_not_exist_1.jsonl || true
echo "== 3. a patch that applies but breaks the closed record schema (adds /note)"
$PY -X utf8 06_patch.py quakes.jsonl $ID patch_extra_field.json should_not_exist_2.jsonl || true
echo "== files written (only the good patch may have produced one)"
ls quakes_patched.jsonl should_not_exist_1.jsonl should_not_exist_2.jsonl 2>&1 | sed 's/^ls: cannot access/absent:/'
echo "== the patched record"; grep "\"id\":\"$ID\"" quakes_patched.jsonl
echo "== how many lines differ between the original and the patched file (must be 1)"
diff <(cat quakes.jsonl) <(cat quakes_patched.jsonl) | grep -c '^<' || true
echo "== line counts: original $(wc -l < quakes.jsonl), patched $(wc -l < quakes_patched.jsonl)"
echo "== status counts before and after (jq, streaming)"
for f in quakes.jsonl quakes_patched.jsonl; do printf '%-22s ' $f; $J -c -n 'reduce inputs as $r ({}; .[$r.status] += 1)' $f; done
echo "== the original file was not modified: sha256 $(sha256sum quakes.jsonl | cut -c1-16) (same as in step 3: 35fb9a34a36a7df8)"

Output:
== before
{"id":"aka2026ufzwkj","time":"2026-10-12T18:49:05Z","updated":"2026-10-12T18:50:09Z","mag":1.4,"mag_type":"ml","place":"10 km N of Susitna North, Alaska","region":"Alaska","lon":-149.852,"lat":62.256,"depth_km":14.2,"status":"automatic","tsunami":false,"sig":30,"net":"ak","url":"https://earthquake.usgs.gov/earthquakes/eventpage/aka2026ufzwkj"}
== 1. a good patch
OK   patch patch_good.json applied to aka2026ufzwkj -> quakes_patched.jsonl
== 2. a patch whose test precondition is false
FAIL patch patch_wrong_precondition.json: JsonPatchTestFailed: automatic (<class 'str'>) is not equal to tested value reviewed (<class 'str'>) -> no output file written
== 3. a patch that applies but breaks the closed record schema (adds /note)
FAIL patch patch_extra_field.json: ValueError: patched record fails the schema: Additional properties are not allowed ('note' was unexpected) -> no output file written
== files written (only the good patch may have produced one)
absent: 'should_not_exist_1.jsonl': No such file or directory
absent: 'should_not_exist_2.jsonl': No such file or directory
quakes_patched.jsonl
== the patched record
{"id":"aka2026ufzwkj","time":"2026-10-12T18:49:05Z","updated":"2026-10-12T18:50:09Z","mag":1.5,"mag_type":"ml","place":"10 km N of Susitna North, Alaska","region":"Alaska","lon":-149.852,"lat":62.256,"depth_km":14.2,"status":"reviewed","tsunami":false,"sig":30,"net":"ak","url":"https://earthquake.usgs.gov/earthquakes/eventpage/aka2026ufzwkj"}
== how many lines differ between the original and the patched file (must be 1)
1
== line counts: original 9891, patched 9891
== status counts before and after (jq, streaming)
quakes.jsonl           {"automatic":1799,"reviewed":8092}
quakes_patched.jsonl   {"reviewed":8093,"automatic":1798}
== the original file was not modified: sha256 35fb9a34a36a7df8 (same as in step 3: 35fb9a34a36a7df8)

--- 07_run_all.sh (steps 2 to 6, twice, in fresh folders) ---
# Re-run steps 2 to 6 from the saved snapshot, twice, in two fresh output folders, and compare the checksums of every output.
set -e
PY=./.venv/Scripts/python.exe
for run in 1 2; do
  rm -rf out$run && mkdir out$run
  bash 03_clean.sh > out$run/step3.log
  node 02_validate.mjs > out$run/step2.log
  node 04_report.mjs > out$run/step4.log
  bash 05_crosscheck.sh > out$run/step5.log
  $PY -X utf8 06_patch.py quakes.jsonl aka2026ufzwkj patch_good.json quakes_patched.jsonl > out$run/step6.log
  cp quakes.jsonl report.json strong.jsonl quakes_patched.jsonl out$run/
done
echo "files compared: quakes.jsonl report.json strong.jsonl quakes_patched.jsonl and the five step logs"
for f in quakes.jsonl report.json strong.jsonl quakes_patched.jsonl step2.log step3.log step4.log step5.log step6.log; do
  a=$(sha256sum out1/$f | cut -c1-16); b=$(sha256sum out2/$f | cut -c1-16)
  printf '%-22s %s %s %s\n' $f $a $b "$([ $a = $b ] && echo identical || echo DIFFERENT)"
done

Output:
files compared: quakes.jsonl report.json strong.jsonl quakes_patched.jsonl and the five step logs
quakes.jsonl           35fb9a34a36a7df8 35fb9a34a36a7df8 identical
report.json            b7505866b46d628e b7505866b46d628e identical
strong.jsonl           f1b82fbbe12da043 f1b82fbbe12da043 identical
quakes_patched.jsonl   698e2e3b95544442 698e2e3b95544442 identical
step2.log              10af1acadee43607 10af1acadee43607 identical
step3.log              738c234a8aec5f90 738c234a8aec5f90 identical
step4.log              b7505866b46d628e b7505866b46d628e identical
step5.log              ce919feff480b83b ce919feff480b83b identical
step6.log              09419c95c8c4949e 09419c95c8c4949e identical

--- README.md ---
# Earthquake JSON pipeline (Working with JSON, capstone)

A small, repeatable pipeline: download one snapshot of a public JSON feed, validate it, clean it into JSON Lines,
stream-report it, cross-check the numbers with a second tool, and apply a checked JSON Patch to one record.

## Data and licence

- **Source:** USGS Earthquake Hazards Program, GeoJSON summary feed `all_month`
  (`https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.geojson`).
- **Snapshot:** one request on 2026-10-12; the feed's own clock says generated 2026-10-12T19:05:41Z.
  7,173,808 bytes, 10,105 features, sha256 `37f31144591d663c45ba9a8fb776a3530830258474a455b8441f264b8c44f05a`.
  The feed is refreshed every minute, so a later download is different data.
- **Licence:** USGS states that most of its information is in the public domain and asks for credit. Credit line used here:
  "Source: U.S. Geological Survey". I read this in USGS policy pages found by search on 2026-10-12, not in a statement written
  for this one feed, so check the current USGS terms before reusing the data.

## Steps

| # | Script | What it does | Result on this snapshot |
|---|--------|--------------|-------------------------|
| 1 | `01_fetch.sh` / `01_record.sh` | download once; record size, hash, feed clock, count check | metadata.count equals the number of features |
| 2 | `02_validate.mjs` (Ajv), `02_validate.py` (jsonschema), `02_control.mjs` | validate each feature against `quake.schema.json` (JSON Schema 2020-12); cross-check; control | 10,104 valid, 1 invalid (a null `place`); both validators name the same record; control rejects all 6 deliberately broken copies |
| 3 | `03_clean.sh`, `03_clean.jq` | keep type `earthquake`, flatten to one record per line | 9,891 records; 214 non-earthquakes dropped (72 explosion, 117 quarry blast, 22 ice quake, 2 mining explosion, 1 landslide) |
| 4 | `04_report.mjs` | stream the JSON Lines, validate each record against `record.schema.json`, count | 9,891 valid, 0 invalid, 0 duplicate ids, 58 MB peak memory |
| 5 | `05_crosscheck.sh` | recompute the key numbers with jq from the original GeoJSON and compare | agree; a control with one number changed disagrees |
| 6 | `06_patch.py`, `06_patch.sh` | apply a JSON Patch (RFC 6902) to one record, check it against the schema, write a new file | good patch applied (1 line differs); a false `test` and a schema-breaking patch both leave no output file |
| 7 | `07_run_all.sh` | run steps 2-6 twice from the saved snapshot | every output file and log has the same sha256 both times |

## Headline numbers

- Magnitude: under 2: 6,898; 2-3: 1,713; 3-4: 303; 4-5: 808; 5-6: 161; 6 and over: 8. Strongest: M 7.7, 12 km WSW of Pitaloza Arriba, Panama (2026-10-09T17:56:06Z).
- Status: 1,799 automatic, 8,092 reviewed. Tsunami flag set on 5 records. Depth from -3.43 km to 665.057 km.
- Most records are in Alaska (3,323), then "CA" (2,828).

## Known limits

- The `region` field is the text after the last comma of `place`. USGS writes "CA" for California but "Alaska" in full, so regions are not consistent names. 137 places had no comma and keep the whole text.
- The "(unknown)" rule for a null `place` never fired on this snapshot: the one null-place feature was a quarry blast and was dropped by the type filter.
- The patch demonstration changes a record for teaching; it is not a real seismic correction, and the original file is never modified.
- One snapshot, one machine (Windows 11), Node 24.18.0, Python 3.14.0, jq 1.8.2, Ajv 8.20.0, jsonschema 4.26.0, jsonpatch 1.35.

WHY THIS WORKS AS AN ANSWER
---------------------------
The patch has preconditions (two test operations) so it fails loudly if the record is not what the author expected. The script never writes the output file unless the patch applied AND the patched record passes the closed schema; the three runs show all three outcomes: success with exactly one differing line and the original file unchanged, a failed test with no file, and a legal but schema-breaking add with no file. The two runs of steps 2 to 6 produce files and logs with identical SHA-256 values, which shows the scripts are deterministic on the same snapshot
(it says nothing about a fresh download, which differs every minute). The README records the source, snapshot hash, licence and its second-hand caveat, the steps with results and the known limits, so someone else can judge and repeat the work. Not run: concurrent patches, patching with other libraries, a second snapshot.