Capstone

This chapter uses everything in the course on one real job. You download a public JSON feed, check it against a JSON Schema, clean it into JSON Lines with jq, stream through it with a script to produce a report, check that report with a second tool, change one record with a JSON Patch, and prove you can run the whole thing again and get the same files. The data is the USGS earthquake feed for the past 30 days. Unlike the earlier chapters, the data is not mine or generated: it has real surprises in it, and the pipeline found one that I did not expect.

The Plan

Step 1: The Data and Its Licence

The USGS publishes earthquake summaries as GeoJSON (a JSON format for map data), refreshed every minute, at https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.geojson. The page that documents the feed says each feature has an id, a geometry (a Point with longitude, latitude and depth) and properties such as magnitude, place, time and status. I made one request, with a descriptive User-Agent, and kept the file:

Licence. A capstone should record where data came from and what you may do with it. The USGS pages I could read through a search say that most of its information resides in the public domain and may be used without restriction, with a request for credit (the suggested form is a line such as "Source: U.S. Geological Survey"), and that some third-party material on its sites is copyrighted. A direct fetch of the main USGS copyright page failed (HTTP 403), so I did not read that page myself, and the README says so. Before you reuse the data, read the current USGS terms. The snapshot is not stored with the course; the README records how to fetch another.

Step 2: Validate Before You Trust

A feed you did not write can change without warning. The schema describes only the parts this pipeline depends on, and stays open to new fields, because a feed that adds a property should not break us. An excerpt (Chapters 6 and 7 explain each keyword):

I compiled the schema once with Ajv (strict mode on, ajv-formats for format: "uri") and checked each feature separately so every bad record can be named. Results on the snapshot:

The control matters: a validator that says "valid" to everything would also have produced "10,104 valid". Seeing it reject known-bad copies is what makes the one real failure believable.

Step 3: Clean and Reshape with jq

The raw feature nests the useful fields two levels deep, stores times as milliseconds, and mixes real earthquakes with explosions and quarry blasts. The jq program (Chapter 5) turns each earthquake into one flat record:

I ran it as jq -b -c -f 03_clean.jq quakes.geojson > quakes.jsonl. -c gives one record per line and -b keeps Windows from writing CRLF (Chapter 9). Checks on the output:

• 9,891 records written; 214 features dropped by the type filter (72 explosions, 117 quarry blasts, 22 ice quakes, 2 mining explosions, 1 landslide). 9,891 + 214 = 10,105, so nothing went missing.
• The file contains no carriage-return bytes (0, counted with tr, after I had to fix my first check; see the mistakes).
• The unexpected result: the one feature with a null place is a quarry blast, so the type filter had already dropped it, and my "(unknown)" rule for null places never fired (0 records). I had written the rule to handle the invalid record Step 2 found; on this snapshot it is dead code. It would matter for a feed with a null place on an earthquake, so I kept it and wrote that down rather than claim it was exercised.
• The region rule is crude. It takes the text after the last comma of place. USGS writes "CA" for California but "Alaska" in full, so the biggest regions are "Alaska" (3,323 records) and "CA" (2,828), and 137 places with no comma (for example "Mid-Indian Ridge") keep their whole text. A real analysis would map abbreviations to names.

Step 4: Stream the Lines and Make the Report

The cleaned file is JSON Lines, so a Node script reads it one line at a time (Chapter 9), validates each record against a second, closed schema (additionalProperties: false: this is our own output, so an unknown field is a bug), and tallies. Excerpt:

The script also checks its own arithmetic: valid + invalid + bad + blank equals the number of lines, and the magnitude buckets sum to the valid count (both true).

Step 5: Check the Numbers a Different Way

A report is only as good as its method, so I recomputed the key numbers with jq straight from the original GeoJSON, not from quakes.jsonl, and compared them with report.json: number of earthquakes, the six magnitude buckets, status counts, the tsunami count and the strongest event's id. They agree. As a control I changed one number in the jq output on purpose and the comparison reported a disagreement. Two methods, two inputs, same answer: the cleaning step did not lose or alter any of the things counted.

Step 6: Patch One Record, Safely

Suppose a reviewer confirms record aka2026ufzwkj, so it should become reviewed and its magnitude 1.5. This is a teaching edit on a copy, not a real seismic correction. A JSON Patch (Chapter 8) states it exactly, with preconditions:

The script streams the JSON Lines file, applies the patch to the one record whose id matches (the library applies it to a copy), checks the patched record against the closed record schema, and only then writes the new file; the original is never opened for writing. Three patches, three outcomes:

The third row is why the schema check comes after the patch: JSON Patch knows nothing about your schema, so a patch can be perfectly legal and still produce a record your pipeline rejects. The original quakes.jsonl had the same SHA-256 afterwards.

Step 7: Repeatable

I ran steps 2 to 6 twice from the saved snapshot in two fresh folders. The four data files (quakes.jsonl, report.json, strong.jsonl, quakes_patched.jsonl) and the five step logs all had identical SHA-256 values in both runs. That is a modest claim: both runs read the same snapshot, so it shows the scripts are deterministic. It says nothing about a new download, which will differ because the feed changes every minute. That is why Step 1 records the hash of the snapshot.

The README

A pipeline is only reusable if someone else can tell what it did. Mine states: the source and URL; the snapshot time, size, count and SHA-256; the licence and how I learned it (with the caveat); a table of the seven steps with their results on this snapshot; the headline numbers; and the known limits (the crude region rule, the dead "(unknown)" rule, that the patch is a classroom edit, and the exact tool versions). The full file is in the solution to Exercise 3.

Exercises

Course Complete: What You Can Now Do