jq Reshaping

Chapter 4 was about looking at JSON. This chapter is about changing its shape: turning a list of orders into totals per customer, an array into an object keyed by ID, a nested structure into CSV, or a document with some fields removed. It builds on the filters, options and quoting rules from Working with JSON 4, and uses two sample files: the shop.json from that chapter and a new orders.json of six orders. The chapter ends with a section on what changed in jq 1.7 and 1.8, because several things you will read online about jq are out of date, and one example in jq's own release notes does not hold in the version I tested.

The Sample Data

Building Objects and Arrays

An object literal builds a new object from the current input; an array literal collects every result of its contents into one array. I ran these on orders.json:

The rules: {id} is shorthand for {id: .id}; a key that is not a plain word needs quotes ("order-id"); a key computed from an expression goes in parentheses (("k" + "ey")). Order 4 has an empty items list, so .items | map(...) | add would give null; add // 0 turns that into 0 (the // operator is explained below). Notice the total for order 4 is 0 in the output.

Two ways to combine objects. + merges at the top level, with the right side winning; * merges recursively. I ran both on {a:{b:1,c:2}} and {a:{c:3,d:4}}: + gave {"a":{"c":3,"d":4}} (the whole a replaced, b lost) and * gave {"a":{"b":1,"c":3,"d":4}}.

Entries: Turning Objects into Lists and Back

To work on an object's keys as data, convert it to a list of {key, value} pairs with to_entries, change the list, then convert back with from_entries. with_entries(f) does all three steps:

from_entries is also the way to build an object from an array. Three ways to turn the four products into an object keyed by SKU all gave {"A1":40,"B2":0,"C3":120,"D4":15} for the stock levels: map({(.sku): .stock}) | add, map({key: .sku, value: .stock}) | from_entries, and INDEX(.sku) | map_values(.stock).

What from_entries accepts is narrower than people assume. I tried eleven entry shapes on jq 1.8.2:

So: use key and value exactly (or name/Name, Key/Value), make keys strings with tostring, and be suspicious of a result full of nulls. A wrong value name does not error; it just gives null.

Sorting, Grouping and Picking

group_by(f) sorts by f first and returns an array of arrays, one per distinct value, which is why .[0] inside each group is a handy way to name the group. Each group is a plain array, so length and map work on it directly. unique_by keeps the first item of each group. A group-by result is the usual first step of a "totals per X" report.

Variables, reduce and foreach

EXPR as $name | ... stores a value for use later in the same expression. It does not change the input: . is still the same. It also destructures: .[0] as {customer: $c, items: [{sku: $first}]} | [$c, $first] gave ["ada","A1"], and .items | . as [$a, $b] | [$a.sku, $b.sku] on order 5 gave ["A1","B2"].

reduce folds a stream into one value. foreach does the same but emits each intermediate state:

The pattern .[$key] += 1 works because a missing key is null and null + 1 is 1: it needs no setup. Notice the key order in the second result: reduce keeps the order it first saw each status (shipped, pending, cancelled), while group_by sorted them alphabetically. The two results are the same data in a different order, and compare JSON from two different filters with jq -S (sort keys), as the exercise does. reduce on an empty stream returns the starting value: reduce empty as $x (0; . + 1) gave 0.

Conditionals, the // Operator, try and Early Exit

if COND then A elif COND2 then B else C end is an expression. empty produces no output, which drops the item: map(if .status == "cancelled" then empty else .id end) gave [1,2,3,5,6]. The else branch is optional since jq 1.7 (the missing branch is .): 5 | if . > 3 then "big" end printed "big". In a condition, only false and null are false: I tested [1, null, false, 0, "", [], {}] and got falsy for null and false only. 0, "", [] and {} are all truthy.

a // b ("alternative") means "a, unless it is false, null or empty, then b". Same truthiness rule: [false, null, 0] | map(. // "default") gave ["default","default",0], so a real 0 is kept. map(.items[0].sku // "none") gave ["A1","B2","D4","none","A1","C3"] (order 4 has no items). With a stream on the left it gives all the truthy values: (false, 1, 2) // 9 printed 1 and 2, and [(false, null) // 9] gave [9].

try catches an error. try EXPR alone produces nothing on error; try EXPR catch HANDLER runs the handler on the error message; EXPR? is shorthand for try EXPR:

Early exit: first(f) and limit(n; f) stop producing as soon as they have enough, which matters for infinite streams: [limit(3; repeat("x"))] gave ["x","x","x"]. label $out | 1, 2, break $out, 3 printed only 1 and 2. Do not overuse ? and try: they hide real mistakes (a misspelt filter inside try just silently produces nothing).

Updating in Place: =, |=, +=, //=, del

= assigns a value computed from the whole input; |= updates a location using its own current value (.a |= map(. * 2)). Prefer |= when the new value depends on the old one. Every update returns the whole modified document, not just the part you changed.

Paths: paths, getpath, setpath, walk, pick

A path is an array like ["a","b",1,"c"] that leads to a value. The sample is {"a":{"b":[10,{"c":null}],"d":"x"},"e":true}:

Take two warnings from the table. getpath cannot tell "the value is null" from "there is no such path"; to ask whether a path exists, compare against the list of paths ([paths] | map(. == ["a","b",1,"c"]) | any gave true, and false for a made-up path). And pick manufactures nulls for anything absent, so a typo in a pick looks like data. walk(f) applies a function to every value from the inside out (on [[1,2],[3]], a rule that joins each array's items with commas gave "1,2,3", so the inner arrays were converted before the outer one), which is the tool for changes that apply "everywhere", such as upper-casing all strings or deleting a key at any depth. leaf_paths, which older tutorials use, no longer exists: it was removed in 1.7, and [leaf_paths] gave leaf_paths/0 is not defined. Use paths(scalars).

Output Formats: @csv, @tsv, @sh, @uri and Friends

Format strings turn an array or string into text for another program. Use them with -r so the quotes of the JSON string do not appear. First, CSV with a header row:

@csv quotes strings, leaves numbers bare, writes null as an empty field and true as true. To test that it quotes correctly, I built a file of five items whose names contained a comma, a double quote, a newline, a null and an accented letter, wrote it as CSV with jq, and read the result back with Python's csv module, comparing with the original. All five rows matched. As a control, I built the same rows by joining with commas (no quoting) and the same check reported mismatches on four rows. The first result, as CSV text:

@csv accepts only an array of scalars: giving it an object failed with object ({"sku":"A1","name":"Tea",...}) cannot be csv-formatted, only array, and a row containing an array failed with array (["drink","hot"]) is not valid in a csv row. Flatten first: (.tags // [] | join("|")) turned the tags into one field (and Milk's missing tags into an empty string).

String interpolation is "text \(expression) text", and a format name in front applies that format to each interpolated part: "Total: \(.products | map(.price) | add)" gave Total: 10.63; @uri "https://example.com/search?q=\(.owner.first + " & " + .owner.last)" gave https://example.com/search?q=Ada%20%26%20Lovelace (the literal text is left alone, the interpolated part is encoded); @sh "echo \(.owner.first) \(.shop)" gave echo 'Ada' 'Corner Shop'. Always use @sh when jq output goes into a shell command: it is what stopped the "INJECTED" test string from running (Working with JSON 11 comes back to injection).

Your Own Functions and Modules

def name: body; defines a function that works on its input; def name(a; b): body; takes filter parameters, separated with semicolons. I tested:

A plain parameter like rate is a filter that is re-evaluated each time it is used (so it could be .stock); a $rate parameter is evaluated once to a value. For simple values, $name is the safer form. For real reuse, put definitions in a module, a file ending in .jq. I wrote lib/shoplib.jq with three functions:

import "name" as prefix; keeps the functions behind a prefix (s::money); include "name"; pastes them in with no prefix. -L dir (long form --library-path) says where to look. Without it, jq did not find the module, with a clear message and exit status 3.

What Changed in jq 1.7 and 1.8 (Each One Run)

jq changed more in 2023 to 2026 than in the previous five years, and many online answers assume 1.6. The list below comes from the project's NEWS.md; every row was run on 1.8.2 and the result is shown. A feature marked 1.7 or 1.8 is not there in older versions: if a script must run on an old jq, check jq --version first.

One feature I expected is not there: toarray. [1] | toarray failed with toarray/0 is not defined on 1.8.2, although I had seen it recommended. Another thing to know is the 1.8.0 change in how as binds against operators: [-1 as $x | 1, $x] gave [1,-1] on 1.8.2 (the notes say it used to be [-1,-1]), and 1 | . + 2 as $x | -$x gave -3. If an old script behaves differently, these notes are the first place to look.

Hands-On Exercises