jq Basics

Chapters 2 and 3 handled JSON inside programs. Often you only want to look at some JSON: pick one field from an API response, count the items in a file, list the names that match a condition. Writing a script for each question is slow. jq is a small command-line program that reads JSON, applies a short filter you write, and prints the result. This chapter installs it, teaches the filters you will use every day (selecting, filtering, the main options), connects it to curl, and covers the part that trips people up on Windows: quoting a filter correctly in bash, PowerShell and cmd. The next chapter covers reshaping data.

Installing jq on Windows (Checked)

jq was not installed on this machine, so I installed it into a scratch folder instead of changing the system: download the program from the project's GitHub releases page, verify the download, and put it on the PATH for the session. The release page lists a single file for 64-bit Windows, jq-windows-amd64.exe (1,035,264 bytes), and a text file of checksums (sha256sum.txt).

The two hashes matched, and a deliberately damaged copy did not, which shows the comparison can fail. Always check the hash of a program you download. To make jq available everywhere you would put the .exe in a folder on your PATH (and rename it jq.exe, as above); I did not do that. From here on, jq means this binary. Run jq --version to see which version your shell is actually using.

The Model: Input In, Filter, Output Out

jq reads JSON, runs your filter on each JSON value it reads, and prints every result it produces as pretty-printed JSON. A filter can produce zero, one or many results for a single input, and that idea explains most of jq. All the examples use this sample file, shop.json:

The simplest filter is ., "the input itself", which makes jq . file.json a formatter and validator (it prints the file indented, or reports an error). A filter is the first argument and the file name comes after; with no file name, jq reads standard input. The rest of the chapter is about the filters.

Selecting: Keys, Indexes and []

[] is the "one result per item" filter: .products[] turned one array into four separate objects, and .products[].name then took name from each. This is the "zero, one or many" idea at work. Notice that a missing key gives null, not an error, which is convenient and also a trap: a typo in a key name looks like missing data. The fourth product, Milk, has no tags at all, and that shows up next:

For Milk, .tags is null and [] cannot step through null, so jq printed the five tags from the first three products, then stopped with an error and exit status 5. The ? suffix means "if this fails, quietly produce nothing", and it gave the same five tags with no error. Use ? deliberately: it also hides real mistakes.

Pipes, Commas and Building Output

| feeds the results of the left side into the right side, one at a time, exactly like a shell pipe. , gives several results from the same input. And | binds loosest: .shop | type, (.open|type) does not mean "the type of .shop, then the type of .open". I ran it:

The comma sits inside the right side of the pipe, so (.open|type) ran on the string "Corner Shop" and failed. The fix is to write what you mean: (.shop | type), (.open | type), or .shop, .open | type. When in doubt, add parentheses.

You can also build new values. An object from parts: .products[0] | {name, cost: .price} gave {"name":"Tea","cost":3.5} ({name} is shorthand for {name: .name}). An array from many results: [.products[] | .name | length] gave [3,6,8,4]. These are the building blocks for Chapter 5.

Filtering and the Everyday Functions

select(condition) passes its input through only if the condition is true; map(f) is [.[] | f]. These ran on shop.json:

Two lessons from this table. First, guard against missing keys before calling a function on them: the tags != null and ... in the last row is needed because contains on null would fail for Milk. Second, select produces nothing (not false) when the condition fails, which is why a filter that matches nothing prints nothing at all.

Comparison Traps: Strings Are Not Numbers

jq has a total ordering across types: null < false < true < numbers < strings < arrays < objects. I confirmed it by sorting [1, "1", null, true, false, [], {}] and got [null,false,true,1,"1",[],{}]. The consequence for command-line use: --arg always creates a string, so this filter compares a number with a string:

The first command printed an empty list with exit status 0 and no warning: every number sorts before every string, so 3.5 > "2" is false (jq -n '3.5 > "2"' printed false). Convert with tonumber, or pass the value with --argjson, which parses its argument as JSON. The test jq -n --arg n 5 --argjson m 5 '[$n, $m, ($n == "5"), ($m == 5)]' gave ["5",5,true,true].

The Options You Will Use

-r is the one you will use most, because it makes jq's output usable by other commands: jq -r '.products[].name' prints one name per line, ready for a loop. -e is for scripts: it lets an if test whether a filter found something. Note the three different exit statuses of -e (a "null or false" result is 1, "no output" is 4), and that without -e a filter that matches nothing exits 0.

What jq Does to Your Numbers and Input

I fed jq the awkward inputs from Chapter 1:

The point for this course: jq's handling of numbers and of invalid input is its own, a third behaviour beside JavaScript's and Python's. When data must pass through all three, do not assume they agree. Chapter 1's rule stands: test the actual inputs.

curl into jq

The most common use of jq is looking inside an API response. curl downloads, jq reads from the pipe (the curl side, including its options, is covered in API Testing & Tooling 6 and 7). I started a tiny server on 127.0.0.1 that returns the sample file, a 404 with a JSON error body, and a 502 HTML page, and tried each way of calling it:

Three lessons. A pipe's exit status is the last command's unless you use pipefail, so jq reporting success says nothing about whether the request worked. A server's JSON error body is still valid JSON, so jq prints it as if it were your data; check the status with -f, --fail-with-body or the %{http_code} variable. And "Invalid numeric literal" almost always means the input was not JSON at all: an HTML page or progress text. Look at the raw body first. This is the same mistake as the Unexpected token < error in Chapter 2.

Reading jq's Error Messages

The messages come in three families, and the first words tell you which. parse error means the data is not valid JSON: look at the input. error (at file:line) means the data was fine but the filter did something impossible to it (indexing a number): check the shape of the data, with jq 'type' or jq keys. compile error means the filter itself is malformed: check quoting and parentheses. The exit statuses I saw were 2 (cannot open the file), 3 (compile error) and 5 (parse or run-time error), and, as above, 1 and 4 from -e.

Quoting a Filter on Windows

A jq filter is full of characters the shell wants for itself: |, ", $, parentheses. The right quoting differs per shell. I ran the same filter, .products[] | select(.name == "Tea") | .sku (expected output "A1"), in each:

Two habits make quoting nearly painless on any shell. Keep strings out of the filter with --arg and --argjson, so the filter has no double quotes. And put long filters in a file with -f: the shell never sees them. I tested Windows PowerShell 5.1 only; PowerShell 7 may pass arguments to programs differently, so check the backslash row there before trusting it. Piping into jq from PowerShell also worked: @{ a = 1; b = 'x' } | ConvertTo-Json -Compress | jq -c '.' printed {"a":1,"b":"x"}.

Line Endings on Windows

I looked at the bytes jq wrote for jq -r .shop shop.json redirected to a file: they ended in 0d 0a (a carriage return and a line feed, CRLF), and with -b in 0a alone. So on Windows, jq writes Windows line endings to files and pipes unless you add -b. That matters when the output goes to a Linux program or into a repository that expects LF. In Git Bash, command substitution (x=$(jq -r .shop shop.json)) still gave a string of 11 characters, without a stray carriage return, both with and without -b, so loops over $(jq -r ...) were fine in my test; other shells and tools may not strip it.

Hands-On Exercises