Pointer, Path, Patch

Four small standards cover most of what you need to point at a part of a JSON document and to change it, and they get confused constantly. JSON Pointer names exactly one location. JSONPath searches for many. JSON Patch is a precise list of edits. JSON Merge Patch is a much simpler "here is what the document should look like now" edit. You have already met the first one without knowing it: the instancePath in a validator's errors (Chapter 6) and the $ref paths are JSON Pointers. This chapter explains each one, applies each with real libraries in Node and Python and with jq (Chapters 4 and 5), measures where the libraries disagree with the standards, and ends with how to choose.

The Four Standards at a Glance

JSONPath existed for years as a loose convention with many incompatible versions; RFC 9535 (February 2024) finally standardised it, which matters for the library comparison below. Patch media types are application/json-patch+json and application/merge-patch+json, used with the HTTP PATCH method.

JSON Pointer (RFC 6901)

A pointer is a string of reference tokens, each starting with /: /foo/0 means "the member foo, then its element 0". Two characters need escaping inside a token because they have a meaning: ~ is written ~0 and / is written ~1. When decoding, replace ~1 first and then ~0, otherwise ~01 would be misread as /. Three special cases:

• The empty string "" points to the whole document, but "/" points to the member whose name is the empty string "". They are different.
• Array indexes have no leading zeros: /foo/0 is valid, /foo/01 is not.
• The token - means "the element after the last one". It never points at an existing value, so reading it is an error; it is useful only for adding (JSON Patch below).

RFC 6901 (section 5) gives a test document with awkward keys; I checked my copy and the expected values against the RFC text. I wrote it to a file and resolved eighteen pointers (the twelve from the RFC plus six edge cases) with four implementations, one of them a jq function I wrote:

json-p3, Python's jsonpointer and my jq function matched the RFC on all 18 pointers. fast-json-patch matched on 17 and returned "baz" for /foo/01, accepting a leading zero that the RFC forbids. In the other "no value" cases the libraries differ in how they say so: json-p3 throws a JSONPointer...Error, fast-json-patch returns undefined, Python's jsonpointer raises JsonPointerException (and for /foo/- returns a special EndOfList marker object, not an error). Check for "no value" the way your library reports it.

In jq a pointer is a path, so getpath does the work once the string is converted. My function splits on /, unescapes ~1 then ~0, and treats a digit-only token as an index only when the value there is an array (so the key "0" in an object still works): "/foo/0" gave "bar", and on {"0":"zero-key","list":["a","b"]} the pointers /0 and /list/0 gave "zero-key" and "a". jq itself has no pointer syntax; the reverse conversion (paths | map(tostring) | join("/")) is not safe if keys contain / or ~.

Escaping is the usual source of bugs: /a/b means two levels; /a~1b means the single key a/b. A pointer can also be written in a URI fragment (#/c%25d for the pointer /c%d, percent-encoded as RFC 6901 says; I did not run that form).

JSONPath (RFC 9535)

A JSONPath query starts at $ (the root) and chains segments. The result is a nodelist: zero or more values, each with a known location (RFC 9535 calls it a Normalized Path, written like $['store']['book'][0]). An empty result is valid, not an error.

I ran 21 queries on the bookstore document that RFC 9535 and older JSONPath articles use (four books and a bicycle; store.json), with the expected answer for each written by hand beforehand, in four libraries:

The older libraries are not "wrong" in their own terms: they predate the RFC and each has its own dialect. But jsonpath-plus returning an empty result for a filter it does not understand, with no error, is the dangerous kind of difference: the query looks like it ran and found nothing. The same query text can therefore return different data in different libraries. Pick an RFC 9535 implementation when queries cross a language boundary or are stored, and keep a few known-answer queries as a test. Two details of the RFC the compliant libraries implemented: a comparison of two queries that both select nothing is true ($.store.book[?@.nope == $.store.book[0].nope].price returned all four prices), and JSON null is an ordinary value, not "missing". match() and search() take I-Regexp patterns (RFC 9485), a deliberately small regular-expression dialect; I tested only "J.*" and "Moby".

jq has no JSONPath, but it can do the same work with filters. These jq programs gave the same answers as their JSONPath twins on store.json:

JSON Patch (RFC 6902)

A patch is an array of operations applied in order. Each has an op and a path (a JSON Pointer):

RFC 6902 says a patch is applied as a whole: if any operation fails, the patch as a whole is not successful. I took the examples from the RFC's appendix, A.1 to A.16 except A.13 (a duplicate-op document that an ordinary parser cannot even represent), which I checked against the RFC text (add, remove, replace, move, test, the ~ escape, appending an array as a single value, unrecognised members ignored, and the error cases), added nine of my own (copy, add past the end of an array, remove or replace a missing member, add over an existing member, move into its own child, test 1 against 1.0, an unknown op, replacing the whole document with an empty path) and ran all 24 in three libraries:

The three libraries agreed with each other on every case when configured correctly. The lesson for fast-json-patch: pass true as the third argument (validate operations), or a patch that should have been rejected will go through.

Atomicity is your job. I applied a three-operation patch whose third operation fails (remove /nope) to {"a":1}:

So a patch that fails halfway can leave a half-edited document unless you apply it to a copy and keep the copy only on success. The test operation is also how you do optimistic concurrency ("apply this only if /version is still 7"); in the HTTP example below a leading test on /name let the rest of the patch through.

Patches can also be generated by comparing two documents. Python's jsonpatch.make_patch turned {"a":1,"b":[1,2,3],"c":{"d":4}} and {"a":2,"b":[1,3],"c":{"d":4,"e":5}} into [{"op":"replace","path":"/a","value":2}, {"op":"remove","path":"/b/1"}, {"op":"add","path":"/c/e","value":5}], and applying that patch to the first document reproduced the second. jq has no built-in JSON Patch; you express edits directly (.a = 2 | del(.b[1]) | .c.e = 5), and Chapter 5's setpath and delpaths are the primitives.

JSON Merge Patch (RFC 7396)

A merge patch is just a JSON document shaped like the part of the target you want to change. The rules are short: if the patch is not an object, it replaces the target entirely; if it is an object, each member is merged in; a member whose value is null deletes that member; an array in the patch replaces the whole array. I ran the RFC's Appendix A table of 15 examples (checked against the RFC text) plus three of my own (18 cases) in the Node library, the Python library and a jq function I wrote from the RFC's pseudo-code: all three gave the RFC's result on all 18. Some of them:

Merge patch is pleasant for "update these fields", but it has built-in limits: you cannot set a value to null (that deletes it), you cannot change one element of an array (you must send the whole array), and a patch generated from two documents loses a null: generate({a:1}, {a:1, f:null}) produced {"f":null}, and applying it to {"a":1} gave {"a":1}, not the intended {"a":1,"f":null}. Do not use merge patch where null is meaningful data. Both libraries also changed their input object in place unless I copied it first (apply({a:1}, {a:2}) left the target as {"a":2}).

jq's * operator is not merge patch. I ran four expressions: {"a":"b"} * {"a":null} gave {"a":null} (RFC 7396 would delete a), the nested example gave {"a":"z","c":{"d":"e","f":null}} (the RFC's answer deletes f), {"a":["b"]} * {"a":"c"} gave {"a":"c"} (the same), and + merged only the top level ({"a":{"x":1}} + {"a":{"y":2}} gave {"a":{"y":2}}). So * is a recursive merge that keeps nulls, close to merge patch but not equal. If you need real merge-patch semantics in jq, write the pseudo-code as a function (mine passed all 18 cases).

Patching Over HTTP

The two patch formats are carried by the HTTP PATCH method, with the media type telling the server which one you sent. I wrote a small Node server (made for this chapter, so the status codes are my choices, not a standard) that accepts both and applies the patch to a copy, and called it with curl:

The failing patch left the stored document unchanged because the server applied it to a copy and replaced the stored document only on success: exactly the atomicity from the previous section. Note that the merge patch changed one field and removed another in one small request, while the JSON Patch needed an operation per edit but could append to the array and could check a precondition first.

Choosing

Hands-On Exercises