json1-7  Exercise 3: Error Quality, the Editor Engine, Generated Types and Docs, Compatible Changes
=================================================================================================

Run with Node 24.18.0 (Ajv 8.20.0, ajv-formats 3.0.1, ajv-errors 3.0.0, json-schema-to-typescript 16.0.0, TypeScript 7.0.2, vscode-json-languageservice 6.0.0-next.4) and Python 3.14.0 (jsonschema 4.26.0, json-schema-for-humans 1.5.1), installed in a scratch folder. The editor part runs the JSON language service package (the engine inside VS Code's JSON support), not the VS Code window.

--- t5.mjs (three styles of one-of-these-shapes, and ajv-errors) ---
import { createRequire } from 'node:module'; const require = createRequire(import.meta.url);
const Ajv2020 = require('ajv/dist/2020'); const addFormats = require('ajv-formats'); const ajvErrors = require('ajv-errors');
const events = {
  click: { type: 'object', properties: { type: { const: 'click' }, x: { type: 'integer' }, y: { type: 'integer' } }, required: ['type', 'x', 'y'] },
  key:   { type: 'object', properties: { type: { const: 'key' }, key: { type: 'string', minLength: 1 } }, required: ['type', 'key'] },
  scroll:{ type: 'object', properties: { type: { const: 'scroll' }, dy: { type: 'number' } }, required: ['type', 'dy'] },
};
const bad = { type: 'click', x: 'ten', y: 5 };
const show = (label, v, d) => { const ok = v(d); console.log('## ' + label + ' -> ' + (ok ? 'valid' : 'INVALID, ' + v.errors.length + ' error(s)')); if (!ok) for (const e of v.errors) console.log('     ' + (e.instancePath || '/').padEnd(6) + e.keyword.padEnd(12) + e.message + (e.keyword === 'const' ? ' ' + JSON.stringify(e.params) : '')); };
const mk = o => { const a = new Ajv2020({ allErrors: true, strict: false, ...o }); addFormats(a); return a; };
console.log('document tested: ' + JSON.stringify(bad));
show('oneOf of three event schemas', mk().compile({ oneOf: Object.values(events) }), bad);
// if/then chain keyed on "type"
const chain = { allOf: Object.entries(events).map(([k, s]) => ({ if: { properties: { type: { const: k } }, required: ['type'] }, then: s })), required: ['type'], properties: { type: { enum: Object.keys(events) } } };
show('if/then chain keyed on "type"', mk().compile(chain), bad);
show('  ... same chain, unknown type "hover"', mk().compile(chain), { type: 'hover' });
// Ajv discriminator keyword
const disc = { type: 'object', discriminator: { propertyName: 'type' }, required: ['type'], oneOf: Object.entries(events).map(([k, s]) => ({ ...s, properties: { ...s.properties, type: { const: k } } })) };
show('Ajv { discriminator: true } option', mk({ discriminator: true }).compile(disc), bad);
show('  ... same, unknown type "hover"', mk({ discriminator: true }).compile(disc), { type: 'hover' });
show('  ... same, missing "type"', mk({ discriminator: true }).compile(disc), { x: 1 });
// ajv-errors: custom messages
const a2 = mk(); ajvErrors(a2);
const custom = { type: 'object', properties: { qty: { type: 'integer', minimum: 1, errorMessage: { type: 'quantity must be a whole number', minimum: 'quantity must be at least 1' } } }, required: ['qty'], errorMessage: { required: { qty: 'quantity is required' } } };
const vc = a2.compile(custom);
for (const d of [{}, { qty: 1.5 }, { qty: 0 }, { qty: 2 }]) { const ok = vc(d); console.log('## ajv-errors ' + JSON.stringify(d).padEnd(12) + (ok ? 'valid' : vc.errors.map(e => (e.instancePath || '/') + ' ' + e.keyword + ': ' + e.message).join(' | '))); }

Output:
document tested: {"type":"click","x":"ten","y":5}
## oneOf of three event schemas -> INVALID, 6 error(s)
     /x    type        must be integer
     /     required    must have required property 'key'
     /type const       must be equal to constant {"allowedValue":"key"}
     /     required    must have required property 'dy'
     /type const       must be equal to constant {"allowedValue":"scroll"}
     /     oneOf       must match exactly one schema in oneOf
## if/then chain keyed on "type" -> INVALID, 2 error(s)
     /x    type        must be integer
     /     if          must match "then" schema
##   ... same chain, unknown type "hover" -> INVALID, 1 error(s)
     /type enum        must be equal to one of the allowed values
## Ajv { discriminator: true } option -> INVALID, 1 error(s)
     /x    type        must be integer
##   ... same, unknown type "hover" -> INVALID, 1 error(s)
     /     discriminatorvalue of tag "type" must be in oneOf
##   ... same, missing "type" -> INVALID, 2 error(s)
     /     required    must have required property 'type'
     /     discriminatortag "type" must be string
## ajv-errors {}          / errorMessage: quantity is required
## ajv-errors {"qty":1.5} /qty errorMessage: quantity must be a whole number
## ajv-errors {"qty":0}   /qty errorMessage: quantity must be at least 1
## ajv-errors {"qty":2}   valid

--- t6.py output (Python's view of the same oneOf, and a formatter) ---
oneOf: top-level errors: 1 | ['oneOf'] | {'type': 'click', 'x': 'ten', 'y': 5} is not valid under any
  context (the branches): [(['x'], 'type', "'ten' is not of type 'integer'"), (['type'], 'const', "'key' was expected"), ([], 'required', "'key' is a required property"), (['type'], 'const', "'scroll' was expected"), ([], 'required', "'dy' is a required property")]
  best_match -> {'type': 'click', 'x': 'ten', 'y': 5} is not valid under any of the given schemas | path []
if/then chain: errors: [(['x'], 'type', "'ten' is not of type 'integer'")]
jsonschema.validate with a bad schema -> SchemaError
V(bad schema).is_valid({}) -> raises UnknownType - Unknown type 'strng' for validator with schema:
    {'type': 'strng'}

customer: '' should be non-empty
id: 0 is less than the minimum of 1
items/0/price: 0 is less than or equal to the minimum of 0
items/0/qty: 0 is less than the minimum of 1
items/0/sku: 'a1' does not match '^[A-Z][0-9]$'
status: 'lost' is not one of ['pending', 'shipped', 'cancelled']

--- ls.mjs (language service diagnostics) ---
import { getLanguageService } from 'vscode-json-languageservice';
import { TextDocument } from 'vscode-languageserver-textdocument';
import fs from 'node:fs';
const schemaText = f => fs.readFileSync(f, 'utf8');
const ls = getLanguageService({ schemaRequestService: uri => Promise.resolve(schemaText(uri.replace('file:///', '').replace('mem://', ''))) });
const diag = async (schemaFile, doc, label, extra = {}) => {
  ls.configure({ allowComments: false, schemas: [{ uri: 'mem://' + schemaFile, fileMatch: ['*.json'], schema: JSON.parse(schemaText(schemaFile)) }] });
  const td = TextDocument.create('file:///test.json', 'json', 1, typeof doc === 'string' ? doc : JSON.stringify(doc, null, 2));
  const jd = ls.parseJSONDocument(td);
  const ds = await ls.doValidation(td, jd, { comments: 'error', trailingCommas: 'error' });
  console.log('## ' + label + ' -> ' + (ds.length ? ds.length + ' problem(s)' : 'no problems'));
  for (const d of ds) console.log('     line ' + (d.range.start.line + 1) + ', severity ' + d.severity + ': ' + d.message);
};
const order = { id: 1, customer: 'ada', status: 'pending', date: '2026-10-01', items: [{ sku: 'A1', qty: 2, price: 3.5 }] };
await diag('order.schema.json', order, 'valid order');
await diag('order.schema.json', { ...order, id: 0, status: 'lost', items: [{ sku: 'a1', qty: 0, price: 0 }], note: 1 }, 'invalid order (several mistakes)');
await diag('order.schema.json', { id: 1 }, 'order missing fields');
await diag('order.schema.json', { ...order, date: '2026-13-45' }, 'bad date (format)');
await diag('order.schema.json', '{"id": 1, "customer": "ada",}', 'a trailing comma (syntax)');
await diag('order.schema.json', '// a comment\n{"id": 1}', 'a comment (syntax)');
// does the service understand 2020-12 keywords?
fs.writeFileSync('s2020.json', JSON.stringify({ $schema: 'https://json-schema.org/draft/2020-12/schema', type: 'object', properties: { t: { type: 'array', prefixItems: [{ type: 'string' }, { type: 'integer' }], items: false }, d: { type: 'object', dependentRequired: { a: ['b'] } } }, unevaluatedProperties: false }));
await diag('s2020.json', { t: ['a', 1] }, '2020-12: prefixItems OK');
await diag('s2020.json', { t: ['a', 1, 2] }, '2020-12: prefixItems + items:false, extra item');
await diag('s2020.json', { t: [1, 'a'] }, '2020-12: prefixItems wrong order');
await diag('s2020.json', { d: { a: 1 } }, '2020-12: dependentRequired violated');
await diag('s2020.json', { zzz: 1 }, '2020-12: unevaluatedProperties violated');

Output:
## valid order -> no problems
## invalid order (several mistakes) -> 6 problem(s)
     line 2, severity 2: Value is below the minimum of 1.
     line 4, severity 2: Value is not accepted. Valid values: "pending", "shipped", "cancelled".
     line 8, severity 2: String does not match the pattern of "^[A-Z][0-9]$".
     line 9, severity 2: Value is below the minimum of 1.
     line 10, severity 2: Value is below the exclusive minimum of 0.
     line 13, severity 2: Property note is not allowed.
## order missing fields -> 4 problem(s)
     line 1, severity 2: Missing property "customer".
     line 1, severity 2: Missing property "status".
     line 1, severity 2: Missing property "date".
     line 1, severity 2: Missing property "items".
## bad date (format) -> no problems
## a trailing comma (syntax) -> 4 problem(s)
     line 1, severity 2: Missing property "status".
     line 1, severity 2: Missing property "date".
     line 1, severity 2: Missing property "items".
     line 1, severity 1: Trailing comma
## a comment (syntax) -> 5 problem(s)
     line 2, severity 2: Missing property "customer".
     line 2, severity 2: Missing property "status".
     line 2, severity 2: Missing property "date".
     line 2, severity 2: Missing property "items".
     line 1, severity 1: Comments are not permitted in JSON.
## 2020-12: prefixItems OK -> no problems
## 2020-12: prefixItems + items:false, extra item -> 1 problem(s)
     line 2, severity 2: Array has too many items according to schema. Expected 2 or fewer.
## 2020-12: prefixItems wrong order -> 2 problem(s)
     line 3, severity 2: Incorrect type. Expected "string".
     line 4, severity 2: Incorrect type. Expected "integer".
## 2020-12: dependentRequired violated -> 1 problem(s)
     line 2, severity 2: Object is missing property b required by property a.
## 2020-12: unevaluatedProperties violated -> 1 problem(s)
     line 2, severity 2: Property zzz is not allowed.

--- ls2.mjs ($schema inside the document) ---
import { getLanguageService } from 'vscode-json-languageservice';
import { TextDocument } from 'vscode-languageserver-textdocument';
import fs from 'node:fs';
const ls = getLanguageService({ schemaRequestService: uri => { const f = uri.replace(/^(file:\/\/\/|mem:\/\/)/, ''); try { return Promise.resolve(fs.readFileSync(f, 'utf8')); } catch (e) { return Promise.reject(e.message); } } });
ls.configure({ allowComments: false });   // NO schema association in settings: only the document's own $schema
const run = async (label, text) => { const td = TextDocument.create('file:///x.json', 'json', 1, text); const ds = await ls.doValidation(td, ls.parseJSONDocument(td)); console.log('## ' + label + ' -> ' + (ds.length ? ds.map(d => d.message).join(' | ') : 'no problems')); };
await run('$schema in the file, bad status', JSON.stringify({ $schema: 'mem://order.schema.json', id: 1, customer: 'a', status: 'lost', date: '2026-10-01', items: [] }));
await run('$schema in the file, good document', JSON.stringify({ $schema: 'mem://order.schema.json', id: 1, customer: 'a', status: 'pending', date: '2026-10-01', items: [] }));
await run('$schema in the file points at a missing file', JSON.stringify({ $schema: 'mem://nope.schema.json', id: 1 }));
await run('no $schema at all, nothing configured', JSON.stringify({ id: 'not even checked' }));

Output:
## $schema in the file, bad status -> Value is not accepted. Valid values: "pending", "shipped", "cancelled". | Property $schema is not allowed.
## $schema in the file, good document -> Property $schema is not allowed.
## $schema in the file points at a missing file -> Unable to load schema from 'mem://nope.schema.json': ENOENT: no such file or directory, open 'C:\lwdj\json\c7\nope.schema.json'.
## no $schema at all, nothing configured -> no problems

--- gen.mjs (json-schema-to-typescript) ---
import { compileFromFile } from 'json-schema-to-typescript'; import fs from 'node:fs';
const ts = await compileFromFile('order.schema.json', { bannerComment: '' });
fs.writeFileSync('order.d.ts', ts); console.log(ts);
const ts2 = await compileFromFile('shop.schema.json', { bannerComment: '' });
fs.writeFileSync('shop.d.ts', ts2); console.log('--- shop ---'); console.log(ts2);

Output:
export interface Order {
  id: number;
  customer: string;
  status: "pending" | "shipped" | "cancelled";
  date: string;
  items: Item[];
}
export interface Item {
  sku: string;
  qty: number;
  price: number;
}

--- shop ---
export interface CornerShop {
  shop: string;
  open: boolean;
  owner: {
    first: string;
    last: string;
    phone?: string | null;
    [k: string]: unknown;
  };
  /**
   * @minItems 1
   */
  products: [Product, ...Product[]];
  [k: string]: unknown;
}
export interface Product {
  sku: string;
  name: string;
  price: number;
  stock: number;
  tags?: string[];
  [k: string]: unknown;
}

--- use.ts (eight objects: five wrong in ways a type can catch, two in ways it cannot, one fine) ---
import type { Order } from './order.d.ts';
import type { CornerShop } from './shop.d.ts';

const ok: Order = { id: 1, customer: 'ada', status: 'pending', date: '2026-10-01', items: [{ sku: 'A1', qty: 2, price: 3.5 }] };
const badStatus: Order = { id: 1, customer: 'ada', status: 'lost', date: '2026-10-01', items: [] };
const missing: Order = { id: 1, customer: 'ada', status: 'pending', date: '2026-10-01' };
const extra: Order = { id: 1, customer: 'ada', status: 'pending', date: '2026-10-01', items: [], note: 'x' };
const idZero: Order = { id: 0, customer: '', status: 'pending', date: 'not a date', items: [{ sku: 'a1', qty: -3, price: 0 }] };
const idString: Order = { id: '1', customer: 'ada', status: 'pending', date: '2026-10-01', items: [] };
const shopExtra: CornerShop = { shop: 's', open: true, owner: { first: 'a', last: 'b', nick: 'x' }, products: [{ sku: 'A1', name: 'n', price: 1, stock: 1, anything: true }] };
const shopNoProducts: CornerShop = { shop: 's', open: true, owner: { first: 'a', last: 'b' }, products: [] };
console.log(ok, badStatus, missing, extra, idZero, idString, shopExtra, shopNoProducts);

Output of: npx tsc -p .   (Version 7.0.2, strict)
use.ts(5,52): error TS2322: Type '"lost"' is not assignable to type '"cancelled" | "pending" | "shipped"'.
use.ts(6,7): error TS2741: Property 'items' is missing in type '{ id: number; customer: string; status: "pending"; date: string; }' but required in type 'Order'.
use.ts(7,98): error TS2353: Object literal may only specify known properties, and 'note' does not exist in type 'Order'.
use.ts(9,27): error TS2322: Type 'string' is not assignable to type 'number'.
use.ts(11,95): error TS2322: Type '[]' is not assignable to type '[Product, ...Product[]]'.
  Source has 0 element(s) but target requires 1.

--- documentation: generate-schema-doc --config template_name=md order.schema.json docs/order.md (first 20 lines of the result) ---
# Order

- [1. Property `Order > id`](#id)
- [2. Property `Order > customer`](#customer)
- [3. Property `Order > status`](#status)
- [4. Property `Order > date`](#date)
- [5. Property `Order > items`](#items)
  - [5.1. Order > items > item](#items_items)
    - [5.1.1. Property `Order > items > items items > sku`](#items_items_sku)
    - [5.1.2. Property `Order > items > items items > qty`](#items_items_qty)
    - [5.1.3. Property `Order > items > items items > price`](#items_items_price)

**Title:** Order

|                           |             |
| ------------------------- | ----------- |
| **Type**                  | `object`    |
| **Required**              | No          |
| **Additional properties** | Not allowed |


--- compat.mjs (eleven schema changes, tested in both directions) ---
import { createRequire } from 'node:module'; const require = createRequire(import.meta.url);
const Ajv2020 = require('ajv/dist/2020');
const v = s => new Ajv2020({ strict: false }).compile(s);
const base = (over = {}, extraProps = {}, req = ['id', 'status'], closed = false) => ({ type: 'object', properties: { id: { type: 'integer', minimum: 1 }, status: { enum: ['pending', 'shipped'] }, ...extraProps, ...over }, required: req, ...(closed ? { additionalProperties: false } : {}) });
const O = { id: 1, status: 'pending' }, O2 = { id: 7, status: 'shipped' };
const changes = [
  // [label, old schema, new schema, old-style documents, new-style documents]
  ['add an OPTIONAL property (open schema)',      base(), base({}, { note: { type: 'string' } }),                 [O, O2],                     [O, { ...O, note: 'hi' }]],
  ['add an OPTIONAL property (closed schema)',    base({}, {}, ['id','status'], true), base({}, { note: { type: 'string' } }, ['id','status'], true), [O, O2], [O, { ...O, note: 'hi' }]],
  ['add a REQUIRED property',                     base(), base({}, { note: { type: 'string' } }, ['id', 'status', 'note']), [O, O2],            [{ ...O, note: 'hi' }]],
  ['make a required property optional',           base(), base({}, {}, ['id']),                                    [O, O2],                     [O, { id: 3 }]],
  ['add an enum value',                           base(), base({ status: { enum: ['pending', 'shipped', 'cancelled'] } }), [O, O2],            [O, { id: 2, status: 'cancelled' }]],
  ['remove an enum value',                        base(), base({ status: { enum: ['pending'] } }),                 [O, O2],                     [O]],
  ['widen a range (minimum 1 -> 0)',              base(), base({ id: { type: 'integer', minimum: 0 } }),           [O, O2],                     [O, { id: 0, status: 'pending' }]],
  ['narrow a range (minimum 1 -> 5)',             base(), base({ id: { type: 'integer', minimum: 5 } }),           [O, O2, { id: 2, status: 'pending' }], [O2]],
  ['widen a type (integer -> number)',            base(), base({ id: { type: 'number', minimum: 1 } }),            [O, O2],                     [O, { id: 1.5, status: 'pending' }]],
  ['narrow a type (number -> integer)',           base({ id: { type: 'number', minimum: 1 } }), base(),            [O, { id: 1.5, status: 'pending' }], [O, O2]],
  ['rename a property (id -> orderId)',           base(), { type: 'object', properties: { orderId: { type: 'integer', minimum: 1 }, status: { enum: ['pending', 'shipped'] } }, required: ['orderId', 'status'] }, [O, O2], [{ orderId: 1, status: 'pending' }]],
];
const pad = (s, n) => String(s).padEnd(n);
console.log(pad('change', 46) + pad('OLD documents vs NEW schema', 30) + 'NEW documents vs OLD schema');
console.log(pad('', 46) + pad('(backward compatible?)', 30) + '(forward compatible?)');
for (const [label, oldS, newS, oldDocs, newDocs] of changes) {
  const vo = v(oldS), vn = v(newS);
  oldDocs.forEach(d => { if (!vo(d)) throw new Error('corpus error: an "old" document is not valid under the old schema in: ' + label); });
  newDocs.forEach(d => { if (!vn(d)) throw new Error('corpus error: a "new" document is not valid under the new schema in: ' + label); });
  const back = oldDocs.filter(d => vn(d)).length + '/' + oldDocs.length + ' still valid';
  const fwd = newDocs.filter(d => vo(d)).length + '/' + newDocs.length + ' accepted';
  console.log(pad(label, 46) + pad(back + (oldDocs.every(d => vn(d)) ? '  YES' : '  NO'), 30) + fwd + (newDocs.every(d => vo(d)) ? '  YES' : '  NO'));
}

Output:
change                                        OLD documents vs NEW schema   NEW documents vs OLD schema
                                              (backward compatible?)        (forward compatible?)
add an OPTIONAL property (open schema)        2/2 still valid  YES          2/2 accepted  YES
add an OPTIONAL property (closed schema)      2/2 still valid  YES          1/2 accepted  NO
add a REQUIRED property                       0/2 still valid  NO           1/1 accepted  YES
make a required property optional             2/2 still valid  YES          1/2 accepted  NO
add an enum value                             2/2 still valid  YES          1/2 accepted  NO
remove an enum value                          1/2 still valid  NO           1/1 accepted  YES
widen a range (minimum 1 -> 0)                2/2 still valid  YES          1/2 accepted  NO
narrow a range (minimum 1 -> 5)               1/3 still valid  NO           1/1 accepted  YES
widen a type (integer -> number)              2/2 still valid  YES          1/2 accepted  NO
narrow a type (number -> integer)             1/2 still valid  NO           2/2 accepted  YES
rename a property (id -> orderId)             0/2 still valid  NO           0/1 accepted  NO

WHY THIS WORKS AS AN ANSWER
---------------------------
Error quality: the same wrong document gives 6 errors with a bare oneOf, 2 with an if/then chain and 1 with Ajv's discriminator option, so the schema style is a user-experience decision; Python's oneOf puts the useful error inside .context, and best_match did not pick it.
Editor: the language service reports the same keyword failures with line numbers, understood the 2020-12 keywords I tried (prefixItems, dependentRequired, unevaluatedProperties) even though VS Code's documentation says support is limited, did not report the bad
format: date, and flagged a $schema line inside the document as an extra property because the schema is closed. This is the engine, not a screenshot of the editor. Types: tsc caught enum, missing field, extra field, wrong type and empty non-empty list, and it could
not catch minimum, pattern or format violations, because a type cannot express them: validate real data at runtime as well. Compatibility: every row was computed by running sample documents through both schemas, and the corpus-error checks inside the
script would have stopped it if an 'old' document were not actually valid under the old schema. A handful of documents per change shows the pattern; it does not prove compatibility for a real schema.