fix(relay): encode literal review paths as a YAML tree [H3,H16,V8]

.github/workflows/release.yml

@@ -7,7 +7,7 @@ name: release
 on:
   push:
     tags: ['v*']
-  # The person, 27 Oct: GitHub CI is a second opinion. When it finds something, that becomes an item; it does not
+  # GitHub CI is a second opinion. When it finds something, that becomes an item; it does not
   # hold a release the board already proved. A tag whose Linux jobs failed publishes by hand through publish-proven.
   workflow_dispatch:
     inputs:

docs/formats.md

@@ -130,6 +130,33 @@ A signed board uses event-log format 2; older tools refuse it.
 
 A builder never picks an engine number (#1101). An item that adds a board move adds its own `src/<name>-engine.js` leaf as `export const NAME_ENGINE = unassignedEngine();` (importing `unassignedEngine` from `./engine-next.js`); on its branch the leaf reads as one above the highest landed engine. The landing merge writes the next number into the leaf, in landing order, inside that item's merge commit, and sets the `move engine` row above to the highest landed number. A landed number never changes, so recorded moves replay as they were written. Moves are recorded under landed numbers only: code with an unnumbered engine is for its branch's own tests, and a move it records on a real board before landing may later read under another item's number, so the item lands before its moves are used.
 
+The unassigned `COMPACT_VERDICT_REVIEW_ENGINE` leaf adds lossless wire-only YAML review-path trees (#14). A `verify` move uses `review.paths: {format: 'pullboard-review-path-yaml-v1', bytes, sha256, yaml}` only when the complete encoded object is smaller than the literal array. `yaml` is a string containing an actual YAML block mapping, not a JSON object relabeled as YAML. `bytes` and `sha256` bind the UTF-8 JSON of the complete original sorted unique path array. The reader bounds expansion to the existing 32 MiB Git-capture ceiling and checks the exact envelope fields, canonical tree, literal path grammar, expanded byte length and SHA-256 before verification. Stored reviews and snapshots still contain exact arrays: there are no globs, omitted paths, changed notes or relaxed freshness checks. This is not a universal guarantee that every review fits the relay request limit.
+
+After compatible readers are released and adopted, ordinary outbox flushing also compacts an older queued `verify` move's literal paths when that makes its sealed request smaller. It first pulls the ordered journal and settles any existing receipt by the original move ID. Only the unresolved move's wire copy changes: the ID, actor, criterion, source bindings, logical command intent, reverse captures and later queued moves remain intact. A lost acknowledgement is reconciled by the next pull, not by creating a replacement verdict. An unreadable journal leaves the queue unchanged. This is not permission to rewrite an already sealed `pending` send or replay an uncertain effect; those retain their existing custody and recovery rules.
+
+CLI/device processes that replay executable records, including devices that build presentation snapshots, need the landed feature engine before receiving the new representation; an older reader refuses before receipt or cursor advancement. The relay stores sealed records. Browser clients consume device-built presentation snapshots and stored literal arrays, not the Node executable move codec. Browser presentation compatibility remains a separate qualification obligation.
+
+The dependency-free reader accepts only the emitted YAML subset:
+
+- Each nonempty key is one literal path component, double-quoted with canonical `JSON.stringify` escapes. Additionally U+007F–U+009F, U+2028, U+2029, U+FFFE and U+FFFF are escaped as lowercase `\uNNNN` so YAML cannot normalize line breaks or reject non-printable characters. Slashes separate levels, never appear within keys, and punctuation/wildcards have no special meaning.
+- Two spaces indent each level; siblings are sorted by JavaScript string order. A branch ends in `:` and has children. An exact leaf ends in `: null`. LF terminates every line; the empty tree is exactly `{}\n` (one mapping and a trailing LF).
+- If both `node` and `node/child.txt` are reviewed (for example a file/directory transition across revisions), `node` is a branch with `"": null` as its first child. This empty-key marker means exactly the parent path, not the subtree. A branch with only that marker is noncanonical; use a normal leaf instead.
+- No unquoted keys, comments, document markers, aliases, anchors, tags, flow collections (except the empty root), implicit values, duplicate keys, empty branches or alternate escaping/indentation are accepted. Re-encoding must reproduce the exact tree; malformed or ambiguous input is refused rather than repaired. This is not a general YAML parser.
+
+For `['node', 'node.txt', 'node/child.txt', 'src/[literal]*?.js', 'src/line\nbreak.js']`, the `yaml` string is:
+
+```yaml
+"node":
+  "": null
+  "child.txt": null
+"node.txt": null
+"src":
+  "[literal]*?.js": null
+  "line\nbreak.js": null
+```
+
+Landing assigns this feature's engine number normally. Older CLI engines stop before receipt/cursor changes; the hosted browser currently requires exact engine equality and also stops. Release CLI and hosted reader assets together, with the hosted `/relay/engine.js` derived from the same package; do not serve this Node-only codec (`node:crypto` and the native engine leaf) as a browser module. The browser currently consumes presentation snapshots, not executable review args, so no codec import is added there. Browser runtime and mixed-version rollout still need release qualification. Already queued moves are not rewritten by this change; recovery must preserve their original logical intent and ordered identity.
+
 Engine 130 (`MECHANICAL_MERGE_ENGINE`) gives new automatic conflict returns version-two `computedConflict` receipts. Health, fresh conflict returns and landing share conservative mechanical decisions: structural JSON retains independent additions and unchanged-value removals, jointly raised numeric counters keep the larger value, and add-only text unions must pass a syntax-only check for JavaScript. Health classifies declared generated `docs/lifecycle.md` without executing item code; only landing runs its generator in the merged tree before its gate. A failed generator blocks with `LAND_MERGE`, naming the item, file and generator, rather than reopening accepted work as a conflict. Version-one receipts continue to verify their original plain-Git conflict paths on replay; version-two receipts verify mechanical conflict paths. The receipt version is part of the immutable-object cache identity. Fresh authors cannot reuse version-one semantics, and a version-two receipt requires engine 130 before changing a board or replay cursor.
 
 Relay transport capability `apiVersion: 2` is independent of these versions and

src/compact-verdict-review-engine.js

@@ -0,0 +1,3 @@
+/** Literal YAML review-path trees. Older engines stop before replay, never misread coverage. */
+import { unassignedEngine } from './engine-next.js';
+export const COMPACT_VERDICT_REVIEW_ENGINE = unassignedEngine();

src/compact-verdict-review.js

@@ -0,0 +1,120 @@
+/** Literal YAML block trees at the move boundary; stored review paths remain exact strings. */
+import { createHash } from 'node:crypto';
+import { Refused } from './refused.js';
+import { COMPACT_VERDICT_REVIEW_ENGINE } from './compact-verdict-review-engine.js';
+
+const FORMAT = 'pullboard-review-path-yaml-v1';
+// Same maximum as the existing Git path-capture reader, not a worker resource quota.
+const MAX_BYTES = 32 * 1024 * 1024;
+/** Measure the UTF-8 wire size, including non-ASCII filenames. */
+const bytes = text => new TextEncoder().encode(text).length;
+/** Bind the complete expanded JSON array, not just the tree's leaves. */
+const hash = text => createHash('sha256').update(text).digest('hex');
+/** Refuse without guessing coverage or exposing any path. */
+const bad = () => new Refused('BAD_VERDICT_REVIEW', 'the lossless review paths are invalid; retain the original move and restore its exact encoded paths');
+
+/** Match the native review validator's literal repository-relative path grammar. */
+function validPath(path) {
+  return typeof path === 'string' && !path.includes('\0') && path.split('/').every(part => part && part !== '.' && part !== '..');
+}
+
+/** Quote literal keys; escape YAML line separators and non-printable C1 characters too. */
+function quote(key) {
+  return JSON.stringify(key).replace(/[\u007f-\u009f\u2028\u2029\ufffe\uffff]/gu, c => `\\u${c.charCodeAt(0).toString(16).padStart(4, '0')}`);
+}
+
+/** Emit only a canonical YAML block-mapping subset, without recursive calls or prototype keys. */
+function yamlTree(paths) {
+  const root = { children: new Map(), leaf: false };
+  for (const path of paths) {
+    let node = root;
+    for (const part of path.split('/')) {
+      if (!node.children.has(part)) node.children.set(part, { children: new Map(), leaf: false });
+      node = node.children.get(part);
+    }
+    node.leaf = true;
+  }
+  const lines = [], stack = [...root.children.keys()].sort().reverse().map(key => ({ key, node: root.children.get(key), depth: 0 }));
+  while (stack.length) {
+    const { key, node, depth } = stack.pop(), indent = '  '.repeat(depth);
+    lines.push(`${indent}${quote(key)}:${node.children.size ? '' : ' null'}\n`);
+    if (node.children.size) {
+      // Empty component marks this exact path as well as its descendants (file/directory transitions).
+      if (node.leaf) lines.push(`${indent}  "": null\n`);
+      for (const child of [...node.children.keys()].sort().reverse())
+        stack.push({ key: child, node: node.children.get(child), depth: depth + 1 });
+    }
+  }
+  return lines.join('') || '{}\n';
+}
+
+/** Encode a sorted unique literal array without dropping or generalizing any path. */
+export function compactReviewPaths(paths) {
+  if (!Array.isArray(paths) || !paths.every(validPath) || JSON.stringify(paths) !== JSON.stringify([...new Set(paths)].sort())) throw bad();
+  const text = JSON.stringify(paths), size = bytes(text);
+  if (size > MAX_BYTES) throw bad();
+  return { format: FORMAT, bytes: size, sha256: hash(text), yaml: yamlTree(paths) };
+}
+
+/** Read only the emitted subset; never interpret general YAML, aliases, tags or implicit scalars. */
+function treePaths(yaml, limit) {
+  if (yaml === '{}\n') return [];
+  if (!yaml.endsWith('\n')) throw bad();
+  const paths = [], stack = [{ path: '', previous: null }];
+  let size = 2;
+  for (const line of yaml.slice(0, -1).split('\n')) {
+    const row = /^( *)("(?:[^"\\]|\\.)*"):( null)?$/u.exec(line);
+    if (!row || row[1].length % 2) throw bad();
+    const depth = row[1].length / 2;
+    if (depth >= stack.length) throw bad();
+    stack.length = depth + 1;
+    let key;
+    try { key = JSON.parse(row[2]); } catch { throw bad(); }
+    const parent = stack[depth], leaf = row[3] !== undefined;
+    if (quote(key) !== row[2] || parent.previous !== null && parent.previous >= key
+        || key.includes('/') || key === '' && (!leaf || depth === 0) || key !== '' && !validPath(key)) throw bad();
+    parent.previous = key;
+    const path = key === '' ? parent.path : parent.path ? `${parent.path}/${key}` : key;
+    if (leaf) {
+      size += bytes(JSON.stringify(path)) + (paths.length ? 1 : 0);
+      if (size > limit) throw bad();
+      paths.push(path);
+    } else stack.push({ path, previous: null });
+  }
+  return paths.sort();
+}
+
+/** Bound expansion and validate canonical shape, length and digest before native verification. */
+export function expandReviewPaths(value) {
+  if (!value || typeof value !== 'object' || Array.isArray(value)
+      || Object.keys(value).sort().join(',') !== 'bytes,format,sha256,yaml' || value.format !== FORMAT
+      || !Number.isSafeInteger(value.bytes) || value.bytes < 2 || value.bytes > MAX_BYTES
+      || typeof value.sha256 !== 'string' || !/^[0-9a-f]{64}$/u.test(value.sha256) || typeof value.yaml !== 'string') throw bad();
+  const paths = treePaths(value.yaml, value.bytes), text = JSON.stringify(paths);
+  if (bytes(text) !== value.bytes || hash(text) !== value.sha256 || yamlTree(paths) !== value.yaml) throw bad();
+  return paths;
+}
+
+/** Compact only when smaller; preserve the caller's args and every other move field. */
+export function compactReviewMove(move) {
+  if (move.operation !== 'verify' || !Array.isArray(move.args?.[1]?.review?.paths)) return move;
+  const paths = compactReviewPaths(move.args[1].review.paths);
+  if (bytes(JSON.stringify(paths)) >= bytes(JSON.stringify(move.args[1].review.paths))) return move;
+  return { ...move, engine: Math.max(move.engine, COMPACT_VERDICT_REVIEW_ENGINE), args: [move.args[0],
+    { ...move.args[1], review: { ...move.args[1].review, paths } }, ...move.args.slice(2)] };
+}
+
+/** Stop a YAML envelope mislabeled as an older feature engine before receipt or cursor writes. */
+export function requireCompactReviewEngine(move) {
+  const paths = move?.operation === 'verify' ? move.args?.[1]?.review?.paths : undefined;
+  if (paths !== undefined && !Array.isArray(paths) && move.engine < COMPACT_VERDICT_REVIEW_ENGINE)
+    throw new Refused('ENGINE_VERSION', 'compact review paths require the lossless review-path engine; do not replay or discard this move with an older reader');
+}
+
+/** Expand only wire paths; the board and its freshness projector still receive the original array. */
+export function expandReviewArgs(move, args) {
+  requireCompactReviewEngine(move);
+  if (move.operation === 'verify' && args[1]?.review && !Array.isArray(args[1].review.paths))
+    args[1].review.paths = expandReviewPaths(args[1].review.paths);
+  return args;
+}

src/engine.js

@@ -47,6 +47,11 @@ import {
 } from './submit-invariants.js';
 import { MERGE_ONLY_VERDICT_ENGINE } from './merge-only-verdict-engine.js';
 import { requireCoordinatorHold } from './coordinator-hold.js';
+import {
+  compactReviewMove,
+  expandReviewArgs,
+  requireCompactReviewEngine,
+} from './compact-verdict-review.js';
 
 import {
   MOVE_REGISTRY,
@@ -155,12 +160,13 @@ export function prepareEngineMove(board, operation, args, { id = randomUUID(), a
   if (operation === 'editMilestone' && values[1] && 'target' in values[1]) values[1].target = canonicalMilestoneTarget(values[1].target);
   const move = { version: 1, engine: ENGINE_VERSION, id, operation, args: values, ...(actor === undefined ? {} : { actor }) };
   // JSON is the wire format: optional undefined fields become absent on every replica alike.
-  return JSON.parse(JSON.stringify(move));
+  return JSON.parse(JSON.stringify(compactReviewMove(move)));
 }
 
 /** Stop before interpreting actors or operations that a newer engine may have changed. */
 export function requireSupportedEngine(move) {
   if (Number.isSafeInteger(move?.engine) && move.engine > ENGINE_VERSION) throw new Refused('ENGINE_VERSION', `move engine version ${move.engine} is newer than this pullboard engine version ${ENGINE_VERSION}; upgrade pullboard before applying the relay order`);
+  requireCompactReviewEngine(move);
 }
 
 /** Validate protocol identity before a future engine or malformed input can alter any replica. */
@@ -174,7 +180,7 @@ function validateMove(move) {
 
 /** Restore only the frozen criterion callback; no receiver runs another machine's Git or shell. */
 export function executableArgs(move, board) {
-  const args = structuredClone(move.args);
+  const args = expandReviewArgs(move, structuredClone(move.args));
   if (move.operation === 'rehomeItem' && args[1]?.sourceMoveId !== move.id) throw new Refused('RELAY_MOVE', 'a re-home source MOVEID does not match its sealed move');
   if (move.operation === 'appendFact') {
     if (!args[1] || typeof args[1] !== 'object' || Array.isArray(args[1])) throw new Refused('RELAY_MOVE', 'a fact needs its typed arguments; use the current Pullboard engine');

src/relay-outbox.js

@@ -20,6 +20,7 @@ import {
   gzipSync,
 } from 'node:zlib';
 import * as store from './board.js';
+import { compactReviewMove } from './compact-verdict-review.js';
 import {
   engineReceipt,
   executableArgs,
@@ -672,9 +673,12 @@ async function sendQueued(link, key) {
   let sent = 0;
   for (const entry of outbox?.moves ?? []) {
     if ((entry.refs ?? []).some((ref) => outbox.moves.some((other) => other.move.id === ref.add))) break;
-    const { move, lost } = resolveMove(entry, outbox);
+    const { move: queued, lost } = resolveMove(entry, outbox);
     if (lost.length) break;
-    if (localRecords(root, (board) => engineReceipt(board, move.id))) continue;
+    if (localRecords(root, (board) => engineReceipt(board, queued.id))) continue;
+    // Older writers queued literal paths. Compact only the unsent wire copy: the original intent and
+    // reverse captures remain exact, and an acknowledged move is settled by its existing receipt.
+    const move = compactReviewMove(queued);
     const sequence = outbox.baseSequence + sent + 1;
     const sealed = await sealedRecord(key, move, state, 'move', sequence);
     const registration = seatReceipt(move, sequence, sealed);