Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions docs/INVARIANTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,21 @@ The *rejection* half — "a rejected op leaves the doc untouched", stated in the
**Why:** Offline peers must create entities without a central allocator.
**Enforced by:** `insert_workflow` derives every carried ID inside the applier from the immutable `op_id`, scope, kind, and raw ID (ADR-031); producers send raw workflows and perform no allocation against document state. General producer randomness remains **UNGUARDED — see roadmap**.

**Logged exception (2026-09-22, ADR-033; amended and ratified 2026-09-23):**
a link's derived id is the ONE carried-ID kind whose numeric type
this rule's "53-bit random mint" clause is worth stating explicitly for.
ComfyUI_frontend's `LinkId` is a branded `number`, unlike the
string-or-number `NodeId`, so `insert_workflow`'s usual pure string
derivation cannot satisfy it; `remap.ts`'s `derivedLinkId` mints a candidate
purely from the op's own content (SHA-256 of the ADR-031 seed, folded into
the full `[1, Number.MAX_SAFE_INTEGER]` range) with **no document read and
no retry** — an earlier revision retried against `doc.ts`'s
`persistedLinkIds`, which review found made link identity depend on local
arrival order (KA-2), so that retry and `persistedLinkIds` were both
removed. See `docs/decisions/EXCEPTIONS.md`'s KA-5 row and ADR-033's
"Amendment (2026-09-23)" section for the full reasoning, the quantified
residual collision probability, and the ratification record.

### KA-6 — Raw struct updates flow host → follower one-way only
**Rule:** Followers never write the shared doc.
**Why:** Reverse-flow Yjs structs from independently edited docs can corrupt or diverge state.
Expand Down
260 changes: 260 additions & 0 deletions docs/decisions/ADR-033-insert-workflow-numeric-link-ids.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs/decisions/EXCEPTIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,4 @@ is retained as history, not authorization to accept unprojectable writes.
| 2026-08-21 (PROPOSED) | KA-11 | **The snapshot read surface's fail-closed gate is CONDITIONAL**: `src/read.ts` refuses a document carrying content under an unreadable schema, but returns its empty value for a document carrying nothing, where `project()` refuses that document too. Deliberate — ADR-004's follower reads a root-less document before its first frame and there is no content there to mis-key — but it is a read path that answers where a sibling read path throws, so it is recorded rather than left in source comments. Normative text: Amendment A12. The divergence is disposition-only; both paths use `readSchemaVersion` as the predicate. Mitigated by asking "carries content" against the struct store rather than the v1 root names, so a v2 that renamed its roots is refused rather than read as empty. | `src/read.ts` `assertSnapshotReadable`/`carriesContent`; schema §10 + Amendment A12 | Owner: UNASSIGNED. Sunset PROPOSED, not agreed: revisit when `SCHEMA_VERSION` first moves past 1, since a real v2 makes the disposition split observable to a deployed follower for the first time. |
| ~~2026-08-27 (PROPOSED)~~ **RESOLVED 2026-08-28** | KA-3 | ~~**A promoted `connect` (`grow.promoted: true`, schema Amendment A15) is LWW-gated here on `("input", to_node, "grow", name)` and moves the materialized slot's `grow_id` to the winning link; comfy-cli's `_apply_connect` gates neither.**~~ **Sunset condition met: comfy-cli PR #818 (amendment v1.5, `ba0b0b92abcc86b01e8a6704d07088f92afe7aa7`) gates the same register, spelled with the FULL declared name on both sides, and moves `grow_id` to the winner.** Struck rather than deleted so the exception's life stays legible. Both spell the register identically (`_write_target`), so the divergence is in GATING only and is observable only under concurrency comfy-cli's single-writer apply never sees — the corpus session it generated replays green. Gating is required for §2.5 convergence (two connects into one declared input contend for one slot exactly as two concrete connects do, A1). | `src/applier.ts` `claimPromotedInput`; schema Amendment A15 | Owner: UNASSIGNED. Sunset PROPOSED, not agreed: comfy-cli `_lww_gate`s promoted grows on the same register (asked for in the PR that lands A15). |
| 2026-09-21 (APPROVED) | FC-10 | `set_node_field` is a package-local, provisional operation because post-creation title and flag changes otherwise have no non-clobbering write path. The owner requested consolidation on the general per-field architecture rather than the competing title-only operation. Its field set and value types are closed, its incarnation semantics match `set_widget`, and a language-neutral conformance session is included here; the upstream comfy-cli vocabulary has not yet adopted the kind. | `src/types.ts`, `src/applier.ts`, `src/stamps.ts`, `src/collab-trace.ts`; schema Amendment A21; ADR-032 | Owner: Christian Byrne. Sunset: amend comfy-cli's frozen vocabulary and pin the adopted definition here, reconciling the name/payload if upstream differs. |
| ~~2026-09-22 (APPROVED)~~ **AMENDED AND APPROVED 2026-09-23** | KA-5 | ~~`insert_workflow`'s numeric link-id mint consulted mutable document state (a document-verified retry), unlike every other id this op derives.~~ Christian Byrne's review of that design (PR #245, inline on `src/remap.ts:65`) found it made link identity depend on LOCAL ARRIVAL ORDER — two replicas applying the same concurrent op set in opposite orders could project different link ids for the same logical links, the exact class of bug KA-4/KA-2 exist to rule out — and asked for either a fix plus explicit ratification, or a return to pure derivation. **This row now describes the amended design: `derivedLinkId` mints PURELY from the op's own content (`opId`, `scope`, the raw link), with NO document read, no caller-supplied reservation, and no retry.** It spends the FULL safe-integer range (`[1, Number.MAX_SAFE_INTEGER]`, 53 bits, widened from the original's 52-bit window) rather than checking for a collision. ComfyUI_frontend's `LinkId` (a branded `number`, unlike the string-or-number `NodeId`) is still why a link needs a numeric mint at all — ADR-031's pure STRING scheme still can't satisfy it — but the mint no longer tries to guarantee collision-freedom against the document; it accepts a quantified residual risk instead. Residual probability: two independently derived 53-bit candidates collide with probability `1/(2^53-1) ≈ 1.11×10⁻¹⁶` per pair; for `k` links minted into one document the birthday-bound chance ANY two collide is `≈ k²/2^54` — about `2.2×10⁻¹²` at `k=200` (this package's own scale test), about `5.5×10⁻⁵` even at an unrealistic `k=1,000,000` — the same order of "cosmically unlikely, not impossible" risk this package already accepts for UUID4 `op_id` collisions (KA-2). A REALIZED collision is not specially detected: it is caught by `applier.ts`'s pre-existing `acceptsLink`/`seenLinks` apply-time id-collision handling (the same ordinary, order-independent LWW-gated check every other `insert_workflow` id kind already goes through), rejecting the whole op `link_id_collision` byte-identically rather than silently reassigning a different id — this check is REACHABLE via realistic input now (not merely via deliberate id-space exhaustion, since there is no more exhaustion to construct). **Separately, a pre-existing, unresolved gap** (out of scope for this amendment): a `connect` op's client-minted `link_id` can collide with an `insert_workflow`-derived numeric id, and `applier.ts`'s `claimLinkIdentity` does not guard against it at all, silently overwriting the `insert_workflow` link when `connect` arrives second — this predates and is unaffected by this amendment either way (the removed retry only ever protected the OPPOSITE arrival order, and only when the `connect` had already landed on that replica). See ADR-033's "Amendment (2026-09-23)" section for the full writeup, including this cross-mechanism note. | `src/remap.ts` `derivedLinkId`; `src/applier.ts` `prepareInsertedWorkflow`, `acceptsLink`, `claimLinkIdentity`; ADR-033 | Owner: Christian Byrne. The amended pure-derivation design was ratified after exact-head review confirmed that the document-state retry was removed, opposite-order convergence coverage was added, and realized collisions remain loud and byte-identical. Sunset (unchanged from the original row): revisit only if ComfyUI_frontend widens `LinkId` to `string \| number` (matching `NodeId`), at which point ADR-031's string form becomes sufficient for links too and this exception can be retired outright. |
24 changes: 20 additions & 4 deletions src/applier.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1007,7 +1007,16 @@ function prepareInsertedWorkflow(op: InsertWorkflowOp): Record<string, unknown>
}
validateRawGraphIds(wf["nodes"] as unknown[], (wf["links"] as unknown[] | undefined) ?? [], "workflow");
validateDefinitionInputs((subgraphs as unknown[] | undefined) ?? []);
return remapInsertedWorkflowIds(wf as unknown as import("./types.js").WorkflowJSON, op.op_id) as unknown as Record<string, unknown>;
// ADR-033 (amended): every id this op derives, including a link's numeric
// id, is a PURE function of the op's own content — no document state is
// read here. See `remap.ts`'s `derivedLinkId` for why `LinkId` (a branded
// `number`, unlike the string-or-number `NodeId`) needed a numeric mint at
// all, and for why the mint no longer checks the target document (KA-5
// exception, `docs/decisions/EXCEPTIONS.md`).
return remapInsertedWorkflowIds(
wf as unknown as import("./types.js").WorkflowJSON,
op.op_id,
) as unknown as Record<string, unknown>;
}

function applyInsertWorkflow(doc: Y.Doc, op: InsertWorkflowOp, catalog?: WidgetCatalog): SuccessfulOutcome {
Expand Down Expand Up @@ -1177,9 +1186,16 @@ function updateInsertedWorkflowMeta(
const currentNode = numericId(meta.get("last_node_id")) ?? 0;
const maxNode = Math.max(currentNode, ...nodeWrites.map(([, id]) => numericId(id) ?? currentNode));
if (maxNode > currentNode) mset(meta, "last_node_id", maxNode);
const currentLink = numericId(meta.get("last_link_id")) ?? 0;
const maxLink = Math.max(currentLink, ...linkWrites.map((link) => numericId(link[0]) ?? currentLink));
if (maxLink > currentLink) mset(meta, "last_link_id", maxLink);
// `last_link_id` is deliberately NOT advanced from `linkWrites` (KA-5:
// "document high-water marks are advisory, never allocators"). Every
// inserted link's own id is now a large, arbitrary-looking derived number
// (ADR-033 — `remap.ts`'s `derivedLinkId`, needed only because ComfyUI_
// frontend's `LinkId` is a branded `number`), not a small sequential one,
// so folding it into the high-water mark would pin the doc's bookkeeping
// field to that value forever after the FIRST insert_workflow — a new,
// surprising side effect this line never had before (the pre-fix STRING
// derived id already failed `numericId` and left this a no-op for every
// inserted link).
}

// ---------------------------------------------------------------------------
Expand Down
83 changes: 78 additions & 5 deletions src/remap.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,65 @@ function derivedDefinitionId(opId: string, scope: string, original: unknown): st
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-4${hex.slice(13, 16)}-8${hex.slice(17, 20)}-${hex.slice(20, 32)}`;
}

/**
* A newly derived link id must be a real `number` — ComfyUI_frontend's
* `LinkId` is `number & { __brand }`, unlike `NodeId` (`string | number`), so
* the string form `derivedId` gives every other kind cannot satisfy it
* (ComfyUI_frontend#18458).
*
* `derivedLinkId` folds the digest into the FULL safe-integer range
* (`[1, Number.MAX_SAFE_INTEGER]`, i.e. `[1, 2^53 - 1]`) rather than the
* narrower 52-bit window an earlier revision used, so the mint spends every
* bit of entropy `LinkId`'s numeric type can safely hold (ADR-033 amendment,
* review discussion on PR #245). 32 hex characters (128 bits) are taken from
* the digest before the modulus is applied so the reduction to 53 bits stays
* close to uniform; sha256Hex's 64-character output has plenty to spare.
*/
const LINK_ID_HEX_DIGITS = 32; // 128 bits, folded down to 53 by SAFE_INTEGER_MODULUS below.

/** `2^53 - 1` as a `BigInt`, i.e. `Number.MAX_SAFE_INTEGER` — every candidate lands in `[1, MAX_SAFE_INTEGER]`. */
const SAFE_INTEGER_MODULUS = BigInt(Number.MAX_SAFE_INTEGER);

/**
* Mint a numeric id for a link `insert_workflow` carries — PURELY as a
* function of the op's own content (`opId`, `scope`, `original`), with NO
* document-state dependency (ADR-033 amendment, superseding the
* document-verified retry mint the original revision of this PR shipped).
*
* Christian Byrne's review of the original revision (PR #245, inline on this
* file) found that the retry-against-`taken` design made link identity
* depend on LOCAL ARRIVAL ORDER: if two link seeds shared the same first
* candidate, applying them A-then-B minted A the natural candidate and B the
* next one, while applying B-then-A minted B the natural candidate and A the
* next one — two replicas that apply the same op set in different orders
* could project genuinely different link ids for the same logical links,
* which is exactly the class of bug KA-4 (deterministic/idempotent
* projection) and KA-2 (order-independent identity) exist to rule out.
*
* The fix removes the retry (and the document read it required) entirely:
* the SAME op, applied to ANY document state, in ANY order relative to any
* other op, always derives the SAME numeric id — because nothing but the
* op's own fields feeds the hash. This restores ADR-031's "producers do not
* inspect document state or remap ids" for links too, at the cost of the
* "checked collision-free against this document" property the retry design
* offered: two derived ids (or a derived id and something already persisted)
* can now coincide, with probability bounded by the ~53 bits of entropy
* below (see the KA-5 row in `docs/decisions/EXCEPTIONS.md` for the exact
* figure and why it is an accepted residual risk rather than a defect).
* `applier.ts`'s existing `acceptsLink` still catches a REALIZED collision
* against an already-persisted link at apply time (the same LWW-gated check
* every other `insert_workflow` id kind already goes through) — that check
* is ordinary id-collision handling this package has always had, not new
* state-dependence introduced here, and it resolves by the op-order-
* independent `[base_version, actor, op_id]` stamp comparison, not by
* arrival order.
*/
function derivedLinkId(opId: string, scope: string, original: unknown): number {
const seed = derivedId(opId, scope, "link", original);
const hex = sha256Hex(seed).slice(0, LINK_ID_HEX_DIGITS);
return Number((BigInt(`0x${hex}`) % SAFE_INTEGER_MODULUS) + 1n);
}

function linkEndpoints(link: unknown): [unknown, unknown] | undefined {
if (Array.isArray(link) && link[1] !== undefined && link[3] !== undefined) return [link[1], link[3]];
if (typeof link === "object" && link !== null) {
Expand Down Expand Up @@ -51,7 +110,7 @@ export function linkHasMissingEndpoint(link: unknown, hasNode: (id: unknown) =>
return endpoints !== undefined && (!hasNode(endpoints[0]) || !hasNode(endpoints[1]));
}

function remapInputs(inputs: unknown, linkIds: Map<string, string>, droppedLinkIds: Set<string>): void {
function remapInputs(inputs: unknown, linkIds: Map<string, number>, droppedLinkIds: Set<string>): void {
if (!Array.isArray(inputs)) return;
for (const input of inputs) {
if (typeof input !== "object" || input === null || !("link" in input)) continue;
Expand All @@ -73,7 +132,7 @@ function remapInputs(inputs: unknown, linkIds: Map<string, string>, droppedLinkI
function remapLinkIdArray(
records: unknown,
field: string,
linkIds: Map<string, string>,
linkIds: Map<string, number>,
droppedLinkIds: Set<string>,
): void {
if (!Array.isArray(records)) return;
Expand Down Expand Up @@ -121,7 +180,7 @@ function remapGraph(
): void {
const nodes = graph["nodes"] as WorkflowNode[];
const nodeIds = new Map<string, string>();
const linkIds = new Map<string, string>();
const linkIds = new Map<string, number>();
// Numeric and string aliases normalized by validation name the same node.
for (const node of nodes) nodeIds.set(normalizedId(node.id), derivedId(opId, scope, "node", node.id));
const droppedLinkIds = new Set<string>();
Expand All @@ -134,7 +193,7 @@ function remapGraph(
});
for (const link of links) {
const id = linkId(link);
linkIds.set(normalizedId(id), derivedId(opId, scope, "link", id));
linkIds.set(normalizedId(id), derivedLinkId(opId, scope, id));
}

for (const node of nodes) {
Expand Down Expand Up @@ -175,7 +234,21 @@ function remapGraph(
}
}

/** Deterministically remap every id carried by an insertion op, including nested definition graphs. */
/**
* Deterministically remap every id carried by an insertion op, including
* nested definition graphs.
*
* Every id this function derives — node, link, group, and definition — is a
* PURE function of (`opId`, `scope`, `kind`, `original`) alone (ADR-031,
* amended for links by the ADR-033 revision on PR #245). No document, no
* caller-supplied reservation set, and no state from any other call factors
* into the result: the same `(wf, opId)` pair always produces byte-identical
* output, on any replica, regardless of what else that replica's document
* holds or what order concurrent ops arrive in. This is what makes the
* result safe to compute before touching the target document at all —
* `applier.ts` calls this once per `insert_workflow` application with no
* document in scope yet.
*/
export function remapInsertedWorkflowIds(wf: WorkflowJSON, opId: string): WorkflowJSON {
const out = structuredClone(wf) as WorkflowJSON & Record<string, unknown>;
out.links ??= [];
Expand Down
Loading
Loading