diff --git a/docs/INVARIANTS.md b/docs/INVARIANTS.md index d73106f..b9fda1b 100644 --- a/docs/INVARIANTS.md +++ b/docs/INVARIANTS.md @@ -43,6 +43,21 @@ The *rejection* half — "a rejected op leaves the doc untouched", stated in the **Why:** Offline peers must create entities without a central allocator. **Enforced by:** `insert_workflow` derives every carried ID inside the applier from the immutable `op_id`, scope, kind, and raw ID (ADR-031); producers send raw workflows and perform no allocation against document state. General producer randomness remains **UNGUARDED — see roadmap**. +**Logged exception (2026-09-22, ADR-033; amended and ratified 2026-09-23):** +a link's derived id is the ONE carried-ID kind whose numeric type +this rule's "53-bit random mint" clause is worth stating explicitly for. +ComfyUI_frontend's `LinkId` is a branded `number`, unlike the +string-or-number `NodeId`, so `insert_workflow`'s usual pure string +derivation cannot satisfy it; `remap.ts`'s `derivedLinkId` mints a candidate +purely from the op's own content (SHA-256 of the ADR-031 seed, folded into +the full `[1, Number.MAX_SAFE_INTEGER]` range) with **no document read and +no retry** — an earlier revision retried against `doc.ts`'s +`persistedLinkIds`, which review found made link identity depend on local +arrival order (KA-2), so that retry and `persistedLinkIds` were both +removed. See `docs/decisions/EXCEPTIONS.md`'s KA-5 row and ADR-033's +"Amendment (2026-09-23)" section for the full reasoning, the quantified +residual collision probability, and the ratification record. + ### KA-6 — Raw struct updates flow host → follower one-way only **Rule:** Followers never write the shared doc. **Why:** Reverse-flow Yjs structs from independently edited docs can corrupt or diverge state. diff --git a/docs/decisions/ADR-033-insert-workflow-numeric-link-ids.md b/docs/decisions/ADR-033-insert-workflow-numeric-link-ids.md new file mode 100644 index 0000000..b36945d --- /dev/null +++ b/docs/decisions/ADR-033-insert-workflow-numeric-link-ids.md @@ -0,0 +1,260 @@ +# ADR-033: numeric link ids for `insert_workflow` + +- **Status:** Amended and accepted — see "Amendment (2026-09-23)" below. The original decision (2026-09-22, below) shipped a document-state-verified retry mint; Christian Byrne's review of that revision (PR #245) found it broke link identity's order-independence and asked for either a fix plus explicit sign-off on the residual risk, or a return to pure derivation. This amendment takes the pure-derivation path and was ratified after exact-head review of that fix and its opposite-order convergence coverage. **The "Accepted" disposition below describes the ORIGINAL decision only** and is superseded by this accepted amendment. +- **Date:** 2026-09-22 (original), amended 2026-09-23 +- **Decider:** Christian Byrne +- **Source:** ComfyUI_frontend#18458 (interim 32-bit-hash review) and the Slack thread it was raised in; the amendment's source is Christian Byrne's PR #245 review (inline comment on `src/remap.ts:65`, review summary "Not merge-ready: link identity becomes arrival-order-dependent"). + +## Context + +ADR-031 remaps every id `insert_workflow` carries — node, link, group, and +nested-definition id — to a deterministic **string** derived from the op +envelope id, graph scope, id kind, and original id +(`insert:::link:`). That is a correct fit for a +`NodeId` (`string | number` on the frontend) but not for a `LinkId`, which +ComfyUI_frontend brands as `number & { __brand: 'LinkId' }`. The frontend's +interim fix (`ecsFollowerAdapter.ts`'s `resolveLinkId`) hashed the derived +string into a number with FNV-1a folded into a 32-bit space. Two independent +reviewers (Christian Byrne, dante01yoon) found this genuinely broken, not +merely approximate: + +- The 32-bit hash space is small enough that two distinct derived ids collide + at a measured, reproducible rate (~0.01% at 1k live derived links, ~1% at + 10k, birthday-bound), silently replacing one link with another with no + error reported — indistinguishable from the original "links silently + dropped" bug this was fixing. +- The resolved numeric id is what `graph.serialize()` writes to the saved + workflow file. A reserved id range chosen to sit above any real + `last_link_id` does not survive a save/reload: the persisted file now + contains a genuine numeric link id inside that "reserved" range, and + nothing bounds real link ids below it, so a LATER `insert_workflow` can + collide with a real, previously-synthetic-now-real id. A session-local + allocator has the same reload hole from the opposite direction: a fresh + session's reservation map is empty and does not know the reloaded id is + taken. + +Both reviewers converged on the same root fix: mint the numeric identity +where the link is minted (this package), not on every frontend peer that +happens to read it. + +## Decision + +`remap.ts`'s `derivedLinkId` mints a real JS `number` for every link id +`insert_workflow` derives — at the top-level graph scope AND at every +subgraph-definition interior scope, so the fix does not regress +comfy-multi-player#230's `remapLinkIdArray`/definition-interior handling. +Every other kind `insert_workflow` derives (node, group, definition) is +UNCHANGED: still the ADR-031 string, still a pure function of +(`opId`, `scope`, `kind`, `original`) with no document-state dependency. + +The link-id mint is NOT a wider hash gambled to be "unlikely enough" — the +same failure mode being fixed. It is: + +1. **Hashed**: SHA-256 of the same `insert:::link:` + seed ADR-031 already uses, keeping 52 bits (well inside + `Number.MAX_SAFE_INTEGER`) as the first candidate. +2. **Verified against every id the caller can see**, not assumed collision-free + by width alone: the caller (`applier.ts`) passes `derivedLinkId` (via + `remapInsertedWorkflowIds`'s `reservedLinkIds` parameter) every numeric + link id already persisted in the target document — top-level AND every + subgraph definition's interior links, at every nesting depth + (`doc.ts`'s `persistedLinkIds`) — plus every id `derivedLinkId` has already + minted earlier in the SAME `insert_workflow` call, so two links in one + insertion can never collide with each other either. +3. **Retried, bounded**: on a collision the mint deterministically tries the + next candidate (re-hashing with an incremented attempt counter) up to + `MAX_LINK_ID_MINT_ATTEMPTS` (256) times. Any realistic document — even one + with millions of live links — succeeds on the first or second attempt; + the bound exists so a pathological/adversarial reservation set fails + loudly (`link_id_collision`, byte-identical, `op_id` not consumed) in + bounded work rather than looping. + +This directly fixes the reviewers' reproduction: two independent +`insert_workflow` ops (distinct `op_id`s) inserting links that would +otherwise hash to the same value now resolve to distinct ids, because the +second op's mint sees the first op's already-committed numeric id in +`persistedLinkIds` and retries past it — the exact cross-op collision the +32-bit interim hash reproduced. + +## Deviation from KA-5 / ADR-031 + +Every other id `insert_workflow` derives is intentionally **pure**: a +function of the op's own fields alone, with producers/the applier never +inspecting document state to decide an id (KA-5's "IDs are collision-free +without coordination" and ADR-031's "Producers do not inspect document +state or remap ids"). `derivedLinkId` breaks that purity for links, and only +for links: a truly information-theoretic collision-free encoding of an +unbounded set of possible (`opId`, link) pairs into a fixed 53-bit numeric +space is impossible by construction (pigeonhole), unlike the string form, +whose output space is unbounded and therefore never needs to consult +anything besides the op itself. Squeezing that identity into a small, +externally-imposed numeric type (`LinkId`) is only possible by ALSO checking +it against real state — so this package does, rather than pretending a wide +hash is "collision-free" the way the interim frontend fix did. This is +logged as a deliberate exception in `docs/decisions/EXCEPTIONS.md` (KA-5 row), +with the residual gap named explicitly: two REPLICAS that have never +exchanged this op and each independently mint a colliding candidate from +their own (different) locally-visible document state could in principle +diverge — bounded by the same ~2^-52-per-pair probability that makes a UUID4 +`op_id` collision (already a foundational assumption throughout this +package, KA-2) an accepted risk elsewhere in this same system. + +## Consequences + +- ComfyUI_frontend can delete its `resolveLinkId`/`fnv1a` hashing entirely + (tracked in ComfyUI_frontend#18458) and read `LinkId` straight off the doc. +- `link_id_collision`'s existing insert-time collision check + (`applier.ts`'s `acceptsLink`) is no longer reachable via any + realistically-shaped input for LINKS specifically, because the mint now + actively avoids everything that check would have caught. It remains + reachable, and is retained, via the id-space-exhaustion path above; the + test suite covers exhaustion directly rather than asserting an + unreachable natural collision. +- `updateInsertedWorkflowMeta`'s `last_link_id` high-water-mark bookkeeping + deliberately does NOT fold in these new large numeric ids (see the comment + at its call site) — they are not meant to look like the next sequential + id, and KA-5 already treats this field as advisory only. +- `SCHEMA_VERSION` is unchanged: no document root or reserved key changes, + only the VALUES `insert_workflow` was already free to choose for a link id. + +## Amendment (2026-09-23): pure derivation, no document-state retry + +**This amendment supersedes the "Decision" and "Deviation from KA-5 / ADR-031" +sections above for the CURRENT behavior.** They are kept as written for +history — they are what actually shipped in the original PR #245 revision — +but `derivedLinkId` no longer works the way they describe. + +### What review found + +Christian Byrne's inline blocker on `src/remap.ts:65`: + +> Blocker: `taken` makes identity depend on local arrival order. If two link +> seeds share the first candidate, A→B yields A=x/B=y while B→A yields +> B=x/A=y, so replicas applying the same op set can project different link +> IDs. Please add an opposite-order convergence case and obtain explicit +> owner ratification for the collision-free ID invariant (KA-5) exception, +> or retain pure derivation. + +This is exactly right, and it is a sharper failure than the interim frontend +hash this ADR originally set out to fix: the retry-against-`taken` design +made the FINAL numeric id a function of arrival order whenever two link +seeds' natural (zeroth) candidates coincided, which the original "Decision" +section's item 2 ("verified against every id the caller can see") describes +as a feature rather than naming as this cost. + +### The amended decision + +`derivedLinkId` (`src/remap.ts`) mints PURELY from the op's own content — +`opId`, `scope`, and the raw link's original id/tuple — with **no document +read, no caller-supplied reservation set, and no retry**. `remapInsertedWorkflowIds` +no longer accepts a `reservedLinkIds` parameter; `applier.ts`'s +`prepareInsertedWorkflow` no longer calls `doc.ts`'s (now-deleted) +`persistedLinkIds`. The same `(wf, opId)` pair always produces byte-identical +output — including every link's numeric id — on any replica, in any +document state, regardless of what other ops have or have not been applied +yet, at the top-level scope AND at every subgraph-definition interior scope. +This restores ADR-031's "producers do not inspect document state or remap +ids" for links, closing the gap this ADR's original "Deviation" section +opened. + +The mint also widens its candidate from 52 to the FULL safe-integer range: +it hashes the ADR-031 seed with SHA-256, takes 32 hex characters (128 bits, +comfortably enough to keep the reduction below close to uniform), and folds +that down via `% (Number.MAX_SAFE_INTEGER)` into `[1, Number.MAX_SAFE_INTEGER]` +— `[1, 2^53 - 1]`, all of it, rather than the narrower `2^52` window the +original revision used. `Number.isSafeInteger` (used elsewhere in this +package, e.g. `doc.ts`'s deleted `persistedLinkIds` and `applier.ts`'s +`numericId`) accepts exactly this range, and it round-trips losslessly +through `JSON.stringify`/`JSON.parse` and Yjs storage, which is what a +saved-and-reloaded workflow file and a `Y.Map` value both need. + +### Residual collision probability + +With `N = 2^53 - 1 = 9,007,199,254,740,991` possible candidates and +(to first order, since the modulus reduction is close to uniform) two +INDEPENDENTLY derived candidates, the probability that a specific pair +collides is `1/N ≈ 1.11 × 10⁻¹⁶`. For `k` derived link ids all drawn from +this space, the birthday-bound probability that ANY two collide is +approximately `k² / (2N)`. At `k = 200` (this package's own scale test) that +is about `2.2 × 10⁻¹²`; at `k = 1,000,000` (unrealistically many links in one +document) it is about `5.5 × 10⁻⁵` — still far below anything a real +workflow reaches. This is the same order of magnitude this package already +accepts for a UUID4 `op_id` collision (KA-2, 122 bits of entropy per id, +so an equivalent `k²/(2N)` bound at real-world `k` is smaller still, but +both are "cosmically unlikely, not impossible" risks this package already +lives with elsewhere). + +A REALIZED collision — a freshly derived link id that equals one already +persisted in the target document, or one already derived earlier in the +SAME `insert_workflow` call — is not specially detected or avoided by the +mint (that would reintroduce the document read this amendment removes). It +is caught, if and when it happens, by the ordinary apply-time id-collision +handling every other `insert_workflow` id kind already goes through +(`applier.ts`'s `acceptsLink`, and the `seenLinks` intra-op duplicate check +ahead of it) — the whole op is rejected `link_id_collision`, byte-identically +and without consuming the `op_id` (KA-4), rather than the op silently +picking a different id for the colliding link. This trades a vanishingly +small chance of a LOUD, safe rejection for the arrival-order-dependent +SILENT divergence the retry design risked — the right trade, since a loud +failure that never fires in practice is strictly better than a silent one +that can. + +### Cross-mechanism interaction with `connect` (a separate, pre-existing risk) + +A `connect` op's client-minted `link_id` can, in principle, coincide with a +numeric id `insert_workflow` derives for an unrelated link — a SEPARATE risk +from the one this amendment addresses, raised independently in the Slack +thread linked above. `applier.ts`'s `claimLinkIdentity` (the `connect` +register) does not check for this at all: it deletes and replaces whatever +currently occupies that numeric key with no LWW gate when no prior +`("link", …)`-register stamp exists for it, which is always true for a key +an `insert_workflow` link occupies (that link's stamp lives under a +DIFFERENT register, `("insert_workflow_link", key)`). This asymmetry +predates this ADR and this amendment equally: the ORIGINAL document-verified +retry mint only ever avoided this collision in the one arrival order where +the `connect` had already landed before the `insert_workflow` mint ran on +that replica (in which case `persistedLinkIds` saw it and the retry skipped +past it, applying `insert_workflow` cleanly with a different id); in the +opposite order (`insert_workflow` lands first, `connect` arrives later) the +retry design offered no protection at all, because it cannot see a future +op's client-chosen id — `claimLinkIdentity` would silently overwrite the +`insert_workflow` link exactly as it does today. That is itself a real, +order-dependent divergence between replicas that was already present before +this amendment. + +The amendment does not fix this — it is explicitly out of scope for this PR, +and fixing it would mean changing `claimLinkIdentity` to recognize an +`insert_workflow`-sourced incumbent, not anything in `remap.ts`. What the +amendment changes is narrower: removing the retry removes the one arrival +order (`connect`-then-`insert_workflow`) where the OLD design offered +partial, order-dependent protection, so that order now also rejects loudly +(`link_id_collision`) rather than silently succeeding with a different id. +The genuinely dangerous order (`insert_workflow`-then-`connect`, silent +overwrite) is unaffected by this amendment either way, and both orders +require an engineered or astronomically unlucky collision between a +client-chosen `link_id` and a 53-bit hash output to occur at all. This is +recorded here, honestly, as a known, unresolved, pre-existing gap — not +claimed as fixed and not newly introduced by this amendment. + +### Consequences of the amendment + +- The "Consequences" section's `acceptsLink`/`link_id_collision`-reachability + bullet above is corrected: that check IS reachable via realistic input + now (any real collision reaches it, since nothing avoids one), not merely + via deliberate id-space exhaustion — there is no more exhaustion to + construct, because there is no more retry sequence. +- `test/insert-workflow.test.ts`'s "numeric link id minting (ADR-033, pure + derivation)" block and + `test/insert-workflow-numeric-link-ids.regression.test.ts`'s + "opposite-order convergence" block replace the retry/exhaustion-focused + tests the original revision added, per Christian's request for an + opposite-order convergence case. +- The **Glossary**'s "Reservation set" entry below no longer applies to the + shipped mint; it is kept as history since the "Decision" section above + still uses the term. + +## Glossary + +- **Derived id:** ADR-031's deterministic string, `insert::::`. +- **Candidate:** the single hashed numeric value `derivedLinkId` computes for one link (post-amendment: there is exactly one, never retried). +- **Reservation set (historical, pre-amendment only):** the ids the original retry-based mint call was required not to choose — persisted document ids plus ids already minted in the same call. Not part of the amended design. diff --git a/docs/decisions/EXCEPTIONS.md b/docs/decisions/EXCEPTIONS.md index 567e2f8..0579b87 100644 --- a/docs/decisions/EXCEPTIONS.md +++ b/docs/decisions/EXCEPTIONS.md @@ -21,3 +21,4 @@ is retained as history, not authorization to accept unprojectable writes. | 2026-08-21 (PROPOSED) | KA-11 | **The snapshot read surface's fail-closed gate is CONDITIONAL**: `src/read.ts` refuses a document carrying content under an unreadable schema, but returns its empty value for a document carrying nothing, where `project()` refuses that document too. Deliberate — ADR-004's follower reads a root-less document before its first frame and there is no content there to mis-key — but it is a read path that answers where a sibling read path throws, so it is recorded rather than left in source comments. Normative text: Amendment A12. The divergence is disposition-only; both paths use `readSchemaVersion` as the predicate. Mitigated by asking "carries content" against the struct store rather than the v1 root names, so a v2 that renamed its roots is refused rather than read as empty. | `src/read.ts` `assertSnapshotReadable`/`carriesContent`; schema §10 + Amendment A12 | Owner: UNASSIGNED. Sunset PROPOSED, not agreed: revisit when `SCHEMA_VERSION` first moves past 1, since a real v2 makes the disposition split observable to a deployed follower for the first time. | | ~~2026-08-27 (PROPOSED)~~ **RESOLVED 2026-08-28** | KA-3 | ~~**A promoted `connect` (`grow.promoted: true`, schema Amendment A15) is LWW-gated here on `("input", to_node, "grow", name)` and moves the materialized slot's `grow_id` to the winning link; comfy-cli's `_apply_connect` gates neither.**~~ **Sunset condition met: comfy-cli PR #818 (amendment v1.5, `ba0b0b92abcc86b01e8a6704d07088f92afe7aa7`) gates the same register, spelled with the FULL declared name on both sides, and moves `grow_id` to the winner.** Struck rather than deleted so the exception's life stays legible. Both spell the register identically (`_write_target`), so the divergence is in GATING only and is observable only under concurrency comfy-cli's single-writer apply never sees — the corpus session it generated replays green. Gating is required for §2.5 convergence (two connects into one declared input contend for one slot exactly as two concrete connects do, A1). | `src/applier.ts` `claimPromotedInput`; schema Amendment A15 | Owner: UNASSIGNED. Sunset PROPOSED, not agreed: comfy-cli `_lww_gate`s promoted grows on the same register (asked for in the PR that lands A15). | | 2026-09-21 (APPROVED) | FC-10 | `set_node_field` is a package-local, provisional operation because post-creation title and flag changes otherwise have no non-clobbering write path. The owner requested consolidation on the general per-field architecture rather than the competing title-only operation. Its field set and value types are closed, its incarnation semantics match `set_widget`, and a language-neutral conformance session is included here; the upstream comfy-cli vocabulary has not yet adopted the kind. | `src/types.ts`, `src/applier.ts`, `src/stamps.ts`, `src/collab-trace.ts`; schema Amendment A21; ADR-032 | Owner: Christian Byrne. Sunset: amend comfy-cli's frozen vocabulary and pin the adopted definition here, reconciling the name/payload if upstream differs. | +| ~~2026-09-22 (APPROVED)~~ **AMENDED AND APPROVED 2026-09-23** | KA-5 | ~~`insert_workflow`'s numeric link-id mint consulted mutable document state (a document-verified retry), unlike every other id this op derives.~~ Christian Byrne's review of that design (PR #245, inline on `src/remap.ts:65`) found it made link identity depend on LOCAL ARRIVAL ORDER — two replicas applying the same concurrent op set in opposite orders could project different link ids for the same logical links, the exact class of bug KA-4/KA-2 exist to rule out — and asked for either a fix plus explicit ratification, or a return to pure derivation. **This row now describes the amended design: `derivedLinkId` mints PURELY from the op's own content (`opId`, `scope`, the raw link), with NO document read, no caller-supplied reservation, and no retry.** It spends the FULL safe-integer range (`[1, Number.MAX_SAFE_INTEGER]`, 53 bits, widened from the original's 52-bit window) rather than checking for a collision. ComfyUI_frontend's `LinkId` (a branded `number`, unlike the string-or-number `NodeId`) is still why a link needs a numeric mint at all — ADR-031's pure STRING scheme still can't satisfy it — but the mint no longer tries to guarantee collision-freedom against the document; it accepts a quantified residual risk instead. Residual probability: two independently derived 53-bit candidates collide with probability `1/(2^53-1) ≈ 1.11×10⁻¹⁶` per pair; for `k` links minted into one document the birthday-bound chance ANY two collide is `≈ k²/2^54` — about `2.2×10⁻¹²` at `k=200` (this package's own scale test), about `5.5×10⁻⁵` even at an unrealistic `k=1,000,000` — the same order of "cosmically unlikely, not impossible" risk this package already accepts for UUID4 `op_id` collisions (KA-2). A REALIZED collision is not specially detected: it is caught by `applier.ts`'s pre-existing `acceptsLink`/`seenLinks` apply-time id-collision handling (the same ordinary, order-independent LWW-gated check every other `insert_workflow` id kind already goes through), rejecting the whole op `link_id_collision` byte-identically rather than silently reassigning a different id — this check is REACHABLE via realistic input now (not merely via deliberate id-space exhaustion, since there is no more exhaustion to construct). **Separately, a pre-existing, unresolved gap** (out of scope for this amendment): a `connect` op's client-minted `link_id` can collide with an `insert_workflow`-derived numeric id, and `applier.ts`'s `claimLinkIdentity` does not guard against it at all, silently overwriting the `insert_workflow` link when `connect` arrives second — this predates and is unaffected by this amendment either way (the removed retry only ever protected the OPPOSITE arrival order, and only when the `connect` had already landed on that replica). See ADR-033's "Amendment (2026-09-23)" section for the full writeup, including this cross-mechanism note. | `src/remap.ts` `derivedLinkId`; `src/applier.ts` `prepareInsertedWorkflow`, `acceptsLink`, `claimLinkIdentity`; ADR-033 | Owner: Christian Byrne. The amended pure-derivation design was ratified after exact-head review confirmed that the document-state retry was removed, opposite-order convergence coverage was added, and realized collisions remain loud and byte-identical. Sunset (unchanged from the original row): revisit only if ComfyUI_frontend widens `LinkId` to `string \| number` (matching `NodeId`), at which point ADR-031's string form becomes sufficient for links too and this exception can be retired outright. | diff --git a/src/applier.ts b/src/applier.ts index 18ddfed..702142e 100644 --- a/src/applier.ts +++ b/src/applier.ts @@ -1007,7 +1007,16 @@ function prepareInsertedWorkflow(op: InsertWorkflowOp): Record } validateRawGraphIds(wf["nodes"] as unknown[], (wf["links"] as unknown[] | undefined) ?? [], "workflow"); validateDefinitionInputs((subgraphs as unknown[] | undefined) ?? []); - return remapInsertedWorkflowIds(wf as unknown as import("./types.js").WorkflowJSON, op.op_id) as unknown as Record; + // ADR-033 (amended): every id this op derives, including a link's numeric + // id, is a PURE function of the op's own content — no document state is + // read here. See `remap.ts`'s `derivedLinkId` for why `LinkId` (a branded + // `number`, unlike the string-or-number `NodeId`) needed a numeric mint at + // all, and for why the mint no longer checks the target document (KA-5 + // exception, `docs/decisions/EXCEPTIONS.md`). + return remapInsertedWorkflowIds( + wf as unknown as import("./types.js").WorkflowJSON, + op.op_id, + ) as unknown as Record; } function applyInsertWorkflow(doc: Y.Doc, op: InsertWorkflowOp, catalog?: WidgetCatalog): SuccessfulOutcome { @@ -1177,9 +1186,16 @@ function updateInsertedWorkflowMeta( const currentNode = numericId(meta.get("last_node_id")) ?? 0; const maxNode = Math.max(currentNode, ...nodeWrites.map(([, id]) => numericId(id) ?? currentNode)); if (maxNode > currentNode) mset(meta, "last_node_id", maxNode); - const currentLink = numericId(meta.get("last_link_id")) ?? 0; - const maxLink = Math.max(currentLink, ...linkWrites.map((link) => numericId(link[0]) ?? currentLink)); - if (maxLink > currentLink) mset(meta, "last_link_id", maxLink); + // `last_link_id` is deliberately NOT advanced from `linkWrites` (KA-5: + // "document high-water marks are advisory, never allocators"). Every + // inserted link's own id is now a large, arbitrary-looking derived number + // (ADR-033 — `remap.ts`'s `derivedLinkId`, needed only because ComfyUI_ + // frontend's `LinkId` is a branded `number`), not a small sequential one, + // so folding it into the high-water mark would pin the doc's bookkeeping + // field to that value forever after the FIRST insert_workflow — a new, + // surprising side effect this line never had before (the pre-fix STRING + // derived id already failed `numericId` and left this a no-op for every + // inserted link). } // --------------------------------------------------------------------------- diff --git a/src/remap.ts b/src/remap.ts index 278bb3d..4abb66d 100644 --- a/src/remap.ts +++ b/src/remap.ts @@ -10,6 +10,65 @@ function derivedDefinitionId(opId: string, scope: string, original: unknown): st return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-4${hex.slice(13, 16)}-8${hex.slice(17, 20)}-${hex.slice(20, 32)}`; } +/** + * A newly derived link id must be a real `number` — ComfyUI_frontend's + * `LinkId` is `number & { __brand }`, unlike `NodeId` (`string | number`), so + * the string form `derivedId` gives every other kind cannot satisfy it + * (ComfyUI_frontend#18458). + * + * `derivedLinkId` folds the digest into the FULL safe-integer range + * (`[1, Number.MAX_SAFE_INTEGER]`, i.e. `[1, 2^53 - 1]`) rather than the + * narrower 52-bit window an earlier revision used, so the mint spends every + * bit of entropy `LinkId`'s numeric type can safely hold (ADR-033 amendment, + * review discussion on PR #245). 32 hex characters (128 bits) are taken from + * the digest before the modulus is applied so the reduction to 53 bits stays + * close to uniform; sha256Hex's 64-character output has plenty to spare. + */ +const LINK_ID_HEX_DIGITS = 32; // 128 bits, folded down to 53 by SAFE_INTEGER_MODULUS below. + +/** `2^53 - 1` as a `BigInt`, i.e. `Number.MAX_SAFE_INTEGER` — every candidate lands in `[1, MAX_SAFE_INTEGER]`. */ +const SAFE_INTEGER_MODULUS = BigInt(Number.MAX_SAFE_INTEGER); + +/** + * Mint a numeric id for a link `insert_workflow` carries — PURELY as a + * function of the op's own content (`opId`, `scope`, `original`), with NO + * document-state dependency (ADR-033 amendment, superseding the + * document-verified retry mint the original revision of this PR shipped). + * + * Christian Byrne's review of the original revision (PR #245, inline on this + * file) found that the retry-against-`taken` design made link identity + * depend on LOCAL ARRIVAL ORDER: if two link seeds shared the same first + * candidate, applying them A-then-B minted A the natural candidate and B the + * next one, while applying B-then-A minted B the natural candidate and A the + * next one — two replicas that apply the same op set in different orders + * could project genuinely different link ids for the same logical links, + * which is exactly the class of bug KA-4 (deterministic/idempotent + * projection) and KA-2 (order-independent identity) exist to rule out. + * + * The fix removes the retry (and the document read it required) entirely: + * the SAME op, applied to ANY document state, in ANY order relative to any + * other op, always derives the SAME numeric id — because nothing but the + * op's own fields feeds the hash. This restores ADR-031's "producers do not + * inspect document state or remap ids" for links too, at the cost of the + * "checked collision-free against this document" property the retry design + * offered: two derived ids (or a derived id and something already persisted) + * can now coincide, with probability bounded by the ~53 bits of entropy + * below (see the KA-5 row in `docs/decisions/EXCEPTIONS.md` for the exact + * figure and why it is an accepted residual risk rather than a defect). + * `applier.ts`'s existing `acceptsLink` still catches a REALIZED collision + * against an already-persisted link at apply time (the same LWW-gated check + * every other `insert_workflow` id kind already goes through) — that check + * is ordinary id-collision handling this package has always had, not new + * state-dependence introduced here, and it resolves by the op-order- + * independent `[base_version, actor, op_id]` stamp comparison, not by + * arrival order. + */ +function derivedLinkId(opId: string, scope: string, original: unknown): number { + const seed = derivedId(opId, scope, "link", original); + const hex = sha256Hex(seed).slice(0, LINK_ID_HEX_DIGITS); + return Number((BigInt(`0x${hex}`) % SAFE_INTEGER_MODULUS) + 1n); +} + function linkEndpoints(link: unknown): [unknown, unknown] | undefined { if (Array.isArray(link) && link[1] !== undefined && link[3] !== undefined) return [link[1], link[3]]; if (typeof link === "object" && link !== null) { @@ -51,7 +110,7 @@ export function linkHasMissingEndpoint(link: unknown, hasNode: (id: unknown) => return endpoints !== undefined && (!hasNode(endpoints[0]) || !hasNode(endpoints[1])); } -function remapInputs(inputs: unknown, linkIds: Map, droppedLinkIds: Set): void { +function remapInputs(inputs: unknown, linkIds: Map, droppedLinkIds: Set): void { if (!Array.isArray(inputs)) return; for (const input of inputs) { if (typeof input !== "object" || input === null || !("link" in input)) continue; @@ -73,7 +132,7 @@ function remapInputs(inputs: unknown, linkIds: Map, droppedLinkI function remapLinkIdArray( records: unknown, field: string, - linkIds: Map, + linkIds: Map, droppedLinkIds: Set, ): void { if (!Array.isArray(records)) return; @@ -121,7 +180,7 @@ function remapGraph( ): void { const nodes = graph["nodes"] as WorkflowNode[]; const nodeIds = new Map(); - const linkIds = new Map(); + const linkIds = new Map(); // Numeric and string aliases normalized by validation name the same node. for (const node of nodes) nodeIds.set(normalizedId(node.id), derivedId(opId, scope, "node", node.id)); const droppedLinkIds = new Set(); @@ -134,7 +193,7 @@ function remapGraph( }); for (const link of links) { const id = linkId(link); - linkIds.set(normalizedId(id), derivedId(opId, scope, "link", id)); + linkIds.set(normalizedId(id), derivedLinkId(opId, scope, id)); } for (const node of nodes) { @@ -175,7 +234,21 @@ function remapGraph( } } -/** Deterministically remap every id carried by an insertion op, including nested definition graphs. */ +/** + * Deterministically remap every id carried by an insertion op, including + * nested definition graphs. + * + * Every id this function derives — node, link, group, and definition — is a + * PURE function of (`opId`, `scope`, `kind`, `original`) alone (ADR-031, + * amended for links by the ADR-033 revision on PR #245). No document, no + * caller-supplied reservation set, and no state from any other call factors + * into the result: the same `(wf, opId)` pair always produces byte-identical + * output, on any replica, regardless of what else that replica's document + * holds or what order concurrent ops arrive in. This is what makes the + * result safe to compute before touching the target document at all — + * `applier.ts` calls this once per `insert_workflow` application with no + * document in scope yet. + */ export function remapInsertedWorkflowIds(wf: WorkflowJSON, opId: string): WorkflowJSON { const out = structuredClone(wf) as WorkflowJSON & Record; out.links ??= []; diff --git a/test/insert-workflow-numeric-link-ids.regression.test.ts b/test/insert-workflow-numeric-link-ids.regression.test.ts new file mode 100644 index 0000000..41ac59d --- /dev/null +++ b/test/insert-workflow-numeric-link-ids.regression.test.ts @@ -0,0 +1,412 @@ +/** + * `insert_workflow` numeric link ids (ADR-033, amended). + * + * ComfyUI_frontend's `LinkId` is `number & { __brand: 'LinkId' }`, unlike the + * string-or-number `NodeId`, so `remap.ts`'s usual string `derivedId` + * (`insert:::link:`) cannot satisfy it. The interim + * frontend fix (ComfyUI_frontend#18458, `ecsFollowerAdapter.ts`'s + * `resolveLinkId`) hashed that string into a 32-bit number and was found by + * two reviewers to be genuinely broken, not merely approximate: two + * independent derived ids collided at a measured ~0.01-1% rate at realistic + * link counts, AND the resolved id is what `graph.serialize()` persists, so + * a "reserved range" chosen above `last_link_id` does not survive a + * save/reload — the saved file now contains a genuine numeric link id + * inside that range. + * + * The FIRST revision of this PR fixed that by minting a 52-bit candidate and + * retrying past any collision with a numeric link id already persisted in + * the target document. Christian Byrne's review (PR #245, inline on + * `src/remap.ts:65`) found that design itself broken in a different way: + * checking the mint against document state made link identity depend on + * LOCAL ARRIVAL ORDER. If two link seeds shared the same first candidate, + * applying them A-then-B minted A the natural candidate and B the next one + * in the retry sequence, while applying B-then-A minted B the natural + * candidate and A the next one — two replicas that apply the same op set in + * different orders could project genuinely different link ids for the same + * logical links. + * + * `remap.ts`'s `derivedLinkId` now mints PURELY from the op's own content — + * `opId`, `scope`, and the raw link — spending the full safe-integer range + * (53 bits, `[1, Number.MAX_SAFE_INTEGER]`) so a genuine collision is an + * accepted, quantified residual risk rather than something the mint tries to + * avoid by reading state. See + * `docs/decisions/ADR-033-insert-workflow-numeric-link-ids.md` and the KA-5 + * row in `docs/decisions/EXCEPTIONS.md`. + * + * `test/insert-workflow.test.ts`'s "numeric link id minting (ADR-033, pure + * derivation)" block covers the single-op determinism and realized-collision + * cases directly against `occupiedIdVectors`' existing node/definition + * siblings. This file covers the harder cross-cutting properties: multiple + * `insert_workflow` ops in one session (sequential and same-batch), scale + * within one op, the project → mint round trip, comfy-multi-player#230's + * definition-interior promoted-linkIds threading with the new numeric ids, + * and — the regression this whole amendment exists for — opposite-order + * convergence across two independently seeded replicas. + */ +import { describe, expect, it } from "vitest"; +import * as Y from "yjs"; + +import { applyOps, mint, project, type Op, type WidgetCatalog, type WorkflowJSON } from "../src/index.js"; +import { appliedMap } from "../src/doc.js"; + +const catalog: WidgetCatalog = { + types: { + Src: { widget_order: [] }, + Sink: { widget_order: [] }, + Def: { widget_order: [] }, + }, +}; + +let seq = 0; +function opEnvelope(overrides: Partial = {}): Op { + const op_id = ("n" + String(seq++).padStart(4, "0")).padEnd(32, "0"); + return { op: "insert_workflow", op_id, actor: "a", base_version: 1, stamp: [1, "a"], ...overrides } as unknown as Op; +} + +/** All top-level projected link tuples, typed loosely for id/label access. */ +function links(wf: WorkflowJSON): Array<[unknown, unknown, unknown, unknown, unknown, unknown]> { + return (wf.links ?? []) as never; +} + +function linkNamed(wf: WorkflowJSON, label: unknown): [unknown, unknown, unknown, unknown, unknown, unknown] { + const found = links(wf).find((link) => link[5] === label); + if (!found) throw new Error(`no projected link labeled ${String(label)}`); + return found; +} + +describe("insert_workflow numeric link ids (ADR-033, pure derivation)", () => { + it("mints a real number for a top-level link while node ids stay derived strings", () => { + const doc = mint({ nodes: [], links: [] }, catalog); + const op = opEnvelope({ workflow: { nodes: [{ id: 1, type: "Src" }, { id: 2, type: "Sink" }], links: [[10, 1, 0, 2, 0, "L"]] } }); + + expect(applyOps(doc, [op], catalog).outcomes[0]).toMatchObject({ outcome: "applied" }); + const wf = project(doc, catalog); + expect(typeof linkNamed(wf, "L")[0]).toBe("number"); + for (const node of wf.nodes!) expect(typeof node.id).toBe("string"); + }); + + it("two independent insert_workflow ops with distinct op_ids derive independent link ids", () => { + // Distinct `op_id`s feed distinct hash seeds (`derivedId` folds `opId` + // into the seed), so two independently authored ops naturally land on + // different 53-bit candidates — not because either mint avoids the + // other (it reads no document state at all), but because the space is + // wide enough that an accidental collision between UNRELATED ops is the + // same order of unlikely as a UUID4 `op_id` collision (KA-2). + const doc = mint({ nodes: [], links: [] }, catalog); + const opA = opEnvelope({ + workflow: { + nodes: [{ id: 101, type: "Src" }, { id: 102, type: "Sink" }], + links: [[201, 101, 0, 102, 0, "op-a-link"]], + }, + }); + const opB = opEnvelope({ + workflow: { + nodes: [{ id: 103, type: "Src" }, { id: 104, type: "Sink" }], + links: [[201, 103, 0, 104, 0, "op-b-link"]], + }, + }); + + expect(applyOps(doc, [opA], catalog).outcomes[0]).toMatchObject({ outcome: "applied" }); + expect(applyOps(doc, [opB], catalog).outcomes[0]).toMatchObject({ outcome: "applied" }); + + const wf = project(doc, catalog); + expect(wf.nodes).toHaveLength(4); + const linkA = linkNamed(wf, "op-a-link"); + const linkB = linkNamed(wf, "op-b-link"); + expect(linkA[0]).not.toBe(linkB[0]); + // Op A's wire must not have been replaced by op B's: both endpoints + // resolve to their OWN op's nodes. + const nodeIds = new Set(wf.nodes!.map((n) => n.id)); + expect(nodeIds.has(linkA[1] as string)).toBe(true); + expect(nodeIds.has(linkB[1] as string)).toBe(true); + }); + + it("two insert_workflow ops delivered in the SAME batch still derive independent link ids", () => { + // Delivered as ONE `applyOps` batch rather than two sequential calls. + // Neither op's mint threads any state from the other — each derives its + // link id purely from its own (opId, scope, original) — so batching + // changes nothing about the result. + const doc = mint({ nodes: [], links: [] }, catalog); + const opA = opEnvelope({ + workflow: { nodes: [{ id: 111, type: "Src" }, { id: 112, type: "Sink" }], links: [[301, 111, 0, 112, 0, "batch-a"]] }, + }); + const opB = opEnvelope({ + workflow: { nodes: [{ id: 113, type: "Src" }, { id: 114, type: "Sink" }], links: [[301, 113, 0, 114, 0, "batch-b"]] }, + }); + + const result = applyOps(doc, [opA, opB], catalog); + expect(result.outcomes).toEqual([ + { op_id: opA.op_id, outcome: "applied" }, + { op_id: opB.op_id, outcome: "applied" }, + ]); + const wf = project(doc, catalog); + expect(linkNamed(wf, "batch-a")[0]).not.toBe(linkNamed(wf, "batch-b")[0]); + }); + + it("mints distinct ids for a large batch of links inserted in one op", () => { + const opId = "cc11df0c31f9440b9385ac8e01e099b2"; + const count = 200; + const nodes = Array.from({ length: count * 2 }, (_, i) => ({ id: `n${String(i)}`, type: i % 2 === 0 ? "Src" : "Sink" })); + const rawLinks = Array.from({ length: count }, (_, i) => [1000 + i, `n${String(i * 2)}`, 0, `n${String(i * 2 + 1)}`, 0, `link-${String(i)}`]); + const doc = mint({ nodes: [], links: [] }, catalog); + const op = opEnvelope({ workflow: { nodes, links: rawLinks }, op_id: opId }); + + // Each of the 200 raw link ids seeds a distinct hash input, so distinct + // final candidates are the overwhelmingly likely outcome at 53 bits of + // entropy — not a guarantee any dedup mechanism enforces (there is + // none left), just what the birthday bound predicts for 200 draws from + // a ~9-quadrillion-value space. + expect(applyOps(doc, [op], catalog).outcomes[0]).toMatchObject({ outcome: "applied" }); + const wf = project(doc, catalog); + expect(wf.links).toHaveLength(count); + const ids = wf.links!.map((link) => (link as unknown[])[0]); + expect(new Set(ids).size).toBe(count); + for (const id of ids) expect(typeof id).toBe("number"); + }); + + it("round-trips numeric link ids byte-for-byte through project → mint with no duplication or corruption", () => { + const opId = "ff11df0c31f9440b9385ac8e01e099b2"; + const doc = mint({ nodes: [], links: [] }, catalog); + const op = opEnvelope({ + workflow: { + nodes: [{ id: 1, type: "Src" }, { id: 2, type: "Sink" }, { id: 3, type: "Def" }], + links: [[701, 1, 0, 2, 0, "L1"]], + definitions: { + subgraphs: [ + { + id: "Def", + nodes: [{ id: "a", type: "Src" }, { id: "b", type: "Sink" }], + links: [{ id: 702, origin_id: "a", origin_slot: 0, target_id: "b", target_slot: 0, type: "X" }], + }, + ], + }, + }, + op_id: opId, + }); + expect(applyOps(doc, [op], catalog).outcomes[0]).toMatchObject({ outcome: "applied" }); + const first = project(doc, catalog); + + const reminted = mint(first, catalog); + const second = project(reminted, catalog); + expect(second).toEqual(first); + }); + + it("subgraph-interior promoted linkIds thread numeric ids end-to-end (comfy-multi-player#230 non-regression)", () => { + // Mirrors `test/insert-workflow-promoted-input-linkids.regression.test.ts` + // but through the REAL applier + project path, asserting the ids + // `remapLinkIdArray` threads into a definition's own `inputs[].linkIds` + // are the SAME real numbers as `links[].id` — exactly what ComfyUI_ + // frontend's `agentSubgraphHostSlots.ts` (`Map` keyed by + // `link.id`) needs to resolve a promoted widget. + const opId = "0011df0c31f9440b9385ac8e01e099b2"; + const doc = mint({ nodes: [], links: [] }, catalog); + const op = opEnvelope({ + workflow: { + nodes: [{ id: 100, type: "PromotedDef", inputs: [{ name: "text", type: "STRING", link: null, widget: { name: "text" } }] }], + links: [], + definitions: { + subgraphs: [ + { + id: "PromotedDef", + name: "Promoted text", + inputs: [{ name: "text", type: "STRING", linkIds: [34] }], + outputs: [], + nodes: [ + { id: 10, type: "Src", inputs: [], outputs: [{ name: "STRING", type: "STRING", links: [34] }] }, + { id: 11, type: "Def", inputs: [{ name: "text", type: "STRING", link: 34, widget: { name: "text" } }], outputs: [] }, + ], + links: [{ id: 34, origin_id: 10, origin_slot: 0, target_id: 11, target_slot: 0, type: "STRING" }], + }, + ], + }, + }, + op_id: opId, + }); + + expect(applyOps(doc, [op], catalog).outcomes[0]).toMatchObject({ outcome: "applied" }); + const wf = project(doc, catalog); + const definition = ( + wf.definitions as { subgraphs: Array<{ inputs: Array<{ linkIds: unknown[] }>; links: Array<{ id: unknown }> }> } + ).subgraphs[0]!; + + expect(definition.links).toHaveLength(1); + const realLinkId = definition.links[0]!.id; + expect(typeof realLinkId).toBe("number"); + // The promoted input names the SAME numeric id, not a stringified or + // otherwise-derived stand-in — a `Map` keyed by + // `definition.links[].id` resolves `inputs[0].linkIds[0]` directly. + expect(definition.inputs[0]!.linkIds).toEqual([realLinkId]); + }); + + it("keeps a definition-interior link fed by the subgraph IO sentinels numeric (sentinel non-regression)", () => { + const opId = "1122df0c31f9440b9385ac8e01e099b2"; + const doc = mint({ nodes: [], links: [] }, catalog); + const op = opEnvelope({ + workflow: { + nodes: [ + { + id: 100, + type: "SentinelDef", + inputs: [{ name: "text", type: "STRING", link: null, widget: { name: "text" } }], + outputs: [{ name: "IMAGE", type: "IMAGE", links: [] }], + }, + ], + links: [], + definitions: { + subgraphs: [ + { + id: "SentinelDef", + name: "Sentinel", + inputs: [{ name: "text", type: "STRING", linkIds: [16] }], + outputs: [{ name: "IMAGE", type: "IMAGE", linkIds: [17] }], + nodes: [ + { id: 27, type: "Def", inputs: [{ name: "text", type: "STRING", link: 16, widget: { name: "text" } }], outputs: [] }, + { id: 8, type: "Src", inputs: [], outputs: [{ name: "IMAGE", type: "IMAGE", links: [17] }] }, + ], + links: [ + { id: 16, origin_id: -10, origin_slot: 0, target_id: 27, target_slot: 1, type: "STRING" }, + { id: 17, origin_id: 8, origin_slot: 0, target_id: -20, target_slot: 0, type: "IMAGE" }, + ], + }, + ], + }, + }, + op_id: opId, + }); + + expect(applyOps(doc, [op], catalog).outcomes[0]).toMatchObject({ outcome: "applied" }); + const wf = project(doc, catalog); + const definition = ( + wf.definitions as { + subgraphs: Array<{ + inputs: Array<{ linkIds: unknown[] }>; + outputs: Array<{ linkIds: unknown[] }>; + links: Array<{ id: unknown; origin_id: unknown; target_id: unknown }>; + }>; + } + ).subgraphs[0]!; + + expect(definition.links).toHaveLength(2); + const promotedLink = definition.links.find((link) => link.origin_id === -10)!; + const exposedLink = definition.links.find((link) => link.target_id === -20)!; + expect(typeof promotedLink.id).toBe("number"); + expect(typeof exposedLink.id).toBe("number"); + expect(definition.inputs[0]!.linkIds).toEqual([promotedLink.id]); + expect(definition.outputs[0]!.linkIds).toEqual([exposedLink.id]); + }); + + it("an exact replay (same op_id, same payload) is a true no-op and does not re-mint the link id", () => { + const opId = "2233df0c31f9440b9385ac8e01e099b2"; + const doc = mint({ nodes: [], links: [] }, catalog); + const op = opEnvelope({ + workflow: { nodes: [{ id: 1, type: "Src" }, { id: 2, type: "Sink" }], links: [[801, 1, 0, 2, 0, "L"]] }, + op_id: opId, + }); + + expect(applyOps(doc, [op], catalog).outcomes[0]).toMatchObject({ outcome: "applied" }); + const firstProjection = project(doc, catalog); + const mintedId = linkNamed(firstProjection, "L")[0]; + const beforeReplay = Buffer.from(Y.encodeStateAsUpdate(doc)); + + expect(applyOps(doc, [op], catalog).outcomes[0]).toMatchObject({ outcome: "no-op" }); + expect(Buffer.from(Y.encodeStateAsUpdate(doc)).equals(beforeReplay)).toBe(true); + expect(appliedMap(doc).has(op.op_id)).toBe(true); + const afterReplay = project(doc, catalog); + expect(linkNamed(afterReplay, "L")[0]).toBe(mintedId); + }); + + describe("opposite-order convergence (the regression Christian Byrne's review found)", () => { + /** + * Apply `ops` in order to a FRESH `Y.Doc` forked from `seed` via + * `Y.applyUpdate` (KA-10: independent replicas fork from one common + * snapshot, never re-seed independently) and return the projection. + */ + function applyToFreshReplica(seed: Uint8Array, ops: Op[]): WorkflowJSON { + const doc = new Y.Doc(); + Y.applyUpdate(doc, seed); + for (const op of ops) { + expect(applyOps(doc, [op], catalog).outcomes[0]).toMatchObject({ outcome: "applied" }); + } + return project(doc, catalog); + } + + it("two concurrent insert_workflow ops project byte-identically in either arrival order", () => { + // This is the exact shape of Christian's blocker: two CONCURRENT ops + // (never exchanged with each other's replica before either is + // applied), each carrying links, replayed on two independent replicas + // in opposite order. Under the retry-against-`taken` design this PR's + // first revision shipped, a shared first candidate between a link in + // opA and a link in opB would resolve to opA=x/opB=y on one replica + // and opB=x/opA=y on the other — a genuine, silent projection + // divergence between two replicas that applied the identical op set. + // Pure derivation makes that impossible by construction: neither op's + // mint reads anything but its own content, so its result cannot + // depend on which replica, or which order, it is evaluated in. + const seed = Y.encodeStateAsUpdate(mint({ nodes: [], links: [] }, catalog)); + const opA = opEnvelope({ + workflow: { + nodes: [{ id: 101, type: "Src" }, { id: 102, type: "Sink" }, { id: 103, type: "Sink" }], + links: [ + [201, 101, 0, 102, 0, "a1"], + [202, 101, 0, 103, 0, "a2"], + ], + }, + }); + const opB = opEnvelope({ + workflow: { + nodes: [{ id: 501, type: "Src" }, { id: 502, type: "Sink" }], + links: [[601, 501, 0, 502, 0, "b1"]], + }, + }); + + const forward = applyToFreshReplica(seed, [opA, opB]); + const reverse = applyToFreshReplica(seed, [opB, opA]); + + // Not merely "both applied successfully" -- the WHOLE projected + // document, including every link's numeric id, is byte-for-byte + // identical regardless of order (`project()` sorts by id, so this + // equality is meaningful even though the two replicas wrote their + // Y.Map entries in opposite orders). + expect(forward).toEqual(reverse); + expect(linkNamed(forward, "a1")[0]).toBe(linkNamed(reverse, "a1")[0]); + expect(linkNamed(forward, "a2")[0]).toBe(linkNamed(reverse, "a2")[0]); + expect(linkNamed(forward, "b1")[0]).toBe(linkNamed(reverse, "b1")[0]); + }); + + it("two insert_workflow ops sharing a definition converge identically in either order", () => { + // Extends the same property to a subgraph-definition interior scope + // (this PR's other cross-cutting surface): opA inserts a definition + // and an instance of it; opB is an unrelated top-level insert. Both + // the top-level and the definition-interior link ids must agree + // across orders. + const seed = Y.encodeStateAsUpdate(mint({ nodes: [], links: [] }, catalog)); + const opA = opEnvelope({ + workflow: { + nodes: [{ id: 900, type: "Def" }], + links: [], + definitions: { + subgraphs: [ + { + id: "Def", + nodes: [{ id: "x", type: "Src" }, { id: "y", type: "Sink" }], + links: [{ id: 41, origin_id: "x", origin_slot: 0, target_id: "y", target_slot: 0, type: "X" }], + }, + ], + }, + }, + }); + const opB = opEnvelope({ + workflow: { nodes: [{ id: 701, type: "Src" }, { id: 702, type: "Sink" }], links: [[801, 701, 0, 702, 0, "top-level"]] }, + }); + + const forward = applyToFreshReplica(seed, [opA, opB]); + const reverse = applyToFreshReplica(seed, [opB, opA]); + expect(forward).toEqual(reverse); + + const defLinkId = (wf: WorkflowJSON): unknown => + (wf.definitions as { subgraphs: Array<{ links: Array<{ id: unknown }> }> }).subgraphs[0]!.links[0]!.id; + expect(defLinkId(forward)).toBe(defLinkId(reverse)); + expect(linkNamed(forward, "top-level")[0]).toBe(linkNamed(reverse, "top-level")[0]); + }); + }); +}); diff --git a/test/insert-workflow.test.ts b/test/insert-workflow.test.ts index cb458d4..4db2989 100644 --- a/test/insert-workflow.test.ts +++ b/test/insert-workflow.test.ts @@ -11,7 +11,17 @@ * the op_id gate, and byte-identical doc on rejection. * * ID allocation belongs to the applier and is derived from the immutable op - * id plus each raw id, never from mutable document state. + * id plus each raw id alone, never from mutable document state — including + * for links. An earlier revision of this PR made a link's numeric id + * (ComfyUI_frontend#18458 — `LinkId` is a branded `number`, unlike the + * string-or-number `NodeId`) an exception to that rule, retrying against the + * target document's already-persisted numeric link ids to avoid a collision. + * Review (Christian Byrne, PR #245) found that retry made link identity + * depend on local arrival order, so `remap.ts`'s `derivedLinkId` mints + * PURELY from the op's own content instead (ADR-033, amended), spending the + * full safe-integer range (53 bits) so a genuine collision is an accepted, + * quantified residual risk (`docs/decisions/EXCEPTIONS.md`'s KA-5 row) + * rather than something the mint tries to avoid by reading state. */ import { describe, expect, it } from "vitest"; import * as Y from "yjs"; @@ -96,6 +106,21 @@ function replaySnapshot(seed: Uint8Array, ops: Op[]): WorkflowJSON { return project(doc, catalog); } +/** + * One link's numeric id, as `remap.ts`'s `derivedLinkId` mint would produce + * it for (`opId`, `"root"`, `rawLinkId`) — a pure function of those two + * inputs alone (ADR-033, amended), so this is exactly what a real apply of + * the same (opId, rawLinkId) will also compute, regardless of document + * state or arrival order. + */ +function derivedLinkIdForTest(opId: string, rawLinkId: number): number { + const remapped = remapInsertedWorkflowIds( + { nodes: [{ id: "a", type: "Src" }, { id: "b", type: "Sink" }], links: [[rawLinkId, "a", 0, "b", 0, "x"]] } as unknown as WorkflowJSON, + opId, + ) as unknown as { links: Array<[number, ...unknown[]]> }; + return remapped.links[0]![0]; +} + const bytes = (doc: Y.Doc): Buffer => Buffer.from(Y.encodeStateAsUpdate(doc)); function ids(wf: WorkflowJSON): unknown[] { @@ -408,7 +433,10 @@ describe("insert_workflow: happy path", () => { const wf = project(doc, catalog); expect(ids(wf).slice(0, 5)).toEqual([1, 2, 3, 4, 6]); expect(ids(wf).slice(5).every((id) => typeof id === "string" && id.includes(op.op_id))).toBe(true); - expect(linkIds(wf)[1]).toEqual(expect.stringContaining(op.op_id)); + // Link ids are minted numeric (ADR-033 — `remap.ts`'s `derivedLinkId`), + // unlike the op-id-embedding derived STRING every other insert_workflow + // id keeps: ComfyUI_frontend's `LinkId` is a branded `number`. + expect(typeof linkIds(wf)[1]).toBe("number"); expect(defIds(wf).find((id) => id !== "def-1")).toMatch(/^[0-9a-f-]{36}$/); // Inserted nodes project with their widgets resolved through the catalog. const n100 = wf.nodes!.find((n) => n.pos?.[0] === 0)!; @@ -583,30 +611,38 @@ describe("insert_workflow: happy path", () => { it("drops both boundary directions and retains only the sibling link and its exact slot references", () => { const opId = "0123456789abcdef0123456789abcdef"; - const siblingLinkId = "insert:0123456789abcdef0123456789abcdef:root:link:903"; const sourceNodeId = "insert:0123456789abcdef0123456789abcdef:root:node:701"; const sinkNodeId = "insert:0123456789abcdef0123456789abcdef:root:node:702"; + const workflow = { + nodes: [ + { id: 701, type: "Src", title: "boundary source", outputs: [{ name: "out", links: [901, 903] }] }, + { id: 702, type: "Sink", title: "boundary sink", inputs: [{ name: "in", link: 902 }, { name: "sibling", link: 903 }] }, + ], + links: [ + [901, 701, 0, 999, 0, "inserted-to-external"], + [902, 999, 0, 702, 0, "external-to-inserted"], + [903, 701, 0, 702, 1, "sibling"], + ], + }; const doc = mint({ nodes: [], links: [] }, catalog); - const op = insertOp( - { - nodes: [ - { id: 701, type: "Src", title: "boundary source", outputs: [{ name: "out", links: [901, 903] }] }, - { id: 702, type: "Sink", title: "boundary sink", inputs: [{ name: "in", link: 902 }, { name: "sibling", link: 903 }] }, - ], - links: [ - [901, 701, 0, 999, 0, "inserted-to-external"], - [902, 999, 0, 702, 0, "external-to-inserted"], - [903, 701, 0, 702, 1, "sibling"], - ], - }, - { op_id: opId }, - ); + const op = insertOp(workflow, { op_id: opId }); seq--; + // `remap.ts`'s `derivedLinkId` is a pure function of (opId, scope, + // original) alone (ADR-033, amended), so the candidate this direct call + // computes for the surviving link (903, "sibling") is exactly what the + // real apply below will also compute — independent of what else the + // target doc holds. + const remapped = remapInsertedWorkflowIds(workflow as unknown as WorkflowJSON, opId) as unknown as { + links: Array<[number, ...unknown[]]>; + }; + const siblingLinkId = remapped.links.find((link) => link[5] === "sibling")![0]; + expect(typeof siblingLinkId).toBe("number"); + expect(applyOps(doc, [op], catalog).outcomes[0]).toMatchObject({ outcome: "applied" }); - const workflow = project(doc, catalog); - expect(workflow.links).toEqual([[siblingLinkId, sourceNodeId, 0, sinkNodeId, 1, "sibling"]]); - expect(workflow.nodes).toEqual([ + const projected = project(doc, catalog); + expect(projected.links).toEqual([[siblingLinkId, sourceNodeId, 0, sinkNodeId, 1, "sibling"]]); + expect(projected.nodes).toEqual([ expect.objectContaining({ id: sourceNodeId, outputs: [{ name: "out", links: [siblingLinkId] }] }), expect.objectContaining({ id: sinkNodeId, @@ -822,14 +858,6 @@ describe("insert_workflow: rejection (KA-4 byte identity, op_id absent from appl links: [], }, }, - { - kind: "link", - code: "link_id_collision", - seed: { - nodes: [{ id: 1, type: "Src" }, { id: 2, type: "Sink" }], - links: [["insert:c641df0c31f9440b9385ac8e01e099b2:root:link:901", 1, 0, 2, 0, "incumbent link"]], - }, - }, { kind: "definition", code: "definition_conflict", @@ -867,6 +895,60 @@ describe("insert_workflow: rejection (KA-4 byte identity, op_id absent from appl }); } + describe("numeric link id minting (ADR-033, pure derivation)", () => { + it("derives the identical numeric id whether or not that id already exists in the target document", () => { + // The whole point of the amended design: the candidate is a function + // of (opId, rawLinkId) ALONE. Computing it against an empty workflow + // (no document in scope at all) and computing it via a real apply + // into a doc that starts with unrelated content must agree exactly — + // there is no document read left in `derivedLinkId` to make them + // differ. + const opId = "d641df0c31f9440b9385ac8e01e099b2"; + const expected = derivedLinkIdForTest(opId, 901); + + const doc = mint({ nodes: [{ id: 1, type: "Src" }, { id: 2, type: "Sink" }], links: [[555, 1, 0, 2, 0, "unrelated"]] }, catalog); + const op = insertOp( + { nodes: [{ id: 701, type: "Src" }, { id: 702, type: "Sink" }], links: [[901, 701, 0, 702, 0, "candidate link"]] }, + { op_id: opId }, + ); + seq--; + + expect(applyOps(doc, [op], catalog).outcomes[0]).toMatchObject({ outcome: "applied" }); + const insertedLink = (project(doc, catalog).links as unknown[][]).find((link) => link[5] === "candidate link")!; + expect(insertedLink[0]).toBe(expected); + }); + + it("rejects, rather than avoids, a genuine collision with a numeric link id already persisted in the doc", () => { + // Christian Byrne's blocker on the original revision: making the mint + // AVOID an occupied candidate is exactly what made link identity + // arrival-order-dependent. The amended mint never reads the document, + // so an incumbent occupying the derived candidate is no longer + // side-stepped — it is caught by `acceptsLink`'s ordinary, pre-existing + // id-collision check (the same one every other `insert_workflow` id + // kind already goes through), which rejects the WHOLE op rather than + // silently reassigning a different id to the colliding link. + const opId = "d641df0c31f9440b9385ac8e01e099b2"; + const occupied = derivedLinkIdForTest(opId, 901); + + const doc = mint( + { nodes: [{ id: 1, type: "Src" }, { id: 2, type: "Sink" }], links: [[occupied, 1, 0, 2, 0, "incumbent link"]] }, + catalog, + ); + const before = bytes(doc); + const beforeProjected = project(doc, catalog); + const op = insertOp( + { nodes: [{ id: 701, type: "Src" }, { id: 702, type: "Sink" }], links: [[901, 701, 0, 702, 0, "candidate link"]] }, + { op_id: opId }, + ); + seq--; + + expect(rejectedOutcomeWithIndex(applyOps(doc, [op], catalog))).toMatchObject({ index: 0, code: "link_id_collision" }); + expect(bytes(doc).equals(before)).toBe(true); + expect(project(doc, catalog)).toEqual(beforeProjected); + expect(appliedMap(doc).has(op.op_id)).toBe(false); + }); + }); + it("routes dangling link endpoints through the existing unknown-node no-op path", () => { const doc = mint(baseWorkflow(), catalog); const op = insertOp({ nodes: [], links: [[200, 100, 0, 999, 0, "X"]] }); diff --git a/test/ka4-rejection-byte-identity.test.ts b/test/ka4-rejection-byte-identity.test.ts index 970d493..0de8db9 100644 --- a/test/ka4-rejection-byte-identity.test.ts +++ b/test/ka4-rejection-byte-identity.test.ts @@ -215,6 +215,14 @@ const CASES: Row[] = [ }, }, { + // ADR-033 (amended): `derivedLinkId` is now a PURE function of the op's + // own content, with no retry against document state (removing the + // arrival-order dependence Christian Byrne's review of the original + // revision found — see `remap.ts`'s `derivedLinkId` doc comment). A + // single occupied candidate is therefore sufficient to reach this code, + // exactly like the node/definition rows above: no id-space-exhaustion + // construction is needed or possible anymore, since there is no retry + // sequence to exhaust. kind: "insert_workflow", why: "remapped link id collides with the live tree", code: "link_id_collision", @@ -222,13 +230,16 @@ const CASES: Row[] = [ ({ op: "insert_workflow", ...env(), - workflow: { nodes: [{ id: 2, type: "Src" }, { id: 3, type: "Sink" }], links: [[7, 2, 0, 3, 0, "X"]] }, + workflow: { nodes: [{ id: 2, type: "Src" }, { id: 3, type: "Sink" }], links: [[901, 2, 0, 3, 0, "X"]] }, }) as Op, seed: (op) => { const workflow = baseWorkflow(); - const remapped = remapInsertedWorkflowIds((op as { workflow: WorkflowJSON }).workflow, op.op_id); - const [remappedLinkId] = remapped.links![0] as [string, number, number, number, number, string]; - workflow.links!.push([remappedLinkId, 2, 0, 3, 0, "X"]); + const remapped = remapInsertedWorkflowIds((op as { workflow: WorkflowJSON }).workflow, op.op_id) as unknown as { + links: Array<[number, ...unknown[]]>; + }; + const remappedLinkId = remapped.links[0]![0]; + workflow.nodes!.push({ id: 50, type: "Src" }, { id: 51, type: "Sink" }); + workflow.links!.push([remappedLinkId, 50, 0, 51, 0, "incumbent"]); return workflow; }, },