diff --git a/.agents/checks/catalog-pinning.md b/.agents/checks/catalog-pinning.md index 85a1c92b..995aa134 100644 --- a/.agents/checks/catalog-pinning.md +++ b/.agents/checks/catalog-pinning.md @@ -32,8 +32,11 @@ Apply .agents/checks/catalog-pinning.md (KA-12, FC-10). The widget catalog (fixtures/catalog.json) is pinned by sha256 at mint; the golden conformance corpus and lww-vectors are SHA-pinned and one-way. Flag any change that edits a pinned fixture without a corresponding manifest/SHA update, cites -the catalog/vocabulary by a moving branch instead of a SHA, or would let the -corpus drift without being regenerated in CI (verify:corpus). +the catalog/vocabulary by a moving branch instead of a SHA, or bypasses +catalog-corpus-check=verify:corpus, the CI hash-verification gate. Preserve the +one-way-corpus-rule: expected outputs come from the pinned upstream generator +or explicitly documented manual fixtures, never from the applier under test. +Regeneration is a separate manual procedure; CI verifies the retained hashes. ``` @@ -43,6 +46,7 @@ carries it in a folded scalar, where every space is a legal line break. This blo audited against the code for [#80](https://github.com/Comfy-Org/comfy-multi-player/issues/80) and verified without a wording change. - + + > Before reporting PASS for any check above, apply [vacuity.md](vacuity.md): P0 to every check, P1 to any guard this change adds, P10 to what that guard's test asserts on, P2 to any tool you ran, and P7 to any run you quote. diff --git a/.agents/checks/dep-secrets-scan.md b/.agents/checks/dep-secrets-scan.md index f78ed64d..c3ef42be 100644 --- a/.agents/checks/dep-secrets-scan.md +++ b/.agents/checks/dep-secrets-scan.md @@ -6,14 +6,18 @@ Detect known CVEs in dependencies and leaked secrets. Applies to any change; esp 1. Probe tools: `npm --version`, `gitleaks version`. If neither is available, skip and report the install hints (`npm i -g npm`, gitleaks: https://github.com/gitleaks/gitleaks#installing). If only one is available, run it and note the other was skipped. 2. **Dependency audit** (npm): + ```bash npm audit --json; echo "npm audit exit: $?" ``` + Do not suppress stderr or discard the exit status: `npm audit` exits non-zero when it finds advisories, but a non-zero exit with no parseable JSON means the tool itself failed — report that as an error, not a clean result. Map severity `critical`→critical, `high`→major, `moderate`→minor, `low`→nitpick. Report package, version, advisory title, CVE, and patched version. 3. **Secrets** (gitleaks): + ```bash gitleaks detect --no-banner --report-format json --report-path /tmp/gitleaks.json --source .; echo "gitleaks exit: $?" ``` + `gitleaks` exits `1` when it finds secrets and `2` on execution error — never collapse these to a pass. If the run errors or the report cannot be parsed, report it as an indeterminate/failed scan, not "no issues found". All secret findings are critical. Report file/line, rule, a redacted match, and advise removal plus credential rotation. ## Repo-specific emphasis diff --git a/.agents/checks/error-handling.md b/.agents/checks/error-handling.md index 9c1d1e93..319b3952 100644 --- a/.agents/checks/error-handling.md +++ b/.agents/checks/error-handling.md @@ -15,8 +15,8 @@ The `src/**` `path_instructions` entry is an umbrella over every profile, so it load-bearing sentence is this profile's mutate-before-throw rule, so the anchor for it lives here: if the generated block loses that requirement, CI fails rather than the bot quietly stopping to ask for it. The needle is space-free because the YAML carries it in a folded scalar. That block's content was -moved byte-for-byte from a hand-written config and has **not** been audited — it still cites issue -#10 as open, and #10 is closed; see +moved byte-for-byte from a hand-written config and has **not** been audited — it still cites +issue #10 as open, and #10 is closed; see [#80](https://github.com/Comfy-Org/comfy-multi-player/issues/80). diff --git a/.coderabbit.yaml b/.coderabbit.yaml index 70632d36..ad017516 100644 --- a/.coderabbit.yaml +++ b/.coderabbit.yaml @@ -38,8 +38,12 @@ reviews: conformance corpus and lww-vectors are SHA-pinned and one-way. Flag any change that edits a pinned fixture without a corresponding manifest/SHA update, cites the catalog/vocabulary by a moving branch instead of a - SHA, or would let the corpus drift without being regenerated in CI - (verify:corpus). + SHA, or bypasses catalog-corpus-check=verify:corpus, the CI + hash-verification gate. Preserve the one-way-corpus-rule: expected + outputs come from the pinned upstream generator or explicitly + documented manual fixtures, never from the applier under test. + Regeneration is a separate manual procedure; CI verifies the retained + hashes. - path: "scripts/**" # source: .agents/checks/purity.md instructions: >- diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index b2503b84..b46030e2 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -10,7 +10,7 @@ derives tree-wide IDs from `op_id`; producers submit raw IDs and do not inspect ## Guard gaps - ~~**Positive purity assertion (#22):**~~ **CLOSED.** `scripts/check-purity.mjs` asserts directly that the declared and resolved production dependency roots are exactly `{yjs}`, and `scripts/check-import-graph.mjs` asserts the same per source module via `src-runtime-dep-is-yjs-only`. A cross-language parity guard is still only warranted if a second-language implementation is proposed; the preferred policy remains no second implementation (tracked in the separate bullet below). -- **Property tests (#24):** cover arbitrary op sequences, legal causal permutations, retries, batch boundaries, actor counts, byte-identical idempotency, and schema-version failure on the normal read path. Narrowed by #38: the normal read path now HAS a schema gate (`project()` → `assertReadableSchema`), enumerated case-by-case in `test/schema-version-on-read.test.ts`, and **KA-11** no longer points here for it. What remains open under #24 is the PROPERTY-based form — the enumerated cases are hand-written, not generated. +- **Property tests (#24):** **PARTIALLY CLOSED.** `test/pbt-convergence.test.ts` generates actor counts, legal causal permutations, retries, batch boundaries, convergence, and byte-identical double-apply idempotency. `test/pbt-mint-project-roundtrip.test.ts` generates workflows and op streams for mint/apply/project round trips and retry determinism. These are fixed-seed samples, not exhaustive coverage. The normal read-path schema gate is covered case-by-case, not property-based, in `test/schema-version-on-read.test.ts`; generated malformed/non-current schema layouts remain a gap. The round-trip suite also still needs coverage guards proving its optional metadata, definitions, slots, and widget features were exercised. - **Catalog-SHA lint (#22):** ~~fail on moving branch/tag citations in vocabulary and catalog references.~~ **Landed** as `scripts/check-pins.mjs` + `docs/upstream-pins.json` + `test/upstream-pins.test.ts`. Offline in CI; `npm run check:pins -- --verify-remote` additionally proves each pinned SHA still resolves upstream and each cited section still exists at it, and reports INCONCLUSIVE rather than passing when it cannot reach GitHub. Remaining, and the reason KA-12 keeps a scoped UNGUARDED marker: `meta.catalog_version`'s own sha256 shape is asserted by `test/catalog-sha-binding-integration.test.ts` but is not rejected at either write site — neither `mint()` (`src/mint.ts`) nor the module-private `initDoc()` helper (`src/doc.ts`) validates it, so `mint(workflow, catalog, "main")` still succeeds. That is Q4, not this lint. - **Conformance-corpus provenance (#23):** record generator repository and commit SHA, command, environment, and manifest; regenerate and diff in CI. - **Cross-language parity guard (#22):** if the single-package decision is deliberately revisited, both implementations must pass shared golden vectors before merge. diff --git a/docs/adr/0001-op-based-crdt-v1.md b/docs/adr/0001-op-based-crdt-v1.md index 67449fb8..359b9042 100644 --- a/docs/adr/0001-op-based-crdt-v1.md +++ b/docs/adr/0001-op-based-crdt-v1.md @@ -40,9 +40,9 @@ authority" is therefore not a design axis; the axes that matter are (a) applier 4. **Persistence is separate from merge.** Every writer applies then writes; ensure no pending ops before a write; never pull the DB as a higher authority (that collapses back to de-facto full-document last-writer-wins). -5. **Defer the logical clock past V1.** A scalar `base_version` gives deterministic convergence for - V1; a vector/Lamport/HLC clock (faithful causality across long offline branches) is deferred and - requires op-sharing + replay first. See `src/types.ts` (`[base_version, actor, op_id]`) and +5. **Logical-clock decision superseded.** V1 originally deferred a logical clock. [ADR-0005](0005-lamport-ordering-v1-migration.md) + supersedes that choice: producers mint `base_version` as a durable, creator-owned Lamport counter, + while the total winner key remains `[base_version, actor, op_id]`. See `src/types.ts` and `src/stamps.ts`. 6. **Applier stays a pure, portable package** with no client-idiosyncratic local state and no DOM/framework/server-only deps, so it runs identically in browser and host (and a future peer). @@ -96,11 +96,10 @@ flowchart LR re-minted). - **Ordering is independent of arrival order.** The canonical winner of two concurrent ops is computed from the op stamp `[base_version, actor, op_id]`, which every replica evaluates - identically offline — never from receive order. `base_version` is a scalar version cursor the host - advances on apply, not a causal clock; `actor` and `op_id` are pure tie-breakers applied only - after it. The known limitation — a scalar cursor cannot faithfully represent causality across long - independent offline branches — is the deferred logical-clock item, out of V1 scope. Convergence - tests must deliver concurrent ops in both arrival orders and assert an identical winner. + identically offline — never from receive order. [ADR-0005](0005-lamport-ordering-v1-migration.md) + supersedes V1's scalar-cursor design: `base_version` is now a durable, creator-owned Lamport + counter; `actor` and `op_id` remain deterministic tie-breakers. Convergence tests must deliver + concurrent ops in both arrival orders and assert an identical winner. - **Invalid-op batch semantics: valid-prefix commit, abort remainder.** Ops in a batch apply in stamp order. The first invalid op aborts the remainder: the applied valid prefix is committed and persisted, `base_version` advances only for committed ops, and the unprocessed suffix (from the @@ -156,8 +155,8 @@ flowchart LR - **Positive:** future peer-to-peer / LAN, hostless multiplayer on hosted or enterprise deployments, multi-agent replay, offline and double-offline all remain reachable from the V1 shape. Ops give replay, observability, and cheap deltas. -- **Cost:** raw-ops persistence needs new DB tables; reconnect seeding is naive in V1; a proper - logical clock and conflict UX are deferred. +- **Cost:** raw-ops persistence needs new DB tables; reconnect seeding is naive in V1; conflict UX + is deferred. The original logical-clock deferral was superseded by ADR-0005. - **Non-goals for V1:** vector clock, node-level conflict UX, separate-repo extraction of the applier (the package boundary is kept, but extraction into a standalone repo is not required yet). @@ -179,6 +178,6 @@ flowchart LR - **Room host** — the server node process holding the in-memory Yjs doc for a workflow; a V1 convenience, not a conflict authority. - **Headless peer** — the agent replica running with no browser attached. -- **base_version** — scalar version pointer; advances on apply. A V1 ordering aid, not the - permanent conflict key. +- **base_version** — the numeric element of the winner key; originally a V1 scalar version pointer, + now a durable, creator-owned Lamport counter under ADR-0005. - **LWW** — last-writer-wins; here resolved by the op stamp, never by client-id or a DB read. diff --git a/docs/api-contract-proposal.md b/docs/api-contract-proposal.md index 0413e5e7..ab8043a4 100644 --- a/docs/api-contract-proposal.md +++ b/docs/api-contract-proposal.md @@ -9,10 +9,16 @@ Two halves: - **Part 2 — open questions.** Not settled. This is what the meeting is for. Each has a recommendation and the trade-off it costs. -Grounded against the source at `6793d75`, `docs/multiplayer-schema.md`, and the -op vocabulary in comfy-cli. Nothing on a default branch consumes this package -yet — the server doc host lives on an unmerged branch — so a contract change is -still cheap. That will not stay true. +Originally grounded against the source at `6793d75`, +`docs/multiplayer-schema.md`, and the op vocabulary in comfy-cli. +**Consumer-status correction, 2026-09-18:** the newer +[ADR-006](decisions/ADR-006-publish-to-npm-pin-exact-versions.md) records that +the frontend, cloud `services/agent/dochost` sidecar, and `examples/dochost` +consumed the published `0.1.0` package at its 2026-08-22 acceptance. Consumers +pin exact published versions and must be upgraded deliberately. The former +claim that no default-branch consumers existed and contract changes were cheap +is obsolete. This correction does not assert their current deployed versions +or revalidate every historical proposal below. --- diff --git a/docs/decisions/ADR-001-single-shared-applier.md b/docs/decisions/ADR-001-single-shared-applier.md index d1b98fb8..ddb13d0b 100644 --- a/docs/decisions/ADR-001-single-shared-applier.md +++ b/docs/decisions/ADR-001-single-shared-applier.md @@ -14,7 +14,7 @@ The deployed cloud architecture runs this TypeScript package in a stateless Node `@comfyorg/comfy-multi-player` is the one implementation of op-to-document semantics. -- Browser and Node host consume the same package pinned by immutable git SHA. +- Browser and Node host consume the same package at the same exact published npm version (ADR-006). - Do not add op-to-document logic in Go or another language. - Keep applier, projection, and mint pure and portable, with `yjs` as the only runtime dependency. - If a second implementation is ever unavoidable, record an invariant exception first and require shared golden-vector parity before use. diff --git a/docs/decisions/ADR-005-read-only-snapshot-surface.md b/docs/decisions/ADR-005-read-only-snapshot-surface.md index e550aa7c..b2256dcb 100644 --- a/docs/decisions/ADR-005-read-only-snapshot-surface.md +++ b/docs/decisions/ADR-005-read-only-snapshot-surface.md @@ -38,7 +38,7 @@ entrypoint: `readGraph`, `readMeta`, `docCatalogPin`, `hasNode`, constant. Every entry maps to a demonstrated call site above; nothing was added for symmetry. -The surface is enforced, not conventional. Four mechanical properties, each +The surface is enforced, not conventional. Five mechanical properties, each proved by attempting the violation in `test/readonly-surface.test.ts`: 1. **No live handle escapes.** Nothing reachable from a return value is a diff --git a/docs/decisions/ADR-006-publish-to-npm-pin-exact-versions.md b/docs/decisions/ADR-006-publish-to-npm-pin-exact-versions.md index fcfb7a33..e250f3ca 100644 --- a/docs/decisions/ADR-006-publish-to-npm-pin-exact-versions.md +++ b/docs/decisions/ADR-006-publish-to-npm-pin-exact-versions.md @@ -12,6 +12,11 @@ stabilizing and no registry release existed. The package is now published to npm as `@comfyorg/comfy-multi-player@0.1.0`, so consumers no longer need a git checkout or an install-time `prepare` build. +At this decision's 2026-08-22 acceptance, `0.1.0` was the published release. +The repository manifest is currently version `0.2.1` with the +`GPL-3.0-only` license; this current-state note does not announce or perform a +new npm release. + All three consumers use the registry release: ComfyUI frontend, the cloud `services/agent/dochost` sidecar on `main`, and the `examples/dochost` example merged in PR #27. Git-SHA pinning has therefore been retired in practice. @@ -19,7 +24,8 @@ merged in PR #27. Git-SHA pinning has therefore been retired in practice. ## Decision Publish `@comfyorg/comfy-multi-player` to the npm registry and have every -consumer pin an exact published version. The current shared version is `0.1.0`. +consumer pin an exact published version. At acceptance, the shared published +version was `0.1.0`; the repository manifest is currently `0.2.1`. Version updates remain explicit, reviewable changes in each consumer. The frontend, cloud doc-host, and examples MUST use the same exact package version @@ -33,8 +39,9 @@ where they rely on identical applier behavior (ADR-001). do not use version ranges for the shared applier. - Publishing a new version is an explicit maintainer action with its own cadence, separate from merging repository changes. -- Version `0.1.1` is pending to carry the package's license metadata after the - GPL-3.0 license change lands. +- The repository manifest and `LICENSE` now identify the package as + `GPL-3.0-only`; the earlier pending `0.1.1` license-metadata plan is historical + and no longer current. - Consumers install the prebuilt registry artifact and no longer need to allow this package's git-dependency `prepare` script. diff --git a/docs/decisions/EXCEPTIONS.md b/docs/decisions/EXCEPTIONS.md index 5121f914..1d6ba83a 100644 --- a/docs/decisions/EXCEPTIONS.md +++ b/docs/decisions/EXCEPTIONS.md @@ -6,7 +6,7 @@ Deliberate deviations from `docs/INVARIANTS.md` require explicit maintainer revi | --- | --- | --- | --- | --- | | YYYY-MM-DD | KA-0 / FC-0 | Decision and approver | Exact files, release, or deployment | Date or removal condition | | ~~2026-08-21 (PROPOSED)~~ **RESOLVED 2026-08-21** | KA-4 | ~~Four `connect` rejections mutate before validating (issue #10).~~ **Sunset condition met: PR #34 landed.** The block asserting the bug went red exactly as designed, the four rows moved into `CASES` in `test/ka4-rejection-byte-identity.test.ts`, and `KNOWN_KA4_VIOLATIONS` is now empty. Struck rather than deleted so the exception's full life stays legible. | (was `src/applier.ts` `connect` paths) | Expired. Amendment A9 closes the cloneable-but-unstorable, `link_id`, and `removed_links` write-order holes; Amendment A10 closes cycles; Amendment A14 closes the `link_type` shape hole before any write. | -| 2026-08-21 (PROPOSED) | KA-4 | **Permanent convergence carve-outs**, schema §2.5 items 4-6 and 8 / Amendment A6, and the RULE that generates them: a rejection is arrival-order-dependent exactly when the check raising it READS THE DOCUMENT and sits below an early return that consumes the `op_id` (a delete-wins return or the interior null return). Under §4 abort-remainder such a rejection also decides whether the rest of the batch applies, which turns a bookkeeping difference into a projection difference. Item 7's `add_node` exception was resolved by Amendment A7's node-presence stamp gate and its pin flipped positive. Every OP-ONLY precondition is hoisted above those returns and does NOT carve out. Measured in both orders with converging controls. | `src/applier.ts` `applyConnect` / `applySetWidget` / `resolveInteriorNode`; schema §2.5 items 4-6 and 8 | **No sunset — deliberate.** A6 rejects three alternatives with reasons; closing any remaining item either re-opens issue #10 or changes the concrete-input register's semantics, which is a vocabulary change needing a comfy-cli counterpart. Remaining items are pinned by still-diverges tests. Revisit only via a new amendment. | +| 2026-08-21 (PROPOSED) | KA-4 | **Permanent convergence carve-outs**, schema §2.5 items 4-6 and 8 / Amendment A6, and the RULE that generates them: a rejection is arrival-order-dependent exactly when the check raising it READS THE DOCUMENT and sits below an early return that consumes the `op_id` (a delete-wins return or the interior null return). Under §4 abort-remainder such a rejection also decides whether the rest of the batch applies, which turns a bookkeeping difference into a projection difference. Item 7's `add_node` exception was resolved by Amendment A7's node-presence stamp gate and its pin flipped positive. Every OP-ONLY precondition is hoisted above those returns and does NOT carve out. Measured in both orders with converging controls. | `src/applier.ts` `applyConnect` / `applySetWidget` / `resolveInteriorNode`; schema §2.5 items 4-6 and 8 | Owner: UNASSIGNED. Sunset PROPOSED, not agreed: revisit only through a new amendment; A6 rejects three alternatives with reasons, and closing a remaining item either re-opens issue #10 or changes concrete-input register semantics, requiring a comfy-cli vocabulary counterpart. | | 2026-08-21 (PROPOSED; scope widened 2026-08-27 by schema Amendment A15 — a NESTED promoted host write resolves its `instance_path` through the same guard and is rejected the same way) | KA-3 | This applier REJECTS an interior write into a shared subgraph definition where comfy-cli FORKS it, and it counts instances by resolved definition where comfy-cli's `engine._count_instances` matches the literal id string. Pre-declared by schema §5.3, which keeps forking OPEN; recorded here so the divergence has an owner and a sunset rather than living only in source comments. | `src/doc.ts` `countDefinitionInstances`/`definitionAliases`; `src/applier.ts` §5.3 guard | Owner: UNASSIGNED. Sunset PROPOSED, not agreed: the schema §5.3 forking amendment lands and both implementations adopt it. | | 2026-08-21 (PROPOSED) | KA-11 | **The snapshot read surface's fail-closed gate is CONDITIONAL**: `src/read.ts` refuses a document carrying content under an unreadable schema, but returns its empty value for a document carrying nothing, where `project()` refuses that document too. Deliberate — ADR-004's follower reads a root-less document before its first frame and there is no content there to mis-key — but it is a read path that answers where a sibling read path throws, so it is recorded rather than left in source comments. Normative text: Amendment A12. The divergence is disposition-only; both paths use `readSchemaVersion` as the predicate. Mitigated by asking "carries content" against the struct store rather than the v1 root names, so a v2 that renamed its roots is refused rather than read as empty. | `src/read.ts` `assertSnapshotReadable`/`carriesContent`; schema §10 + Amendment A12 | Owner: UNASSIGNED. Sunset PROPOSED, not agreed: revisit when `SCHEMA_VERSION` first moves past 1, since a real v2 makes the disposition split observable to a deployed follower for the first time. | | ~~2026-08-27 (PROPOSED)~~ **RESOLVED 2026-08-28** | KA-3 | ~~**A promoted `connect` (`grow.promoted: true`, schema Amendment A15) is LWW-gated here on `("input", to_node, "grow", name)` and moves the materialized slot's `grow_id` to the winning link; comfy-cli's `_apply_connect` gates neither.**~~ **Sunset condition met: comfy-cli PR #818 (amendment v1.5, `ba0b0b92abcc86b01e8a6704d07088f92afe7aa7`) gates the same register, spelled with the FULL declared name on both sides, and moves `grow_id` to the winner.** Struck rather than deleted so the exception's life stays legible. Both spell the register identically (`_write_target`), so the divergence is in GATING only and is observable only under concurrency comfy-cli's single-writer apply never sees — the corpus session it generated replays green. Gating is required for §2.5 convergence (two connects into one declared input contend for one slot exactly as two concrete connects do, A1). | `src/applier.ts` `claimPromotedInput`; schema Amendment A15 | Owner: UNASSIGNED. Sunset PROPOSED, not agreed: comfy-cli `_lww_gate`s promoted grows on the same register (asked for in the PR that lands A15). | diff --git a/docs/mutation-testing.md b/docs/mutation-testing.md index da96deb5..39e78bde 100644 --- a/docs/mutation-testing.md +++ b/docs/mutation-testing.md @@ -1,107 +1,37 @@ # Mutation testing -Stryker measures whether the test suite detects behavioral regressions in the load-bearing CRDT code. The scope is `src/applier.ts`, `src/stamps.ts`, `src/project.ts`, `src/doc.ts` and `src/mint.ts`; tests, `src/index.ts` (re-exports only), `src/types.ts` (declarations), `src/exhaustive.ts` (a compile-time helper whose one runtime line is a `throw`) and `src/migrate.ts` are not mutated. Two semantic files remain unmeasured and should join the glob next: `src/migrate.ts` (kept out only because PR #30 owned it while this branch was written — #30 has merged and made it the fail-closed read gate) and `src/schema-version.ts`, which #60 added after this glob was set and which now holds the ONE definition of the schema read that `migrate()` and `project()` share. +Stryker measures whether the test suite detects behavioral regressions in the load-bearing CRDT code. The configured scope is `src/applier.ts`, `src/stamps.ts`, `src/project.ts`, `src/doc.ts`, and `src/mint.ts`. Tests, re-export-only `src/index.ts`, declarations in `src/types.ts`, and the compile-time helper `src/exhaustive.ts` are not mutated. `src/migrate.ts` and `src/schema-version.ts` are semantic files that should join the scope in a separately measured change. -`src/doc.ts` and `src/mint.ts` were added in MUT-GLOB-KA4-1 and had never been mutated before that. Do not narrow the glob back: the two files carry the schema §1 doc layout, the §1.2 opaque-widgets routing, the §5.3 shared-definition instance count and the §9 bootstrap-snapshot path, and every one of those was unmeasured while the score read 80.00%. +`src/doc.ts` and `src/mint.ts` joined the scope in MUT-GLOB-KA4-1. Do not narrow it back: those files carry the schema §1 document layout, §1.2 opaque-widget routing, §5.3 shared-definition instance count, and §9 bootstrap-snapshot path. ## The score only means something because the run is pinned -Stryker classifies each mutant `Killed`, `Survived`, `Timeout`, `NoCoverage` or an error, and **scores a `Timeout` as detected**. That is right for a mutant that genuinely cannot terminate (`while (…)` mutated to `while (true)`) and wrong for a mutant that merely ran slowly because the machine was busy — and the report cannot tell those apart on its own. With `timeoutMS` and `concurrency` left at their defaults, the score is therefore a function of host load, and it moves in the flattering direction: **more contention, more timeouts, higher score.** - -Measured on this repo, four runs of one commit on one machine, two configurations by two load levels ("contended" = 20 CPU spinner processes for the duration of the run): - -| Config | Contended | Killed | Timeout | Survived | NoCov | Score | -| --- | --- | --- | --- | --- | --- | --- | -| pinned | no | 732 | 8 | 154 | 31 | **80.00%** | -| pinned | **yes** | 733 | 7 | 154 | 31 | **80.00%** | -| unpinned | no | 732 | 8 | 154 | 31 | 80.00% | -| unpinned | **yes** | 732 | **21** | **141** | 31 | **81.41%** | - -A fifth run of the pinned config on Node 22 — the version `mutation.yml` uses — reproduced the pinned rows exactly, including every per-file column, so the number does not depend on the Node major either. - -The pinned rows are identical to two decimals, and their `Survived` and `NoCoverage` sets are element-for-element identical — the survivor *list*, which is what a coverage-gap audit actually consumes, does not move. Only one non-terminating loop mutant swapped between `Killed` and `Timeout`, and both count as detected, so it is score-neutral. - -The unpinned rows show what the pins are for. Unpinned and idle the number happens to agree; unpinned and loaded it gains 1.41 points, because 13 mutants that really do survive — seven on `validateEnvelope`'s `typeof` guard at `src/applier.ts:203`, six in `project.ts`, none of them anywhere near a loop — exceed the default 5s budget and are scored as kills. - -`stryker.config.mjs` pins `timeoutMS`, `timeoutFactor`, `concurrency` and `coverageAnalysis` for exactly this reason. **Do not unpin them.** A score produced with different values is not comparable to any score recorded here. - -`dryRunTimeoutMinutes` is pinned separately from the mutation score knobs. It -only bounds Stryker's initial unmutated Vitest pass; it does not decide whether -any mutant is `Killed`, `Survived` or `Timeout`. The scheduled workflow reached -Stryker's default 5 minute initial dry-run ceiling on 2026-08-31 before it could -write `reports/mutation/mutation.json`, so the report checker and artifact -upload failed only as downstream symptoms. Raising this dry-run cap keeps the -nightly from failing before the measured mutation phase while preserving the -per-mutant `timeoutMS`, `timeoutFactor`, `concurrency` and `coverageAnalysis` -pins that make scores comparable. - -Incremental mode reuses results only when Stryker proves that the mutant and its -covering tests are unchanged. CI restores `reports/stryker-incremental.json` from -a cache keyed by the operating system, `package-lock.json`, and -`stryker.config.mjs`; changing a dependency or any pinned measurement setting -therefore forces a full run. Each workflow run uses a unique cache save key and -restores the newest compatible report, avoiding GitHub Actions cache-key -immutability while still carrying results forward. - -## Current baseline - -Measured 2026-08-21 on the pinned settings over the five-file glob, at `main` = `9e3e38e`: -**85.24% overall** across 1328 mutants (`applier.ts` 80.55%, `doc.ts` 94.09%, `mint.ts` 88.68%, -`project.ts` 87.43%, `stamps.ts` 89.00%), with 1126 killed / 6 timeout / 168 survived / 28 -no-coverage. The break threshold is **84**, 1.24 points under the measured score for the reason #56 -recorded when it set the three-file threshold to 79 rather than to the 80.00% it had just measured: -a threshold equal to the measurement passes with zero margin and goes red the first time a sibling -PR adds an uncovered line, reporting "mutation score regression" when it means "new code arrived". -Raise the threshold whenever the score is raised; the headroom is for new code, not for measurement -noise, which the pins have removed. The timeouts-as-survivors floor — the score if every timeout -were really a survivor — is 84.79%, which is also above the threshold, so the pass does not depend -on how the 6 timeouts are classified. - -**Host conditions, stated because this number is load-sensitive by construction.** Both runs below -were taken on a 28-core host at 1-minute load averages of **2.16** (baseline) and **0.85** (branch), -Node 25.9.0, pins untouched. An earlier attempt at this measurement was made while the host carried -five concurrent agents at load 15-19; one of its runs was killed by the OOM killer, and none of that -attempt's figures are reported here. Refusing to publish a number measured under those conditions is -the correct outcome, not a gap. - -Per-scope movement, kept because the *shape* of the change matters more than the headline. Both -"before" columns are a real run of the SAME five-file glob against `origin/main` `9e3e38e` (passed -on the command line with `--mutate`, pins untouched), so the only difference between the columns is -this branch: - -| Scope | Before (`9e3e38e`) | After | What moved | -| --- | --- | --- | --- | -| five-file overall | 79.92% (1031/1290) | **85.24%** (1132/1328) | both rows below | -| `applier.ts` + `stamps.ts` + `project.ts` | 81.52% (803/985) | 82.74% (815/985) | the KA-4 rejection sweep (`test/ka4-rejection-byte-identity.test.ts`) | -| `doc.ts` + `mint.ts` | 74.75% (228/305, first ever run) | 92.42% (317/343) | `test/doc-mint-mutation-survivors.test.ts` | - -Reproducibility of the five-file glob, measured rather than asserted. At the previous base this -branch was run twice, at load averages 2.99 and 8.90: identical to two decimals and in every -per-file killed/timeout/survived/no-coverage column. At the base before that it was run twice more -and the `Killed`, `Timeout`, `Survived` and `NoCoverage` **sets** were element-for-element -identical, symmetric difference zero. What is stable is the survivor *list*, not merely the -percentage. The overall score does move between bases — 85.31% at `32ab1f2`, 85.24% here — but only -because `main` itself moved: #60 added one `project.ts` survivor. That is exactly why a "before" is -re-measured at the same commit rather than quoted from a previous run. - -The `doc.ts` + `mint.ts` mutant total rises from 305 to 343 because the fix itself adds code, so the -"after" column is measured over more surface, not less. - -The three-file scope reads 81.61% here rather than the 80.00% (740/925) `main` recorded when the pin -landed, because #67 and #31 have since added tests and code. That is the point of re-measuring the -baseline at the same commit rather than quoting a stored number: a "before" from a different `main` -is not a before. - -Adding files to the glob moves the headline for two reasons at once — new mutants, and new tests — so compare per-file columns, never the single overall number, when judging whether a change helped. - -Earlier figures on this page do not survive. **63.70%** (once recorded here) and **74.81%** (recorded in the workspace) were produced with the knobs unpinned and are void — not low, not high, just not measurements of the test suite. - -One more figure is withdrawn rather than merely stale. An earlier revision of this branch cited -**81.41%** as the "after" for the three-file scope. That number is `main`'s **unpinned, under -contention** three-file figure from the two-by-two table above — a documented measurement artifact, -not a measurement of this branch. Do not re-cite it. - -The triple **84.38% / 88.01% / 74.59%** was previously cited here as the proof of load-sensitivity. It is withdrawn, and for a sharper reason than staleness: those three runs differed in their `coverageAnalysis`/`timeoutMS`/`concurrency` *flags*, not in host load, so they never evidenced the load claim they were offered for. They do show the score is a function of the knobs. The load half is the two-by-two table above, measured here at fixed settings. +Stryker classifies each mutant as `Killed`, `Survived`, `Timeout`, `NoCoverage`, or an error, and scores a `Timeout` as detected. That is appropriate for a genuinely non-terminating mutant and misleading for one that merely exceeded its budget on a busy host. Host-dependent timeout and worker defaults can therefore move the score in the flattering direction by reclassifying survivors as timeouts. + +`stryker.config.mjs` pins `timeoutMS`, `timeoutFactor`, `concurrency`, and `coverageAnalysis` for this reason. A score produced with other values is not comparable. `dryRunTimeoutMinutes` separately bounds the initial unmutated Vitest pass and does not classify individual mutants. + +The scheduled workflow reached Stryker's default five-minute initial dry-run +ceiling on 2026-08-31 before writing `reports/mutation/mutation.json`. +The missing checker report and artifact were downstream symptoms. Raising +`dryRunTimeoutMinutes` lets the unmutated suite finish without changing +per-mutant classification settings. + +Incremental mode reuses a result only when Stryker determines that the mutant and its covering tests are unchanged. CI keys the incremental report by the operating system, dependency lock, and Stryker configuration, so changing a dependency or pinned measurement setting starts a full run. + +Each workflow run uses a unique cache save key and restores the newest compatible +report, avoiding immutable cache-key reuse while carrying results forward. + +## Baseline policy + +No measured baseline is published here without retained output from `npm run check:mutation-report`. The available archived report belongs to a different historical checkout and configuration, so it cannot validate the figures formerly quoted here or in `stryker.config.mjs`. + +The removed tables and withdrawn measurements remain in Git history; they are +not a current baseline and must not be re-cited without their matching report, +configuration, source revision, host conditions, and passing checker receipt. + +The configured break threshold remains **84**. Keep a threshold below a freshly checked baseline rather than equal to it: zero margin turns the first uncovered line in a sibling change into a nominal score regression. Raise the threshold when the checked baseline rises. That headroom is for new code, not measurement noise. + +Adding files to the mutation glob changes both the mutant population and the tests that run. Compare checked per-file outcomes, not only the overall score, when judging a change. ## Running it @@ -112,21 +42,16 @@ npm run test:mutation npm run check:mutation-report ``` -Node 22 or newer. +Use Node 22 or newer. Do not quote a score unless the checker passes and its output is retained with the baseline evidence. -`check:mutation-report` is the fail-closed half of the fix. It re-derives the score from `reports/mutation/mutation.json`, prints `Timeout` as its own outcome next to the "timeouts-as-survivors floor" the score cannot fall below, and exits: +`check:mutation-report` re-derives the score from `reports/mutation/mutation.json`, reports `Timeout` separately, and also computes a floor with every timeout treated as a survivor. It exits: | Exit | Meaning | | --- | --- | -| 0 | conclusive, at or above the break threshold | -| 1 | conclusive, below the break threshold | -| 2 | **INCONCLUSIVE** — no report, fewer than 500 mutants, more than 2% of detected mutants were timeouts, or the report carries no `thresholds.break` to judge against | - -INCONCLUSIVE is not a pass. Re-run on a quieter machine; do not record the score. - -Stryker writes local HTML and JSON reports plus the incremental report under -`reports/`; generated reports and `.stryker-tmp/` are ignored by git. -`.github/workflows/mutation.yml` runs nightly and by manual -`workflow_dispatch`, not on every pull request; it caches the incremental report -and uploads `reports/mutation/` as a build artifact so a failing run leaves -something to read. +| 0 | Conclusive and at or above the break threshold | +| 1 | Conclusive and below the break threshold | +| 2 | **INCONCLUSIVE**: no report, fewer than 500 mutants, more than 2% of detected mutants timed out, or no `thresholds.break` value | + +INCONCLUSIVE is not a pass. Re-run on a suitable host and retain the checker output; do not record the score. + +Stryker writes local HTML and JSON reports plus the incremental report under `reports/`. Generated reports and `.stryker-tmp/` are ignored. `.github/workflows/mutation.yml` runs nightly and by manual dispatch, caches the incremental report, and uploads `reports/mutation/` as an artifact. diff --git a/fixtures/MANIFEST.json b/fixtures/MANIFEST.json index e87c8e25..6d5fb0df 100644 --- a/fixtures/MANIFEST.json +++ b/fixtures/MANIFEST.json @@ -14,7 +14,8 @@ }, "findings.json": { "generator": "crdt-spike/gen_fixtures.py: workflow_ops probe capture", - "sha256": "c9015f64f66e5fe842671cc808c363ddb007abb78f9e5919c339cd2c35297efe" + "notes": "Historical probe capture, not regenerated. The clear-scope prose was manually corrected on 2026-09-18 against test/clear-empty-removed-nodes.test.ts and the current applier; source request: https://github.com/Comfy-Org/ComfyUI_frontend/pull/16644#discussion_r3914401872. Executable vectors are unchanged.", + "sha256": "d1462ed6786745e843f651dea8889b189f875f41358d14fa3cb7fca1b18d691e" }, "lww-vectors.json": { "generator": "crdt-spike/gen_fixtures.py: workflow_ops.apply_op both-order vectors", diff --git a/fixtures/findings.json b/fixtures/findings.json index 4e57155b..b94ce258 100644 --- a/fixtures/findings.json +++ b/fixtures/findings.json @@ -5,7 +5,7 @@ }, { "topic": "clear-scope", - "detail": "`clear` empties nodes, links, and groups but NOT `extra` and NOT definitions.subgraphs; last_node_id/last_link_id are preserved by design (id-reuse guard). _widget_stamps also survive a clear." + "detail": "`clear.removed_nodes` is the authoritative target set: an empty list removes no nodes, while listed node IDs and their incident links are removed. `clear` empties groups but NOT `extra` and NOT definitions.subgraphs; last_node_id/last_link_id are preserved by design (id-reuse guard). _widget_stamps also survive a clear." }, { "topic": "Q6-subgraph-op-scope", @@ -16,4 +16,4 @@ "delete-interior-node": "node 57/27 not found in workflow. Nodes in this workflow: 57 (f2fdebf6-dfaf-43b6-9eb2-7f70613cfdc1), 9 (EmptyLatentImage), 2943932432288920 (LoadImage), 1991397258626261 (SaveImage). Use an id from `comfy workflow slots` / `ls-nodes` \u2014 never rebuild it." } } -] \ No newline at end of file +] diff --git a/stryker.config.mjs b/stryker.config.mjs index 2d61b232..e6a3e821 100644 --- a/stryker.config.mjs +++ b/stryker.config.mjs @@ -6,17 +6,13 @@ * knobs left at their defaults the mutation score is a function of host load * rather than of the test suite — and it moves in the flattering direction: the * busier the machine, the more mutants time out, the HIGHER the score. - * Measured on this tree, unpinned: 80.00% idle and 81.41% under 20 CPU - * spinners, because 13 real survivors were reclassified as timeouts and scored - * as kills. Do not unpin these without re-measuring; any score produced with + * Do not unpin these without a checked measurement; any score produced with * different values is not comparable. * * `coverageAnalysis: "all"` rather than `"perTest"`: `"all"` runs every test * against every mutant, so the outcome cannot depend on Stryker's per-test - * coverage attribution. On this tree the two agree on an idle host (identical - * 154-survivor set), and `"all"` costs about 2.4x the wall time; it is pinned - * because it removes a variable, not because `perTest` was observed to - * mis-classify here. + * coverage attribution. It is pinned because it removes a variable, not + * because `perTest` is assumed to mis-classify here. * * @type {import('@stryker-mutator/api/core').PartialStrykerOptions} */ @@ -55,21 +51,12 @@ export default { timeoutFactor: 1.5, // Fixed worker count so the measurement does not vary with core count. concurrency: 4, - // Measured 2026-08-21 on the pinned settings above, over the five-file glob: - // 85.24% overall (1328 mutants; 1126 killed / 6 timeout / 168 survived / 28 - // no-coverage), on an idle host (load average 0.85). The SAME glob run - // against the parent commit 9e3e38e scores 79.92% over 1290 mutants — a real - // baseline run, not a stored figure — so the delta is this branch and nothing - // else. See docs/mutation-testing.md. - // - // Threshold sits strictly UNDER the measured score, not at it — the rule #56 - // wrote in when it set the three-file threshold to 79 rather than to the - // 80.00% it had just measured (it raised it from 60; it never shipped 80). A + // Threshold must sit strictly under a checker-validated score, not at it. A // threshold equal to the measurement passes with zero margin and turns red on // the first uncovered line any sibling PR adds, a failure that would say // "mutation score regression" while meaning "new code arrived". Raise it // whenever the score is raised; the margin is for new code, NOT for - // measurement noise, which is now zero. + // measurement noise. Pinning alone does not prove noise is zero. thresholds: { break: 84, low: 84,