Skip to content

docs(adr): CRDT-RECOVERY-0029 — sequence-gap recovery replays against the existing follower doc - #16160

Merged
christian-byrne merged 17 commits into
mainfrom
adr/0024-seq-gap-replay-never-wipe
Oct 12, 2026
Merged

christian-byrne merged 17 commits into
mainfrom
adr/0024-seq-gap-replay-never-wipe

Conversation

@christian-byrne

@christian-byrne christian-byrne commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds ADR-CRDT-RECOVERY-0029 for in-app agent CRDT sequence-gap recovery and amends ADR-GRAPH-DOCUMENT-0024 so both records state the same current replacement rule.

Changes

  • Decisions 1-3 record shipped behavior: a sequence gap withholds the uncertain update and replays against the existing follower doc. Only a lineage break replaces it: doc_reset, a reset synthesized after a re-minting or conflicting doc_reseed_result, or a subscribe to another workflow_id.
  • Decisions 4-6 require the follower to use the host's existing lineage_seq: store it with the Y.Doc, send it as known_lineage_seq, and compare it on updates, resets, and subscribe acknowledgements. Until then, a lost reset during a gap remains an accepted exposure.
  • ADR-GRAPH-DOCUMENT-0024 now names the same exposure and future amendment point.
  • The ADR stays Proposed because the follower does not yet check lineage_seq on updates or acknowledgements.
  • Private planning records remain only as non-governing provenance.

Review Focus

Whether decisions 1-3 match the current follower and whether decisions 4-6 match the generated and host doc-frame contracts.

Validation

tsx scripts/check-adrs.ts, ADR link validation, Oxfmt on the changed docs, and git diff --check pass after merging current main.

@christian-byrne
christian-byrne requested a review from a team August 28, 2026 03:27
@dosubot dosubot Bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Aug 28, 2026
@coderabbitai

coderabbitai Bot commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough
📝 Walkthrough

Walkthrough

The PR adds ADR-0027 for lineage-gated replay during CRDT sequence-gap recovery. It defines generation validation, same-generation delta replay, reset-before-replacement behavior, deferred contracts, and adds the ADR to the accepted index.

Changes

Sequence-gap recovery decision

Layer / File(s) Summary
Document recovery policy
docs/adr/0027-seq-gap-recovery-is-replay-never-wipe.md, docs/adr/README.md
ADR-0027 defines immutable Y.Doc generation IDs, same-generation state-vector replay, reset handling for invalid generations, deferred contracts, rejected alternatives, and related references. The ADR index lists it as accepted on 2026-08-28.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested reviewers: drjkl


Merge Risk: 🟡 Moderate · up to ba02f

The documentation change records the sequence-gap recovery rule, but its ADR number conflicts with another open change. Resolve the numbering and update all references before merging to avoid ambiguous or broken documentation.


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
End-To-End Regression Coverage For Fixes Inconclusive The available context lists the changed files as documentation files under docs/adr/, so the source-file condition does not appear to apply. However, the PR title and commit subjects are not provide… Provide the PR title and commit subjects, then reassess the explicit bug-fix-language condition.
✅ Passed checks (6 passed)
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Website End-To-End Regression Coverage Passed The available PR metadata shows changes only in docs/adr/README.md and docs/adr/0027-seq-gap-recovery-is-replay-never-wipe.md. No files under apps/website/src/ or apps/website/public/ changed,…
Adr Compliance For Entity/Litegraph Changes Passed The changed-file list contains only docs/adr/README.md and docs/adr/0027-seq-gap-recovery-is-replay-never-wipe.md. No file under src/lib/litegraph/, src/ecs/, or a graph-entity-related path is…
Title check Passed The title clearly identifies the primary behavior: sequence-gap recovery replays against the existing follower document. The ADR identifier conflicts with the 0027 slot shown in the changes, but the t…
Description check Passed The description provides a detailed summary, review focus, validation results, design decisions, risks, and known ADR-slot conflict. It uses a “Validation” section instead of the template’s “Verificat…

Full details: End-To-End Regression Coverage For Fixes

Explanation

The available context lists the changed files as documentation files under docs/adr/, so the source-file condition does not appear to apply. However, the PR title and commit subjects are not provided. The custom check requires an inconclusive result when commit subjects are unavailable, rather than guessing whether bug-fix language is present.


✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch adr/0024-seq-gap-replay-never-wipe


Comment @coderabbitai help to get the list of available commands.

coderabbitai[bot]
coderabbitai Bot previously approved these changes Aug 28, 2026
@github-actions github-actions Bot added the risk:R1 PR risk grade (advisory shadow check; grader-owned) label Aug 28, 2026

@DrJKL DrJKL left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one correctness issue in the proposed recovery invariant. Details inline.

@DrJKL DrJKL left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Proposed committable language for the recovery invariant.

@christian-byrne

Copy link
Copy Markdown
Contributor Author

Note

Agent-authored update.

  • ADR now distinguishes a lost doc_reset from an ordinary sequence gap.
  • Status preserves ADR-019's provisional 2026-08-30 auto-ratification gate.
Full context for agent readers

Commit 696abf2f5 folds the validated detail from closed duplicate #16183 into ADR-0024. It records that only an explicit lineage break authorizes Y.Doc replacement after projector notification; an ordinary transport gap never does. It also carries the DQ-47 disposition from issue 56, links ADR-019's bounded scalar-v1, reset/remint deferral, and deployed-proof tradeoffs, and leaves transport cursor, missed-reset precedence, and replacement barrier semantics deferred to DQ-24.

Validation: git diff --check passed. Documentation-only change; no runtime tests apply.

@christian-byrne

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@christian-byrne
christian-byrne force-pushed the adr/0024-seq-gap-replay-never-wipe branch from 696abf2 to 30bd71a Compare August 31, 2026 22:58
@christian-byrne

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor
⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@christian-byrne christian-byrne assigned DrJKL and unassigned DrJKL and christian-byrne Sep 1, 2026
Addresses review feedback:\nhttps://github.com//pull/16160#discussion_r3879129889
Addresses review feedback:\nhttps://github.com//pull/16160#discussion_r3883679146
@christian-byrne christian-byrne changed the title docs(adr): 0024 — seq-gap recovery is state-vector replay, never wipe the follower doc docs(adr): 0027 — seq-gap recovery validates lineage before replay Sep 1, 2026

@DrJKL DrJKL left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two issues remain before this ADR is internally consistent.

  1. ADR-0024 says an explicit doc_reset is the sole replacement path. This ADR also replaces the document when the generation ID is missing or cannot be verified. Update ADR-0024 in this PR, or state exactly which clause this ADR supersedes. Otherwise both rules will remain live.
  2. The ADR-019 and DQ-47 links return 404 for repository readers, so they cannot verify the decisions that govern this record. Move the governing material to an accessible source or make this ADR self-contained and treat those links as optional history.

The numeric prefix should be reassigned from the latest main immediately before merge. I recommend Lineage-Gated Replay (LGR) as the stable name across renumbering. The repository still requires the numeric ADR prefix, so the acronym complements the number rather than replacing it.

The inline suggestions tighten the prose, distinguish the proposed protocol from the current implementation, remove expired weekend process detail, and align the title and index.

Comment thread docs/adr/README.md Resolved
@huang47 huang47 removed the risk:R1 PR risk grade (advisory shadow check; grader-owned) label Sep 2, 2026
christian-byrne added a commit that referenced this pull request Sep 2, 2026
- renumber 0027 -> 0028 to resolve the identifier collision with #16160
  (0027-seq-gap-recovery-is-replay-never-wipe.md), which is also open
- rewrite Current implementation to reflect #15381 (merged): both listing
  and deletion now branch on the assetsEnabled/assetDeletionEnabled flags,
  not just the legacy history path
- fold the 2026-08-29 update into the numbered Decision list instead of
  keeping it as a separate dated section
- rename Current Gaps -> Implementation gaps and correct it: decision 7
  (history deletion gated on every record deletion succeeding) is verified
  unimplemented in useMediaAssetActions.deleteAssets on main; only the
  legacy (assetsEnabled=false) path deletes history today
- replace 'unwind' in the meta-review reference bullet with the actual
  non-atomic/no-rollback semantics decision 7 states
- fix three References links: the old asset-record-content-split paths
  in Comfy-Org/ideation-sharing 404; repoint to the real
  asset-deletion/intended/{index,logical,scenarios}.md paths pinned to
  ab6246440c3234fe315e4fc36145c818e5309868

All rewritten claims verified against origin/main source
(useMediaAssetActions.ts, assetsStore.ts) and the referenced repo's
current tree before applying.

Addresses: #16102 (review)
DrJKL
DrJKL previously requested changes Sep 4, 2026

@DrJKL DrJKL left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two consistency issues from my prior review remain.

ADR-0024 says an explicit doc_reset is the sole ordinary follower-document replacement path, while this ADR also requires replacement when generation validation is missing or fails. The records need one rule. The ADR-019 and DQ-47 references also still return 404, so the governing claims are inaccessible.

Maintainability-skill finding: no new finding beyond these duplicated architectural authorities. The renumbering to ADR-0029 is internally consistent. The branch conflicts with the current ADR index and must be updated from main.

Checks: frozen install, Oxfmt, and git diff --check passed; both external references returned 404. Documentation-only typecheck and tests were not run.

state vector.
2. The follower keeps its existing Y.Doc and applies the returned delta only after the
host confirms the same generation.
3. A missing, unverifiable, or different generation is a lineage break. The follower

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ADR-0024 still says an explicit doc_reset is the sole ordinary follower-document replacement path, while this rule also replaces the document when generation validation is missing or fails. Update ADR-0024 here or state exactly which clause this ADR supersedes so both accepted records define one replacement rule.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You were right that the two records disagreed, and checking the code first turned up something worse: the ADR disagreed with the implementation, not just with ADR-0024.

What the follower actually does, on current main:

  • layoutFollowerBridge.ts:283-295 — a seq gap withholds the uncertain frame, dispatches doc_gap, and calls resubscribe(). The doc is kept and the subscribe carries its existing state vector.
  • layoutFollowerBridge.ts:342 — dropDocForNewLineage() is reached from exactly two callers: onDocReset (:333) and a subscribe() naming a different workflow_id (:174-175).

So an explicit doc_reset (or a cross-workflow subscribe) is the only replacement path, and a gap never replaces the doc. That is ADR-GRAPH-DOCUMENT-0024, implemented as written.

The old decision 3 here said "a missing, unverifiable, or different generation is a lineage break". No frame carries a generation ID — grep -rn generation src/workbench/extensions/agent/crdt --include=*.ts returns nothing outside tests, and docFrameClient.ts:420-425 sends only workflow_id and state_vector_b64. Read literally, "missing generation" is always true, so that clause mandated replacing the doc on every gap: the opposite of decision 2, of the title, and of the code.

Rewritten in 73b5690 to split the two:

  • Decision 1-3, "Current rule (implemented)", restates ADR-GRAPH-DOCUMENT-0024. This ADR now says explicitly that it supersedes no clause of it.
  • Decision 4-6, "Required protocol change (not implemented)", is the generation ID as a proposed wire change. Decision 5 says ADR-GRAPH-DOCUMENT-0024 gets amended when it ships, not before, and there is an explicit line that decision 5 must not be read as "absent generation means lineage break".

ADR-GRAPH-DOCUMENT-0024 gained a matching paragraph in its reconnect section plus a Glossary qualifier, so both records now name the same single rule and the same open exposure.

One thing I did not paper over: status went from Accepted back to Proposed. You asked for Accepted earlier because the index said Accepted, but main no longer carries that row, and the substance of this ADR is a protocol change nothing implements. Marking that Accepted while the code does something else is the failure mode this thread caught. Happy to flip it back if you read it differently.

Also worth flagging, since it bears on decision 6 being an accepted risk rather than a theoretical one: #16372 shows a rejected Yjs update mutating the replica before throwing, with the state vector advancing past it, and #16353 shows a resubscribe carrying a vector that already covers a withheld update, so the host returns no delta. Neither is in scope here, but both are the same ambiguous-recovery surface.

## References

- [ADR-0003](0003-crdt-based-layout-system.md) — centralized layout management with CRDT.
- [ADR-019 — CRDT weekend final call](https://github.com/christian-byrne/in-app-agent-program/blob/main/decisions/ADR-019-crdt-weekend-final-call.md)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reference and the DQ-47 reference below still return 404 for repository readers. Move the governing material to an accessible source, or make this ADR self-contained and identify these links as optional history.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, and the cause is worse than a bad path. Both targets exist but sit in private repos:

  • christian-byrne/in-app-agent-program — .private is true. decisions/ADR-019-crdt-weekend-final-call.md is really there, but only I can see it.
  • christian-byrne/blocked-on-christian — also .private: true. Issue 56 is "URGENT DQ-47: choose one CRDT sequence-gap ADR vehicle and owner".

So they 404 for every reader of this repo and govern nothing here. Same shape as #16191, where the cited ADR-017 turned out to live in that same private repo.

A third link was dead for an unrelated reason: [ADR-0003](0003-crdt-based-layout-system.md) pointed at a filename that no longer exists — main renamed it to CRDT-LAYOUT-0003-crdt-layout-intent-and-local-measurement.md. pnpm adr:check rejects the old form outright as a legacy numbered reference.

Fixed in 9c28f37:

  • Both private links dropped as references. What they settled that still governs is now stated inline — the deferred-contracts section names the three open contracts itself instead of pointing at a tracker nobody can open.
  • They survive only as a trailing "Non-public history (not governing)" note that says plainly the records are unreadable from this repo and confer no authority. I would rather name the provenance and mark it unreachable than quietly delete it, but tell me if you would rather it go entirely.
  • The ADR-0003 reference now points at the renamed file.

The ADR is self-contained now: nothing in the decision depends on a link a reader cannot open.

Resolves the docs/adr/README.md index conflict in favor of main's
TOPIC-NNNN identifier scheme. The new ADR's index row is re-added under
its renamed identifier in the following commit.
The follower replaces its doc in exactly two places -- an explicit
doc_reset, and a subscribe naming a different workflow_id. A sequence
gap withholds the frame and resubscribes with the existing doc's state
vector; it never replaces the doc. That matches ADR-GRAPH-DOCUMENT-0024.

The previous decision 3 ("a missing, unverifiable, or different
generation is a lineage break") contradicted that code: no frame carries
a generation ID today, so read literally it mandated replacing the doc
on every gap. Split the decision into the implemented rule and the
proposed protocol change, state that this ADR supersedes no clause of
ADR-GRAPH-DOCUMENT-0024, and amend that ADR to name the same exposure so
both records define one replacement rule. Status drops to Proposed
because the generation ID is not implemented.

Also renames the file to the TOPIC-NNNN scheme now on main and restores
its index row.

Addresses review feedback:
#16160 (comment)
Both cited sources live in private repositories
(christian-byrne/in-app-agent-program and
christian-byrne/blocked-on-christian), so they 404 for every reader of
this repository and govern nothing here. Everything they settled that
still applies is stated inline, and they are kept only as a clearly
labelled non-governing provenance note.

The ADR-0003 reference was also dead: that file was renamed to
CRDT-LAYOUT-0003-crdt-layout-intent-and-local-measurement.md on main.

Addresses review feedback:
#16160 (comment)
@christian-byrne christian-byrne changed the title docs(adr): 0029 — seq-gap recovery validates lineage before replay docs(adr): CRDT-RECOVERY-0029 — sequence-gap recovery replays against the existing follower doc Sep 11, 2026
@github-actions

github-actions Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

🧩 Custom Nodes Ecosystem Matrix: 🟡 Withheld — could not measure

Executes the frontend JS of every registry pack that ships any, against this change, in both renderers. The eight ecosystem-matrix shard checks are a harness self-test; the pack verdict is matrix-verdict.

This candidate changes nothing the matrix measures, so every expensive stage was skipped.

Reproduce locally
# build one shard's fixture, then run it
python3 scripts/registry-census/build_matrix.py --shard 1/4
MATRIX_RENDERER=vue MATRIX_VUE=1 \
  npx vitest run --config vitest.matrix.config.mts

Full docs: scripts/registry-census/README.md

🌾 Fallow: ✅ Passed

Fallow audits the diff for duplication, complexity and dead code. It gates on new findings only, so when the audit completes, a red is something this PR introduced or moved — never inherited backlog. A red can also mean the audit itself did not finish; the details below say which happened.

No new findings in the changed files.

Reproduce locally
# the base the gate compared against
npx fallow audit --root . --quiet --format json \
  --changed-since ae3afc4705a12efcaecde072e5299be0befc4767

Full docs: ADR-DEVEX-LINT-0015

DrJKL
DrJKL previously requested changes Sep 15, 2026

@DrJKL DrJKL left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: changes requested

The two prior consistency findings are addressed: the ADR now separates the shipped replacement rule from the unimplemented generation-ID proposal, and its governing rationale no longer depends on private references.

One integration blocker remains. This head still conflicts with current main in docs/adr/README.md (mergeable_state: dirty; merge-tree conflict reproduced against origin/main at 53d8eb9f). Please update the branch, retain both ADR index additions while resolving the table, and rerun the claimed merged-tree checks. The PR description currently says this conflict was already cleared, but the live head is still based on dd1867ba.

I also left one committable suggestion for the protocol wording: not every accepted frame carries a sequence number.

Checks: live head fetched and confirmed as 9c28f37a; current-main merge conflict reproduced; adr:check passed via the direct tsx entry point; Oxfmt passed on all three changed files; git diff --check passed; CI is green for the head. Documentation and maintainability lanes completed. Fallow, unit tests, and typecheck are not applicable to this Markdown-only diff.

## Context

The in-app agent's CRDT follower integrates Yjs updates from one authoritative doc
host. Every frame carries a sequence number, so transport loss can leave a gap.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doc_subscribed and doc_ops_result accept seq only optionally, while awareness has no sequence field, so “Every frame” overstates the protocol. Narrow this to the frame whose sequence actually drives gap detection:

Suggested change
host. Every frame carries a sequence number, so transport loss can leave a gap.
host. Every `doc_update` carries a sequence number, so transport loss can leave a
gap.

@DrJKL DrJKL added agent Comfy Agent product, integration, tests or product documentation agent-crdt Agent document synchronization, graph projection or CRDT operations agent-docs Agent product architecture, contracts or documentation labels Sep 24, 2026
@DrJKL
DrJKL dismissed coderabbitai[bot]’s stale review September 24, 2026 01:56

Dismissed at DrJKL's request during the Agent PR survey. This does not mark the findings as fixed or approve the PR.

@DrJKL DrJKL left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on 2483eb74148bab979eb98bb80254edaf7568d880.

I pushed two commits to unblock this ADR:

  • a4206bc5 merges current main and keeps both new ADR index rows.
  • 2483eb74 replaces the obsolete generation-ID proposal with the existing lineage_seq contract and fixes the sequence-number wording.

Relevance check: #18700, #20172, and closed #16375 do not replace this decision. The host and generated contract already carry lineage_seq, but the frontend still uses it only on doc_reset. It does not read it from updates or subscribe acknowledgements and does not send known_lineage_seq. The ADR is still needed to record the missed-reset exposure and the follower change.

The earlier findings are resolved: the ADR and GRAPH-DOCUMENT-0024 state one current replacement rule, private records do not govern the decision, the ADR-index conflict is gone, and the protocol wording matches the code.

Checks: ADR validation, ADR links, Oxfmt, and git diff --check passed locally. Required CI passed on this head, including lint, typecheck, Fallow, unit, and E2E gates.

@DrJKL
DrJKL dismissed stale reviews from themself October 12, 2026 08:49

Resolved on 2483eb7: ADR-0024 and CRDT-RECOVERY-0029 now state one current replacement rule, and private records no longer govern the decision.

@christian-byrne
christian-byrne added this pull request to the merge queue Oct 12, 2026
Merged via the queue into main with commit ef9b2de Oct 12, 2026
85 checks passed
@christian-byrne
christian-byrne deleted the adr/0024-seq-gap-replay-never-wipe branch October 12, 2026 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent Comfy Agent product, integration, tests or product documentation agent-crdt Agent document synchronization, graph projection or CRDT operations agent-docs Agent product architecture, contracts or documentation flag-exempt risk:medium PR risk grade (advisory shadow check; grader-owned) size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants