Skip to content

feat(control-plane): add recoverable shared-authority foundation - #3669

Merged
huangruiteng merged 10 commits into
mainfrom
feat/stage3-recoverable-execution
Sep 1, 2026
Merged

feat(control-plane): add recoverable shared-authority foundation#3669
huangruiteng merged 10 commits into
mainfrom
feat/stage3-recoverable-execution

Conversation

@wchwawa

@wchwawa wchwawa commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

TLDR

Stage 3 adds the recoverable-execution reference lifecycle on top of the merged authority core and aggregate/provider seam: renew, release, expiry-plus-grace reclaim, atomic completion with successors, stale-executor fencing, store-lineage fencing, and exact receipt replay.

This revision also widens the original NoKV proposal into the provider-neutral LoopX Shared Control-Plane Authority and Pluggable State Providers v0 RFC. LoopX remains the only semantic authority; file, NoKV, and PostgreSQL are storage-provider profiles behind that authority. Agents never write provider storage directly.

What this PR delivers

  • Preserves the contributor's cohesive Stage 3 lifecycle and evidence work.
  • Integrates current main, where native task-lease acquire is owned by the TypeScript control-plane boundary. The Python Stage 3 executor remains a coverage-only reference slice and does not become a second production authority.
  • Publishes file store_identity only under the cross-process lock through complete temp write, file fsync, atomic rename, parent-directory fsync, and exact file:<32 lowercase hex> validation.
  • Adds deterministic concurrent-create, short-write, crash/retry, directory-fsync retry, and invalid-identity tests.
  • Defines one semantic authority and three provider profiles:
    • built-in file parity backend;
    • optional NoKV shared-store provider;
    • optional PostgreSQL provider behind an authenticated LoopX authority service.
  • Records the staged delivery plan: TypeScript-owned store contract and file parity, parallel NoKV/PostgreSQL provider work, one-way shadow parity, one-Goal/two-Agent TEST ONLY canary, then a single authority-source flip. No long-lived dual-write or dual-master mode.

Product and architecture judgment

The user problem is not "put LoopX files in a database". It is preserving one legal, auditable chain for claim, lease, Turn, quota, settlement, and receipts when multiple Agents or hosts advance one Goal. The reusable boundary is therefore:

Agent -> LoopX authority -> typed transactions -> file | NoKV | PostgreSQL

Provider revisions, atomic persistence, cursors, and durable readback belong to the store contract. Goal/Todo meaning, legal transitions, fencing, idempotency, and receipt meaning stay in LoopX. This keeps the current slice valuable without prematurely promoting its reference executor or coupling the public RFC to a private workflow deployment.

Delivery boundary

The owner accepts this as a foundation/reference-contract slice. That acceptance does not make the modules a shipped production caller and does not waive correctness or review gates. A real authority-source promotion still requires the provider-neutral TypeScript transaction boundary, local-writer fencing, authorization publishing, projection flip, rollback, retention/capacity decisions, shadow parity, and the bounded canary.

Validation

  • Exact head: 6caffb99851c4bf2c573f414b595848d93123b7e.
  • Focused Python coordination suite: 180/180 passed.
  • TypeScript control-plane suite: 295/295 passed.
  • Strict mypy: 7 coordination source files passed.
  • Ruff: changed coordination/governance/test paths passed.
  • TypeScript control-plane typecheck: passed.
  • Documentation publication smoke: passed.
  • Risk-based premerge: 18/18 selected checks passed, including the public/private boundary scan.
  • Strict change-quality receipt: cqr_b3ed99cd008026702379, exact fingerprint b3ed99cd008026702379a0d87d129b8ab0a038fddda58b7d78ad798ccdd79ee3, verified valid with no unresolved blocker.

The prior head's reported live NoKV restore/SIGKILL evidence was not rerun in this revision. It remains qualification evidence from the contributor, not a new independent claim by this update. New GitHub checks must pass on the pushed head.

Remaining gate

This changes control-plane runtime/reference code and is not eligible for admin self-merge. After the exact-head validations pass, it requires a fresh independent review before merge.

@wchwawa

wchwawa commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator Author

@huangruiteng Stage 3(可恢复执行权)就绪,请评审。

三句话版本:§1.2 horizon 行完整落地——renew/release/reclaim/complete 四动词 + stale-fence 规则 + store-lineage 绑定围栏,全部委托 Stage 1 core、跑在 Stage 2 seam 上,NoKV 侧零代码改动(0.11 的 workspace_incarnation_id 直接承载围栏,live 对真实 restore 验证)。reclaim 组合经三方案对真实 core 的 battery 选出(朴素 acquire-first 死于 owner_conflicts_with_claim;胜出形态=最小特权委托 unclaim + 普通 claim 组合,新租约过真实 holder gate)。完成记录逐字段过生产 durable-completion 投影 seam。

三个需要你决策的点都写进了 RFC:§12 Q1 的拟议答案(lifecycle+completion 一起、内部有序);§7.1 封存段修正案(三方案 live 实测:head 缩 19 倍/重发布少 7.5 倍/延迟平坦,未生效等你的 Q5 裁决,切片本身仍跑 retain_all_v0);§6.4 的续约推进 todo_revision(关闭侦察审计发现的 renew-vs-reclaim rebase 空洞)。

证据:102 个焦点测试(含逐边时钟边界与被替代写回证明)、全量 1917 过(4=已知环境固有)、strict mypy 15 文件、live 十二行 file↔NoKV parity + 真实 restore 围栏行 + 61 秒 SIGKILL 生命周期演练 + K≤8 并发包络实测。独立复跑还揪出两个 NoKV 存储面缺陷(NoKV-Lab/NoKV#492 恢复发布楔死、#493 写窗口 SIGKILL 损坏 manifest),已如实上报并写进 §11——协调层语义两案均保持正确,但生产 canary 可用性依赖修复。

@wchwawa
wchwawa requested a review from huangruiteng August 26, 2026 12:00

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

结论:REQUEST_CHANGES。我按 exact head d34b5df0b46fa87ca1466c6e5b55a16282d96d6a 读完了13个改动文件和完整 diff。整体的 fence/CAS 构型是清晰的,但当前有四个会阻断合并的问题:可以用非有限或负的 reclaim grace 提前夺走有效 lease;evidence 合同会把本地绝对路径和任意隐私分类持久化进共享 head;同一 loopx_coordination_head_v0 无迁移地增加必填 store_binding,所有 Stage 2 旧 head 会在 load 时直接失效;以及新增的运行时命令面仍没有任何 LoopX 产品调用者。

动机

旧行为只能 claim_work:lease 过期后没有合同化的续约、释放、接管和完成路径,旧执行者的写回也没有一条被 exact epoch fence 终态拒绝的端到端证明。这个 PR 试图把 renew_work / release_work / reclaim_work / complete_work 一起建在 Stage 1 authority core 和 Stage 2 provider CAS 之上,并用 store_binding 防止 restore 后的冻结字节自动获得新现场权限。这个问题值得在同一个生命周期切片内解决;但§7.1 的 sealed-segment 方案仍是 owner 未决的后续设计,它不是本次 retain_all_v0 运行时的必要修复,建议单独保持为设计决策而不扩大当前交付面。

改动思路

命令从 sample_*_envelope / _semantic_request 进入,先对每个动词做 closed-field 校验并计算 request digest;然后 apply 加载和验证 head,通过 _store_binding_fence 对比 provider lineage,再路由到五个 transition。holder 动词共用 _held_lease_context,依次检查 todo revision、lease fence、holder 和 authority-clock liveness;reclaim_work 在过期+宽限后先做最小的 delegated clear-claim,再复用普通 acquire+claim;complete_work 把 done 记录、lease 退役和 successor 创建放进同一次 CAS。成功路径是 claim → renew → expiry+grace → reclaim 铸造 epoch+1 → 旧 fence 写回被拒绝 → 新 holder 完成并原子创建 successor;负路径是任何 reload 看到 lineage 不符都返回 store_lineage_mismatch,不进入 CAS。

这个分层本身合理,且 holder opening 与 acquire+claim tail 的提取比重复实现更容易证明。但当前精确 head 上,git greptests/examples/ 之外找不到 CoordinationAuthorityExecutor(...) 或 bootstrap 的生产调用者;所以现在只证明了一个可运行的参考实现,还没有从 LoopX 主机动作到可观测共享状态的 shipped caller path。

具体改动

精确 merge-base…head diff 为 13 个文件、+2212/-119:4 个 production coordination 文件 +894/-77,7 个 test/example 文件 +959/-20,中英文 RFC +359/-22,没有 generated 或 mechanical move。

关键代码讲解

  • CoordinationAuthorityExecutor._semantic_request (executor.py:285) 为每个 verb 定义闭合字段集,绑定 actor/goal/command 并将 transport 排除在语义 digest 之外;这让 exact operation replay 与传输重试分离。
  • _held_lease_context_reclaim_transition (executor.py:661, 869) 分别拥有 holder verb 的共享开场和过期接管;前者将 liveness 判断作为 typed snapshot 交给 core,后者在 delegated clear 后返回普通获取路径。
  • _complete_transition (executor.py:967) 在 holder gate 后检查 successor 冲突,调用 core terminal fence,然后在一个 next head 里写 done 记录、evidence、移除 lease 并创建 revision-0 successors;validated_head 反向验证 done/continuation/successor 闭包。
  • FileCoordinationProvider.store_identity / NoKVCoordinationProvider.store_identity 为 provider 提供 lineage id,_store_binding_fence (executor.py:560) 在初次 load 和每次 CAS reload 后都重验。file 实现的 copy-with-identity 残余风险已在 RFC 说明,NoKV 使用 workbench incarnation 作权威边界。

对主干的风险

  1. [P1] 宽限配置可以直接绕过 lease 有效期。 executor.py:281 只做 float(reclaim_grace_seconds),没有 finite/non-negative 校验;_reclaim_transitionexpired_for < grace 在 grace=NaN 时永返 false。我在刚 claim 的600秒有效 lease 上传入 NaNreclaim_work 立即 applied 并把 owner 从 agent-a 换成 agent-b;grace=-1 时也能在过期前0.5秒接管。这与 RFC “宽限下限是最大时钟偏差、偏差只能延迟不能提前接管”直接矛盾。最小修复是在构造/配置边界拒绝 bool、负数、NaN±inf,并添加“active lease 在任何非法 grace 下不得被接管”的负例。

  2. [P1] evidence 边界会持久化本地/私有路径,且 privacy_class 是任意字符串。 executor.py:402-413head.py:293-305 只要 pointer 和 privacy class 非空就接受。我使用绝对 pointer /private/example/secret.log 和拼错的 privacy_class=publci 执行 complete_work,结果是 applied,两个值都进入了共享 head。这破坏 RFC 对“无 absolute path / 只有经评审的 opaque pointer+privacy class”的公开/私有边界,也违反 typed-state 要求。最小修复是复用或定义一个 provider-neutral evidence schema:pointer 必须是可携带的 artifact id/URI,显式拒绝绝对路径和 file://;privacy class 使用闭合 enum,并对路径、未知分类和 private-pointer/public-class 错配添加负例。

  3. [P1] 在不更换 schema version 的前提下将 store_binding 变为必填字段,旧 v0 head 无法升级。 head.py:31-44 仍声明 loopx_coordination_head_v0,但 _HEAD_FIELDS 已强制新字段,validated_head 先做 exact-field 校验。我用合并基线会生成的旧 v0 shape(唯一差异是没有 store_binding)读取,结果是 HeadValidationError: coordination head fields do not match v0;所有 Stage 2 已持久 head 都会在任何命令前失效。不能通过“自动绑到当前 provider”修复,因为那会给 restore copy 授权。最小修复应是新 schema version + 显式、operator-reviewed 的 re-bootstrap/migration 路径,并用真实旧 v0 fixture 验证“旧 head 被识别为需迁移,不是无分类崩溃;新 binding 只能通过显式操作产生”。

  4. [P1] 新命令面没有 active production caller。 精确 head 上的全库搜索只在 tests/examples 中构造 CoordinationAuthorityExecutorgoal_state_shadow 也没有 LoopX 运行时调用者。因此本 PR 新增的 859 行 executor/head 产品逻辑没有一条用户/主机动作能到达,所谓 Stage 3 只有参考与 live-example 证据,并未 ship。按当前 control-plane scope-fit 合同,test-only 覆盖的未使用运行时是阻塞项,除非 owner 显式接受 coverage-only 边界。最小修复是二选一:把它连到一个明确 opt-in 的共享权威 runtime/CLI,增加从入口到持久 receipt 的 e2e;或把这一切保留为 experimental/example 提供者,记录 owner 接受的 coverage-only 范围,不将其宣称为已交付的 production capability。

除上述问题外,typed status/continuation/command closed sets 设计正确,命令和拒绝文案保持了 goal-neutral;没有把 machine-enforced obligation 误称为 guidance。但 grace 下限未机器强制、未分类的 privacy class 以及 v0 默认读取行为静默改变,分别不满足 guidance-vs-obligation、typed-state 和 behavior-change-disclosure 镜头。最强回归场景是:一个错配 NaN grace 的 authority 在 holder 的600秒 lease 刚建立时就接受 peer reclaim,旧 holder 继续工作但所有写回突然被新 epoch 拒绝;爆炸半径是该 authority 下的所有 shared goals,现有 receipt 能审计但不能恢复被抢占的现场工作。

我的整体评价

我独立运行了 102 个 coordination 测试(全过)、改动面 ruff(过)、strict mypy 15 文件(过)、git diff --check(过),以及 file-provider live matrix 的12行(全过);GitHub 上11个已跑 check 也均为 success/expected skip。我没有使用 NoKV 凭证重跑真实 restore/SIGKILL 行,所以作者报告的 NoKV live 证据对本次独立评审仍是 unverified。

正面看,旧 fence 的终态拒绝、reclaim 复用真 holder gate、completion+successor 原子性和 reload 后重做 lineage fence 都有较完整的正负测试,内部 helper 提取也是有价值的 bounded future-facing refactor。但在未先关掉上述权限/隐私/迁移边界、并给出 active caller 或 owner 接受的 coverage-only 决策前,我不认为这个 exact head 可以进主干。修复后请保留目前的时钟边界、并发和失响应 replay 覆盖,再加上上述三类负例和一条产品入口 e2e,我可以对新 exact head 复评。

English verdict: REQUEST_CHANGES on exact head d34b5df: reject non-finite or negative reclaim grace, enforce a portable typed evidence/privacy schema, version and migrate the new store binding, and provide an active production caller or an owner-accepted coverage-only boundary; 102 focused tests, ruff, strict mypy, diff-check, and the 12-row file-provider matrix passed, while NoKV live evidence was not independently rerun.

The RFC section 1.2 horizon row, delivered over the Stage 1 core and the
Stage 2 aggregate: renew_work, release_work, reclaim_work, complete_work,
stale-fence rejection on every fence-carrying verb, and the store-lineage
binding fence. Every domain decision stays delegated to authority_core;
the executor owns exactly what the core refuses - expiry adjudication
against its own clock and the loaded head's expires_at, the reclaim grace
window, envelope normalization, and the aggregate writeback.

Verb compositions (selected by a three-way battery against the real
core; the naive acquire-first reclaim dies on owner_conflicts_with_claim):
renew delegates to LeaseRenewCommand and advances todo_revision so the
validity interval is revision-covered - a reclaim carrying pre-renew
observations conflicts instead of surviving the internal rebase; release
clears the claim under the live holder gate and then releases, deleting
the lease entry while last_lease_epoch keeps the no-ABA watermark;
reclaim is a standing delegation to eligible agents - authority-clock
expiry plus a configurable grace window, then the minimal delegated
clear_claim followed by the exact claim_work composition, minting
epoch+1 and recording whom it superseded; complete drives the core
COMPLETE terminal fence, retires the lease in the same transition,
writes the explicit continuation the local durable-completion seam
demands, and creates successor todos atomically with inherited context.

The head codec grows conditionally: done records carry
completion_continuation (mandatory), optional no_followup /
successor_todo_ids / evidence pointer with the local contradiction
rules; receipts validate per-verb closed sets; the status vocabulary is
pinned to open|done (previously unvalidated); a lease attached to a done
todo fails closed; store_binding is required and bootstrap embeds the
provider-issued identity - a head restored into a different store
lineage fails every command closed with store_lineage_mismatch until an
explicit re-bootstrap, on both the file provider (exclusive-create
identity file) and the NoKV adapter (the never-reused
workspace_incarnation_id, verified live).

A holder gate rejects a correct fence in the wrong hands as typed
not_lease_holder, keeping owner_conflicts_with_claim unreachable and
fail-closed. Clock-boundary tests pin the grace window edge for edge;
the superseded-executor proof shows every stale-fence write terminally
rejected with the head untouched; the completed record round-trips
through the production project_durable_completion_outcome seam.

Signed-off-by: wchwawa <wch19961116@gmail.com>
The eight-scenario live matrix grows to twelve shared rows plus one
NoKV-specific row, all driven by the production executor: renew extends
the active lease without minting an epoch, an expired lease is
reclaimed with epoch+1 and the superseded owner recorded, every write
the superseded executor sends with its old fence is a typed
stale_lease_fence rejection, and completion creates a claimable
successor atomically while the parent's lease retires. The matrix
clock is explicit and advanceable because expiry adjudication is the
authority's own decision; the provider underneath stays real.

The NoKV-only row exercises the binding fence against a REAL restore:
a head bootstrapped in workbench A, committed, snapshotted, and
restored into workbench B refuses every command on B with typed
store_lineage_mismatch (the never-reused workspace_incarnation_id
differs) while A keeps serving - restored bytes never grant live
authority. Parity is computed over shared rows, with provider-specific
rows reported separately. Probes and the deterministic provider carry
the store-binding bootstrap signature.

Signed-off-by: wchwawa <wch19961116@gmail.com>
Both language mirrors gain the normative Stage 3 surface: sections
5.2-5.5 define the four verbs' closed field sets, preconditions, and
transitions; 5.6 states the stale-fence rule (a stale fence is a
terminal typed rejection the internal rebase never crosses); 6.4
assigns expiry adjudication to the applying authority's own clock with
the reclaim grace window, requires renewal to advance todo_revision so
the validity interval is revision-covered, and specifies the
store-lineage binding fence over the new provider store_identity()
verb, including the file provider's copied-directory residual and the
NoKV incarnation identity as the authoritative fence.

Section 7.1 proposes the sealed-receipt-segment amendment the Section 7
text requires before receipts may leave the single document - with the
live three-design battery evidence (19x smaller head, 7.5x less
republish, flat latency, +3% round trips, exactly-once sealing from
NoKV's root-scoped publication-id replay) - explicitly not in force
until the owner decides Q5. Section 12 Q1 records the proposed answer
(lifecycle and completion together, sequenced internally). The Section
11 status subsection carries the measured gates: twelve-row live
parity plus the real-restore fence row, the ~61 s SIGKILL lifecycle
drill, the K<=8 concurrency envelope, and the two NoKV storage-plane
defects the independent reruns surfaced (recovery-publication wedge at
a dead lease's epoch; metadata manifest corruption from a mid-write
SIGKILL), reported upstream rather than worked around silently.

Signed-off-by: wchwawa <wch19961116@gmail.com>
…-claim tail

The three holder verbs adjudicated the same opening (prechecks, stale
fence, holder gate, liveness snapshot) in copy; reclaim's tail repeated
the claim composition verbatim. Both are now single functions, which
makes the contract's claim that a reclaimed lease passes the same true
holder gate as any first claim structurally enforced rather than
textual. In complete_work the holder gate now precedes the successor-id
check: ownership is adjudicated before payload semantics, so a
non-holder learns nothing about successor-id availability.

Signed-off-by: wchwawa <wch19961116@gmail.com>
Signed-off-by: wchwawa <wch19961116@gmail.com>
Signed-off-by: wchwawa <wch19961116@gmail.com>
Signed-off-by: wchwawa <wch19961116@gmail.com>
@wchwawa
wchwawa force-pushed the feat/stage3-recoverable-execution branch from d34b5df to aecf4ee Compare August 27, 2026 01:49
@wchwawa

wchwawa commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator Author

@huangruiteng I updated #3669 on exact head aecf4eec80298d0218d35f93b8737e0ff77d697c and reran the review from RED through real NoKV recovery.

The first three requested changes are fixed:

  1. Invalid reclaim grace is rejected at construction: bool, negative, NaN, both infinities, and conversion overflow. The original active-lease takeover reproducer is now a negative test.
  2. Evidence is restricted to a privacy-bound portable artifact URI, SHA-256 digest, and closed privacy enum. Absolute/file/provider paths, URI metadata, traversal, and privacy mismatches cannot enter the shared head.
  3. The head is now v1. Real Stage 2 v0 bytes receive typed head_schema_migration_required, then require an explicit operator-attested store binding. Migration never infers authority from the loading provider.

Adversarial TDD found and closed one additional migration authority hole: a v0 head could previously disagree with its retained claim receipt, then reuse a lease epoch or let an unproved owner complete. Migration now reconstructs the Stage 2 producer contract from retain-all receipts and rejects any actor, revision, lease id/epoch/expiry, claim/lease relation, or authority-sequence drift as corruption rather than migration.

I did not fake-fix the fourth request with a diagnostic CLI or dual write. The former is not a product caller; the latter creates non-atomic split authority. The correct provider-aware runtime needs the RFC's unresolved migration/rollback, local-writer fence, authorization publisher, projection, provider deployment, and retention decisions. The PR and visible-governance status now say coverage-only and request your explicit decision: accept this as the Stage 3 reference-contract slice, or keep the PR blocked for a separately scoped promotion/runtime slice.

Final evidence: focused coordination 123/123; strict mypy, ruff, CLI budget, TypeScript 200/200; probes 9/9; premerge 17/17; real file/NoKV parity 12/12 plus restore fence; real SIGKILL/reopen chain 8/8; real v0 NoKV CAS migration 5/5. NoKV main stayed unchanged. The six local macOS dashboard fixture failures reproduce identically on exact origin/main; latest main Linux CI is green, and this PR's fresh CI is the full-repository gate.

NoKV #492 and #493 remain explicit production-promotion holds; no adapter workaround was added.

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

结论:REQUEST_CHANGES。我按 exact base cabe2f96af130bbf94821dc9335996d3ae69966f、exact head aecf4eec80298d0218d35f93b8737e0ff77d697c 重新读完 16 个改动文件和完整 diff。上一轮的非法 reclaim grace、evidence 可携带性/隐私类型、以及 v0 无迁移三项已经实质修复;当前仍有一个新发现的文件 provider 持久化正确性阻塞项,并且 production caller / coverage-only 边界仍未获得 owner 接受。因此这个 exact head 还不能合并。

动机

Stage 2 只有 claim,执行进程死亡后没有合同化的 renew/release/reclaim/complete,也没有用 exact lease epoch 终态拒绝旧执行者写回的完整证明。这个 PR 将四个生命周期动词、过期加 grace 的接管、completion+successors 的单次 CAS、以及 provider store-lineage fence 建在既有 authority core / aggregate seam 上。问题本身值得作为一个内聚切片解决;§7.1 sealed-segment 仍然只是 owner 未决设计,当前 retain_all_v0 不应被它扩大。

改动思路

executor.py 把五个 verb 的 closed envelope 归一化成 semantic request,load head 后做 store-binding fence,再由 _held_lease_context_reclaim_transition_complete_transition 组合 Stage 1 typed transitions;每次 CAS reload 后重验 lineage。head.py 将 aggregate 升为 v1,增加 typed migration-required 和显式 v0→v1 migration,并把 done/evidence/continuation/successor/lease 关系收紧为闭合不变量。file_provider.py 与 NoKV example provider 暴露 store identity,examples 扩展为 file/NoKV parity、restore fence 和 lifecycle matrix;RFC 双语镜像与 visible_governance.py 明确目前为 coverage-only。

正路径是 claim → renew → expiry+grace → peer reclaim 铸造 epoch+1 → 旧 fence 写回被拒绝 → 新 holder complete 并原子创建 successor。负路径包括非法 grace、不可携带或隐私错配 evidence、旧 v0 typed migration hold、迁移 lineage 不一致、store identity 不一致、以及 stale/non-holder fence;这些路径现在大多已有明确 typed rejection。

具体改动

精确 diff 为 16 个文件、+3136/-214。生产面包括 executor.pyhead.pyfile_provider.pygoal_state_shadow.pyvisible_governance.py;验证面包括 4 个既有 coordination test/probe 文件、新增的 test_coordination_recoverable_execution.py 与真实 Stage 2 v0 fixture;example 面包括 NoKV README/provider/probes/live matrix;规范面包括中英文 RFC。没有机械搬移或 generated artifact。

关键 symbol map:

  • CoordinationAuthorityExecutor._semantic_request 定义每个 verb 的闭合字段和可重放 operation digest;_held_lease_context 统一 holder opening;_reclaim_transition 在 authority-clock expiry+grace 后走 delegated clear-claim 再复用普通 acquire+claim;_complete_transition 在 holder gate 后原子写 done、退役 lease 并建 successor。
  • validated_head / v0→v1 migration 负责 receipt、todo、lease、continuation、evidence 和 authority-revision 闭包;新 migration fixture 证明 legacy holder lineage,避免把 restored bytes 自动绑定到当前 provider。
  • FileCoordinationProvider.store_identity 和 NoKV store_identity 为 executor 的 _store_binding_fence 提供 lineage id。这里 file 实现仍有下面的 durability/concurrency bug。

对主干的风险

  1. [P1] file provider 会把“身份文件已创建”提前当成“身份已经完整持久化”,并且接受任意非空残片作为合法 lineage。 file_provider.py:117-136 直接对最终 store-identity 路径做 O_EXCL,然后单次 os.write。并发调用者可以在创建者写入前读到空文件;创建者在 write/fsync 前崩溃会永久留下空文件;短写会被当作成功,读取侧又只检查 non-empty。我在 exact head 上确定性复现了三种结果:并发第二个调用者得到 store identity file is empty(第一个随后成功);注入首次写入崩溃后,重试永久得到同一 empty-file 错误;注入短写后,两个独立目录都返回相同 identity f,使 store-lineage fence 失去唯一性。最小修复是在跨进程锁内生成,通过同目录临时文件执行完整写循环、fsync 文件、原子 rename、fsync 父目录,并在读取侧验证完整 file:<32 lowercase hex> 格式;请加并发创建、短写、创建中崩溃/重试的确定性测试。仅在最终路径上 O_EXCL 不能同时满足“竞争者只看到完整值”和“崩溃可恢复”。

  2. [P1 / owner decision] 新命令面仍没有 active LoopX production caller。 exact head 上在 tests/examples/ 之外仍找不到 CoordinationAuthorityExecutor(...) 的构造或可达 bootstrap;PR 现在诚实标为 coverage-only,也解释了 diagnostic CLI 和 dual-write 都不是安全捷径,但 owner 尚未明确接受“这次只合并 reference contract、production promotion 后置”的边界。按当前 scope-fit 合同,约 1.2k 行生产 coordination runtime 只有测试/example 调用仍是合并阻塞项。请由 owner 明确接受 coverage-only Stage 3 切片,或在另一个经过迁移、local-writer fence、授权、projection flip/rollback 评审的可达入口中完成 promotion;不应以临时 CLI 或非原子双写假装解决。

上一轮前三个阻塞项的复核结果是正面的:reclaim grace 现在拒绝 bool/负数/NaN/±inf/转换溢出;evidence 现在是 artifact://public|private/<opaque-id> + SHA-256 + closed privacy enum;head 已升 v1,旧 v0 得到 typed migration hold,且显式迁移验证 retained claim/receipt/lease/authority revision lineage。typed state、domain-neutral error text、行为变化披露和 machine obligation 表述均已对齐;没有把 must 降格成 guidance。

爆炸半径方面,identity 残片或空文件会让整个目录中的所有 shared goals 永久 fail-closed;更危险的是合法格式未验证的短写残片可能在不同目录碰撞,从而让本应拒绝的 restored copy 通过 lineage fence。receipt 能审计业务 transition,但不能修复 provider identity 的错误授权根。

我的整体评价

我在 exact-head archive 上独立跑了 123 个 focused coordination tests(全过)、contract probes 9/9、file-provider lifecycle 12/12 和 git diff --check;GitHub 11 个 checks 均为 success/expected skip。NoKV live 因本机未设置 NOKV_COORDINATION_LIVE 未独立重跑,所以作者发布的 NoKV restore/SIGKILL 证据在本轮仍属于 read-only corroboration,而不是我的独立执行证据。

scope-fit / code-volume 复核:四个生命周期 verb 同属一个 authority change reason,holder opening 和 acquire+claim tail 的提取是有覆盖、可逆的 bounded future-facing refactor;把它们拆成更小 PR 会让原子 completion 与 stale-fence 证明跨切片。相反,production promotion 依赖未决授权/迁移/投影边界,应该继续后置,不应在本 PR 追加大范围 host 接线。除了修复 file identity 原子持久化和取得 owner 对 coverage-only 的明确裁决,没有发现值得顺手扩大的相邻重构。

请修复 file identity 的完整写入、原子发布、目录持久化与格式验证,再补上述三类负例;同时请 owner 对 coverage-only 边界给出明确合并决定。完成后我会按新 exact head 复审。

English verdict: REQUEST_CHANGES on exact head aecf4eec80298d0218d35f93b8737e0ff77d697c. The prior grace, portable typed-evidence, and versioned-migration blockers are fixed. The file provider still publishes its final identity path before a complete durable value exists, treats a single possibly-short os.write as success, and accepts any non-empty fragment as a valid lineage; deterministic concurrency, crash, and short-write repros show both permanent failure and cross-directory identity collision. Also, the coverage-only runtime boundary still needs explicit owner acceptance or a separately reviewed production promotion path. 123 focused tests, 9/9 probes, the 12-row file matrix, diff hygiene, and all current GitHub checks passed; NoKV live was not independently rerun in this environment.

Comment thread loopx/control_plane/coordination/file_provider.py Outdated
@huangruiteng

Copy link
Copy Markdown
Owner

Owner direction for the next revision:

  1. I accept Stage 3 as a foundation/reference-contract slice under a broader provider-neutral LoopX control-plane store RFC. This acceptance does not promote the current modules to a shipped production caller and does not waive correctness gates.
  2. The RFC should now define one LoopX semantic authority service with three provider profiles: built-in file parity, optional NoKV shared storage, and an optional PostgreSQL-backed service deployment. Agents call LoopX authority APIs; they never write provider storage directly.
  3. Keep semantic operations such as Turn admission, claim/lease fencing, quota, and idempotent settlement in LoopX. The store contract owns only atomic persistence/readback primitives, provider revisions, event/projection cursors, receipts, and audit durability.
  4. The immediate merge blockers are therefore bounded: rebase onto current main/the native TypeScript task-lease acquire boundary, and fix the unresolved file store_identity publication bug with lock + complete temp write + fsync + atomic rename + directory fsync + strict format validation and deterministic concurrency/crash/short-write tests.
  5. After this slice, the NoKV provider owner will lead NoKV qualification and contribute to the shared store boundary; LoopX core/PostgreSQL owners will lead the provider-neutral TypeScript transaction boundary, file-provider parity/promotion, and PostgreSQL service/provider. Joint shadow parity and a one-goal/two-agent TEST ONLY canary gate any authority-source flip.

I will update this PR on top of the contributor commits so attribution is preserved. The public RFC and PR will stay provider-neutral and will not include private workflow-system, messaging, or work-item payload details.

Signed-off-by: huangruiteng <huangrt01@163.com>
Signed-off-by: huangruiteng <huangrt01@163.com>
Signed-off-by: huangruiteng <huangrt01@163.com>
@huangruiteng huangruiteng changed the title feat(coordination): Stage 3 - recoverable execution ownership feat(control-plane): add recoverable shared-authority foundation Sep 1, 2026
@huangruiteng

Copy link
Copy Markdown
Owner

Exact head 6caffb99851c4bf2c573f414b595848d93123b7e is ready for fresh review.

The remaining requested changes are closed:

  • integrated current main while preserving task_lease_acquire.ts as the default acquire authority;
  • fixed file store-identity publication and resolved the inline thread;
  • widened the RFC from a NoKV-specific proposal to one LoopX semantic authority with file, NoKV, and PostgreSQL provider profiles;
  • kept the Python Stage 3 executor coverage-only and recorded the TypeScript/file, NoKV, PostgreSQL, shadow-parity, canary, and authority-flip workstreams.

Independent validation on the exact head: Python 180/180, TypeScript 295/295, strict mypy, ruff, TypeScript typecheck, docs smoke, premerge 18/18, and strict change-quality receipt cqr_b3ed99cd008026702379 verified valid.

@wchwawa Please check that the generalized RFC preserves the original NoKV contribution and gives the NoKV provider owner a clear qualification track. A fresh independent maintainer review is still required before merge; this update is not requesting an admin bypass.

@huangruiteng
huangruiteng requested a review from wujc12 September 1, 2026 04:18
@huangruiteng
huangruiteng dismissed stale reviews from themself September 1, 2026 04:19

Addressed on exact head 6caffb9; fresh independent review requested.

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

动机

这个 PR 为 LoopX 的共享控制面补齐可恢复执行所有权的参考合同:在既有 claim_work 之外增加 renew、release、过期 reclaim、stale-fence 终态拒绝和 completion+successor 原子提交,并用 provider lineage 防止复制或 restore 的旧字节自动获得新现场权限。它仍明确是 coverage-only reference boundary,不改变当前本地运行时的默认权威路径。

改动思路

实现把语义权威和存储 provider 分开。CoordinationAuthorityExecutor 负责闭合命令字段、operation digest、holder/fence/liveness 判定和 typed transition;file、NoKV 以及 RFC 中规划的 PostgreSQL 只负责持久化 reviewed transaction。每次 load/CAS reload 都重验 store_binding。head 升级为 v1,并通过显式 migration 识别旧 v0,而不是把 restored bytes 隐式绑定到当前 provider。

正向路径是 claim → renew → 到期并越过 grace → peer reclaim 铸造 epoch+1 → 旧 executor 的 fence 写回被拒绝 → 新 holder complete 并在同一次 CAS 创建 successor。负向路径包括非法 grace、不可携带或隐私分类不匹配的 evidence、旧 schema migration hold、store-lineage mismatch、non-holder 和 stale fence;这些路径均 fail closed。

具体改动

  • executor.py 增加并组合 renew_workrelease_workreclaim_workcomplete_work_held_lease_context 统一 holder 动词的 revision/fence/holder/liveness 开场,_reclaim_transition 复用普通 acquire+claim,_complete_transition 原子写 done、退役 lease 与创建 successor。
  • head.py 将 aggregate 升为 loopx_coordination_head_v1,收紧 todo/lease/receipt/continuation/evidence 闭包,并提供显式 v0→v1 migration 与真实旧 fixture。
  • file_provider.py 修复上一 exact head 的 blocker:store identity 现在在跨进程锁内通过临时文件完整写入、file fsync、atomic replace 和 directory fsync 发布;读取严格校验 file:<32 lowercase hex>。新测试覆盖并发创建、短写、rename 前崩溃重试、directory fsync 失败收敛和非法残片。
  • NoKV example 的 provider/probes/live matrix 扩展到 Stage 3 生命周期、receipt replay、stale fence 和 restore-lineage fence;visible_governance.py 与双语 RFC 明确这些模块是 coverage-only,production promotion 仍需独立评审迁移、writer fencing、授权、rollback、projection 和 provider qualification。

对主干的风险

没有发现新的阻断性问题。上一轮的非法 reclaim grace、portable typed evidence、schema migration 和 file identity 原子持久化问题均已实质修复。当前变更规模很大,但 lifecycle verbs、fencing、migration 与 provider conformance 属于同一个共享 authority change reason;拆开会使 completion 原子性和 stale-fence 证明跨越不完整切片。相反,生产接线没有被顺手扩大,代码和治理表都诚实声明 coverage-only,因此不会改变现有默认运行路径。

主要残余风险是生产 promotion 尚未发生,以及 NoKV live restore/SIGKILL 证据本轮未在本机独立重跑;它们属于后续 promotion gate,而不是这个已明确 coverage-only reference slice 的隐式能力。另一个暂时性风险是 GitHub 的主 pytest 与 non-blocking Sonar workflow 在评审时仍运行中;本地 focused suite 与所有已完成远端门禁均通过,因此结论可发布,但合并仍应等待 required checks 完成。

精确 head:6caffb99851c4bf2c573f414b595848d93123b7e。独立验证:136 个 focused coordination tests 全过;Ruff 通过;contract probes 9/9;file-provider lifecycle 12/12;git diff --check 通过。NoKV live 因 NOKV_COORDINATION_LIVE 未设置而未独立执行。

我的整体评价

APPROVE。此前最危险的 store identity 不完整发布/短写碰撞已经通过原子持久化、严格格式校验和确定性负例关闭;其余权限、隐私和迁移边界也保持 fail closed。这个 exact head 作为 coverage-only 的 shared-authority reference foundation 已具备合并质量,production authority promotion 应继续作为独立工作流处理。

English verdict: APPROVE — no blocking findings at exact head 6caffb9. The prior file-identity durability blocker is fixed with locked temp-file publication, complete writes, file/directory fsync, atomic replacement, strict format validation, and deterministic concurrency/crash/short-write tests. Focused coordination tests (136 passed), Ruff, 9/9 contract probes, the 12-row file-provider lifecycle matrix, and diff checks are green; NoKV live was not independently rerun, and merge should still wait for currently pending required GitHub checks.

@huangruiteng
huangruiteng merged commit 7e43e6a into main Sep 1, 2026
11 checks passed
@huangruiteng
huangruiteng deleted the feat/stage3-recoverable-execution branch September 1, 2026 04:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants