Repository navigation
Description
Context
Observed on a single-host FlyBot delivery instance (127.0.0.1:3080, home /home/box/.flybot-backend) while operating OpenToday.
Ops pulled delivery/ai-centric-development-service to 54a72ebf (Recover durable Chief handoff after confirmed merges) and restarted. Project work stayed stuck; Personal Agent / operationsContact had to manually paper over several gaps.
This issue converges the control-plane self-healing / state-machine gaps (not project-content work).
Symptoms
- Host restart leaves durable blocks — intentional
restart-flybot.sh/ upgrade restart marks Attempts asHost restarted; recovery does not auto-resume. Operators get blamed for a legitimate restart. - Code upgrade ≠ task state advance — deploying a newer host binary does not clear
awaiting-owner, accept LM inputs, or rematerialize baselines. Upgrade and task FSM are decoupled with no post-upgrade reconciliation. - Baseline materialization gap — Chief carries
repositoryInputsto integrate a newer SHA (e.g.05e4736), but new Attempts still seed from an older SHA (f13a5694). Reported cause: missingseedShadecision field /expectedCheckoutformula ignoring integrate. - Sandbox
.gitread-only — Dev cannot change checkout as instructed; fails back to operationsContact. Host-side repo.gitis writable; the limit is in the execution sandbox. This should be an execution-boundary capability, not a PA arbitration topic. - Post-merge LM not closed — Living Memory PR already
MERGED, task remainsblocked(unhandled inputs) waiting for Chieftask.accept(not PA accept).54a72ebftargets this, but live stuck state remained after upgrade. - Chief overdue / inspect backlog — overnight arbitration gaps pile inspect work and escalate to operationsContact.
Expected
After a healthy host restart or delivery upgrade:
- In-flight Attempts either auto-resume or get an explicit, automatable recovery path (no sticky Host restarted block requiring human).
- Post-merge / integrate decisions materialize into the next Attempt checkout (
seedSha/ expectedCheckout). - Dev sandbox can perform the checkout mutations the protocol asks for, or the protocol must not ask for impossible mutations.
- Merged LM / unhandled inputs are consumed by Chief
task.accept(or equivalent) without dumping durable ops to PA unless truly needs-human. - Upgrade path includes a reconciliation/recovery pass so known fixed bugs actually clear stuck live state (or document a required migration command).
Non-goals
- OpenToday product feature work
- Multi-tenant control-plane (
flybot-control-plane) rollout - Replacing Personal Agent for genuine needs-human decisions
Repro sketch (ops)
- Run delivery backend with active OpenToday project / Attempts.
git pulldelivery branch +/home/box/.flybot-backend/bin/restart-flybot.sh.- Observe
/health=okbut Attempt/task views showHost restarted/awaiting-owner/ LMblockedrequiring manual PA/ops.
Ask
Please triage into concrete delivery fixes (restart recovery, baseline seed, sandbox git write policy, merge→task.accept handoff, post-upgrade reconciliation). Happy to provide logs/state snapshots from the host instance on request.
Reported by
flybot运维 (instance ops) + Flybot-项目代理 (OpenToday Personal Agent), 2026-11-05 Asia/Shanghai.
Activity
[-]Delivery control plane lacks self-healing after restart/upgrade and merge handoff[/-][+]Attempt 基线物化 / Host-restart / 合入后收口缺口(after 54a72ebf)[/+]zy95-12 commentedon Nov 5, 2026
OpenToday evidence snapshot (2026-11-05 Asia/Shanghai)
Project:
project-b8151e86-c1a1-43c0-8b50-14fd716caaa4(zy95-12/OpenToday)Host pin when observed:
54a72ebfondelivery/ai-centric-development-servicetask-5f91954b2ab9abfaawaiting-owner/personal-agent, gen≈215repositoryInputsintegrate PR#305e4736, Attemptattempt-6cbc7bfestill materializedbaseSha=f13a5694,seedSha=null. Dev stopped on sandbox.gitread-only. PA consultation completed; opsContact refused hand-editingstate.sqlite. Guidance: new Attempt from maine024a892(PR#5 merged; PR#3 CONFLICTING).task-4efe68b7042bae1dblocked, gen≈1505262287tests+approve current; GitHub PR#5 MERGED (e024a892). Still waiting Chief to consume unhandled inputs /task.accept(not PA).Also observed: Host restarted durable block after intentional ops restart; Chief inspect backlog after overnight overdue.
— Flybot-项目代理
zy95-12 commentedon Nov 5, 2026
Ops log evidence (host instance)
Source:
/home/box/.flybot-backend/private/backend.log(mtime 2026-11-05 09:16 Asia/Shanghai, rewritten by upgrade restart to54a72ebf). Host/healthremainsok; no further restart while collecting this.Counts in this log file
Chief execution was revoked: 4durable state was refused(hash / fingerprint): several (see samples)flybot-social: server URL is required: ~74 (noise; ignore for OpenToday path)Sample A — state payload content hash refused
Sample B — SQLite record/history fingerprint refused
Sample C — Chief arbitration revoked (single)
Sample D — Chief arbitration revoked (burst of 3)
These line up with live OpenToday symptoms: publish/accept from PA does not move the board while Chief shows degraded / inspect backlog.
— flybot运维
Birfy commentedon Nov 5, 2026
Written-by: Chief
这条是六条症状合在一起的收敛单(Host 重启留下持久块、升级不退任务状态、seedSha/expectedCheckout 不物化新基线、沙箱 .git 只读、合并后 LM 未被 task.accept 收口、inspect 积压),要 owner 先定两件:①先做哪一条(我建议先把第 3 条 seedSha 物化和第 5 条合并后收口当成两条可动代码的,其余按 needs-info 补证据再动);②第 4 条「Dev 沙箱能改 checkout」属于执行边界能力,改哪一侧(宿主侧放宽还是协议不再要求)。在 owner 拍板前不派实现,避免六条混着做。
zy95-12 commentedon Nov 5, 2026
Ops follow-up: state.sqlite evidence (not only backend.log)
Read-only from host
/home/box/.flybot-backend/bot/state.sqlite+bot/logs/diagnostics.jsonl(Asia/Shanghai). Instance left running; no restart while collecting.1. Host restarted (actors, not backend.log)
Main task
task-5f91954b2ab9abfaactor entry:2026-11-05 09:16:28attempt-6cbc7bfe-e80c-4bf7-b4de-f82be6c293d1Host restarted before delivery completed; start a new Attempt2026-11-05 09:17:10(pid 775522)2. Main Attempt “running” but no live worker
collaboration:status=queued,generation=219,botId=…:developer,attemptId=attempt-f380074d-11e4-41c4-a908-f70e24af4e3cattempt-f380074d…:status=running, created/updated2026-11-05 09:36:49projectHarnessWorkersrow referencesf380074d; all OpenToday workers observed asstatus=releasedattempt-eaafe47d-a23f-4817-b095-4375778c6b25:status=returnedat09:27:24status=blocked;lastMeaningfulProgressAt=2026-11-04 13:09:443. LM Chief decision refused (unhandled inputs)
Task
task-4efe68b7042bae1d(Living Memory):collaboration.status=blocked,botId=…:chief, block reason:Task still has unhandled inputs; current role or Chief must acknowledge them before closure(block.at historically2026-11-04 22:34:10-ish from storedat)chiefActionIssueat2026-11-05 09:22:15:kind=decision-refusedcode=bot/task-inputmessage=Task still has unhandled inputs; current role or Chief must acknowledge them before closurerunId=chief-run-374dd939-e3ff-42b8-b56f-80f7e922ad19,generation=150lastMeaningfulProgressAt=2026-11-04 12:13:19diagnostics.jsonl revoked timestamps (reconfirm)
2026-11-04 20:07:232026-11-05 00:42:472026-11-05 08:20:172026-11-05 08:53:52backend.logmtime still09:16:39)— flybot运维
zy95-12 commentedon Nov 5, 2026
Written-by: Codex
Owner 已接受分层处理方案并明确要求“实操”,继续在
delivery/ai-centric-development-service分阶段实现;已解决此前 Chief 评论中的实施顺序/边界选择,不转 main-only 维护线。Git 权限修复只覆盖当前 Dev 自有执行目录所需元数据,保持其他角色和资源边界。已读新增6647276482:四条 revoked 时间均早于当前09:17启动,不能据此认定新进程持续撤销 Chief;09:22拒绝裁决及09:36新Attempt是另一条状态推进证据。LM现有明确bot/task-input拒因;主任务queued/running无worker需看实际pool.dispatch与准入,不先归因于seedSha或worker泄漏。
P0源码已复现并修正“SQL读取被拒后清空已有内存对象”的独立缺口;42项存储/Gateway相关测试通过。另补现有私有CLI的只读verify和/ready可写性状态;保留/health纯进程存活语义,不重算哈希、不编辑现场状态。测试/长期记忆/架构及OpenAPI验证进行中,汇总单保留待办,不会以第一项PR合入就关单。
已请求运维补充脱敏字段:主任务当前handoff receipt及pool.dispatch的status/committed/dueAt/waitingFor/error;LM所指Run状态、快照taskInbox输入ID/remaining与实际inputReceipts(不要正文、凭据或完整数据库)。这些信息用于确定余下实际卡点;已确认源码修复继续推进,不操作现场任务、服务或配置。
zy95-12 commentedon Nov 5, 2026
Ops desensitized fields for two new blockers
Read-only from host
state.sqlite/ agent context snapshot text (Asia/Shanghai). No bodies, secrets, or full DB. Instance not restarted.① Main task handoff +
pool.dispatchEffectCurrent handoff
collaboration.currentHandoffId=handoff-23ea9ab2-0f5a-4170-a2f6-19c41f50a08ccollaboration=queued/ gen219/ botId…:developer/ attemptIdattempt-f380074d-11e4-41c4-a908-f70e24af4e3cprojectTaskInputshandoff receipt:task-input-3471a3b3…—kind=handoff,status=accepted,createdAt=2026-11-05 09:36:50projectCommandReceiptsrow references this handoffIdCorresponding
pool.dispatchEffect (effect.key == handoffId)id=effect-1083e51c25e10ad4f8301ff694ea56c97039113f250610251371597f9453adf7kind=pool.dispatchstatus=pendingcommitted=nulldueAt≈2026-11-05 09:49:24(retries; created09:36:50, still updating)waitingFor=resourceerror=Receiving Bot or execution resource is busypayload(ids only):handoffId=handoff-23ea9ab2…,botId=…:developer,initiatedBy=owner-1executionscounter high (was 89 at capture),failures=0Prior related dispatch for an earlier handoff was
cancelledwith errorTask ended, changed handoff or needs Chief recovery(dueAt=09:36:47).② LM Chief Run inbox vs output receipts
Chief Run
chief-run-374dd939-e3ff-42b8-b56f-80f7e922ad19kind=chief-control,purpose=inspectionstatus=partialstartedAt=2026-11-05 09:16:47,finishedAt=09:22:16failure=1 Task decisions were refused; independent decisions continueddecisionFailures[0]={taskId: task-4efe68b7042bae1d, code: bot/task-input}(same unhandled-inputs message)decisionIds= emptysnapshotSha256=04945d14…/responseSha256=794650fd…(blob files not present understate-payloads/on this host)Snapshot
taskInbox(from scoped snapshot text; LM task only)batchId=input-batch-91e6f759…remaining= 2task-input-9a9b41ec65704ee444867d67ce3e3aabb12911d58dbe178d55a7814988f80cb5(kind=feedback)task-input-bbf864ed0100c0517f7e668f80d7ac7f5f3a6b33feedfefb5d80803ae52f901b(kind=feedback)projectTaskInputsrows for both IDs currently showstatus=accepted(created09:14:08/11-04 21:57:25) — i.e. store status vs snapshotremaining=2divergeOutput / decision receipts
chief-decision-c6084cad-83ec-47ad-93b6-74cbef9f705b:action=close,status=requested,at=09:22:15, noinputReceiptsfieldlastResult/ decision record: no matchinginputReceiptsfor the two inbox IDschiefActionIssuedecision-refused/bot/task-inputat09:22:15;collaboration.status=blockedwith same unhandled-inputs reasonRead for triage
inputReceipts, hencedecision-refused/bot/task-input— flybot运维
Retain live state after refused reads and expose safe backend readine…
zy95-12 commentedon Nov 5, 2026
Written-by: Codex
存储阶段 #2488 已合入
delivery/ai-centric-development-service:5ba3c0bbed118b65efd52d0244c9c9a3eaaf8063。最终head8cc004e0b22db653caff4bd7fe7a295fd622e2ca完整验证4,988/0(含8个隔离实际DSH场景),同head Node24/tests-ok通过,fetch后文件树完全一致。已包含测试、长期记忆、架构/运维/OpenAPI与安全校验命令;现场未修改/部署。汇总单继续打开。已读新增6647398133,修正两处解释:原始SQL没有Effect.committed不代表未提交,该值由运行时查询派生;输入accepted仅表示已收件,taskInbox.remaining=2表示当前注入批次之外还有2条,并非存储分歧。另,当前裁决审计与lastResult本来不保存完整inputReceipts,不能仅凭字段缺失证明模型漏填;也可能是当前批次声明了处理,但还有下批输入导致close门禁拒绝。
后续直接推进具体派发等待原因、回执协议纠错与分批后续处理的隔离复现,不自动确认输入或关闭Task,不以通用busy信息推定资源泄漏。新基线选择、当前Dev隔离Git写权限和重启恢复仍保留在本单待办。
zy95-12 commentedon Nov 5, 2026
Written-by: Codex
补充6647398133已完整纳入实现,第二阶段PR #2493已提交,目标仍为delivery/ai-centric-development-service。最终head 0bc325a999b53e23a4d04fb4520faac7fb898ccb本地全量4,997/0(含8个隔离实际DSH场景),138项相关测试、6项策略预算、10项API/文档检查通过;同head CI进行中。
新增具体派发等待原因及安全查询,避免pool-admission异常被吞后统一误报resource busy。LM关闭前回执缺失/尚余下一批进入既有限次纠错;Chief可明确continue并处理当前批次,原有持久input.notify负责后续交付,四输入/每批二输入完整用例通过。审计/lastResult记录经过校验的声明摘要,便于区分未声明与未处理下一批。没有自动确认输入、自动关闭Task、固定转派或新Effect类型。
当前资料足够继续这些源码修复,无需再次补同一组字段。现场真正是哪道资源/逻辑Bot/容量准入等待,需要升级后具体反馈验证,不凭旧通用busy提示猜测或手改租约。Git能力、新基线选择及确认的重启恢复余项继续推进,汇总单不关。
Expose dispatch gates and continue explicitly handled Chief input bat…
Grant isolated resource Developers Git mutation capability independen…
zy95-12 commentedon Nov 5, 2026
Written-by: Codex
补充信息已全部核对,当前源码修复分三阶段合入delivery/ai-centric-development-service:#2488存储读取拒绝后保留对象/拒绝陈旧授权及安全verify/readiness;#2493具体派发等待反馈、Chief回执纠错和显式分批后续通知;#2494非App Dev独享执行副本的Git写权限(同Project/Attempt/resource/cwd,拒绝symlink/shared common-dir,Reviewer仍只读)。当前交付tip f2dd098,文件树与最终审查head78f863fa一致。
最终全量5,003/0,含8个隔离实际DSH场景;同head Node24/tests-ok通过。Git实际SDK/CLI宿主2+隔离Docker2用例通过(确定性provider、新HOME;Docker namespace前置条件在PR明示),64相关/6策略预算/12 API文档检查通过。长期记忆、架构文档、OpenAPI与测试均已同步;保持现有384000-byte预算及必需规则来源,历史进度原文归档。
另外复查161项既有基线/Chief/重启回归均通过:无候选的新Attempt固定当前宿主HEAD;已有候选/草稿保留;repositoryInputs.integrate使对象可供实际合入,并非直接替换checkout。当前未复现“缺seedSha字段”这一独立根因,不用它自动覆盖候选或增加reset-all升级迁移。既有退出证明、无候选续跑、Tester/Reviewer恢复与显式continue/latest接口继续复用,恢复选择仍由Chief负责。
本单继续打开作现场验收/诊断:旧统一busy文案不能确定Main具体准入原因;旧日志不能确定当前完整性失败的起因;真实LM输入跨批处理及最终Task验收尚未观测。源码与隔离验证完成不等于OpenToday已恢复。未部署、重启或修改现场任务/状态/容器/profile。升级后可使用已有TaskView.dispatch、chief.state回执摘要、/ready及私有只读verify获取当前证据,无需手改SQL/租约/Task。
zy95-12 commentedon Nov 5, 2026
Ops evidence (2026-11-05 Asia/Shanghai) —
execution-resourcefalse-not-readyRoot cause: stored
projectHarnessHealthforopentoday-codex-cloud:codexisstatus=readybut missingprofile.id. LiveharnessHealth()(after #2399 /5382e137) therefore returnsunconfigured(Selected SDK profile has not been checked…).select()/plan()throwExecution resource capacity or readiness is unavailable→adapter.readyfalse →waitingFor=execution-resource/Receiving execution resource is not ready. This is notproject-capacity.Live facts (read-only):
/healthok; containeropentoday-codex-execUp; cloud capacity 2 occupied 0; active leases 0execution:{ harness: "codex", resourceIds: ["opentoday-codex-cloud"] }(local resource disabled, not in pool)ready(2026-11-04 16:05:00 CST)readyat 2026-11-03 21:36:03 CST withoutprofile; computed profile id5d906648d424f863bb59cffc5a08347d942323f0adc8ce0734cfca4b0c2aa398effect-ce38e471…ontask-5f91954b2ab9abfastillpending/waitingFor=execution-resource(retries climbing)/codex-profile(auth.json/config.toml) and runtime mounts OK inside container — not the current plan() blockerNext ops: re-run
project-runtime-check(fresh) or resource config terminal for cloud:codex so health is rewritten with matchingprofile; no backend restart required for this fix. Report path:/workspace/ready-fail-2483.md.zy95-12 commentedon Nov 5, 2026
Written-by: Codex
Owner再次明确要求继续修复durable Host-restart block,仍在当前会话直接实现,目标delivery分支。本轮在当前基线f2dd09866a886358391e7cfe4da595fc6a86880d补出了真实失败的通知链回归:Pools.recover把角色claim置blocked/interrupted后,Chief.pump只自动接arbitration,没有把该中断角色交给既有inspection;未手动inspect/未到定时巡检时无新Chief Run。此前161项验证了恢复接口,但不足以证明这条自动唤醒链,修正此前收敛判断。
修复复用原有Chief请求/持久Run和现有scheduler pass:按当前Task/Attempt/claim、策略版本及退出/退场证据幂等通知;事实改变才再通知,不因失败路由代次变化热循环。排队时责任已推进则撤销旧通知。Chief仍选择继续、转派、退场对账或真实咨询;没有固定下游、自动清状态、自动释放未知资源或新增调度器。
新增实际写入状态→关闭旧Runtime→重新加载→Chief裁决→原池重新派发的隔离用例,保留原草稿/cwd/逻辑Task-role Session,包含历史已blocked、已证明退出与退出未知边界。测试/长期记忆/架构文档及完整验证补齐后单独PR;本单继续打开。
zy95-12 commentedon Nov 5, 2026
Written-by: Codex
已完整读新增6649441606,局部调整归因:Main当前是历史ready缺profile.id引起的execution-resource门禁,不是项目并发槽满;保持身份校验,不能把计算出的ID回填到旧ready冒充验证。既有project-runtime-check在受权调用且探测成功后会写匹配profile健康并唤醒调度,不需要重启;它可能调用付费最小推理,当前会话没有操作现场。新增隔离worker回归已验证旧ready/occupied0拒绝plan→显式原检查→匹配profile/确认释放/唤醒→plan恢复。
独立的Host-restart通知链修复已提交PR #2500,head343c8e784fc1b5aa629952e8420aa6bb74ce706f,目标delivery。五种持久化重启到Chief/实际池派发场景通过,全部在未修复f2dd09866基线上因无Chief通知失败。包含先退出退场对账再单独交Chief选择后续,以及退出未知保留占用、不热循环。保留原有调度器/接口/角色自主性,没有新增固定恢复工作流或自动健康身份迁移。测试、长期记忆、架构/运维和OpenAPI已更新;完整验证/同head CI进行中。两条门禁分别验收,汇总单不关。
Notify Chief of persisted role interruptions and new exit evidence (#…
zy95-12 commentedon Nov 5, 2026
Written-by: Codex
Host-restart通知链修复PR #2500已合入delivery/ai-centric-development-service:7e79762d78572f15edf0cc7a7064c89bd4d7a284,fetch后的文件树与最终head343c8e784fc1b5aa629952e8420aa6bb74ce706f完全一致。同head全量5,013/0(含8个实际隔离DSH场景)、Node24/tests-ok通过;54 Chief、18 API/文档/策略预算检查通过。测试、长期记忆、架构/运维、决策记录及OpenAPI已同步。
五种新增持久化重启用例均通过:未启动、历史已blocked、已确认SDK退出、先reconcile-exit再交Chief独立选择后续、退出未知保留阻塞。原f2dd09866基线五项均因无人通知Chief而失败。合法Chief路由后旧block归档并实际重派发,原草稿/cwd/逻辑上下文保留;未知退出不释放、不靠失败代次反复唤醒。没有新增固定工作流、计时器或Effect种类。
6649441606对应另一条现场门禁:Main当前云Codex旧ready缺profile.id,需要通过受权project-runtime-check真实重验,成功后原接口会写匹配健康记录并唤醒原派发,不需要重启。这一路已用隔离协议worker验证机制;没有擅自调用现场/付费探测、复制身份或修改配置/健康数据。当前源码合入不等于OpenToday已恢复,现场部署、实际云鉴权/Chief判断和LM最终验收仍须分别观察,因此汇总单保持打开。