Skip to content

test(interactive): freeze C01–C12 complex acceptance and verify scripted Host scenarios - #12

Merged
rockyshi1993 merged 24 commits into
mainfrom
test/interactive-complex-c01-c12-acceptance
Nov 12, 2026
Merged

rockyshi1993 merged 24 commits into
mainfrom
test/interactive-complex-c01-c12-acceptance

Conversation

@rockyshi1993

@rockyshi1993 rockyshi1993 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Scope

Preserve the C01–C24 acceptance assets as test/documentation contributions, without changing production Core, Program ABI or Store schema.

  • 14 scripted tests cover C01–C04, C10–C11 and C13–C20.
  • Five ChatGPT-authored responses were recorded after inspecting the exact Pending identities and are replayed through the actual FlowKit Program, SQLite and Trusted Host.
  • The replay verifies exact requestId/taskDigest binding, dependent Agent turns, invalid-schema retry, signed receipt acceptance, SQLite close/reopen and terminal state.

Important evidence boundary

The five responses are recorded ChatGPT-authored fixtures, not live model API calls inside the CI job. GitHub tool identity is not provider-native session attestation. The test does not prove native Grok Build/Codex JS tools or cross-principal Host isolation. C12/S13 and applicable C22/C23 native-host gates remain BLOCKED.

The test recreates a temporary SQLite database per CI run and reopens it within that run; it does not share one physical DB across separate model turns/Actions runs.

C04's dedicated fixture uses retries=1 (two attempts); default retries=2 / three attempts is covered separately by main Coordinator tests.

Test gate

Normal CI requires all five recorded responses. The Pending-preparation path is available only when explicitly invoked with FLOWKIT_C01_PREPARE=1; an incomplete fixture cannot silently pass normal CI.

Verification

  • Updated-base PR HEAD: bede343d6226daa391f2857d737e5486cd567db1 (refresh after the current review changes).
  • The prior updated-base PR run passed 7/7; this test-gate change requires a fresh exact-HEAD run.
  • This PR contains docs, fixtures, examples and tests only; no production Core, ABI or Store schema changes.

Note

Medium Risk
The coordinator start() snapshot change affects run creation and idempotency semantics at a concurrency-sensitive boundary; the rest is mostly tests and documentation.

Overview
Adds C01–C24 interactive acceptance documentation and scripted Host test assets (fixtures, c01-live-chatgpt workflow, m6-host.mjs worker bridge), and wires tests/integration/*.test.mjs into npm test plus a new CI step that runs tests/integration/m6-e2e.test.mjs under FLOWKIT_M6_ISOLATION (with npm ci on the Host-isolation job).

Fixes R04: coordinator.start() now deep-copies state and startIntent before the first await, so caller mutations after invoking start no longer affect persisted checkpoints or startKey idempotency.

Also commits a large Phase 2 / M6 evidence layer—local verification report, conformance audit R1/R2, formal source register, trusted-host V2 design/review/acceptance docs, and roadmap updates—without expanding Core ABI beyond the coordinator snapshot change.

Reviewed by Cursor Bugbot for commit b008d86. Bugbot is set up for automated code reviews on this repo. Configure here.

Copy link
Copy Markdown
Contributor Author

C01–C12 独立源码与实测证据复审(2026-11-11)

复审基线

  • 精确测试分支 HEAD:33c9418fd01d9fe42f6686507477f61f659bfabc;基线 main ea2bb10893a9be2e5de3561fc9a660a6977ae15c。本次仅复核测试/说明文档,没有修改核心生产代码,也没有合并 PR。
  • Actions 42963831991:7/7 completed/success,Ubuntu 24.04/Windows 2025 × Node 22.13/24.22 各 364/364 PASS、0 FAIL、0 SKIP,Linux 两项 Host 跨 UID 隔离及网站构建通过。日志确认 C01/C02/C03/C04/C10/C11 的结构化 FLOWKIT_COMPLEX_ACCEPTANCE 及 C01_LIVE_CHATGPT_COMPLETED。
  • 复核 six new complex-c01-c12.test.mjs 用例与主分支的 crash/coordinator/trusted-host/s07-capacity/host-oversize 负例;真实 ChatGPT 记录来自 c01-live-chatgpt.test.mjs + 五次依次提交 fixture 的 GitHub commit,不是 364 次模型调用。
  • 五次答复对应独立提交 SHA:1d515fe0(requirements)、031eb433(architecture)、f0fd5700(implementation)、7363bd17(review invalid schema)、abe592e8(review retry valid)。提交历史顺序可核对;但 GitHub commit 作者/消息与固定 fixture 并不能独立证明模型提供商或原生 Session 身份。

逐项裁决

ID 协议/脚本结果 真实模型/原生结果 备注
C01 PASS ChatGPT Bridge PASS / Native BLOCKED 4 串行 Agent、2 Conditions,五次依 Pending 提交的答复,其中审查 Agent 重试一次;脚本 35 Journal,Bridge 39 Journal
C02 PASS PARTIAL / Native BLOCKED ready=false、缺失 ready FK_R002、reviewApproved=false 在脚本中验证;真实 ChatGPT 仅经过 true/approved
C03 PASS ChatGPT Bridge PASS(受控错误注入) / Native BLOCKED reviewApproved 首次字符串错误 → attempt=2 新 requestId → 正确布尔值;未把非法业务值写进 State
C04 PASS 真实模型 BLOCKED 失败两次 → FK_R004;只在脚本运行,未演示模型持续失败
C05 main 既有 SIGKILL PASS 真实模型组合 PARTIAL 多个崩溃点在 main 真子进程验证;Bridge 仅 CI Job 内关闭/重开 Store
C06 main 真子进程 CAS/丢响应 PASS 真实模型并发 BLOCKED 不应把单个模型 answer 的重复 Ack 当作真实多会话竞争
C07 PARTIAL Native BLOCKED main 有 policy/Program 回归,完整 source+Skill 漂移证据仍属 M6 Draft #7
C08 PASS(既有签名/权限) Native BLOCKED Linux 跨 UID、签名拒绝和最小披露已验证;未证 Grok 进程/Windows ACL
C09 PASS(既有容量/超限) Native BLOCKED 非法大回执拒绝、有界失败与容量终态有 CI;没有 Grok 自然超限事件
C10 PASS(脚本) 非必需 36 Agent、255 Journal、全量 Ack 36、Checkpoint 缓存 16
C11 PASS(脚本) 非必需 budget=1/128 最终 State 同哈希,Journal 85/35 分别合法
C12 不适合脚本作为替代 BLOCKED 未有 Grok Build/Codex 原生模型工具日志和受信 Host 独立认证模型会话

复审中发现的重要证据边界

  1. C01 “5 次模型输出”有顺序提交与 CI replay 记录,但不是五次在同一原生 Host Session 内自动调度。两次 GitHub Actions 之间并未共用一个 SQLite DB。最终四平台 364/364 反复使用同一份 fixture,不能计作 20 次模型输出。
  2. C03 第一轮无效类型系人为安排的负例,不能声称模型自发生成错误后自我修复;测试证明协议正确处置特定非法结构。
  3. C05–C09 复用已存在的主分支回归,不意味着新加了相应真实模型端到端场景。C07 总体仍为 PARTIAL。全套 C01–C12 不能给统一 PASS。
  4. 冻结的矩阵将 C02/C04/C05/C08/C12 的真实模型门禁写为待实测,符合现有 P0-A 能力范围;未不当引入 Human/Action/Loop/Parallel。整个 PR 只包含 6 个测试/文档/示例文件,未修改生产源码。

审查结论与后续

测试代码、协议用例与 CI 通过;复杂场景矩阵验收属于“协议层多数 PASS、真实模型 PARTIAL、原生 S13 BLOCKED”。当前没有从这些测试发现新的确定性 P0-A Core 缺陷。
保留 PR #12 Draft/Open 以便追加真实 Grok Build 的 C01/C03/C05/C08/C12 原生日志;M6 PR #7 同样保留 Draft。下一步优先证明 Grok Build 原生工具调用与 Host 独立验证的 Provider/Session 来源,随后补真实 false 分支、真实重试耗尽、SIGKILL 模型恢复与多会话并发。没有依据时不得声称“全部复杂场景已通过”。

Add current-main M6 scripted Host integration coverage, root-discovery fixtures, Linux cross-UID CI, and the full Interactive delivery roadmap. This is test/CI/documentation only and does not complete native-host C12/S13 or M6 final acceptance.
Add the reviewed Trusted Host V2 threat model, deployment profiles, security acceptance matrix, and explicit Phase 2 limitations. This is documentation-only: it does not implement Host isolation, Grok native provenance, Windows ACL enforcement, or Phase 3/4 runtime changes. F-02 and F-04 remain pending real validation; F-07 formal V0.3/V0.6-R5 conformance remains blocked. Merge does not authorize Phase 3 or claim C12/S13/M6 acceptance.
…dit plan

Record independently validated Requirements V0.3 and Technical Design V0.6-R5 originals, limited Core PoC results and Phase 2 traceability roadmap. Reconcile the previously confirmed M1–M6 P0-A plan and existing M1–M5 implementation. Documentation only: no Core/API/Program ABI/Store schema changes, no Phase 3 authorization, no C12/S13 or full conformance PASS claim.
* docs(interactive): add source-backed T/R/S crosswalk and initial P0-A gap findings

* docs(interactive): document Host/Public terminal State projection contract difference

* docs(interactive): record async Store contract divergence from confirmed V0.6-R5

* docs(interactive): add Phase 2 R2 code-evidence review and R04 mutation findings

* docs(interactive): link R04 independent regression PR and retrigger Phase 2 CI
…idence

* docs(interactive): record local verification of PR 16 and 17

The report keeps the PR heads, commands, R04 failure output, the
snapshot fix result, and the merge recommendation. Raw decisive logs
are stored beside it. No product code changes.

* docs(interactive): record that Actions for PR 16-19 are still queued

The 13:15 +08 reconfirm keeps all four draft PRs unmerged.
Run status is queued with an empty conclusion. That is not a matrix pass
and not a proven Actions outage.
start() copied caller-owned State and startIntent only after
loadInteractiveProgram resolved. That await yields even when the loader
is already settled, so a caller could change both objects before the
Checkpoint and startIntentHash were created. Copy both with the existing
bounded canonical snapshot first, and preserve the initial-State capacity
error. Later Store writes use those copies.

Includes the R04 regression tests from PR #17.
@rockyshi1993
rockyshi1993 marked this pull request as ready for review November 12, 2026 08:23

@rockyshi1993 rockyshi1993 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainer self-review completed against the current-main merge tree and exact PR HEAD. The eight unique files are documentation, fixtures, workflow example and tests only; no production Core/ABI/Store schema changes. The scripted C01-C04/C10-C11/C13-C20 assertions and evidence-tier limitations were reviewed. The recorded ChatGPT-response test now requires all five fixtures in normal CI; explicit FLOWKIT_C01_PREPARE=1 is the only incomplete-preparation path. Exact HEAD Actions run 43181841181 passed all 7/7 jobs, including 386/386 on the four OS/Node verify matrix cells. C12/S13 and native Host gates remain BLOCKED and are not claimed as passed.

@rockyshi1993
rockyshi1993 merged commit 8ccb530 into main Nov 12, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant