Skip to content

fix(runtime): retry transient startup exits - #3599

Merged
huangruiteng merged 1 commit into
huangruiteng:mainfrom
cocolord:codex/effect-runtime-startup-retry-20260825
Aug 26, 2026
Merged

fix(runtime): retry transient startup exits#3599
huangruiteng merged 1 commit into
huangruiteng:mainfrom
cocolord:codex/effect-runtime-startup-retry-20260825

Conversation

@cocolord

Copy link
Copy Markdown
Contributor

Summary

  • include managed-runtime startup in the existing retry-safe request budget;
  • retry one transient runtime_exited_before_ready outcome without expanding
    the total request attempt count beyond two;
  • preserve the original typed startup diagnostic for non-retry-safe requests
    and after the second failed startup;
  • add deterministic tests for no retry, successful second startup, and bounded
    double failure.

Why

Unrelated PRs repeatedly failed the native Windows lifecycle job at
test_retry_safe_typed_write_recovers_after_unexpected_runtime_exit.
The request path retried socket/request failures but invoked _start_runtime
outside that retry boundary, so a transient early runtime exit bypassed the
retry-safe contract.

This is an independent runtime fix. It does not change the behavior or review
scope of the affected documentation, Todo, or benchmark PRs.

Validation

  • 25 passed in tests/control_plane/test_effect_runtime_integration.py;
  • 77 passed, 6 skipped across the focused runtime and Windows lifecycle
    equivalent set;
  • Ruff and Python compile checks passed;
  • loopx canary premerge --from-git-diff: 9/9 selected checks passed;
  • LoopX public/private boundary scan passed;
  • git diff --check passed.

Future-Facing Scope

The bounded pass was applied at the existing request retry owner. No new
runtime lifecycle state, retry counter, compatibility wrapper, or scheduler
surface was added.

@cocolord

Copy link
Copy Markdown
Contributor Author

@huangruiteng This independent runtime fix addresses the repeated Windows runtime_exited_before_ready crash-recovery flake seen on unrelated PRs #3558 and #3587. It keeps one shared two-attempt retry budget and preserves non-retry-safe diagnostics. Please review when CI finishes.

@cocolord cocolord left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

@huangruiteng

Exact head: 3599@d9d790be39c44a2b7df5d6a8bc270216ec5f56ee

English verdict: APPROVE. This patch puts managed-runtime startup failures
inside the existing two-attempt retry boundary for retry-safe requests, while
preserving immediate typed startup diagnostics for non-retry-safe requests and
after the final attempt. Focused local validation and all exact-head GitHub
checks, including Linux and Windows, pass.

详细中文评审

动机

当前 Effect runtime 请求路径已经允许 retry_safe=True 的幂等 typed effect
在连接或请求阶段失败后重试一次,但 _start_runtime(...) 位于该异常边界之外。
因此,runtime 被终止后首次重建若在 ready 之前短暂退出,会直接抛出
EffectRuntimeStartupError(runtime_exited_before_ready),没有消费调用方已经声明
安全的第二次尝试。这个缺口会让本应具备 crash recovery 语义的 typed write 在
Windows 生命周期测试中偶发失败,并阻塞与 runtime 本身无关的 PR。

最小修复应位于现有 effect_runtime_request 重试权威中,而不是在测试里放宽断言、
给 Windows 加等待时间,或新增第二套 runtime supervisor。本 PR 采用了这个最小
边界。

改动思路

入口仍是 effect_runtime_result(...),它将 retry_safe 原样交给
effect_runtime_request(...)。后者继续使用既有的单层循环:安全请求最多两次,
非安全请求一次。唯一行为变化是把缺失 runtime 时的 _start_runtime(...) 调用
纳入同一循环的启动异常处理。

正向路径:第一次启动抛出 EffectRuntimeStartupError,且调用声明
retry_safe=True;请求层记录该异常、删除可能残留的 info 文件并进入第二次尝试。
第二次启动成功后,调用仍使用同一个 request_id 进入 _request_with_info(...)
并返回原有结果。

负向路径有两条。retry_safe=False 时,第一次启动异常立即原样抛出;安全请求的
第二次启动仍失败时,也立即抛出第二次的 typed startup error。代码没有把启动失败
转换成模糊的通用 request error,也没有把总尝试次数扩展到两次以上。

具体改动

  • loopx/control_plane/effect_runtime.py:在 effect_runtime_request 的现有尝试
    循环中捕获 EffectRuntimeStartupError。只有第一次且 retry_safe=True 时才
    清理 stale info 并继续;其余情况原样抛出。
  • tests/control_plane/test_effect_runtime_integration.py:新增参数化边界测试,
    覆盖非安全调用一次失败、安全调用两次均失败、以及安全调用第二次启动成功。

关键代码讲解

  1. effect_runtime_request 仍是 retry policy 的唯一 owner。启动与请求传输共享同一
    for attempt in range(2 if retry_safe else 1),因此不存在“启动重试两次后请求
    再重试两次”的乘法放大。
  2. EffectRuntimeStartupError 保持现有 typed diagnostic contract;
    runtime_exited_before_readydiagnostic_code 不被字符串匹配或 prose
    heuristic 分类。该修复对所有启动错误应用同一 caller-declared retry-safe 规则,
    不加入平台或产品专用分支。
  3. info_path.unlink(missing_ok=True) 与既有请求失败恢复路径一致,清除可能由失败
    runtime 留下的 endpoint 元数据,再由第二次尝试重新读取或启动。
  4. test_request_startup_retry_boundary 以调用语义定义期望次数和结果,不依赖真实
    进程竞态:False/失败 为一次、True/失败 为两次、True/第二次成功 为两次。

对主干的风险

未发现阻塞项。行为变化已明确限制为 caller 标记 retry_safe=True 且 managed
runtime 启动失败的情况;非安全 effect 不会被重复执行。残余风险是安全调用在一次
真实启动故障后会多等待一次 startup deadline,这是既有“最多两次”恢复预算的预期
代价,而不是无界重试。

typed-state lens:启动失败继续由 EffectRuntimeStartupError
diagnostic_code 表达,没有引入布尔状态或 substring 规则。domain-neutrality
lens:实现只讨论 runtime/request/retry,不含 benchmark 或产品专用措辞。
behavior-change disclosure lens:PR 正文准确说明安全请求现在会对启动失败消费第二次
尝试。guidance-vs-obligation lens:retry_safe 仍是机器执行的调用契约,不被写成
可选建议。

验证结果:

  • tests/control_plane/test_effect_runtime_integration.py25 passed
  • 相关 runtime / Windows 生命周期等价测试:77 passed, 6 skipped(实现阶段)。
  • Ruff、Python compile、git diff --check origin/main...HEAD:通过。
  • LoopX public/private boundary scan:2 个候选文件 clean。
  • loopx canary premerge --from-git-diff:9/9 selected checks 通过(实现阶段)。
  • GitHub exact-head:Linux pytest、Windows PowerShell、DCO、dependency review、
    release build 与 SonarCloud 全部通过;publish-only jobs 按预期 skipped。

我的整体评价

这是一个聚焦且可逆的根因修复:它复用现有 retry owner 和两次预算,没有新增 runtime
生命周期状态、兼容层、平台特判或平行 supervisor。66 行新增测试覆盖了最重要的正负
边界,8 行生产改动保持原有 typed error 和非安全调用语义。future-facing pass 已在
正确边界完成,无需进一步抽象。综合代码、失败路径、跨平台 CI 和 exact-head
readback,本轮结论为 APPROVE;请维护者复审并合入该独立 runtime 修复。

Let retry-safe Effect runtime requests spend their existing second attempt
when a managed runtime exits before readiness. Preserve immediate typed
startup diagnostics for non-retry-safe calls and bound startup attempts to
two.

Signed-off-by: lusendong.6789 <lusendong.6789@bytedance.com>
Co-authored-by: TRAE CLI <noreply@bytedance.com>
@cocolord
cocolord force-pushed the codex/effect-runtime-startup-retry-20260825 branch from d9d790b to e3702fb Compare August 26, 2026 03:57
@cocolord

Copy link
Copy Markdown
Contributor Author

@huangruiteng PR #3599 has been rebased onto current main@e8175d4e9ffec578837906ac4af92e9b607acd36.

Exact head for re-review: e3702fbda7c61552b187c55a63d888978d546d3e.

Conflict resolution

The conflict came from merged PR #3541 moving _read_info() and _start_runtime() into the request loop's typed-error try boundary, while this PR had added startup retry handling around the older out-of-try call site.

The resolution keeps the current-main owner structure and preserves both contracts:

  • EffectRuntimeRemoteError remains immediately non-retryable.
  • EffectRuntimeStartupError consumes the existing second attempt only when the caller declared retry_safe=True and the first attempt failed.
  • non-retry-safe startup failures and second startup failures preserve and re-raise the original typed diagnostic.
  • startup and request transport still share one bounded two-attempt loop; no nested retry budget or parallel supervisor was introduced.

The final diff remains limited to:

  • loopx/control_plane/effect_runtime.py
  • tests/control_plane/test_effect_runtime_integration.py

Validation

  • Effect runtime integration: 25 passed.
  • Focused startup retry/diagnostic cases: 5 passed.
  • Adjacent typed startup and start-goal CLI coverage: 59 passed, 1 deselected.
    • The deselected doctor-health assertion depends on writing the shared global registry; under this sandbox it failed only because the registry write probe returned Operation not permitted. The runtime-specific assertions passed.
  • Ruff: passed.
  • Python compile: passed.
  • git diff --check origin/main...HEAD: passed.
  • Commit topology: current origin/main is the exact parent and the branch contains one logical commit.
  • DCO and commit trailers: passed; the rewritten commit has a Signed-off-by trailer and ends with exactly one Co-authored-by: TRAE CLI <noreply@bytedance.com>.
  • Standard premerge canary: 9/9 selected checks passed, including maintainability, control-plane/canary risk smokes, diff hygiene, compile, and public/private boundary checks; 0 manual holds.

GitHub now reports MERGEABLE; fresh exact-head CI is running. Please re-review this exact head once required checks complete.

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

English verdict: APPROVE. I found no blocking issue at exact head e3702fbda7c61552b187c55a63d888978d546d3e. The change keeps managed-runtime startup inside the existing caller-declared two-attempt boundary, preserves typed startup diagnostics, and leaves non-retry-safe requests single-attempt.

动机

当前 effect_runtime_request(...) 已经允许 retry_safe=True 的调用在连接或响应阶段失败后消费第二次尝试,但 _start_runtime(...) 抛出的 typed startup error 会越过该预算直接失败。于是 runtime 意外退出后,首次重建如果在 ready 前再次短暂退出,幂等 effect 也无法使用调用方已经声明安全的恢复机会。这个修复应当落在现有 request retry owner 内,而不是通过放宽 Windows 测试、增加平台特判或新建并行 supervisor 来掩盖竞态。

改动思路

PR 保留单一的 for attempt in range(2 if retry_safe else 1) 权威,把 EffectRuntimeStartupError 纳入同一边界。第一次启动失败且调用标记为 retry-safe 时,代码清除可能残留的 info 文件并进入第二次尝试;非安全调用或第二次启动失败则原样抛出 typed startup error。请求 request_id 仍在循环外生成,因此启动恢复没有引入新的请求身份,也没有形成“启动两次后再请求两次”的乘法重试。

具体改动

  • loopx/control_plane/effect_runtime.py:将 remote error 与 startup error 分开处理;remote semantic error继续立即抛出,startup error只在第一次且 retry_safe=True 时继续,尝试总数仍由既有循环限制为最多两次。
  • tests/control_plane/test_effect_runtime_integration.py:参数化覆盖三条语义边界:非安全调用一次失败、安全调用连续两次失败、安全调用第二次启动成功;同时验证最终 diagnostic_code 仍为 runtime_exited_before_ready

我还检查了 _start_runtime(...)_request_with_info(...) 和主要 effect_runtime_result(...) 调用面:实现没有用字符串匹配判断错误,没有把 remote typed error 错当成本地启动故障,也没有为特定产品、benchmark 或平台增加第二套状态权威。代码量与活跃调用点匹配;8 行生产改动复用既有 retry seam,66 行测试集中证明正负边界,无需新增抽象。

对主干的风险

未发现阻塞项。默认行为变化只发生在调用方已声明 retry_safe=True 且 managed runtime 尚未进入 serving state 的路径;retry_safe=False 继续立即返回首个 typed diagnostic。所有 startup diagnostic 目前共享这一 bounded retry 规则,其中 lock/ready timeout 在持续故障时可能多消耗一次启动截止时间;这是本 PR 明确采用的现有两次尝试预算,且不会无界放大。最值得继续观察的是 Windows 上真实进程退出/重建竞态,但 exact-head Windows job 与 Linux pytest 均已通过。

评审 lenses:typed-state 由 EffectRuntimeStartupError.diagnostic_code 保持,没有 substring denylist 或散落布尔分类;domain-neutrality 保持 runtime/request 语义;behavior change 已在 PR Summary/Why 中披露;retry_safe 是机器执行的调用契约,不是被描述成可选 guidance。

验证与 readback:

  • exact-head 本地 focused suite:43 passed(effect runtime integration、turn-journal readiness、scheduler-state runtime integration)。
  • Ruff 与 Python compile:通过。
  • 额外诊断矩阵确认 startup error 在 retry-safe 路径最多尝试两次,并保留最终 typed code。
  • GitHub exact-head checks:9 success、0 failure、0 pending;包括 Linux pytest、Windows PowerShell、DCO、dependency review、release build 与 SonarCloud。
  • exact head 的父提交就是当前 origin/main,diff 仅包含 2 个目标文件;DCO trailer 完整。

我的整体评价

这是一个聚焦、可逆且边界正确的 runtime 根因修复。它让 caller-declared retry-safe 语义覆盖 startup 阶段,同时保留非安全调用、typed diagnostics、request identity 与总尝试次数;测试从语义而非进程偶发输出定义期望。future-facing pass 已在正确 owner 上完成,附近没有需要一并抽取的第二套权威。综合代码路径、负向边界、focused validation 与 exact-head CI,本轮结论为 APPROVE;合并仍应遵循仓库保护与 maintainer policy。

@huangruiteng
huangruiteng merged commit 601b07b into huangruiteng:main Aug 26, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants