Repository navigation
rlf-codex-rl 5.8：RL gate 判定器 rlf_rl_gate + 证据链 + attempt-1 教训 - #231
Closed
michaellchung wants to merge 2 commits into
Closed
michaellchung wants to merge 2 commits into
michaellchung wants to merge 2 commits into
Conversation
This was referenced Oct 11, 2026
michaellchung
added a commit
that referenced
this pull request
Oct 11, 2026
….8 通过证据（原 #231–#236 恢复合并） (#237) * rlf-codex-rl: 5.8 gate runner + evidence chain (rlf_rl_gate), docs * rlf-codex-rl: 5.8 gate attempt 1 failed at trainer cutlass compile; evidence + lessons * launcher: passthrough YETO_RLF_* envs + rlf_modal_provider as Modal sandbox provider * rlf(modal): sandbox idle_timeout follows ttl_s Gate attempt 5: a CTF VM was reclaimed mid-episode by Modal's 300s idle kill while the model was in a long generation turn (no exec running), the next exec raised NotFoundError, the episode aborted without trusted finish and the fail-closed preflight discarded the group. Default idle_timeout now follows ttl_s (hard cap still bounds leaks); overridable via the new RlfModalVmConfig.idle_timeout_s field. Co-authored-by: Cursor <cursoragent@cursor.com> * rlf-codex-rl 5.8 attempt 2: training attention backend override + tape-surviving gate evidence - --rl-training-attention-backend {auto,flash,fused,unfused}: recipe default overridable (flash-only recipes refuse); gate run uses unfused after the L40S flash-attn cute JIT crd2idx compile failure (failure class trainer_kernel_compile, attention fwd, LoRA-independent) - rlf_rl_gate: policy-update proof from the island event tape rl_publication payload hashes (代拍板 v2) — survives island teardown via --modal-tape-volume; adapter-file path kept; policy_update_source recorded in evidence - CPU tests: 4 new backend-override + 5 new tape-collection tests Co-authored-by: Cursor <cursoragent@cursor.com> * rlf-codex-rl 5.8 attempt 3: --session-server-workers flag gate attempt 2 (rlf-gate-20261011-4) passed the attention-backend override (AttnBackend.unfused on the island) but died on a session-server import storm: Miles defaults --session-server-workers to 32, and 32 parallel sglang imports on the 4-core L40S island missed the 300 s ready window (same failure class as compaction G3 run c). Yeti had no flag for the worker count; plumb it cli -> launcher -> island -> AgentConfig -> Miles argv, launch-side >=1 validation, translation allowlist. The gate run drops to 8 workers (rollout is 4 groups x 4 samples). Silver lining: --modal-tape-volume + completed-groups in ~/yeto-output worked — the island tape was pulled despite the failed run. Co-authored-by: Cursor <cursoragent@cursor.com> * evidence: gate attempt 3 root causes (unfused OOM, sandbox idle reclaim) Co-authored-by: Cursor <cursoragent@cursor.com> * rl gate: sample outcomes ride the event tape (ports engine) Run rlf-gate-20261011-7 SUCCEEDED (2 rounds trained, reward 0.125) but the gate could not judge: on the ports engine the island checkpoint only ever holds an empty completed-group queue, so per-sample reward/success died with the island. queue_completed_groups now also emits one compact rl_sample_outcome event per sample (fail-closed: skips non-numeric reward, success=None is rejected by the judge), and the gate collects samples from --tape-events when --completed-groups is absent. 3 new tests. Co-authored-by: Cursor <cursoragent@cursor.com> * rl gate: accept rl_trajectory_reward tape records (ports engine) rlf-codex-rl 5.8 attempt 8 lesson: the ports launcher binds --rollout-all-samples-process-path to rollout_meta_hook.extract_rollout_metadata, so the legacy queue_completed_groups hook (054c20d5) never runs there and the tape carried zero rl_sample_outcome events. The meta hook already reports per-sample task_id/reward/success which the driver tapes as rl_trajectory_reward behind --rl-observe-timeline; teach the gate collector to accept both event shapes (sample_index fills index), same fail-closed rules. Co-authored-by: Cursor <cursoragent@cursor.com> * rlf-codex-rl: 5.2/5.8 pass — GPU RL gate verdict=True (rlf-gate-20261011-9) Five criteria on the ports engine tape (32 rl_trajectory_reward records, 2 rounds x 16 samples): reward_sum=6.0, reward_std=0.390, success_count=6, advantage_abs_sum=7.0, policy_update payload hashes differ (v0/v1), collector=miles_island. Decision log: -6/-7/-8 root causes (cuDNN fused refusal, ports empty completed-group queue, all-samples hook occupied by extract_rollout_metadata). Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: michael <78113554+chowhsu@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
范围
rlf-codex-rl 5.8（RL 训练门禁）切片：
yeto/rl/eval/rlf_rl_gate.py：RL gate 判定器——读训练运行证据链（事件磁带、奖励/成功样本、guardrail），输出 pass/fail 判定与证据包tests/test_rlf_rl_gate.py：14 个用例覆盖判定逻辑（缺证据 fail-closed、证据齐全 pass、磁带空洞拒绝等）docs/RLF_CODEX_RUN.md补充 5.8 运行手册evidence/rlf/decision-log.md、acceptance-status.json对应 openspec change：
rlf-codex-rl（任务 5.8）。测试证据
依赖关系
rlf-pr1-core（rlf-codex-rl 核心：trusted flag、Qwen3-30B-A3B profile、RLF 入口接线与 baseline runner #230）——gate 判定器导入yeto.rl.rlf.*（catalog/contracts/modal），docs 增量写在 PR1 的RLF_CODEX_RUN.md之上；请先合 rlf-codex-rl 核心：trusted flag、Qwen3-30B-A3B profile、RLF 入口接线与 baseline runner #230Made with Cursor