Repository navigation
rlf-codex-rl：5.2/5.8 通过——GPU RL gate verdict=True（rlf-gate-20261011-9） - #236
Closed
michaellchung wants to merge 1 commit into
Closed
michaellchung wants to merge 1 commit into
michaellchung wants to merge 1 commit into
Conversation
…011-9) Five criteria on the ports engine tape (32 rl_trajectory_reward records, 2 rounds x 16 samples): reward_sum=6.0, reward_std=0.390, success_count=6, advantage_abs_sum=7.0, policy_update payload hashes differ (v0/v1), collector=miles_island. Decision log: -6/-7/-8 root causes (cuDNN fused refusal, ports empty completed-group queue, all-samples hook occupied by extract_rollout_metadata). Co-authored-by: Cursor <cursoragent@cursor.com>
michaellchung
added a commit
that referenced
this pull request
Oct 11, 2026
….8 通过证据（原 #231–#236 恢复合并） (#237) * rlf-codex-rl: 5.8 gate runner + evidence chain (rlf_rl_gate), docs * rlf-codex-rl: 5.8 gate attempt 1 failed at trainer cutlass compile; evidence + lessons * launcher: passthrough YETO_RLF_* envs + rlf_modal_provider as Modal sandbox provider * rlf(modal): sandbox idle_timeout follows ttl_s Gate attempt 5: a CTF VM was reclaimed mid-episode by Modal's 300s idle kill while the model was in a long generation turn (no exec running), the next exec raised NotFoundError, the episode aborted without trusted finish and the fail-closed preflight discarded the group. Default idle_timeout now follows ttl_s (hard cap still bounds leaks); overridable via the new RlfModalVmConfig.idle_timeout_s field. Co-authored-by: Cursor <cursoragent@cursor.com> * rlf-codex-rl 5.8 attempt 2: training attention backend override + tape-surviving gate evidence - --rl-training-attention-backend {auto,flash,fused,unfused}: recipe default overridable (flash-only recipes refuse); gate run uses unfused after the L40S flash-attn cute JIT crd2idx compile failure (failure class trainer_kernel_compile, attention fwd, LoRA-independent) - rlf_rl_gate: policy-update proof from the island event tape rl_publication payload hashes (代拍板 v2) — survives island teardown via --modal-tape-volume; adapter-file path kept; policy_update_source recorded in evidence - CPU tests: 4 new backend-override + 5 new tape-collection tests Co-authored-by: Cursor <cursoragent@cursor.com> * rlf-codex-rl 5.8 attempt 3: --session-server-workers flag gate attempt 2 (rlf-gate-20261011-4) passed the attention-backend override (AttnBackend.unfused on the island) but died on a session-server import storm: Miles defaults --session-server-workers to 32, and 32 parallel sglang imports on the 4-core L40S island missed the 300 s ready window (same failure class as compaction G3 run c). Yeti had no flag for the worker count; plumb it cli -> launcher -> island -> AgentConfig -> Miles argv, launch-side >=1 validation, translation allowlist. The gate run drops to 8 workers (rollout is 4 groups x 4 samples). Silver lining: --modal-tape-volume + completed-groups in ~/yeto-output worked — the island tape was pulled despite the failed run. Co-authored-by: Cursor <cursoragent@cursor.com> * evidence: gate attempt 3 root causes (unfused OOM, sandbox idle reclaim) Co-authored-by: Cursor <cursoragent@cursor.com> * rl gate: sample outcomes ride the event tape (ports engine) Run rlf-gate-20261011-7 SUCCEEDED (2 rounds trained, reward 0.125) but the gate could not judge: on the ports engine the island checkpoint only ever holds an empty completed-group queue, so per-sample reward/success died with the island. queue_completed_groups now also emits one compact rl_sample_outcome event per sample (fail-closed: skips non-numeric reward, success=None is rejected by the judge), and the gate collects samples from --tape-events when --completed-groups is absent. 3 new tests. Co-authored-by: Cursor <cursoragent@cursor.com> * rl gate: accept rl_trajectory_reward tape records (ports engine) rlf-codex-rl 5.8 attempt 8 lesson: the ports launcher binds --rollout-all-samples-process-path to rollout_meta_hook.extract_rollout_metadata, so the legacy queue_completed_groups hook (054c20d5) never runs there and the tape carried zero rl_sample_outcome events. The meta hook already reports per-sample task_id/reward/success which the driver tapes as rl_trajectory_reward behind --rl-observe-timeline; teach the gate collector to accept both event shapes (sample_index fills index), same fail-closed rules. Co-authored-by: Cursor <cursoragent@cursor.com> * rlf-codex-rl: 5.2/5.8 pass — GPU RL gate verdict=True (rlf-gate-20261011-9) Five criteria on the ports engine tape (32 rl_trajectory_reward records, 2 rounds x 16 samples): reward_sum=6.0, reward_std=0.390, success_count=6, advantage_abs_sum=7.0, policy_update payload hashes differ (v0/v1), collector=miles_island. Decision log: -6/-7/-8 root causes (cuDNN fused refusal, ports empty completed-group queue, all-samples hook occupied by extract_rollout_metadata). Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: michael <78113554+chowhsu@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
范围
-9 跑的通过证据，rlf-codex-rl 收官切片：
evidence/rlf/gate-20261011-9.json：GPU RL gate 判定文件（verdict=True）。ports 引擎磁带 32 条rl_trajectory_reward（2 轮 × 16 样本）：reward_sum=6.0、reward_std=0.390、success_count=6、advantage_abs_sum=7.0、policy_update payload 哈希 v0/v1 相异，collector=miles_island——五条判定标准全过acceptance-status.json：5.2/5.8 验收状态更新为通过openspec/changes/rlf-codex-rl/tasks.md：勾选 5.2/5.8decision-log.md：补 -6/-7/-8 根因（cuDNN fused 拒绝、ports 空 completed-group 队列、all-samples hook 被extract_rollout_metadata占用——后者即 rlf-codex-rl 5.8：gate 磁带收集器同时接受 rl_trajectory_reward 事件（ports 引擎关键修复） #235 的修复点）对应 openspec change：
rlf-codex-rl。安全确认
gate-20261011-9.json为纯 JSON 文本（file校验 + json.loads 解析通过），全文 0 处 token/secret/password/api_key 字样；已经过 write_evidence 密钥洗脱，入库前二次扫描确认无密钥值。测试证据
依赖关系
rlf-pr6-tape-reward-records（rlf-codex-rl 5.8：gate 磁带收集器同时接受 rl_trajectory_reward 事件（ports 引擎关键修复） #235）——判定文件正是 rlf-codex-rl 5.8：gate 磁带收集器同时接受 rl_trajectory_reward 事件（ports 引擎关键修复） #235 收集器接受的rl_trajectory_reward磁带记录的产物；合并顺序：rlf-codex-rl 核心：trusted flag、Qwen3-30B-A3B profile、RLF 入口接线与 baseline runner #230 → rlf-codex-rl 5.8：RL gate 判定器 rlf_rl_gate + 证据链 + attempt-1 教训 #231 → rlf-codex-rl 5.8：--rl-training-attention-backend 与 --session-server-workers 两个训练侧 flag #233 → rlf-codex-rl 5.8：样本级 reward/success 上事件磁带，gate 从磁带收集 outcome #234 → rlf-codex-rl 5.8：gate 磁带收集器同时接受 rl_trajectory_reward 事件（ports 引擎关键修复） #235 → 本 PRMade with Cursor