Skip to content

rlf-codex-rl：5.2/5.8 通过——GPU RL gate verdict=True（rlf-gate-20261011-9） #236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
michaellchung wants to merge 1 commit into from
Closed

rlf-codex-rl：5.2/5.8 通过——GPU RL gate verdict=True（rlf-gate-20261011-9） #236

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 14 additions & 14 deletions evidence/rlf/acceptance-status.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
{
"package_admission": "pass",
"modal_vm_dockerd": "pass",
"ctf_image_build": "pass",
"target_health": "pass",
"trusted_reference_flag": "pass",
"codex_trajectory_metadata": "pass_cpu_tests",
"compaction_evidence": "run18 (s1-runs/.../run18-out): 12K window/4K reserve, compaction fired at 10320 tokens, summary_ok, context 8210->2176, episode continued to 12 turns + submit; run14: qwen3_cyber 24K profile active with correct no-fire gating",
"compaction_real_firing": "pass",
"cpu_reward_gae_gate": "pass",
"modal_gpu_rl_gate": "attempted_failed_trainer_cutlass",
"qwen3_baseline": "pass",
"ctf_image_build": "pass",
"ctf_lease_evidence": "evidence/rlf/modal-ctf-lease-e2e.log (vm sb-01M4MEKE66M30GA31EZBGEBRR0, destroyed, describe=gone)",
"ctf_lease_vm_e2e": "pass",
"failure_class": "resolved_transport_single_recv",
"swe": "deferred",
"modal_gpu_rl_gate": "pass",
"modal_gpu_rl_gate_evidence": "evidence/rlf/gate-20261011-9.json (rlf-gate-20261011-9, Modal H200 ports+Qwen3.5-4B rev 851bf6e8 LoRA r16 unfused, 2轮×4组×4样本: reward_sum=6.0 std=0.390 success_count=6 adv_abs_sum=7.0 policy_update=True, collector=miles_island; tape run9-tape, head logs run9-head-logs.tgz, island+head torn down)",
"modal_vm_dockerd": "pass",
"online_harness": "codex_only",
"trusted_flag_evidence": "evidence/rlf/modal-ctf-trusted-recvfix2.log (sandbox sb-01M4MB62RAE1FR0Y3HDGWZJRX7, terminated)",
"ctf_lease_vm_e2e": "pass",
"ctf_lease_evidence": "evidence/rlf/modal-ctf-lease-e2e.log (vm sb-01M4MEKE66M30GA31EZBGEBRR0, destroyed, describe=gone)",
"package_admission": "pass",
"qwen3_baseline": "pass",
"qwen3_baseline_evidence": "run10 (s1-runs/s21-rlf-baseline-20261011a/run10-out): ezshell success reward=1.0 (2 turns), ret2win failed reward=0.0 (9 turns, wrong win addr); success_count=1, reward mean=0.5 std>0; run8: ret2win 36-turn 31K-token max_seq_len truncation (5.7 trigger evidence)",
"compaction_real_firing": "pass",
"compaction_evidence": "run18 (s1-runs/.../run18-out): 12K window/4K reserve, compaction fired at 10320 tokens, summary_ok, context 8210->2176, episode continued to 12 turns + submit; run14: qwen3_cyber 24K profile active with correct no-fire gating",
"modal_gpu_rl_gate_evidence": "s1-runs/s21-rlf-baseline-20261011a/gate-launch7.log (rlf-gate-20261011-3): rollout+RLF reward path live; Megatron train step cutlass crd2idx compile error; artifacts lost with island (pull-before-stop lesson recorded)"
"swe": "deferred",
"target_health": "pass",
"trusted_flag_evidence": "evidence/rlf/modal-ctf-trusted-recvfix2.log (sandbox sb-01M4MB62RAE1FR0Y3HDGWZJRX7, terminated)",
"trusted_reference_flag": "pass"
}
8 changes: 8 additions & 0 deletions evidence/rlf/decision-log.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,11 @@
3. 其余不变（LoRA r16 attention、seq 8192、GRPO 2×4×4、worker 8）。预算预登记 ≤$6。

第五跑（rlf-gate-20261011-7）head 形态（代拍板）：本地 preflight 线程守卫把内核线程计入 root（~1000），真实用户线程仅 ~1841 仍卡 2800 线；按用户指示改用 `--controller head`（Nebius eu-north1 无卡 head）+ `--preflight-threads off`——理由：guard 的目的是保护跑 driver 的机器，head 模式下 driver 在 Nebius VM，本地只剩瘦客户端，本地计数是不适用的机器。Modal 令牌按 s19-critic-g3.sh 模式走环境变量进 head（不落盘）。跑完先从 head 拉回产物再 sky down。

## 2026-10-11 5.8 gate 第六至八跑（-7/-8/-9）与最终通过

- **-6（fused/L40S/0.8B）**：训练步 cuDNN fused 报 "No dot product attention backend is available"——flash(cute JIT 拒)、fused(cuDNN 拒)、unfused(OOM) 三后端在 L40S/sm89 全灭。代拍板：换 H200×1(141G)+unfused+Qwen3.5-4B（rev 851bf6e8 M1 钉版；0.8B 连续两跑 reward 全零过不了 success_count）。
- **-7（H200+4B）**：训练 SUCCEEDED（raw_reward 0.125→0.375）但判定卡证据缺口：ports 引擎 completed_groups 恒空，样本 outcome 随岛销毁。修 054c20d5（legacy hook 写磁带）。
- **-8**：训练 SUCCEEDED（0.5→0.1875）但仍零 outcome——ports 启动配置里 --rollout-all-samples-process-path 被 rollout_meta_hook.extract_rollout_metadata 占用（config.py:834），054c20d5 的 queue_completed_groups 只在 legacy 非 ports 路径注册，修复打在了不执行的位置。根因教训：**改代码前先验证目标函数在真实配置下被调用**（日志 2164 行一锤定音）。
- **-9（通过）**：judge 改收 driver 侧现成的 rl_trajectory_reward 磁带事件（rl-fn-codex-rollout 1.0 管线，--rl-observe-timeline 打开，ec656e4e）。磁带 32 条轨迹事件（16样本×2轮），五判据全过：reward_sum=6.0、std=0.390、success_count=6、advantage_abs_sum=7.0、policy_update=True（v0/v1 payload hash 相异），collector=miles_island，verdict=True。**rlf-codex-rl 5.2/5.8 通过。**
- 奖励曲线（4B，ezshell+ret2win 2×2）：-7: 0.125→0.375；-8: 0.5→0.1875；-9: 0.25→0.125。ret2win 全程零（符合设计，30B 也解不出）；轮间波动为采样噪声，gate 不考趋势。
Loading
Loading