rlf-codex-rl 5.8：RL gate 判定器 rlf_rl_gate + 证据链 + attempt-1 教训

docs/RLF_CODEX_RUN.md

@@ -110,3 +110,68 @@ Known unverified (first GPU run must confirm): Miles `resolve_serving`
 coverage of the `qwen3` tito family, LoRA-free `SessionServerConfig`
 (`lora_rank=None`), and A100-80GB scheduling latency on Modal.
 
+
+## Real-GPU RL gate (5.8): Miles GRPO on `qwen35_08b`
+
+The gate proves the RLF -> Miles GRPO -> GAE -> policy-update chain on real
+GPU with the small `qwen35_08b` profile (0.8B, the S17-verified harness
+stack; LoRA, 1xGPU). The 30B capability surface is covered separately by the
+5.6 baseline; 30B RL training is a follow-up item (design decision
+2026-10-11). Tasks: `fixture-ctf-ezshell` (solvable in ~2 turns, provides
+success) and `fixture-ctf-ret2win` (provides reward variance). Codex-only;
+AgentFlow is offline-only and refused at admission.
+
+### Runner and commands
+
+The gate runner is `yeto.rl.eval.rlf_rl_gate`. Island side, in order:
+
+    # 1. admit the packages, write Miles --prompt-data (flags never logged)
+    python -m yeto.rl.eval.rlf_rl_gate prompt-data \
+      --tasks-dir "$YETO_RLF_CTF_TASKS_DIR" --flags-json /trusted/flags.json \
+      --out /run/rlf_gate_prompt.jsonl --repeat 2
+
+    # 2. launch the Miles GRPO island (qwen35_08b, LoRA, 1xGPU):
+    #    --data /run/rlf_gate_prompt.jsonl --codex-backend-profile qwen35_08b
+    #    --reward-function rlf_success   (-> --custom-rm-path yeto.rl.adapters.miles.rewards.rlf_success)
+    #    --advantage-estimator grpo (fixed by island_entry)
+    #    env: YETO_HARNESS_ENVIRONMENT_PROVIDER=yeto.cloud.modal_reward_env:rlf_modal_provider
+    #         YETO_RLF_CTF_TASKS_DIR / YETO_RLF_CTF_FLAGS_JSON (Modal Secret, never in logs)
+
+    # 3. after training, collect real artifacts, judge, write evidence
+    python -m yeto.rl.eval.rlf_rl_gate gate \
+      --completed-groups /run/completed_groups.pt \
+      --adapter-before /run/adapter_init --adapter-after /run/adapter_final \
+      --step 2 --flags-json /trusted/flags.json --out /run/rlf_gate_evidence.json
+
+### Hard criteria (`yeto.rl.rlf.contracts.rl_gate`)
+
+Non-empty reward, `reward_std > 0`, `success_count > 0`,
+`advantage_abs_sum > 0`, `policy_update == true`. All-zero or constant
+reward MUST fail. The verdict is announced only for evidence collected
+from the real island artifacts (`collector="miles_island"`); CPU/injected
+collections exercise the orchestration but can never announce a pass.
+
+### Evidence chain (`yeto.rlf.rl_gate.evidence/v1`)
+
+- reward/success: the island completed-groups checkpoint
+  (`--completed-groups-path`) — the samples the learner actually trained
+  on, each with the `rlf_success` reward and `metadata.success` (W4 grade);
+- advantage: recomputed per GRPO group with `contracts.group_advantages`
+  (the designated computation of record; never hand-filled);
+- policy update: `contracts.policy_update_evidence` over a per-tensor
+  summary of the initial vs final LoRA adapter (代拍板: the Miles fork
+  exports no per-step snapshot hook; this proves parameters changed without
+  a fork patch, at the cost of not attributing the change to a specific
+  optimizer step, which the gate does not require);
+- the JSON carries reward detail, success, `advantage_abs_sum`, and the
+  before/after policy digests; trusted flags are scrubbed
+  (`rlf.modal.scrub`) and only their sha256 appears in `config_digest`.
+
+### Stop conditions
+
+Stop and treat as infrastructure (never scored): serving not healthy within
+15 min, or per-episode HARD deadline (30 min) exceeded repeatedly. Stop
+with gate failure (the chain is broken, investigate before re-running):
+`rl_gate` raising on any criterion, a sample missing `metadata.success`,
+or unchanged adapter digests after training. A passing run archives the
+evidence JSON next to the island artifacts.

evidence/rlf/acceptance-status.json

@@ -6,7 +6,7 @@
   "trusted_reference_flag": "pass",
   "codex_trajectory_metadata": "pass_cpu_tests",
   "cpu_reward_gae_gate": "pass",
-  "modal_gpu_rl_gate": "pending",
+  "modal_gpu_rl_gate": "attempted_failed_trainer_cutlass",
   "qwen3_baseline": "pass",
   "failure_class": "resolved_transport_single_recv",
   "swe": "deferred",
@@ -16,5 +16,6 @@
   "ctf_lease_evidence": "evidence/rlf/modal-ctf-lease-e2e.log (vm sb-01M4MEKE66M30GA31EZBGEBRR0, destroyed, describe=gone)",
   "qwen3_baseline_evidence": "run10 (s1-runs/s21-rlf-baseline-20261011a/run10-out): ezshell success reward=1.0 (2 turns), ret2win failed reward=0.0 (9 turns, wrong win addr); success_count=1, reward mean=0.5 std>0; run8: ret2win 36-turn 31K-token max_seq_len truncation (5.7 trigger evidence)",
   "compaction_real_firing": "pass",
-  "compaction_evidence": "run18 (s1-runs/.../run18-out): 12K window/4K reserve, compaction fired at 10320 tokens, summary_ok, context 8210->2176, episode continued to 12 turns + submit; run14: qwen3_cyber 24K profile active with correct no-fire gating"
+  "compaction_evidence": "run18 (s1-runs/.../run18-out): 12K window/4K reserve, compaction fired at 10320 tokens, summary_ok, context 8210->2176, episode continued to 12 turns + submit; run14: qwen3_cyber 24K profile active with correct no-fire gating",
+  "modal_gpu_rl_gate_evidence": "s1-runs/s21-rlf-baseline-20261011a/gate-launch7.log (rlf-gate-20261011-3): rollout+RLF reward path live; Megatron train step cutlass crd2idx compile error; artifacts lost with island (pull-before-stop lesson recorded)"
 }

evidence/rlf/decision-log.md

@@ -40,3 +40,12 @@
 - 机制验证另用 12K 窗口/4K reserve/单轮 4096（触发线 8192）：run18 第 8 轮 10320 tokens 触发，summary_ok=true，segment 0 kept_steps=1，context 8210→2176 切换后继续 4 轮并以 submit 正常结束。segment identity 与共享 reward 合并由 CPU 测试锁定（test_codex_bridge_compaction / test_rlf_rollout_entry）。
 - 途中修复：baseline runner 未预创建 segment sessions（已补，含 teardown 清理）；for_profile 与覆盖参数冲突（kwargs 优先）；budget==max_seq_len 桥约束（profile 24576 需配 context_length 24576）。
 - 观察：该模型 thinking 轮常超 2K tokens，单轮截断（harness TITO 完整性规则）是长 episode 的主要死因；8K 窗口对该模型不可行。
+
+## 2026-10-11 5.8 GPU RL gate 首跑（失败：训练端 cutlass 编译，非 RLF 链路）
+
+- 配置：L40S×1，qwen35_08b，GRPO 2 轮×4 组×4 样本，LoRA r16，rlf_success reward，rlf_modal_provider，任务 ezshell+ret2win。
+- 进展：岛启动、RLF provider 环境装配、rollout 正常产样本（prefill batch 4 并发）——RLF→Codex rollout→reward 链路在训练路径上真实工作。
+- 失败点：Megatron 训练步 cutlass MLIR 编译错误 `unable to compute crd2idx ... cute.layout "(?):(?{i64 div=8})"`，island rank 0 exit 1，launcher 按固定名单 0s 恢复超时停跑（exit 4）。
+- 教训：岛停止前 tape/checkpoint 未拉回（"no pulled Modal tape file"），本跑 rollout 的 reward 证据随岛销毁。下次开卡：跑挂时先拉回 ~/yeto-output/ 与 ~/yeto-rl/ 再停。
+- failure class：trainer_kernel_compile（与 RLF 无关；与模型/LoRA shape 相关）。
+- 途中修复（已提交 3ff8e93e）：launcher 透传 YETO_RLF_* 前缀、rlf_modal_provider 入 MODAL_SANDBOX_PROVIDERS、岛侧任务目录 /root/sky_workdir 路径、sandbox token 走 Modal Secret。

tests/test_rlf_rl_gate.py

@@ -0,0 +1,214 @@
+"""RLF real-GPU RL gate runner orchestration (task 5.8): fake learner / fake
+provider only — no GPU, no Miles, no torch, no Modal.
+
+These tests cover the orchestration (collection -> gate rows -> rl_gate ->
+evidence).  The injected collection can never announce a passing verdict;
+only the real ``miles_island`` collector on the GPU run may do that.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import pytest
+
+from yeto.rl.eval.rlf_rl_gate import (
+    EVIDENCE_SCHEMA,
+    GateConfig,
+    GateEvidenceError,
+    build_gate_rows,
+    collect_policy_update,
+    collect_rollout_samples,
+    prepare_prompt_data,
+    run_gate,
+    write_evidence,
+)
+
+FLAGS = {"fixture-ctf-ezshell": "flag{ezshell-secret}", "fixture-ctf-ret2win": "flag{ret2win-secret}"}
+
+
+def _sample(group, index, reward, success, task_id="fixture-ctf-ezshell"):
+    return {"reward": reward,
+            "metadata": {"success": success, "rlf_task_id": task_id,
+                         "rlf_episode_id": f"ep-{group}-{index}",
+                         "rlf_trajectory_id": f"traj-{group}-{index}",
+                         "grader_evidence": {"trusted": True}}}
+
+
+def _payload(groups):
+    return {"schema_version": 1, "completed_groups": groups}
+
+
+def _passing_groups():
+    # ezshell group: one success (variance + success_count); ret2win group: all fail.
+    return [[_sample(0, 0, 1.0, True), _sample(0, 1, 0.0, False),
+             _sample(0, 2, 0.0, False), _sample(0, 3, 0.0, False)],
+            [_sample(1, 0, 0.0, False, "fixture-ctf-ret2win"),
+             _sample(1, 1, 0.0, False, "fixture-ctf-ret2win")]]
+
+
+def _update_loader(before, after):
+    tables = {"before": before, "after": after}
+    return lambda path: tables[str(path)]
+
+
+def _update(updated=True):
+    before = {"lora_a": 0.5, "lora_b": -0.25}
+    after = {"lora_a": 0.5 + (0.01 if updated else 0.0), "lora_b": -0.25}
+    return collect_policy_update("before", "after", step=2,
+                                 loader=_update_loader(before, after))
+
+
+def _config(tmp_path):
+    return GateConfig(tasks_dir=str(tmp_path), flags=FLAGS)
+
+
+# -- collection from the completed-groups checkpoint -------------------------
+
+def test_collect_rollout_samples_reads_real_checkpoint_shape():
+    samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(_passing_groups()))
+    assert len(samples) == 6
+    assert samples[0] == {"group_index": 0, "index": 0, "task_id": "fixture-ctf-ezshell",
+                          "reward": 1.0, "success": True}
+    assert sum(s["success"] for s in samples) == 1
+
+
+def test_collect_flattens_compaction_segments_sharing_one_outcome():
+    group = [[_sample(0, 0, 1.0, True), _sample(0, 0, 1.0, True)], _sample(0, 1, 0.0, False)]
+    samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload([group]))
+    assert [s["reward"] for s in samples] == [1.0, 0.0]
+
+
+def test_collect_fails_closed_on_missing_success_or_reward():
+    bad = _sample(0, 0, 1.0, True)
+    del bad["metadata"]["success"]
+    with pytest.raises(GateEvidenceError, match="metadata.success"):
+        collect_rollout_samples("ckpt", loader=lambda _p: _payload([[bad]]))
+    bad = _sample(0, 0, 1.0, True)
+    bad["reward"] = None
+    with pytest.raises(GateEvidenceError, match="numeric reward"):
+        collect_rollout_samples("ckpt", loader=lambda _p: _payload([[bad]]))
+    with pytest.raises(GateEvidenceError, match="no completed groups"):
+        collect_rollout_samples("ckpt", loader=lambda _p: _payload([]))
+
+
+# -- advantages and policy update are computed, never hand-filled ------------
+
+def test_gate_rows_compute_grpo_group_advantages():
+    samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(_passing_groups()))
+    rows = build_gate_rows(samples, _update())
+    first = [r["advantage"] for r in rows if r["metadata"]["group_index"] == 0]
+    assert first == [0.75, -0.25, -0.25, -0.25]  # reward minus group mean
+    assert all(r["policy_update"] for r in rows)
+
+
+def test_policy_update_unchanged_params_marks_rows_not_updated():
+    samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(_passing_groups()))
+    rows = build_gate_rows(samples, _update(updated=False))
+    assert not any(r["policy_update"] for r in rows)
+
+
+# -- the five gate criteria ---------------------------------------------------
+
+def test_all_five_criteria_pass_but_injected_collection_never_announces(tmp_path):
+    samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(_passing_groups()))
+    update = _update()
+    rows = build_gate_rows(samples, update)
+    evidence = run_gate(rows, collector="injected", config=_config(tmp_path), update=update)
+    gate = evidence["gate"]
+    assert gate["passed"] and gate["result"]["reward_std"] > 0
+    assert gate["result"]["success_count"] == 1
+    assert gate["result"]["advantage_abs_sum"] > 0
+    # CPU orchestration only: the 5.8 gate is never announced off the real run.
+    assert evidence["verdict"] is False
+    assert "never announced" in evidence["verdict_note"]
+
+
+def test_all_zero_rewards_fail():
+    groups = [[_sample(0, i, 0.0, False) for i in range(4)]]
+    samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(groups))
+    evidence = run_gate(build_gate_rows(samples, _update()), collector="miles_island")
+    assert evidence["verdict"] is False and evidence["gate"]["passed"] is False
+    assert "reward_std_positive" in evidence["gate"]["error"]
+    assert "success_count_positive" in evidence["gate"]["error"]
+    assert "advantage_abs_sum_positive" in evidence["gate"]["error"]
+
+
+def test_constant_nonzero_rewards_fail():
+    groups = [[_sample(0, i, 1.0, True) for i in range(4)]]
+    samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(groups))
+    evidence = run_gate(build_gate_rows(samples, _update()), collector="miles_island")
+    assert evidence["verdict"] is False
+    assert "reward_std_positive" in evidence["gate"]["error"]
+
+
+def test_no_policy_update_fails_even_with_real_collector():
+    samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(_passing_groups()))
+    update = _update(updated=False)
+    rows = build_gate_rows(samples, update)
+    evidence = run_gate(rows, collector="miles_island", update=update)
+    assert evidence["verdict"] is False
+    assert "policy_update" in evidence["gate"]["error"]
+    pu = evidence["policy_update_evidence"]
+    assert pu["updated"] is False and pu["before_digest"] == pu["after_digest"]
+
+
+def test_real_collector_passing_run_announces_verdict(tmp_path):
+    samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(_passing_groups()))
+    update = _update()
+    rows = build_gate_rows(samples, update)
+    evidence = run_gate(rows, collector="miles_island", config=_config(tmp_path), update=update)
+    assert evidence["verdict"] is True and evidence["schema"] == EVIDENCE_SCHEMA
+    assert evidence["policy_update_evidence"]["before_digest"] != \
+        evidence["policy_update_evidence"]["after_digest"]
+
+
+# -- evidence on disk: content and flag hygiene --------------------------------
+
+def test_write_evidence_scrubs_flags_and_keeps_detail(tmp_path):
+    config = _config(tmp_path)
+    samples = collect_rollout_samples("ckpt", loader=lambda _p: _payload(_passing_groups()))
+    rows = build_gate_rows(samples, _update())
+    rows[0]["metadata"]["note"] = f"grader saw {FLAGS['fixture-ctf-ezshell']} then submitted"
+    evidence = run_gate(rows, collector="injected", config=config, update=_update())
+    out = write_evidence(evidence, tmp_path / "evidence.json", secrets=config.flags.values())
+    text = out.read_text()
+    for flag in FLAGS.values():
+        assert flag not in text
+    assert "***" in text
+    loaded = json.loads(text)
+    assert loaded["schema"] == EVIDENCE_SCHEMA
+    assert {r["reward"] for r in loaded["rows"]} == {0.0, 1.0}
+    assert "flag{" not in json.dumps(loaded["config_digest"])  # digest only
+
+
+def test_config_requires_flags_for_gate_tasks(tmp_path):
+    with pytest.raises(GateEvidenceError, match="no trusted flag"):
+        GateConfig(tasks_dir=str(tmp_path), flags={"fixture-ctf-ezshell": "flag{x}"})
+
+
+# -- prompt data preparation ----------------------------------------------------
+
+def _package(root, task_id):
+    (root / "public").mkdir(parents=True)
+    (root / "public" / "README.md").write_text(f"# {task_id}\nObtain the flag.")
+    (root / "task.yaml").write_text(f"task_id: {task_id}\ngrader_mode: flag\n")
+    (root / "MANIFEST.json").write_text(json.dumps({"task_id": task_id, "grader_mode": "flag"}))
+
+
+def test_prepare_prompt_data_admits_gate_tasks(tmp_path):
+    for task_id in FLAGS:
+        _package(tmp_path / task_id, task_id)
+    out = prepare_prompt_data(_config(tmp_path), tmp_path / "prompt.jsonl")
+    rows = [json.loads(line) for line in Path(out).read_text().splitlines() if line.strip()]
+    assert len(rows) == 2 * len(FLAGS)  # repeat=2 per task
+    assert {r["metadata"]["rlf_task_id"] for r in rows} == set(FLAGS)
+    assert all(r["metadata"]["benchmark"] == "rlf" for r in rows)
+
+
+def test_prepare_prompt_data_rejects_unadmitted_package(tmp_path):
+    _package(tmp_path / "fixture-ctf-ezshell", "fixture-ctf-ezshell")
+    from yeto.rl.rlf.contracts import AdmissionError
+    with pytest.raises(AdmissionError):
+        prepare_prompt_data(_config(tmp_path), tmp_path / "prompt.jsonl")