rlf-codex-rl 核心：trusted flag、Qwen3-30B-A3B profile、RLF 入口接线与 baseline runner

README.md

@@ -185,6 +185,7 @@ Tested — a completed Yeto fine-tuning run on real hardware:
 |---|---|---|
 | `gemma4` | `google/gemma-4-12B-it` | 66 |
 | `qwen3-8b` | `Qwen/Qwen3-8B` | 17 |
+| `qwen3-30b-a3b` | `Qwen/Qwen3-30B-A3B` | 61 |
 | `qwen35-4b` | `Qwen/Qwen3.5-4B` | 8 |
 | `qwen35-9b` | `Qwen/Qwen3.5-9B` | 18 |
 | `qwen35-9b-base` | `Qwen/Qwen3.5-9B-Base` | 18 |

docs/RLF_CODEX_RUN.md

@@ -0,0 +1,112 @@
+# RLF → Yeto Codex RL runbook
+
+The online RL harness is Codex only. rlenvfactory/AgentFlow may produce or review offline task packages, but its node output is never used as task identity, grader evidence, reward, or rollout input. SWE and the legacy `rl-agentic-reward-env` change remain deferred.
+
+## Apply order
+
+1. Generate or install an rlenvfactory CTF package and retain `public/`, `task.yaml`, `MANIFEST.json`, source commit, package/image digests, and runtime quality-gate evidence.
+2. Admit the package with `yeto.rl.rlf.admit_task_package`; reject packages without a trusted runtime gate.
+3. Acquire one Modal VM lease per `(task_id, episode_id, trajectory_id)`. Keep secrets on the trusted side and tear down the lease after grading.
+4. Run the existing Codex subprocess harness. Copy RLF identities and policy identity into every chain/compaction segment; preserve tokens, logprobs, loss masks, and segment metadata.
+5. Finish and grade on the trusted side, then pass the resulting metadata through the Miles or generic verl preflight. Bare float rewards are rejected.
+6. Run the CPU gate before any paid GPU run. A healthy RL gate must show nonzero reward variance, success count, absolute advantage sum, and policy update.
+
+The baseline profile is Qwen3-30B-A3B thinking mode with a 32K window, 1024 generated tokens per turn, at most 48 turns, and 8192-byte tool output.
+
+## Qwen3 cyber baseline profile (5.6)
+
+The registered backend profile is `qwen3_30b_a3b` (`yeto.rl.codex_backend`,
+allowlisted in `yeto/rl/harness/codex/pins.py`). Pass it per launch with
+`--codex-backend-profile qwen3_30b_a3b`. Its budgets are declared on the
+profile; three of them are launch-time knobs that must be set explicitly
+because their global defaults deliberately stay unchanged:
+
+| Budget | Profile value | How it takes effect |
+| --- | --- | --- |
+| context window | 32768 tokens | `--agent-max-seq-len 32768` (run config `agent_max_seq_len`) |
+| generated tokens per turn | 1024 | `YETO_CODEX_BACKEND_MAX_TOKENS=1024` |
+| turns | 48 | `SECRLENV_MAX_TURNS=48` (worker-passthrough env; global default stays 40) |
+| tool output | 8192 bytes | read from the profile by `codex_harness_agent` (`MAX_TOOL_OUTPUT_BYTES`); profiles without the field keep 32768 |
+
+Keep the 20-30 minute episode deadline (`OPENENV_MAX_ROLLOUT_TIME_SECONDS`,
+default 1800s) and record `max_seq_len_hit`, `end_reason`, turns, model
+tokens, tool calls, and tool-output bytes per episode for the baseline
+analysis. If a flag is reached only after truncation, enable the registered compaction plan: 24K active window, 4K–8K reserve, and at most 2–3 compactions while preserving segment identity.
+
+## Baseline runner (5.6): RLF 形态的 Qwen3-30B-A3B
+
+The runner is `yeto.rl.eval.rlf_baseline` (RLF counterpart of
+`yeto/rl/eval/codex_infer.py`, which covers the TB2 attempt shape). Per run:
+admit the task packages into the RLF catalog, render prompt rows, bring up
+serving, then per row acquire one RLF CTF lease (W4 `RlfModalProvider`),
+drive the training Codex subprocess worker, finish on the trusted side, and
+write the trace. Codex-only; AgentFlow artifacts are refused at admission.
+
+### Serving shape (inside the GPU container)
+
+- Modal `A100-80GB` x1, image = the pinned Miles-ports image
+  (`MILES_NEXT_IMAGE`), model read from the `yeto-fn-models` Volume:
+  `/mnt/yeto-models/hf/Qwen3-30B-A3B/ad44e777` (bf16).
+- `python -m sglang.launch_server --model-path /mnt/yeto-models/hf/Qwen3-30B-A3B/ad44e777 \
+   --trust-remote-code --context-length 32768 --mem-fraction-static 0.9`
+  plus the Miles session server (`session_server_config`, LoRA-free). The
+  profile's parsers/chat template come from Miles' `resolve_serving("qwen3")`.
+- Serving that is not healthy within 15 min aborts the whole run and the
+  summary is written with `aborted.classification = "infrastructure"`
+  (never scored). HARD per-task deadline: 30 min
+  (`--episode-seconds`, default 1800; a tighter lease deadline wins).
+
+### Run
+
+Driver side (needs a local modal client):
+
+    export YETO_CODEX_BUNDLE_DIR=/path/to/codex-bundle        # scripts/fetch_codex_bundle.py
+    export YETO_RLF_CTF_TASKS_DIR=/path/to/tasks              # admitted packages (task_id subdirs)
+    export YETO_RLF_BASELINE_ENVS_JSON=/path/to/envs.json     # see below
+    modal run yeto.cloud.modal_rlf_baseline --config-json run.json
+
+`envs.json` holds the container env, same inputs as the eval island smoke:
+the Codex contract env (`yeto.launcher.codex_container_env(
+codex_bundle_contract(...))`), `TBENCH_REWARD_HMAC_KEY`, the sandbox-only
+`MODAL_TOKEN_ID`/`MODAL_TOKEN_SECRET` (the RLF provider opens one VM per
+episode from inside the container), and `YETO_RLF_CTF_FLAGS_JSON`
+(`{"task_id": "flag{...}"}` — trusted; lives only in the Modal Secret,
+never in the image, the traces, or the logs). `run.json`:
+
+    {"run_name": "qwen3-30b-baseline-01", "repeat": 1,
+     "evidence": {"runtime_quality_gate": {"healthy": true}},
+     "baseline": {"episode_seconds": 1800}}
+
+Evidence lands in the `yeto-rlf-baseline-runs` Volume under the run name.
+In-container / CPU-smoke entry without Modal:
+
+    python -m yeto.rl.eval.rlf_baseline --tasks-dir /root/rlf-tasks \
+        --flags-json @flags.json --evidence '{"runtime_quality_gate": {"healthy": true}}' \
+        --out /tmp/rlf-baseline-out [--repeat N] [--task ID] [--serve external --session-url URL]
+
+### Evidence checklist (per run)
+
+- `traces.jsonl`: one record per episode — RLF identity quadruple,
+  `policy_id` (`Qwen/Qwen3-30B-A3B@ad44e777...`), the task prompt, per-turn
+  completion/context tokens and tool-output bytes, per-tool events
+  (scrubbed command, bounded scrubbed output preview, `flag_seen` marker),
+  submission markers (`after_tool_calls`, `has_flag`, `flag_correct`; the
+  submitted flag value is never recorded), `submit_calls`,
+  `max_model_total_tokens`, `max_seq_len_hit`, compaction count,
+  `end_reason`/`end_kind`, `grader_evidence`, reward/success and the
+  `classification` (`success` / `failed` / `infrastructure` / `rejected`).
+- `summary.json`: success rate over evaluated episodes (infrastructure and
+  rejected episodes are excluded, never scored 0), reward distribution,
+  end-reason counts, `max_seq_len_hit` rate, total compactions,
+  turns/tool-calls/token means and maxima, per-task breakdown.
+
+Stop conditions: serving start timeout (infrastructure abort), the 30-min
+per-task deadline (`timeout` episodes keep reward 0 via the signed outcome
+envelope and are never graded), or every prompt row processed. A
+`max_seq_len_hit` rate above zero is the evidence for enabling the
+registered 24K/4K compaction plan (5.7).
+
+Known unverified (first GPU run must confirm): Miles `resolve_serving`
+coverage of the `qwen3` tito family, LoRA-free `SessionServerConfig`
+(`lora_rank=None`), and A100-80GB scheduling latency on Modal.
+

evidence/rlf/MANIFEST-hf-qwen3-30b-a3b-ad44e777.json

@@ -0,0 +1,192 @@
+{
+ "repo": "Qwen/Qwen3-30B-A3B",
+ "revision": "ad44e777bcd18fa416d9da3bd8f70d33ebb85d39",
+ "files": [
+  {
+   "path": ".gitattributes",
+   "size": 1570,
+   "sha256": "34448b82c17d60fec9b65b1f093c115ddbaadc04beb1b0140b6bfed2e012a930",
+   "expected_sha256": null,
+   "ok": true
+  },
+  {
+   "path": "LICENSE",
+   "size": 11343,
+   "sha256": "832dd9e00a68dd83b3c3fb9f5588dad7dcf337a0db50f7d9483f310cd292e92e",
+   "expected_sha256": null,
+   "ok": true
+  },
+  {
+   "path": "README.md",
+   "size": 16798,
+   "sha256": "1897b7cdf7b5f45ae847b87f9b416d4635fe73228e0daac669a4e1f3e5948858",
+   "expected_sha256": null,
+   "ok": true
+  },
+  {
+   "path": "config.json",
+   "size": 963,
+   "sha256": "2850ddb3bf7aecad20b611e2d44f3077fc8193f4827c93beddd4c02ad63c2297",
+   "expected_sha256": null,
+   "ok": true
+  },
+  {
+   "path": "generation_config.json",
+   "size": 239,
+   "sha256": "2325da0f15bb848e018c5ae071b7943332e9f871d6b60e2ed22ca97d4cb993d2",
+   "expected_sha256": null,
+   "ok": true
+  },
+  {
+   "path": "merges.txt",
+   "size": 1671853,
+   "sha256": "8831e4f1a044471340f7c0a83d7bd71306a5b867e95fd870f74d0c5308a904d5",
+   "expected_sha256": null,
+   "ok": true
+  },
+  {
+   "path": "model-00001-of-00016.safetensors",
+   "size": 3999417504,
+   "sha256": "454e77b346a61bfb201d54df60e15158838cf930617ee135113556204f2802b5",
+   "expected_sha256": "454e77b346a61bfb201d54df60e15158838cf930617ee135113556204f2802b5",
+   "ok": true
+  },
+  {
+   "path": "model-00002-of-00016.safetensors",
+   "size": 3999974192,
+   "sha256": "47f015d6e5bb1782a834d75c06113c5e8c77ddca1cb7daf89686a4ec0deda19e",
+   "expected_sha256": "47f015d6e5bb1782a834d75c06113c5e8c77ddca1cb7daf89686a4ec0deda19e",
+   "ok": true
+  },
+  {
+   "path": "model-00003-of-00016.safetensors",
+   "size": 3997360832,
+   "sha256": "ac0bf5990f2da995c1e8b77a3149dee71900b4fe1b8a614231dea9647193d96b",
+   "expected_sha256": "ac0bf5990f2da995c1e8b77a3149dee71900b4fe1b8a614231dea9647193d96b",
+   "ok": true
+  },
+  {
+   "path": "model-00004-of-00016.safetensors",
+   "size": 3999975056,
+   "sha256": "89b01fd34a683c70fdb7f22c2e7090538d65063335ba0db46f3654642b17d1e5",
+   "expected_sha256": "89b01fd34a683c70fdb7f22c2e7090538d65063335ba0db46f3654642b17d1e5",
+   "ok": true
+  },
+  {
+   "path": "model-00005-of-00016.safetensors",
+   "size": 3999975400,
+   "sha256": "9849eb3584d928e35bf5956ccfd91b64207467e1fad8304aae68dc4d92d6d6d9",
+   "expected_sha256": "9849eb3584d928e35bf5956ccfd91b64207467e1fad8304aae68dc4d92d6d6d9",
+   "ok": true
+  },
+  {
+   "path": "model-00006-of-00016.safetensors",
+   "size": 3999975400,
+   "sha256": "f7a0f1525557d740158fb692986ad43506e1b5901e7406ae65808c6457bc53ad",
+   "expected_sha256": "f7a0f1525557d740158fb692986ad43506e1b5901e7406ae65808c6457bc53ad",
+   "ok": true
+  },
+  {
+   "path": "model-00007-of-00016.safetensors",
+   "size": 3999975472,
+   "sha256": "920702a50f27a009e5f676da3b02978dbd794f6ad7854f301d709136154c1a84",
+   "expected_sha256": "920702a50f27a009e5f676da3b02978dbd794f6ad7854f301d709136154c1a84",
+   "ok": true
+  },
+  {
+   "path": "model-00008-of-00016.safetensors",
+   "size": 3997362064,
+   "sha256": "a85bf0cc8a8047c116172d4c08cf4792eb59d4375032708ba3e1a7ffca0f708a",
+   "expected_sha256": "a85bf0cc8a8047c116172d4c08cf4792eb59d4375032708ba3e1a7ffca0f708a",
+   "ok": true
+  },
+  {
+   "path": "model-00009-of-00016.safetensors",
+   "size": 3999975408,
+   "sha256": "25cd8aaed86b8e17685efb152912f92448ea3fffb1bdc247f257f04d05907604",
+   "expected_sha256": "25cd8aaed86b8e17685efb152912f92448ea3fffb1bdc247f257f04d05907604",
+   "ok": true
+  },
+  {
+   "path": "model-00010-of-00016.safetensors",
+   "size": 3999975400,
+   "sha256": "7c3307ee214797c4bc61c0994f81d690e768150df5f14d18edf84e53ba4810dc",
+   "expected_sha256": "7c3307ee214797c4bc61c0994f81d690e768150df5f14d18edf84e53ba4810dc",
+   "ok": true
+  },
+  {
+   "path": "model-00011-of-00016.safetensors",
+   "size": 3999975408,
+   "sha256": "c658cad2842d36fa4c7c7f726f8515dc5670a3d80c94e13257d4e322ffe61988",
+   "expected_sha256": "c658cad2842d36fa4c7c7f726f8515dc5670a3d80c94e13257d4e322ffe61988",
+   "ok": true
+  },
+  {
+   "path": "model-00012-of-00016.safetensors",
+   "size": 3987400496,
+   "sha256": "8cb898bc5e78492600053d1105c9ff61d7a719f5cc2802df63e41535a652cc26",
+   "expected_sha256": "8cb898bc5e78492600053d1105c9ff61d7a719f5cc2802df63e41535a652cc26",
+   "ok": true
+  },
+  {
+   "path": "model-00013-of-00016.safetensors",
+   "size": 3997353632,
+   "sha256": "599594421f314cb8e8ab4474db0eb490cc6310585067e82d73821254b93319ce",
+   "expected_sha256": "599594421f314cb8e8ab4474db0eb490cc6310585067e82d73821254b93319ce",
+   "ok": true
+  },
+  {
+   "path": "model-00014-of-00016.safetensors",
+   "size": 3999975400,
+   "sha256": "66d3294e9976b5f01d117a0a0bed768f128a8898eb6937e224d051de2961d363",
+   "expected_sha256": "66d3294e9976b5f01d117a0a0bed768f128a8898eb6937e224d051de2961d363",
+   "ok": true
+  },
+  {
+   "path": "model-00015-of-00016.safetensors",
+   "size": 3999975400,
+   "sha256": "2fe000da4fc7399ff6303bb0daec8c415a5d850f5ea53863392758c77fc37664",
+   "expected_sha256": "2fe000da4fc7399ff6303bb0daec8c415a5d850f5ea53863392758c77fc37664",
+   "ok": true
+  },
+  {
+   "path": "model-00016-of-00016.safetensors",
+   "size": 1087928584,
+   "sha256": "0c979e314faf06e94a5e1841abd285d4e26e763c79c1abe7710d543192d9da48",
+   "expected_sha256": "0c979e314faf06e94a5e1841abd285d4e26e763c79c1abe7710d543192d9da48",
+   "ok": true
+  },
+  {
+   "path": "model.safetensors.index.json",
+   "size": 1699758,
+   "sha256": "df0d481ec595c55a0ba58426d517390c6214a566ec4ff1c8fc4bbce9f57b3c24",
+   "expected_sha256": null,
+   "ok": true
+  },
+  {
+   "path": "tokenizer.json",
+   "size": 11422654,
+   "sha256": "aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4",
+   "expected_sha256": "aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4",
+   "ok": true
+  },
+  {
+   "path": "tokenizer_config.json",
+   "size": 9732,
+   "sha256": "d5d09f07b48c3086c508b30d1c9114bd1189145b74e982a265350c923acd8101",
+   "expected_sha256": null,
+   "ok": true
+  },
+  {
+   "path": "vocab.json",
+   "size": 2776833,
+   "sha256": "ca10d7e9fb3ed18575dd1e277a2579c16d108e32f27439684afa0e10b1440910",
+   "expected_sha256": null,
+   "ok": true
+  }
+ ],
+ "n": 26,
+ "total_bytes": 61084187391,
+ "all_ok": true,
+ "wall_s": 386
+}

evidence/rlf/README.md

@@ -0,0 +1,5 @@
+# RLF bridge evidence
+
+The trusted-side bridge requires the rlenvfactory package shape (`public/`, `task.yaml`, `MANIFEST.json`) and an explicit runtime quality gate. AgentFlow node output is deliberately rejected as identity or acceptance evidence. Episodes carry task/trajectory/episode identities, clear secrets during idempotent teardown, and preserve signed grader evidence in neutral reward metadata.
+
+The RL gate rejects empty, constant-reward, zero-advantage or non-updating rollouts. A healthy fixture must show nonzero reward variance, success count, absolute advantage sum, and a policy update.

evidence/rlf/acceptance-status.json

@@ -0,0 +1,20 @@
+{
+  "package_admission": "pass",
+  "modal_vm_dockerd": "pass",
+  "ctf_image_build": "pass",
+  "target_health": "pass",
+  "trusted_reference_flag": "pass",
+  "codex_trajectory_metadata": "pass_cpu_tests",
+  "cpu_reward_gae_gate": "pass",
+  "modal_gpu_rl_gate": "pending",
+  "qwen3_baseline": "pass",
+  "failure_class": "resolved_transport_single_recv",
+  "swe": "deferred",
+  "online_harness": "codex_only",
+  "trusted_flag_evidence": "evidence/rlf/modal-ctf-trusted-recvfix2.log (sandbox sb-01M4MB62RAE1FR0Y3HDGWZJRX7, terminated)",
+  "ctf_lease_vm_e2e": "pass",
+  "ctf_lease_evidence": "evidence/rlf/modal-ctf-lease-e2e.log (vm sb-01M4MEKE66M30GA31EZBGEBRR0, destroyed, describe=gone)",
+  "qwen3_baseline_evidence": "run10 (s1-runs/s21-rlf-baseline-20261011a/run10-out): ezshell success reward=1.0 (2 turns), ret2win failed reward=0.0 (9 turns, wrong win addr); success_count=1, reward mean=0.5 std>0; run8: ret2win 36-turn 31K-token max_seq_len truncation (5.7 trigger evidence)",
+  "compaction_real_firing": "pass",
+  "compaction_evidence": "run18 (s1-runs/.../run18-out): 12K window/4K reserve, compaction fired at 10320 tokens, summary_ok, context 8210->2176, episode continued to 12 turns + submit; run14: qwen3_cyber 24K profile active with correct no-fire gating"
+}

evidence/rlf/cpu-e2e.json

@@ -0,0 +1,27 @@
+{
+  "reward_sum": 1.0,
+  "reward_std": 0.5,
+  "success_count": 1,
+  "advantage_abs_sum": 1.0,
+  "policy_update": true,
+  "passed": true,
+  "evidence": {
+    "advantages": [
+      -0.5,
+      0.5
+    ],
+    "advantage_source": "group_advantages(reward - group mean)",
+    "policy_update": {
+      "step": 0,
+      "updated": true,
+      "before_digest": "1a88ebda8a11cbb6e7c15508650abf744225fca050040070236df580ad547eec",
+      "after_digest": "7f4fe5e9c0d9758e4470943d9f36385a32fd04a3fa957b240a949b81c7a17228",
+      "changed": [
+        "bias"
+      ]
+    },
+    "learner_params_after": {
+      "bias": 0.010000000000000002
+    }
+  }
+}

evidence/rlf/cpu-preflight.json

@@ -0,0 +1 @@
+{"status":"pass","checks":["package admission","trusted finish teardown","reward gate rejects zero/constant","verl metadata gate","codex-only marker"]}