Sync upstream/main (2026-10-14) / 同步上游 main（2026-10-14）

.agents/skills/debug-agentx-runs/SKILL.md

@@ -154,7 +154,7 @@ Use AgentX phase markers, not total Slurm runtime:
 
 ```bash
 grep -E \
-  "Phase warmup progress|WARMUP cache pressure|Phase warmup complete|Phase profiling started|Phase profiling complete|replay_rc=" \
+  "Phase warmup progress|WARMUP cache pressure|Phase warmup complete|Phase profiling started|Phase profiling complete|process_agentic_result" \
   "<LOG_DIR>/benchmark.out"
 date -u
 ```
@@ -199,8 +199,11 @@ Use `scancel` or direct process termination only with explicit approval and a co
 reason. Doing so can bypass cleanup or strand shared-cluster state. Never kill only the
 backend and leave the workflow silently occupying a runner.
 
-After a recipe/config fix, use a targeted e2e dispatch for fast feedback. Reserve another
-official full sweep for the candidate that has passed direct cluster inspection.
+After a recipe/config fix, use a targeted e2e dispatch for fast feedback. Add
+`-f agentx-fast=true` to it for one warmup request per lane and a 20-minute profile. Reserve
+another official full sweep for the candidate that has passed direct cluster inspection.
+The `agentx-fast` PR modifier gives a PR sweep the same short profile, but it needs a primary
+sweep label and blocks artifact reuse while applied, so remove it before that official sweep.
 
 ## 7. Report the live diagnosis
 

.agents/skills/debug-runs/SKILL.md

@@ -24,9 +24,9 @@ paths live in an access-controlled **InferenceX Clusters** Slack canvas, NOT in
 
 Before SSHing to a cluster, look up that cluster's row in the canvas for: **login address**,
 **GHA runner user**, **runner directory**, any **jumpbox / ProxyJump**, whether it's
-**Slurm or bare-metal**, and the **per-node host RAM**. The matching
-`runners/launch_<cluster>.sh` is the source of truth for the exact container image mounts
-and the benchmark command.
+**Slurm or bare-metal**, and the **per-node host RAM**. The cluster's `clusters.<id>` record in
+`inferencex-e2e/configs/runners.yaml` and the launch drivers under `inferencex-e2e/infx/launch/`
+are the source of truth for the exact container image, mounts and benchmark command.
 
 - If you **can't read the canvas** (no Slack access, or unsure), **ask the user** for the
   cluster's SSH target + runner user rather than guessing or pasting infra into the repo.
@@ -46,19 +46,23 @@ and the benchmark command.
 
 ### 1. Trigger (or reuse) the sweep
 
-A PR's sweep is kicked by labels. The `/sweep` comment trigger was removed, so use the label:
+A PR's sweep is kicked by labels. The `/sweep` comment trigger was removed, so apply **exactly one** primary label. With more than one, `check-changelog` fails with `PR has multiple conflicting sweep labels. Pick exactly one.` The [sweep-label reference](../../../.github/AGENT_OPERATIONS.md#sweep-labels-and-reuse) has the full rules.
 
-- **`full-sweep-fail-fast`** runs the full sweep and bails on the first failure per matrix for faster feedback while debugging. It is the **strongly recommended default** and the label that `/nuke` attaches.
-- **`full-sweep-enabled`** runs the full GPU sweep and lets every job complete despite failures. Use it only when a flaky job killing its matrix's in-flight results is unacceptable.
-- To re-trigger a sweep without a new commit, remove and re-add the sweep label.
+- **`full-sweep-fail-fast`** runs a canary first when a point qualifies: the lowest-concurrency non-eval single-node point, else a multi-node AgentX point. Sweeps with only multi-node fixed-sequence or eval points have none and fan out at once. If the canary fails, every other matrix is skipped. Each matrix bails on its first failure for faster feedback while debugging. It is the **strongly recommended default** and the label that `/nuke` attaches.
+- **`full-sweep-enabled`** has the same canary gate but no fail-fast, so every job runs to completion despite failures. Use it only when a flaky job killing its matrix's in-flight results is unacceptable.
+- **`non-canary-full-sweep-enabled`** has neither a canary nor fail-fast, so every job runs to completion despite failures. Use it only when canary gating is also unsuitable.
+- Modifiers need a primary label; alone they start no GPU sweep (`check-changelog` still runs). `all-evals` expands eval selection to every eligible config, keeps throughput, and stays reuse-eligible. `evals-only` suppresses throughput. `agentx-fast` uses one AgentX warmup request per lane and a 20-minute profile. Reuse is rejected while the PR carries `evals-only` or `agentx-fast`.
+- No label trims concurrency. For a lowest-concurrency smoke, dispatch `e2e-tests.yml` (below) with `-f trim-conc=true`.
+- Adding or removing a primary or modifier label cancels the in-progress sweep, and a new one starts only while a primary label remains. To re-trigger a sweep without a new commit, remove and re-add the primary label.
 
 For a **single config** (tightest CI loop, skips the rest of the matrix), dispatch e2e directly:
 
 ```bash
 gh workflow run e2e-tests.yml -f generate-cli-command="test-config --config-key <KEY> --config-file <PATH/to/master.yaml>" -f test-name="debug <KEY>"
 ```
 
-(`generate-cli-command` is the required input. `--target` is NOT a real arg.)
+(`generate-cli-command` carries the matrix selection; the workflow marks it `required: false` because the trusted changelog-dispatch (Klaud) mode omits it, but a manual dispatch without it fails at setup. Its config paths are relative to
+`inferencex-e2e/`, for example `configs/nvidia-master.yaml`. `--target` is NOT a real arg.)
 
 ### 2. Monitor continuously
 
@@ -103,9 +107,14 @@ Steps:
 
 1. Use the job or runner name to identify the node. Look up that cluster's access details in
    the canvas, then SSH in with `ssh -A` when a jumpbox or agent forwarding is involved.
-2. Reproduce the exact benchmark the launcher runs. Read `runners/launch_<cluster>.sh` for
-   the image, container mounts, and the `benchmarks/single_node/<...>.sh` command and env
-   (`IMAGE`, `TP`, `PRECISION`, `EXP_NAME`, `SPEC_DECODING`, …). On Slurm clusters, use
+2. Reproduce the exact benchmark the launcher runs. Single-node jobs go through the srt driver
+   of `python -m infx.launch run` (`inferencex-e2e/infx/launch/drivers/srt/`): it binds the
+   matrix row's `srt-recipe` (`inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/<srt-recipe-dir>/<srt-recipe>`)
+   with `python -m infx.srt_slurm.single_node prepare`, renders the job-local `srtslurm.yaml`
+   from the cluster's record, then submits it through srtctl. Read the bound `recipe.yaml` that
+   `prepare` writes for the image (`model.container`, the master `image`), server args and env,
+   and the cluster record for mounts and the job env (`IMAGE`, `TP`, `PRECISION`,
+   `SPEC_DECODING`, `CONC`, …). On Slurm clusters, use
    `salloc` or `srun` with the squash image. On the **bare-metal `-tw` pools, use `docker run`**
    on the node directly without `srun`.
 3. **Always diff against a working node or working SKU** for reference. Most node failures
@@ -161,10 +170,10 @@ admin-merge on your own judgment.
 Report the two things the user will decide on:
 
 1. **Sweep status.** Is it 100% of full-sweep jobs passing (green), or fail-fast-truncated or partial?
-2. **Perf delta** vs the most recent official `main` run for that SKU. Compare against the
+2. **Perf delta** vs the most recent official (published) results for that SKU. Compare against the
    latest main results, e.g. on inferencex.semianalysis.com
    (`https://inferencex.semianalysis.com/inference?...&i_active=<sku>_<engine>`) or the
-   stored results for that SKU's last main `run-id`.
+   stored results for that SKU's latest published `run-id`.
 
 Present green-ness and the perf comparison, then **wait for the user** to decide whether to merge.
 

.agents/skills/live-logs/SKILL.md

@@ -0,0 +1,65 @@
+---
+name: live-logs
+description: Open a live, per-node log viewer in the browser for a running multi-node sweep job, given a PR number, PR URL, workflow run URL or job URL. Resolves the GitHub job to its Slurm job on the cluster, then streams every worker (prefill / decode / agg), srtctl, Dynamo frontend, Mooncake / etcd service and aiperf log into resizable panes with phase tracking, error highlighting and node memory. Use when the user asks to watch, stream, tail or open the logs of a running sweep, canary or AgentX job.
+---
+
+# Live logs for a sweep job
+
+GitHub Actions buffers multi-node job output until the job ends. This skill streams the
+srt-slurm logs straight from the cluster to a local page instead.
+
+## Run it
+
+```bash
+python3 .agents/skills/live-logs/scripts/live_logs.py <PR# | PR URL | run URL | job URL>
+```
+
+- **PR number or PR URL:** uses the in-progress `Run Sweep` run on the PR head. If none is running, it falls back to the latest one.
+- **Run URL:** uses every in-progress job with a runner.
+- **Job URL:** uses just that job.
+- **Result:** one local viewer per Slurm job, for example the benchmark job and the eval job of the same run. Each opens in the browser, and each URL is printed.
+- **Stop:** `live_logs.py --stop` stops every viewer the script started.
+- **Manual mode:** when you already know the job, pass `--host <ssh target> --job <slurm id> --logdir <.../outputs/<id>/logs>` instead of a target.
+- **Faster start on very large logs:** pass `--history 4000` to fetch only the last 4000 lines of each file.
+- **Requirements:** only the Python standard library, plus `gh` (authenticated) and `ssh` to the cluster login node. The viewer binds to `127.0.0.1` only.
+
+Job resolution relies on srtctl naming the Slurm job after the GitHub runner (for example `b300-dsxe_01`). Jobs named after the worker instead (for example `worker-2` on h200-dgxc) are found through the runner's `gharunnerNN` work directory in `squeue`. `squeue -n <runner>` finds the job, and `sacct` covers a job that has just finished. The logs directory is taken from the job's `StdOut` (srtctl writes `sweep_<job>.log` there), falling back to `<WorkDir>/outputs/<job>/logs`. Single-node jobs have no such Slurm job, and the script says so.
+
+## Cluster login hosts: never put them in the repo
+
+Login addresses are infra details that stay out of this repo; see `$debug-runs`. The script reads the host for each cluster (the runner-name prefix, for example `b300-dsxe`) from either:
+
+- `~/.config/infx-live-logs/clusters.json`, for example `{"b300-dsxe": {"host": "<ssh alias or user@login>"}}`
+- or `INFX_LIVE_LOGS_HOST_B300_DSXE=<ssh target>`, with the cluster name upper-cased and `-` turned into `_`.
+
+If the host is missing, the script names the cluster and exits. Get the login address from the InferenceX Clusters canvas; ask the user for the link, or ask them for the SSH target. Add it to the local config, and never commit it.
+
+## What the page shows
+
+- **Header:**
+  - SSH connection state and Slurm state (state, elapsed time, nodes).
+  - Per-node free host memory and CPU load, refreshed every 20 s. The memory bar turns red below 200 GiB free, which is useful for Mooncake / EFA memory-registration OOMs.
+- **Rows:**
+  1. Engine workers (prefill / decode / agg).
+  2. srtctl sweep, Dynamo frontend, Mooncake master, etcd.
+  3. aiperf / benchmark logs. These appear on their own when the benchmark starts, because new files are discovered every 30 s.
+- **Each pane header:**
+  - The latest phase: weights loaded → KV cache sized → EFA devices up → Mooncake segment registration → autotune → ✅ healthy.
+  - A count of real errors, excluding known noise such as NCCL `ibv_query_port_speed`, the pip resolver notice and the node-exporter TaskProlog message.
+  - Buttons: jump to start / end, ⬇ download that full log (fetched fresh from the cluster, whatever the display settings), copy, and maximize (Esc restores).
+- **Controls:**
+  - Regex filter and "errors only".
+  - "Hide Mooncake metrics noise", on by default. It hides the client metric report blocks, throughput and latency summaries, and dynamo HTTP 200 spam.
+  - `show`: last 4000 lines (the default) or the entire log. The server keeps the whole file, so switching is instant.
+  - **Download all logs:** a zip of every log file (`*.out`, `*.log`, `*.txt`) under the job's `logs/` directory, freshly `tar`-ed on the cluster. Files are always complete, whatever the `show`, filter or `--history` settings. Tachometer metrics are left out.
+  - Theme and reset layout.
+- **Resizing:**
+  - Drag the gutters between panes and rows; double-click a gutter to even out its two panes.
+  - Sizes are saved per layout shape.
+  - Scrolling up in a pane pauses follow mode (amber outline); scrolling back to the bottom resumes it.
+
+## Notes
+
+- One SSH `tail -F` runs per log file, so a huge log doesn't hold up the others. After an SSH drop, that file is re-read from the start, so nothing is lost.
+- The server keeps the full logs, but on connect it sends only the last 4000 matching lines of each; noise and "errors only" are also filtered on the server. This keeps the page responsive with logs of hundreds of MB. "Entire log" asks for confirmation first.
+- The viewer only reads logs. It never touches processes or files on shared hosts.

.agents/skills/live-logs/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Live Logs"
+  short_description: "Stream a sweep job's per-node logs to the browser"
+  default_prompt: "Use $live-logs to open the live per-node logs for this PR or workflow run."