docs(metric-benchmark): sketch Helix submission and result collection

labs/metric-benchmark/docs/architecture.md

@@ -133,6 +133,13 @@ native output of that agent running the benchmark. The
 layout. Invocation, scheduling, and collection are internal to the execution
 implementation; they do not add generic experiment stages.
 
+The [Helix execution proposal](submit-helix-evals.md) separates submission from
+completion checking and download. It assumes the instance and uploaded inputs
+already exist, and includes a focused [flow diagram](assets/helix-execution.svg).
+These are two operations within execution, not additional experiment stages.
+The proposed importer must first resolve the
+[native-evidence compatibility requirements](download-helix-results.md#compatibility-before-implementation).
+
 ## 3. Score analysis
 
 Score analysis consumes the results directory and produces the benchmark's score.

labs/metric-benchmark/docs/download-helix-results.md

@@ -0,0 +1,110 @@
+<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
+<!-- SPDX-License-Identifier: Apache-2.0 -->
+
+# download_helix_results.py — proposal
+
+Check previously submitted Helix runs once. If the complete matrix and its
+artifacts are ready, download the evidence and publish a benchmark results
+directory. **This command is proposed, not implemented.** It does not submit or
+retry evaluations.
+
+## Proposed usage
+
+```bash
+uv run python labs/metric-benchmark/download_helix_results.py \
+  --submission helix-run/submission.json \
+  --output experiment-results
+```
+
+- `--submission`: the record produced by
+  [`submit_helix_evals.py`](submit-helix-evals.md), including the saved input mapping
+  and remote run IDs. It supplies the Helix instance and workspace.
+- `--output`: destination for the complete, locally readable results directory.
+
+Use Helix authentication with permission to read the runs and their artifacts.
+Original executable task and agent directories should not be needed for download;
+collection uses the metadata frozen in the submission record.
+
+## Completion check
+
+First require a recorded submission for every expected case/agent; if submission
+was interrupted, report that it needs to be resumed. Inspect every recorded run
+and all its task members before downloading evidence.
+Compare them with the expected baseline/mutant × task matrix, including task and
+agent identities. A top-level `completed` status or a downloadable summary alone
+does not prove that all expected evidence exists.
+
+| Condition | Proposed behavior |
+| --- | --- |
+| Runs are queued/running, or completed runs are still packaging artifacts | Print progress and “not done running” or “results still being prepared”; exit 2. Run this command again later. |
+| All expected trials are finished and their evidence is ready | Download, validate, publish the results directory; exit 0. |
+| A run is failed/cancelled/expired, or terminal output is missing required trials/evidence | Name the case, agent, and affected tasks; exit 1 without publishing a complete results directory. |
+| Authentication, API, transfer, or validation fails | Report the actionable error; exit 1. Preserve the submission record for retry. |
+
+Report known terminal failures even if other runs are still active. A finished
+Harbor trial with an agent/verifier error is still an observation: retain its
+native error and unavailable measurements. That differs from a missing trial or
+unfinished job. The initial tool does not import partial runs or silently replace
+missing observations with zero. It does not wait indefinitely or cancel other runs.
+
+## Download and output
+
+Fetch each task's native evidence as well as the combined summary. Helix's
+[bundle merger](https://github.com/NVIDIA-NeMo/nemo-helix/blob/04e38c56dcbf1044c0574405c307f7937881bac9/plugins/nemo-evals/src/nemo_evals/taskset_runs/bundle_writer.py)
+links to evidence in per-task bundles; the
+[existing run download](https://github.com/NVIDIA-NeMo/nemo-helix/blob/04e38c56dcbf1044c0574405c307f7937881bac9/plugins/nemo-evals/src/nemo_evals/runs_cli.py)
+downloads the merged bundle only. Copying that bundle alone would leave the
+benchmark dependent on remote evidence.
+
+Map trials using the saved case/task/agent identities and Helix membership records,
+not directory-name guesses. Preserve native trial records, verifier outputs, and
+agent traces, including baseline traces used by coverage. Extract every native
+reward metric and retain error/unavailable outcomes. Do not rewrite native JSON
+to make it match a local run.
+
+The target is the shared [results contract](results-format.md):
+
+```text
+experiment-results/
+  execution-inputs.json
+  execution.json
+  cases/case-0000/execution-results/
+    results.json
+    agents/agent-0000/run-0001/
+      ... intact native trial evidence from the task bundles ...
+```
+
+Storage labels are illustrative; the indexes carry identities and relative
+evidence paths. Preserve the `cases_sha256` binding introduced with the separate
+case definitions, and record Helix execution settings in `execution_sha256`.
+Remote run/job IDs remain available as execution provenance. There is no analysis
+configuration in the downloaded execution identity.
+
+Download into staging and validate before publishing `execution.json` as the
+completion marker. An interrupted transfer must not leave a directory that claims
+to be complete. Repeating collection should reuse a matching complete destination
+or finish the interrupted transfer; it must not merge different submissions.
+Scoring, coverage, and correlation then consume the saved directory as usual.
+A `download_helix_results(...)` library function would provide the same behavior
+as the thin CLI.
+
+## Compatibility before implementation
+
+The flow above is the intended interface, not a claim that Helix bundles currently
+pass our validator. The current reader targets Harbor 0.20.0; the inspected
+[Helix lockfile](https://github.com/NVIDIA-NeMo/nemo-helix/blob/04e38c56dcbf1044c0574405c307f7937881bac9/uv.lock)
+pins 0.24.0. Helix also binds prepared images into task configuration and
+resolves registered agents into runtime configurations. These transformations
+affect checksums, paths, and agent identity checks.
+
+Before implementing collection, obtain a small completed native Helix example:
+one case, a baseline, one mutant, and two tasks, including both runs' frozen plans
+and all task bundles.
+Use it to define a narrow validation of prepared inputs against effective runtime
+inputs and to test native result compatibility. Preserve original evidence and
+the existing local-run checks. Avoid requiring conflicting Harbor versions in the
+same Python environment merely to call the Helix API.
+
+Also test an active run, a terminal partial run, a lost submit response, and an
+interrupted download. These are the acceptance checks for the two-tool interface;
+no Helix jobs or deployment checks have been performed for this proposal.

labs/metric-benchmark/docs/submit-helix-evals.md

@@ -0,0 +1,104 @@
+<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
+<!-- SPDX-License-Identifier: Apache-2.0 -->
+
+# submit_helix_evals.py — proposal
+
+Submit prepared cases to an existing NeMo Helix instance, then exit while Helix
+runs the evaluations. **This is a design sketch; the command and configuration
+below are not implemented.** The [local runner](run-evals.md) remains available.
+
+![Proposed Helix execution: submit prepared inputs, let Helix run the baseline and mutants against tasks, then check and download complete evidence for analysis.](assets/helix-execution.svg)
+
+[Open the diagram](assets/helix-execution.svg).
+
+## Proposed usage
+
+```bash
+uv run python labs/metric-benchmark/submit_helix_evals.py \
+  --cases cases.yaml \
+  --execution helix-execution.yaml \
+  --bindings helix-inputs.yaml \
+  --run-id trial-1 \
+  --output helix-run
+```
+
+- `--cases`: the existing [case definitions](benchmark-format.md#cases-casesyml)
+  and their agent/task manifests. These remain the authority for matrix membership.
+- `--execution`: Helix connection and scheduling settings, illustrated below.
+- `--bindings`: a prep-generated mapping from those case, task, and agent IDs to
+  their uploaded Helix inputs. This is a proposed companion file, not a change to
+  `cases.yaml`.
+- `--run-id`: the execution identifier carried into the downloaded results.
+- `--output`: directory for `submission.json`, the local record used by the
+  [download tool](download-helix-results.md).
+
+Example execution settings, separate from cases and analysis:
+
+```yaml
+backend: helix
+base_url: https://helix.example.com
+workspace: metric-benchmark
+dispatcher_profile: evaluation-workers
+sandbox:
+  profile: evaluation-sandbox
+max_concurrent_task_jobs: 8
+```
+
+The URL and profiles are examples supplied by the Helix operator. Authentication
+uses the operator-supported Helix client configuration; credentials do not belong
+in these files. The concurrency limit applies to each submitted agent's run;
+Helix also enforces instance-wide and caller limits. It is not a total budget
+across all mutants.
+
+## Prepared inputs
+
+Prep has already published each case's tasks as a pinned Helix taskset, built its
+environment images, and made the baseline and mutants executable on that instance.
+The tool does not upload inputs, build images, provision Helix, or construct subsets.
+The instance needs the native taskset-run API, its controller, and working
+Kubernetes dispatcher/sandbox profiles, with access to the prepared images.
+
+The upload manifest connects each case's task inventory to a taskset revision and
+maps each task ID/revision to its member in that taskset. It also connects every
+baseline/mutant ID and revision to a Helix agent target. Submission checks exact
+membership against `cases.yaml`; neither extra remote tasks nor missing mutants
+are accepted. The manifest must bind the published inputs to the prepared source
+identities. Its detailed schema needs one real uploaded example before we fix it.
+
+A registered agent name alone does not establish an immutable version. Helix
+resolves registered agents at submission; the tool must check and save the resolved
+identity against the prepared mapping. Converting arbitrary local agent bundles
+into supported Helix targets remains a prep responsibility.
+
+## Submission and recovery
+
+Submit one Helix taskset run for each **case × (baseline + mutants)**. Helix
+creates parallel task jobs within those runs. Each agent/task pair gets one trial,
+with trial retries disabled; no analysis configuration or subset selection affects execution.
+
+The tool saves an idempotency key and request before submitting each run, then
+records the accepted Helix run ID. `submission.json` contains the instance and
+workspace, benchmark run ID, case and execution identities, prepared-input mapping,
+and remote run IDs. It contains enough saved metadata for later collection without
+the original machine's executable task/agent directories, but no authentication secrets.
+
+Rerunning the same command resumes submission from this record, using the same
+keys under the same Helix caller. A lost response or interruption must not create
+duplicate evaluations. Changed inputs require a new submission directory. Replaying
+a submission does not retry a terminal failed evaluation; a deliberate rerun gets
+a new run ID and directory. There is no automatic polling after submission.
+
+## Python library and implementation boundary
+
+A `submit_helix_evals(...)` library function would implement the same operation;
+the CLI would only load arguments and report its result. The paired download
+function would finish the execution stage. The benchmark needs no new service,
+database, or UI. Analysis continues to consume saved results.
+
+The target is Helix's native
+[`agent-evaluate/runs` API](https://github.com/NVIDIA-NeMo/nemo-helix/blob/04e38c56dcbf1044c0574405c307f7937881bac9/plugins/nemo-evals/src/nemo_evals/api/v2/taskset_runs.py),
+whose [request schema](https://github.com/NVIDIA-NeMo/nemo-helix/blob/04e38c56dcbf1044c0574405c307f7937881bac9/plugins/nemo-evals/src/nemo_evals/api/taskset_run_schemas.py)
+supports stored tasksets, task-job concurrency, and idempotency keys. These source
+links pin the inspected implementation; they do not establish deployment readiness.
+Before implementation, the [download compatibility check](download-helix-results.md#compatibility-before-implementation)
+must establish that its native evidence can satisfy the benchmark contract.

labs/metric-benchmark/docs/tools.md

@@ -59,3 +59,11 @@ defines the output format.
 
 See the [`run_evals.py` command guide](run-evals.md) for arguments, the output
 layout, completed-job reuse, interrupted-job recovery, and the Python API.
+
+### Proposed Helix execution
+
+The documentation-only proposal splits remote execution into
+[`submit_helix_evals.py`](submit-helix-evals.md) and
+[`download_helix_results.py`](download-helix-results.md): submit prepared cases,
+then check completion and collect a local results directory. These commands are
+not implemented; see the [Helix flow diagram](assets/helix-execution.svg).