From 8ac727384dab52826c5c62a9851d3781843895d4 Mon Sep 17 00:00:00 2001 From: mstaats Date: Thu, 29 Oct 2026 15:25:05 +0000 Subject: [PATCH] docs(metric-benchmark): sketch Helix submission and result collection Signed-off-by: mstaats --- labs/metric-benchmark/docs/architecture.md | 7 ++ .../docs/assets/helix-execution.svg | 81 +++++++++++++ .../docs/download-helix-results.md | 110 ++++++++++++++++++ .../docs/submit-helix-evals.md | 104 +++++++++++++++++ labs/metric-benchmark/docs/tools.md | 8 ++ 5 files changed, 310 insertions(+) create mode 100644 labs/metric-benchmark/docs/assets/helix-execution.svg create mode 100644 labs/metric-benchmark/docs/download-helix-results.md create mode 100644 labs/metric-benchmark/docs/submit-helix-evals.md diff --git a/labs/metric-benchmark/docs/architecture.md b/labs/metric-benchmark/docs/architecture.md index 6db7e13f..24741f1e 100644 --- a/labs/metric-benchmark/docs/architecture.md +++ b/labs/metric-benchmark/docs/architecture.md @@ -133,6 +133,13 @@ native output of that agent running the benchmark. The layout. Invocation, scheduling, and collection are internal to the execution implementation; they do not add generic experiment stages. +The [Helix execution proposal](submit-helix-evals.md) separates submission from +completion checking and download. It assumes the instance and uploaded inputs +already exist, and includes a focused [flow diagram](assets/helix-execution.svg). +These are two operations within execution, not additional experiment stages. +The proposed importer must first resolve the +[native-evidence compatibility requirements](download-helix-results.md#compatibility-before-implementation). + ## 3. Score analysis Score analysis consumes the results directory and produces the benchmark's score. diff --git a/labs/metric-benchmark/docs/assets/helix-execution.svg b/labs/metric-benchmark/docs/assets/helix-execution.svg new file mode 100644 index 00000000..71e4bc79 --- /dev/null +++ b/labs/metric-benchmark/docs/assets/helix-execution.svg @@ -0,0 +1,81 @@ + + + + + Metric benchmark execution on Helix — proposal + External prep supplies cases and published inputs. A local submit tool saves run IDs and starts Helix runs for each baseline and mutant. A separate download tool checks every task. If unfinished or failed it reports status; if complete it downloads native evidence and publishes the shared results directory for analysis. + + + + + + Run the metric benchmark on Helix + Proposed interface · two tools within the execution stage + + + EXTERNAL PREP + Prepared cases + published inputs + cases.yaml · uploaded baseline and mutants · pinned tasksets and images · input mapping + + + + LOCAL TOOL 1 + Submit on Helix + One remote run per case × agent + submit_helix_evals.py + + + submit + + EXISTING HELIX INSTANCE + (Baseline + mutants) × eval tasks + Helix schedules parallel task jobs + The local submit command has exited + + + + Saved submission record + submission.json + Run IDs · input identities · task/agent mapping + + + status + artifacts + + + LOCAL TOOL 2 · INVOKE LATER + Check & download + Check every run, task, and artifact + Is the complete matrix ready? + download_helix_results.py + + + Yes: download native evidence and validate + + No + + + Publish the results directory + execution.json · results.json + All native trials and baseline traces + + + Report status and exit + Still running / packaging: try again later + Terminal failure: identify what failed + + + + Existing score analysis + Mutant scores · coverage · correlation + No complete results directory is published + until evidence has been collected and validated. + diff --git a/labs/metric-benchmark/docs/download-helix-results.md b/labs/metric-benchmark/docs/download-helix-results.md new file mode 100644 index 00000000..934ca951 --- /dev/null +++ b/labs/metric-benchmark/docs/download-helix-results.md @@ -0,0 +1,110 @@ + + + +# download_helix_results.py — proposal + +Check previously submitted Helix runs once. If the complete matrix and its +artifacts are ready, download the evidence and publish a benchmark results +directory. **This command is proposed, not implemented.** It does not submit or +retry evaluations. + +## Proposed usage + +```bash +uv run python labs/metric-benchmark/download_helix_results.py \ + --submission helix-run/submission.json \ + --output experiment-results +``` + +- `--submission`: the record produced by + [`submit_helix_evals.py`](submit-helix-evals.md), including the saved input mapping + and remote run IDs. It supplies the Helix instance and workspace. +- `--output`: destination for the complete, locally readable results directory. + +Use Helix authentication with permission to read the runs and their artifacts. +Original executable task and agent directories should not be needed for download; +collection uses the metadata frozen in the submission record. + +## Completion check + +First require a recorded submission for every expected case/agent; if submission +was interrupted, report that it needs to be resumed. Inspect every recorded run +and all its task members before downloading evidence. +Compare them with the expected baseline/mutant × task matrix, including task and +agent identities. A top-level `completed` status or a downloadable summary alone +does not prove that all expected evidence exists. + +| Condition | Proposed behavior | +| --- | --- | +| Runs are queued/running, or completed runs are still packaging artifacts | Print progress and “not done running” or “results still being prepared”; exit 2. Run this command again later. | +| All expected trials are finished and their evidence is ready | Download, validate, publish the results directory; exit 0. | +| A run is failed/cancelled/expired, or terminal output is missing required trials/evidence | Name the case, agent, and affected tasks; exit 1 without publishing a complete results directory. | +| Authentication, API, transfer, or validation fails | Report the actionable error; exit 1. Preserve the submission record for retry. | + +Report known terminal failures even if other runs are still active. A finished +Harbor trial with an agent/verifier error is still an observation: retain its +native error and unavailable measurements. That differs from a missing trial or +unfinished job. The initial tool does not import partial runs or silently replace +missing observations with zero. It does not wait indefinitely or cancel other runs. + +## Download and output + +Fetch each task's native evidence as well as the combined summary. Helix's +[bundle merger](https://github.com/NVIDIA-NeMo/nemo-helix/blob/04e38c56dcbf1044c0574405c307f7937881bac9/plugins/nemo-evals/src/nemo_evals/taskset_runs/bundle_writer.py) +links to evidence in per-task bundles; the +[existing run download](https://github.com/NVIDIA-NeMo/nemo-helix/blob/04e38c56dcbf1044c0574405c307f7937881bac9/plugins/nemo-evals/src/nemo_evals/runs_cli.py) +downloads the merged bundle only. Copying that bundle alone would leave the +benchmark dependent on remote evidence. + +Map trials using the saved case/task/agent identities and Helix membership records, +not directory-name guesses. Preserve native trial records, verifier outputs, and +agent traces, including baseline traces used by coverage. Extract every native +reward metric and retain error/unavailable outcomes. Do not rewrite native JSON +to make it match a local run. + +The target is the shared [results contract](results-format.md): + +```text +experiment-results/ + execution-inputs.json + execution.json + cases/case-0000/execution-results/ + results.json + agents/agent-0000/run-0001/ + ... intact native trial evidence from the task bundles ... +``` + +Storage labels are illustrative; the indexes carry identities and relative +evidence paths. Preserve the `cases_sha256` binding introduced with the separate +case definitions, and record Helix execution settings in `execution_sha256`. +Remote run/job IDs remain available as execution provenance. There is no analysis +configuration in the downloaded execution identity. + +Download into staging and validate before publishing `execution.json` as the +completion marker. An interrupted transfer must not leave a directory that claims +to be complete. Repeating collection should reuse a matching complete destination +or finish the interrupted transfer; it must not merge different submissions. +Scoring, coverage, and correlation then consume the saved directory as usual. +A `download_helix_results(...)` library function would provide the same behavior +as the thin CLI. + +## Compatibility before implementation + +The flow above is the intended interface, not a claim that Helix bundles currently +pass our validator. The current reader targets Harbor 0.20.0; the inspected +[Helix lockfile](https://github.com/NVIDIA-NeMo/nemo-helix/blob/04e38c56dcbf1044c0574405c307f7937881bac9/uv.lock) +pins 0.24.0. Helix also binds prepared images into task configuration and +resolves registered agents into runtime configurations. These transformations +affect checksums, paths, and agent identity checks. + +Before implementing collection, obtain a small completed native Helix example: +one case, a baseline, one mutant, and two tasks, including both runs' frozen plans +and all task bundles. +Use it to define a narrow validation of prepared inputs against effective runtime +inputs and to test native result compatibility. Preserve original evidence and +the existing local-run checks. Avoid requiring conflicting Harbor versions in the +same Python environment merely to call the Helix API. + +Also test an active run, a terminal partial run, a lost submit response, and an +interrupted download. These are the acceptance checks for the two-tool interface; +no Helix jobs or deployment checks have been performed for this proposal. diff --git a/labs/metric-benchmark/docs/submit-helix-evals.md b/labs/metric-benchmark/docs/submit-helix-evals.md new file mode 100644 index 00000000..08f589ef --- /dev/null +++ b/labs/metric-benchmark/docs/submit-helix-evals.md @@ -0,0 +1,104 @@ + + + +# submit_helix_evals.py — proposal + +Submit prepared cases to an existing NeMo Helix instance, then exit while Helix +runs the evaluations. **This is a design sketch; the command and configuration +below are not implemented.** The [local runner](run-evals.md) remains available. + +![Proposed Helix execution: submit prepared inputs, let Helix run the baseline and mutants against tasks, then check and download complete evidence for analysis.](assets/helix-execution.svg) + +[Open the diagram](assets/helix-execution.svg). + +## Proposed usage + +```bash +uv run python labs/metric-benchmark/submit_helix_evals.py \ + --cases cases.yaml \ + --execution helix-execution.yaml \ + --bindings helix-inputs.yaml \ + --run-id trial-1 \ + --output helix-run +``` + +- `--cases`: the existing [case definitions](benchmark-format.md#cases-casesyml) + and their agent/task manifests. These remain the authority for matrix membership. +- `--execution`: Helix connection and scheduling settings, illustrated below. +- `--bindings`: a prep-generated mapping from those case, task, and agent IDs to + their uploaded Helix inputs. This is a proposed companion file, not a change to + `cases.yaml`. +- `--run-id`: the execution identifier carried into the downloaded results. +- `--output`: directory for `submission.json`, the local record used by the + [download tool](download-helix-results.md). + +Example execution settings, separate from cases and analysis: + +```yaml +backend: helix +base_url: https://helix.example.com +workspace: metric-benchmark +dispatcher_profile: evaluation-workers +sandbox: + profile: evaluation-sandbox +max_concurrent_task_jobs: 8 +``` + +The URL and profiles are examples supplied by the Helix operator. Authentication +uses the operator-supported Helix client configuration; credentials do not belong +in these files. The concurrency limit applies to each submitted agent's run; +Helix also enforces instance-wide and caller limits. It is not a total budget +across all mutants. + +## Prepared inputs + +Prep has already published each case's tasks as a pinned Helix taskset, built its +environment images, and made the baseline and mutants executable on that instance. +The tool does not upload inputs, build images, provision Helix, or construct subsets. +The instance needs the native taskset-run API, its controller, and working +Kubernetes dispatcher/sandbox profiles, with access to the prepared images. + +The upload manifest connects each case's task inventory to a taskset revision and +maps each task ID/revision to its member in that taskset. It also connects every +baseline/mutant ID and revision to a Helix agent target. Submission checks exact +membership against `cases.yaml`; neither extra remote tasks nor missing mutants +are accepted. The manifest must bind the published inputs to the prepared source +identities. Its detailed schema needs one real uploaded example before we fix it. + +A registered agent name alone does not establish an immutable version. Helix +resolves registered agents at submission; the tool must check and save the resolved +identity against the prepared mapping. Converting arbitrary local agent bundles +into supported Helix targets remains a prep responsibility. + +## Submission and recovery + +Submit one Helix taskset run for each **case × (baseline + mutants)**. Helix +creates parallel task jobs within those runs. Each agent/task pair gets one trial, +with trial retries disabled; no analysis configuration or subset selection affects execution. + +The tool saves an idempotency key and request before submitting each run, then +records the accepted Helix run ID. `submission.json` contains the instance and +workspace, benchmark run ID, case and execution identities, prepared-input mapping, +and remote run IDs. It contains enough saved metadata for later collection without +the original machine's executable task/agent directories, but no authentication secrets. + +Rerunning the same command resumes submission from this record, using the same +keys under the same Helix caller. A lost response or interruption must not create +duplicate evaluations. Changed inputs require a new submission directory. Replaying +a submission does not retry a terminal failed evaluation; a deliberate rerun gets +a new run ID and directory. There is no automatic polling after submission. + +## Python library and implementation boundary + +A `submit_helix_evals(...)` library function would implement the same operation; +the CLI would only load arguments and report its result. The paired download +function would finish the execution stage. The benchmark needs no new service, +database, or UI. Analysis continues to consume saved results. + +The target is Helix's native +[`agent-evaluate/runs` API](https://github.com/NVIDIA-NeMo/nemo-helix/blob/04e38c56dcbf1044c0574405c307f7937881bac9/plugins/nemo-evals/src/nemo_evals/api/v2/taskset_runs.py), +whose [request schema](https://github.com/NVIDIA-NeMo/nemo-helix/blob/04e38c56dcbf1044c0574405c307f7937881bac9/plugins/nemo-evals/src/nemo_evals/api/taskset_run_schemas.py) +supports stored tasksets, task-job concurrency, and idempotency keys. These source +links pin the inspected implementation; they do not establish deployment readiness. +Before implementation, the [download compatibility check](download-helix-results.md#compatibility-before-implementation) +must establish that its native evidence can satisfy the benchmark contract. diff --git a/labs/metric-benchmark/docs/tools.md b/labs/metric-benchmark/docs/tools.md index c3cf68d4..8459c8a2 100644 --- a/labs/metric-benchmark/docs/tools.md +++ b/labs/metric-benchmark/docs/tools.md @@ -59,3 +59,11 @@ defines the output format. See the [`run_evals.py` command guide](run-evals.md) for arguments, the output layout, completed-job reuse, interrupted-job recovery, and the Python API. + +### Proposed Helix execution + +The documentation-only proposal splits remote execution into +[`submit_helix_evals.py`](submit-helix-evals.md) and +[`download_helix_results.py`](download-helix-results.md): submit prepared cases, +then check completion and collect a local results directory. These commands are +not implemented; see the [Helix flow diagram](assets/helix-execution.svg).