Repository navigation
docs(metric-benchmark): sketch Helix submission and result collection - #159
Draft
mstaats-nvidia wants to merge 1 commit into
Draft
mstaats-nvidia wants to merge 1 commit into
mstaats-nvidia wants to merge 1 commit into
Conversation
asutermo
reviewed
Oct 29, 2026
|
|
||
| ## Compatibility before implementation | ||
|
|
||
| The flow above is the intended interface, not a claim that Helix bundles currently |
Collaborator
There was a problem hiding this comment.
worth matching these once the metrics-benchmark flow is ready? I don't think we necessarily want to pin (folks may want to update or use different versions of harbor)
asutermo
approved these changes
Oct 29, 2026
Signed-off-by: mstaats <mstaats@nvidia.com>
mstaats-nvidia
force-pushed
the
docs/metric-benchmark-helix
branch
from
October 30, 2026 21:38
e73789a to
8ac7273
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The metric benchmark currently runs prepared cases through a local Harbor loop. This documentation-only proposal adds an optional Helix execution workflow with two thin CLIs: submit the prepared agent/task matrix and return; later check all runs and collect a complete local results directory for analysis.
Based on #157's separate cases/execution/analysis configuration. The proposal keeps
cases.yamlunchanged and assumes prep supplies an uploaded-input mapping. It includes per-command guides and an SVG, distinguishes unfinished runs from terminal failures, and identifies the native-evidence compatibility proof required before implementation. No Python implementation, upload/prep tool, new service, or Helix jobs are included.Review order:
labs/metric-benchmark/docs/assets/helix-execution.svglabs/metric-benchmark/docs/submit-helix-evals.mdlabs/metric-benchmark/docs/download-helix-results.mddocs/architecture.mdanddocs/tools.md.Validation: local Markdown links/anchors and SVG XML checked; rendered SVG inspected; Ruff lint/format, copyright, licenses, and whitespace checks pass. Full pytest: 3,025 passed, 29 skipped (optional integrations, platform-specific checks, and live-provider/model tests); no Helix jobs were submitted or deployment access tested.
Rebased onto main after #157 merged; documentation patch unchanged.