Skip to content

Pin and package the released Hermes–Relay–Switchyard compatibility set #8885

Closed as not planned
Closed as not planned
Pin and package the released Hermes–Relay–Switchyard compatibility set#8885
Assignees
cjagwani
Labels
area: architectureArchitecture, design debt, major refactors, or maintainabilityarea: sandboxOpenShell sandbox lifecycle, runtime, config, or recoveryintegration: hermesHermes integration behavior
Milestone
Roadmap: GTC Berlin 2026

Description

@cjagwani
cjagwani
opened on Aug 12, 2026

Parent Epic: #7937

Canonical NemoClaw scope was accepted in #7937 on 2026-08-13. This work item is blocked on one released compatibility tuple.

Outcome

NemoClaw pins and packages one released, mutually compatible Hermes, NeMo Relay, Switchyard, and nvidia.switchyard plugin set for every supported Hermes sandbox architecture. Routing remains disabled by default in this work item.

Current blocker — 2026-08-30

The native architecture is converging, but no complete released tuple exists:

  • Hermes Agent 0.20.6 (v2026.8.27) contains the native in-process Relay loader from Hermes #77915, but the released version still requires nemo-relay>=0.7.1,<0.8 and locks Relay 0.7.2. Hermes #96633 is now ready for review, requires and locks nemo-relay>=0.8.1,<0.9, adapts the typed tool-result and trace-propagation boundaries, and has green required CI on rebased head c3e86f70; it is not yet approved, merged, or released.
  • Relay 0.8.1 is now the latest stable patch. Its signed annotated tag object 97cd439a peels to commit 97466cd4, and its immutable release carries digest-bearing Linux amd64 and arm64 assets. Relay chore(deps): bump actions/download-artifact from 4 to 8 #934 requires signed, trusted dynamic plugins at startup by default without changing the ABI or manifest schema. Hermes #96633 now locks Relay 0.8.1, so the source-level runtime tuple aligns; the complete released tuple remains blocked until that migration is approved, merged, and released, and the Switchyard bundle preserves Relay's default plugin-trust policy.
  • Switchyard feat: recommend optimal NIM model based on detected GPU VRAM #537 is merged. Switchyard [Ubuntu + Docker CE] unable install nemoclaw with the hardware requirement in readme #528 is the current native-plugin integration, but it remains review-required and unreleased. Head 3b3c97b7 now bypasses decoding for unmanaged requests and moves the bundle-packaging regression into the normal test tree; the refreshed CI suite is green. Remaining reviews still include arbitrary inbound metadata-header forwarding, missing final streamed token usage, target request-policy validation, context-window failure classification, and focused interceptor/packaging coverage.
  • Switchyard [Ubuntu + Docker CE] unable install nemoclaw with the hardware requirement in readme #528 has no published, digest-bearing Linux amd64 or arm64 plugin bundles, release-producer provenance, or accepted signing/trust evidence for Relay 0.8.1's secure default. Its documented Relay record must also converge with the manifest-backed activation path used by Hermes.
  • Switchyard test: add vLLM inference smoke test and E2E verification #271 closed unmerged. An accepted successor or equivalent released proof is required for redirect rejection, transport/body/SSE bounds, and diagnostic redaction.

Do not use PR overlays, moving branches, source builds, release candidates, or the frozen demo commit as production selectors.

Scope

  • Audit every crossed Hermes, Relay, and Switchyard release contract from the current NemoClaw pins.
  • Pin exact release tags, source commits, package versions, archive/image digests, installed binary digests, plugin manifest identity, and schema identity.
  • Prove that the selected Hermes lock resolves the exact selected stable Relay version and that Relay satisfies the released plugin manifest range and native API.
  • Require independently released, signed plugin bundles for Linux amd64 and arm64, each bound to its producer repository, tag commit, workflow run and attempt, successful conclusion, archive digest, installed-library digest, and accepted Relay trust evidence.
  • Install the released plugin bundle without leaving source trees, compilers, or plugin build tooling in the final image.
  • Verify license, notice, SBOM, advisory, and producer-provenance coverage for the resolved dependency closure.
  • Prove ordinary supervised Hermes behavior is unchanged when no routing configuration is supplied.
  • Remove the demo-only Hermes source overlays, forced Relay prerelease wheel, and Switchyard source-build stage only after their exact released replacements pass the migration audit.

Structural readiness record

Before changing any production selector, record one candidate tuple with:

  • stable Hermes, Relay, and Switchyard versions and immutable release URLs;
  • Hermes source/archive identity and the Relay version selected by its lock;
  • Relay source identity plus released artifacts for Linux amd64 and arm64;
  • Switchyard source identity plus nvidia.switchyard plugin ID, native API, Relay compatibility range, and disabled default;
  • released plugin archives for Linux amd64 and arm64; and
  • successful producer evidence bound to every consumed native artifact.

A structurally valid record opens the semantic migration audit. It does not prove production readiness or authorize an active selector, activation, merge, or support claim.

Acceptance

  • Maintainers accepted [Epic] Integrate Relay and Switchyard with Classic Hermes #7937 as canonical NemoClaw scope on 2026-08-13.
  • One published compatibility tuple is documented and mutually consistent.
  • Hermes selects the exact stable Relay release required by the tuple.
  • Every consumed artifact is integrity-verified and bound to its successful producer release.
  • Linux amd64 and arm64 plugin bundles are present, provenance-bound, and independently digest-verified before extraction and execution.
  • The final image contains the expected Hermes, Relay, Switchyard plugin, manifest, schema, licenses, and notices.
  • The final image contains no upstream source overlay or build-only toolchain added for the plugin.
  • Default Hermes startup and model calls remain unchanged with routing disabled.
  • Unsupported platform/version combinations fail during build or preflight with an actionable error.
  • The final candidate is re-audited against its immutable tags and published assets rather than earlier PR-head evidence.

Likely implementation surface

  • agents/hermes/Dockerfile.base
  • agents/hermes/Dockerfile
  • agents/hermes/manifest.yaml
  • scripts/update-hermes-agent.sh
  • Hermes image/layout, updater, dependency-pin, artifact-provenance, and secret-boundary tests

Activity

  1. cjagwani
    added this to the Roadmap: GTC Berlin 2026 milestone on Aug 12, 2026
  2. cjagwani
    self-assigned this
    on Aug 12, 2026
  3. cjagwani
    added
    integration: hermesHermes integration behavior
    area: sandboxOpenShell sandbox lifecycle, runtime, config, or recovery
    area: architectureArchitecture, design debt, major refactors, or maintainability
    on Aug 12, 2026
  4. cjagwani
    added a parent issue on Aug 12, 2026

  5. cjagwani commented on Aug 12, 2026

    @cjagwani
    cjagwani
    on Aug 12, 2026
    CollaboratorAuthor

    Packaging gate audit — 2026-08-12

    The released pieces are not yet a consumable native tuple:

    • NeMo Relay 0.7.2 is released and provides the dynamic-plugin host.
    • Switchyard v0.2.0 is released, but its GitHub release has no nvidia.switchyard bundle asset.
    • Switchyard feat: recommend optimal NIM model based on detected GPU VRAM #270 contains the plugin, manifest, schema, and bundle packager, but the crate is publish = false and the current publish workflow does not produce platform plugin archives.
    • Hermes v2026.8.3 does not contain the native loader; Hermes #77915 remains open.

    Therefore this work can proceed in parallel as upstream qualification, but the canonical NemoClaw packaging PR must not pin a PR head, build upstream source, or ship the prototype overlay. Its implementation entrance criteria are:

    1. official Linux amd64 and arm64 plugin archives;
    2. immutable version, digest, license/notice, manifest, and schema identities;
    3. a released Hermes build containing the native loader;
    4. one documented Hermes/Relay/plugin compatibility tuple and a released-Relay load test.

    Once those exist, this issue can add verified archive installation and clean-image rebuild coverage independently of #8896.

  6. cjagwani commented on Aug 14, 2026

    @cjagwani
    cjagwani
    on Aug 14, 2026
    CollaboratorAuthor

    Released-tuple re-audit — 2026-08-14

    The production entrance criteria remain unmet even though Relay advanced:

    • NeMo Relay 0.7.3 is released. Its Linux runtime wheels are immutable and auditable (amd64 SHA-256 8eeaaf8c6a18440e473dd14eb1bb82e56c6514e70adee7456eddf9ce217cff89; arm64 49c70c0a94cebb6cba3dd7521be11f63eac4cd9386881eb29827ac91b1bd780b).
    • Switchyard v0.2.0 still contains no nvidia.switchyard Relay-plugin bundle, manifest, schema, or native plugin library. Switchyard feat: recommend optimal NIM model based on detected GPU VRAM #270 remains open; its latest inspected CI produced no consumable artifacts.
    • The v0.2.0 release commit is not an ancestor of current Switchyard main or feat: recommend optimal NIM model based on detected GPU VRAM #270, so the eventual plugin tag needs a fresh lineage/provenance audit.
    • The latest released Hermes still lacks the native loader. Hermes #77915 remains draft and converts invalid configuration or activation failures into disabled/failed state while Hermes continues, without a stable machine-readable readiness contract.

    Conclusion: this work item remains blocked upstream for a production dependency pin. We can review installer/validator scaffolding in a draft, but canonical NemoClaw must not consume PR heads, moving branches, or source-built overlays.

    Unblock when upstream publishes:

    1. official Linux amd64 and arm64 five-file plugin archives;
    2. immutable archive/member hashes, licenses/notices, manifest/schema, and producer provenance;
    3. a released Hermes loader with an accepted activation/readiness contract; and
    4. a released Hermes/Relay/plugin compatibility load test.

    This is independent of #8896, which is merged, and independent of the dormant configuration foundation in #9012.

  7. cjagwani commented on Aug 24, 2026

    @cjagwani
    cjagwani
    on Aug 24, 2026
    CollaboratorAuthor

    2026-08-24 upstream contract checkpoint

    The loader side has advanced, but the released production tuple is still incomplete:

    The demo should remain on its qualified immutable #270 commit. Canonical packaging must wait for a stable Relay 0.8 release plus released Switchyard plugin archives, digests/provenance, supported architectures, and an exact released-tuple load test. PR heads and source overlays remain out of scope for a supported image.

  8. cjagwani commented on Aug 25, 2026

    @cjagwani
    cjagwani
    on Aug 25, 2026
    CollaboratorAuthor

    Compatibility refresh (2026-08-25):

    Therefore there is still no resolver-valid released Hermes + Relay + Switchyard tuple to pin. The next upstream gate is a reviewed Hermes Relay-0.8 dependency upgrade/release together with stable Relay 0.8 and a released digest-bearing Switchyard plugin bundle. The qualified demo should remain pinned to immutable Switchyard #270 in the meantime.

    Safe-failure dependency order: NVIDIA-NeMo/Switchyard#528 is explicitly stacked on NVIDIA-NeMo/Switchyard#537's bd1488ae head and already consumes its redaction-safe runner summary API. #537 must merge first, followed by retargeting and merging #528 (or an accepted combined merge).

  9. cjagwani commented on Aug 31, 2026

    @cjagwani
    cjagwani
    on Aug 31, 2026
    CollaboratorAuthor

    Readiness update (2026-08-31): Switchyard #528 head 10a0868d now requires Relay 0.8.1 and includes fixes for final streaming token metrics, cross-format request preservation, and MCP tool namespaces; CI is green. It remains REVIEW_REQUIRED and unreleased. Before this issue can select an artifact, review/release still need to establish the Relay/libsy provider-header filtering claim plus selected-target request-policy, context-window, interceptor, and archive-path coverage; signed provenance-bound amd64/arm64 bundles and the Hermes #96633 release are also still required.

  10. cjagwani commented on Sep 1, 2026

    @cjagwani
    cjagwani
    on Sep 1, 2026
    CollaboratorAuthor

    Readiness update (2026-08-31): NeMo Relay 0.8.2 is now the latest stable patch at verified annotated tag object 903807c8 / commit 3b60fdfc, with digest-bearing Linux amd64/arm64 assets. Hermes #96633 remains open and still locks stable 0.8.1, so the production tuple must qualify the exact Relay chosen by the eventual released Hermes lock rather than silently substituting 0.8.2. Switchyard #528 advanced to 5e30df40; its green head now reports redaction-safe in-band/decode stream failures once, but remains review-required and has no signed provenance-bound release bundles. The V5 implementation gate remains closed.

  11. cjagwani commented on Sep 1, 2026

    @cjagwani
    cjagwani
    on Sep 1, 2026
    CollaboratorAuthor

    Readiness update (2026-09-01): Hermes #96633 now requires and locks stable Relay 0.8.2 at head e4123f1a; refreshed required CI is green. This removes yesterday’s source-level 0.8.1-versus-latest-patch mismatch, but the PR remains unapproved, unmerged, and unreleased. NemoClaw PR #9012 was closed as hopelessly out of date and remains historical lifecycle evidence only; it is not the V5 implementation path. The production gate still waits on a released Hermes lock, reviewed/signed Switchyard amd64+arm64 bundles, and the remaining provider/security/E2E gates.

  12. cjagwani commented on Sep 2, 2026

    @cjagwani
    cjagwani
    on Sep 2, 2026
    CollaboratorAuthor

    Relay release delta (2026-09-02): NeMo Relay 0.8.3 is now the minimum trustworthy patch for the released Hermes/Switchyard tuple. Its verified tag object 00680fc6 peels to 843b6f60, and the release publishes digest-bearing Linux amd64/arm64 wheels and CLIs.

    Relay #960 fixes the Python release producer so the compiled native extension embeds the same version as the wheel metadata. The 0.8.1 and 0.8.2 wheels embed 0.8.0 even though their package metadata reports the patch version, so they cannot satisfy NemoClaw's exact runtime-version/provenance attestation. Hermes #96633 still locks 0.8.2 at e4123f1a; it must advance to 0.8.3 or later before merge/release. The V5 gate remains closed, and no candidate JSON should select 0.8.1 or 0.8.2.

  13. cjagwani commented on Oct 27, 2026

    @cjagwani
    cjagwani
    on Oct 27, 2026
    CollaboratorAuthor

    Superseded by #13861 under the replanned #7937 implementation. Closing this earlier planning ticket so the epic has one active implementation set.

  14. cjagwani
    closed this as not plannedon Oct 27, 2026
  15. cjagwani
    removed a parent issue on Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area: architectureArchitecture, design debt, major refactors, or maintainabilityarea: sandboxOpenShell sandbox lifecycle, runtime, config, or recoveryintegration: hermesHermes integration behavior

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions