Repository navigation
Prove adaptive Classic Hermes Switchyard routing with live E2E #13866
Copy link
Copy link
Open
Labels
area: e2eEnd-to-end tests, nightly failures, or validation infrastructureEnd-to-end tests, nightly failures, or validation infrastructurearea: routingRequest routing, policy routing, model selection, or fallback logicRequest routing, policy routing, model selection, or fallback logicarea: sandboxOpenShell sandbox lifecycle, runtime, config, or recoveryOpenShell sandbox lifecycle, runtime, config, or recoveryarea: securitySecurity controls, permissions, secrets, or hardeningSecurity controls, permissions, secrets, or hardeningintegration: hermesHermes integration behaviorHermes integration behavior
Milestone
Description
Parent epic: #7937
Outcome
Deterministic tests and a live managed E2E prove the user-visible Classic Hermes → native Relay → Switchyard adaptive route, including efficient-versus-capable selection, security failures, and restart behavior.
Implementation
- Add source tests for onboarding input validation, configuration generation, judge resolution, stale-file removal, and redaction.
- Add integration tests for the
--switchyard-configpath, Hermes adaptive-route selection, provider and judge binding, readiness, and disabled-default behavior. - Use seeded
weighted-randomonly as an early smoke test that both targets are reachable; do not count it as adaptive-routing acceptance. - Add deterministic classifier tests showing that representative easy work selects the efficient target and representative complex work selects the capable target.
- Reuse the package-contract coverage from the plugin packaging work.
- Extend the existing test(e2e): assert Hermes routing topology #8896 process-topology fixture instead of creating another discovery path.
- Run a live E2E with the real supervised Hermes process and the required OpenShell-managed providers.
- Exercise adaptive buffered and streaming turns, one tool turn, both target selections, and process restart.
- Capture canonical classifier and routing events that identify the judge role and selected target and correlate them to the Hermes turn without secrets or private reasoning.
- Add negative cases for invalid onboarding input, unresolved judge state, invalid generated configuration, unknown routes, missing providers, wrong placeholders, blocked egress, and stale state.
- Scan all captured evidence for credential leakage.
Acceptance criteria
- One supervised Hermes gateway serves the adaptive flow with zero Relay or Switchyard sidecars.
- The public E2E starts through
nemohermes onboard --switchyard-config <file>. - Deterministic cases route representative easy work to the efficient target and representative complex work to the capable target.
- Both real targets return distinguishable responses and appear in canonical routing events.
- The judge role and selected target are observable without exposing prompts, credentials, or private reasoning.
- Normal streaming, a tool turn, and process restart pass.
- The same adaptive route is active after restart.
- Every negative case fails at the intended boundary with bounded, non-secret diagnostics.
- Captured artifacts contain no raw credential value.
Metadata
Metadata
Assignees
Labels
area: e2eEnd-to-end tests, nightly failures, or validation infrastructureEnd-to-end tests, nightly failures, or validation infrastructurearea: routingRequest routing, policy routing, model selection, or fallback logicRequest routing, policy routing, model selection, or fallback logicarea: sandboxOpenShell sandbox lifecycle, runtime, config, or recoveryOpenShell sandbox lifecycle, runtime, config, or recoveryarea: securitySecurity controls, permissions, secrets, or hardeningSecurity controls, permissions, secrets, or hardeningintegration: hermesHermes integration behaviorHermes integration behavior
Activity
[-]Add Classic Hermes Relay and Switchyard integration and live E2E coverage[/-][+]Prove adaptive Classic Hermes Switchyard routing with live E2E[/+]