Skip to content

Prove adaptive Classic Hermes Switchyard routing with live E2E #13866

Open

Description

@cjagwani

Parent epic: #7937

Outcome

Deterministic tests and a live managed E2E prove the user-visible Classic Hermes → native Relay → Switchyard adaptive route, including efficient-versus-capable selection, security failures, and restart behavior.

Implementation

  • Add source tests for onboarding input validation, configuration generation, judge resolution, stale-file removal, and redaction.
  • Add integration tests for the --switchyard-config path, Hermes adaptive-route selection, provider and judge binding, readiness, and disabled-default behavior.
  • Use seeded weighted-random only as an early smoke test that both targets are reachable; do not count it as adaptive-routing acceptance.
  • Add deterministic classifier tests showing that representative easy work selects the efficient target and representative complex work selects the capable target.
  • Reuse the package-contract coverage from the plugin packaging work.
  • Extend the existing test(e2e): assert Hermes routing topology #8896 process-topology fixture instead of creating another discovery path.
  • Run a live E2E with the real supervised Hermes process and the required OpenShell-managed providers.
  • Exercise adaptive buffered and streaming turns, one tool turn, both target selections, and process restart.
  • Capture canonical classifier and routing events that identify the judge role and selected target and correlate them to the Hermes turn without secrets or private reasoning.
  • Add negative cases for invalid onboarding input, unresolved judge state, invalid generated configuration, unknown routes, missing providers, wrong placeholders, blocked egress, and stale state.
  • Scan all captured evidence for credential leakage.

Acceptance criteria

  • One supervised Hermes gateway serves the adaptive flow with zero Relay or Switchyard sidecars.
  • The public E2E starts through nemohermes onboard --switchyard-config <file>.
  • Deterministic cases route representative easy work to the efficient target and representative complex work to the capable target.
  • Both real targets return distinguishable responses and appear in canonical routing events.
  • The judge role and selected target are observable without exposing prompts, credentials, or private reasoning.
  • Normal streaming, a tool turn, and process restart pass.
  • The same adaptive route is active after restart.
  • Every negative case fails at the intended boundary with bounded, non-secret diagnostics.
  • Captured artifacts contain no raw credential value.

Activity

  1. added this to the Roadmap: Q4 2026 milestone on Oct 27, 2026
  2. self-assigned this
    on Oct 27, 2026
  3. added
    area: e2eEnd-to-end tests, nightly failures, or validation infrastructure
    area: routingRequest routing, policy routing, model selection, or fallback logic
    area: sandboxOpenShell sandbox lifecycle, runtime, config, or recovery
    area: securitySecurity controls, permissions, secrets, or hardening
    on Oct 27, 2026
  4. changed the title [-]Add Classic Hermes Relay and Switchyard integration and live E2E coverage[/-] [+]Prove adaptive Classic Hermes Switchyard routing with live E2E[/+] on Oct 27, 2026
  5. added theissue type on Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area: e2eEnd-to-end tests, nightly failures, or validation infrastructurearea: routingRequest routing, policy routing, model selection, or fallback logicarea: sandboxOpenShell sandbox lifecycle, runtime, config, or recoveryarea: securitySecurity controls, permissions, secrets, or hardeningintegration: hermesHermes integration behavior

Type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions