[Epic] Deliver the Hermes Harness Optimization Blueprint with NeMo Platform Insights #8501

Objective

Deliver a reproducible Hermes harness-optimization blueprint that uses Relay telemetry, trace insights, and NeMo Platform optimization capabilities to demonstrate measurable improvements in token consumption, cost, and task accuracy.

The result must include public-safe evaluation data, opinionated task configurations, and enough evidence for another user to reproduce the baseline and optimized results.

Product boundary

This Epic optimizes the agent harness through prompts, tools, task configuration, and supported routing inputs. It does not modify model weights or include model post-training.

Hermes is the initial harness. Expansion to other agent runtimes requires separately accepted scope and evidence.

#7937 owns the supported Relay and Switchyard telemetry and routing capability. This Epic consumes that capability and must not reimplement its collection, credential, policy, or routing contracts.

Scope rules

• Select a public-safe task set that represents meaningful Hermes behavior.
• Define token, cost, and accuracy measurements before running the final evaluation.
• Record the exact Hermes, NemoClaw, model, provider, Relay, Switchyard, prompt, tool, and task-configuration identities.
• Measure the baseline and optimized configurations against the same evaluation boundary.
• Use Relay telemetry and trace insights to identify optimization opportunities.
• Apply an accepted NeMo Platform insight and optimization workflow to the experiment loop.
• Keep raw evaluation inputs, scoring rules, and aggregate results available in an approved public surface.
• Publish opinionated task configurations rather than only reporting experimental observations.
• Report regressions and tradeoffs. Do not present cherry-picked tasks or incompatible baselines as improvement.
• Keep credentials, private prompts, internal data, and conversational content out of published evidence.

Evaluation workflow

Relationships and readiness

Implementation of the task set, scoring method, baseline harness, and configuration format may proceed while #7937 is in progress.

Final telemetry-backed evaluation and completion cannot occur until #7937 provides the accepted Relay and Switchyard capability or records an approved replacement.

Completion criteria

• A public-safe Hermes task set and objective scoring method are published.
• Success thresholds for token consumption, cost, and accuracy are defined before final evaluation.
• A reproducible baseline records the exact software, model, provider, prompt, tool, and configuration identities.
• Relay telemetry and trace insights are captured through the supported capability from [Epic] Integrate Relay and Switchyard with Classic Hermes #7937.
• The accepted NeMo Platform insight and optimization workflow is documented.
• At least one opinionated configuration demonstrates measurable token, cost, and accuracy improvement against the baseline.
• Results include repeated or otherwise statistically defensible measurements appropriate to the task set.
• Regressions, limitations, and tradeoffs are recorded alongside improvements.
• Opinionated Hermes task configurations are published in an approved reusable surface.
• A fresh environment can reproduce the evaluation from the documented configuration and exact artifact identities.
• Published evidence contains no credentials, private data, or private conversational content.

Non-goals

• Model fine-tuning, reinforcement learning, or other model post-training.
• Reimplementing Relay telemetry or Switchyard routing owned by [Epic] Integrate Relay and Switchyard with Classic Hermes #7937.
• Building a general-purpose observability platform.
• Optimizing every Hermes workflow, model, provider, or deployment topology.
• Claiming improvement from private-only data or unreproducible evaluation.
• Expanding the blueprint to every NemoClaw agent runtime.