[Epic] Establish the Post-Training Feedback Loop Blueprint #8568

Objective

Establish a reproducible NemoClaw blueprint that turns agent telemetry and evaluation results into a model-customization feedback loop, validates the resulting model against the same evaluation boundary, and returns an approved model artifact to a supported NemoClaw execution path.

The reference workflow must preserve traceability across the source evidence, dataset, base model, customization configuration, candidate model, evaluation results, and promotion decision.

Product boundary

This Epic owns the NemoClaw-facing integration contract and end-to-end reference workflow. The underlying model-training or customization engine remains an external dependency and is not reimplemented in NemoClaw.

This work changes model behavior through post-training. Prompt, tool, routing, and harness-configuration optimization remain separate.

Required decisions

• Select one supported post-training method and integration path for the reference blueprint.
• Define which telemetry and evaluation outputs may become training inputs.
• Define privacy, PII, consent, retention, and data-filtering requirements.
• Resolve the supported service versus on-premises topology.
• Define evaluation thresholds, regression policy, model promotion, and rollback.
• Define the responsibility boundary between NemoClaw and the external customization system.

Reference workflow

1. Establish a reproducible baseline using a fixed task set and model.
2. Collect eligible agent telemetry and evaluation results.
3. Curate and version the accepted customization dataset.
4. Run one supported model-customization workflow.
5. Evaluate the candidate model against the original baseline and regression boundary.
6. Promote or reject the candidate using documented criteria.
7. Reproduce the complete workflow from recorded artifacts and configuration.

Completion criteria

• One supported workflow connects NemoClaw telemetry and evaluations to a completed model-customization run.
• Input, output, and integration contracts are documented.
• Sensitive information is excluded or handled according to documented requirements.
• Data, model, configuration, and evaluation artifacts are versioned and traceable.
• The baseline and candidate model are evaluated against the same task and scoring boundary.
• Improvement thresholds and unacceptable regressions are defined and evaluated.
• The service versus on-premises topology is explicitly resolved and documented.
• An accepted candidate can be used through a supported NemoClaw execution path.
• Promotion and rollback procedures are demonstrated.
• A fresh environment can reproduce the workflow from the published instructions and exact artifact identities.

Non-goals

• Building or replacing a general-purpose model-training platform.
• Including prompt, tool, routing, or harness-configuration optimization.
• Supporting every post-training method, model, provider, or deployment topology.
• Training directly on unfiltered private conversations or telemetry.
• Automatically promoting a candidate model without evaluation and rollback evidence.