feat(evaluator): unify dataset + task evaluation under one Evaluator by SandyChapman · Pull Request #736 · NVIDIA-NeMo/nemo-platform

SandyChapman · 2026-07-16T20:08:13Z

Draft — seeking team input on the evaluator SDK UX + naming before I polish.

What this does

Merges the two SDK entrypoints — dataset-driven Evaluator and task-driven AgentEvaluator — into a single Evaluator whose methods map 1:1 onto the backend protocol, so in-process ↔ platform and dataset ↔ agent eval differ only by which backend you inject.

Agree, one entrypoint:

Evaluator.submit(target=model|agent|runner, taskset=[...]) # agent eval
Evaluator.submit(target=model|agent|runner, taskset=generate_taskset(metrics, dataset))

Evaluator (public):

run_dataset_eval — metrics over a dataset → backend evaluate_dataset
run_taskset_eval — tasks (each carrying its own metrics) → backend evaluate_taskset
each has a _sync twin; run / run_sync remain as backward-compatible dataset dispatchers.

Backend protocol (EvaluationBackend / SyncEvaluationBackend): evaluate_dataset / evaluate_taskset.

Plugin: the executor implements the full protocol; client.evaluator.as_backend() returns it, so Evaluator(NeMoPlatform(...).evaluator) runs both dataset and taskset eval through the platform. AgentEvaluator becomes a thin deprecated shim; the agent-eval job runner is migrated onto the unified path.

Submitting a live runner

A runner built in-process previously had no wire form at all, so a Codex or Fabric runtime could only ever be evaluated locally — _agent_eval_target_to_spec rejected it outright.

jobs/runner_targets.py adds the inverse of _resolve_target: it maps a live runner onto the target spec that reproduces it job-side. Both directions are platform concerns, so both live in the plugin — the SDK runtimes know nothing about job specs, target kinds, or the wire. They expose their own configuration as ordinary read-only attributes; the plugin decides how that is spelled.

Environment-supplied arguments (work roots, sandbox providers, process factories) never travel — the job supplies its own. State with no wire representation — injected skills, a custom base_dir or codex_bin — raises UnsubmittableRunnerError rather than being dropped, so a submitted run never silently differs from the local one it claims to mirror. Resolved secret values never enter a spec; only references do.

Evaluator(client.evaluator.as_backend()).run_taskset_eval(
    taskset=[...], target=FabricContainerRuntime(config, provider=DockerSandboxProvider(), ...)
)

Fabric container runner on the job path

FabricRunnerTarget gains sandbox (FabricSandboxSpec: provider / image / secrets). Setting it swaps the host FabricAgentRuntime for FabricContainerRuntime, whose evidence contract is a superset (workspace and logs always captured). Fields the sandboxed runtime can't honor — model, non-default timeout_s, capture_trajectory=False — are rejected rather than ignored. The job compiler surfaces a sandboxed runner's harness credentials as from_secret task environment, alongside metric and endpoint secrets.

The plugin backend is remote

Both protocol methods now submit to the platform and read the completed result back from job artifacts. Agent-eval jobs post to their own agent-evaluate collection, which meant parameterizing the job-collection segment in the URL helpers and adding AgentEvaluatorJob (agent-eval payloads don't validate as BaseJob[EvaluateSpec]). Dataset results come from the job's own result JSON — the one representation covering both the single-metric and benchmark shapes.

The "use the plugin but run in-process" mode is removed rather than renamed: in-process evaluation is what Evaluator()'s default LocalBackend is for. That deletes run_local on both executors, their local spec resolvers, and sdk/fs_utils.py, whose only purpose was reading local job results off disk.

Questions for reviewers

Naming — run_dataset_eval / run_taskset_eval on Evaluator, mirroring evaluate_dataset / evaluate_taskset on the backend. Good alignment, or too verbose?
Drop-in — should Evaluator(client=NeMoPlatform(...).evaluator) work without the .as_backend() hop (i.e. make the resource itself satisfy the protocol)? Deliberately deferred — a guard test currently keeps evaluate* off the resource surface.
Sync surface — keep the _sync twins + SyncEvaluationBackend, or go async-only? (Leaning keep: notebook + job-runner callers, and run_sync is Jupyter-safe.)
Deprecation — AgentEvaluator as a warning-emitting shim → Evaluator().run_taskset_eval — path OK?
evaluate_remote — now redundant with evaluate_dataset (same submit; it differs only in being single-metric and using the dedicated aggregate/row-score routes) and has no callers left in src. Remove it, or keep for the narrower result-fetch?

Known gaps (follow-ups)

Skills don't travel. AgentSkill is a set of local directories, so a skill-bearing runner raises on submit. Carrying them needs a fileset upload — the same class of problem cloudpickle solves for metrics. A/B skill evals stay local until then.
Runner secrets inherit the metric-secret convention. The job sets env var env_name from platform secret secret_name, then LocalSecretResolver looks up names derived from secret_name — these only agree when the two match. Pre-existing across the metric path; the runner path now inherits it.
Durable task submit (fire-and-forget returning a job handle, as submit does for datasets) and result-persistence for the new runner targets.

🤖 Generated with Claude Code

github-actions · 2026-07-16T20:21:26Z

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	30397/38407	79.1%	63.7%
Integration Tests	18010/37076	48.6%	21.0%

ngoncharenko · 2026-07-16T21:10:19Z

        return namespace_result(metric_key, result, aggregate_fields)

-    async def evaluate(
+    async def evaluate_metric(


for consistency, should it be evaluate_datasets?

ngoncharenko · 2026-07-16T21:11:26Z

+        Raises:
+            ValueError: If both ``inference_fn`` and ``agent_inference_fn_factory`` are set.
+        """
+        if inference_fn is not None and agent_inference_fn_factory is not None:


also need a check that only one or the other is provided

ngoncharenko · 2026-07-16T21:13:57Z

+            tasks: Tasks to evaluate; each task carries its own metrics.
+            trials: Precomputed trials to score. Mutually exclusive with ``target``.


considering tasks and trials are mutually exclusive, we should expose this, so it's clear before invoking it

ngoncharenko · 2026-07-16T21:23:31Z

-                raise TypeError("metrics must be a Metric or a sequence of Metric objects")
-            return await self._backend.evaluate(
-                metric=metrics,
+    async def run(


should probably become run_dataset_eval since it's specific to that path

ngoncharenko · 2026-07-16T21:31:04Z

+
+            from nemo_evaluator_sdk.execution.evaluator import Evaluator
+
+            evaluator = Evaluator(client.evaluator.as_backend())


are we going to allow sdk to be an interface with platform again via injected backend? At some point in the past (~April), we reverted this change

We should discuss today if we can find time, otherwise Monday. I'm not sure why we wouldn't support this as it aligns with the early design we had to make moving from SDK to Plugin as straight-forward as possible.

ngoncharenko · 2026-07-16T21:51:42Z

Some initial thoughts:

I'm not sure whether combining both eval types under one class resolves ambiguity for a user. Two use cases are fundamentally different from one another to merge them - one is input/output eval, another one is comprehensive multi-turn agent that acts in an env, and we evaluate env state change
- ambiguity now moves from having to choose between two classes to choosing between many functions. Not sure if it will be easy to explain what to choose for someone seeing this for the first time
- combining two classes also goes against single responsibility principle from my perspective
run_dataset_metric_eval / run_dataset_benchmark_eval are a bit verbose

…luator Merges the two SDK entrypoints — dataset-driven `Evaluator` and task-driven `AgentEvaluator` — into a single `Evaluator` whose methods map 1:1 onto the backend protocol, so in-process vs platform and dataset vs agent evaluation differ only by which backend you inject. `Evaluator` (public): - `run_dataset_eval` — metrics over a dataset → backend `evaluate_dataset` - `run_taskset_eval` — tasks carrying their own metrics → backend `evaluate_taskset` - each has a `_sync` twin; `run`/`run_sync` remain as dataset dispatchers. `client.evaluator.as_backend()` returns the plugin executor, so `Evaluator(NeMoPlatform(...).evaluator.as_backend())` runs both dataset and taskset evaluation on the platform. `AgentEvaluator` becomes a deprecated shim and the agent-eval job runner moves onto the unified path. Submit live runners over the wire --------------------------------- A runner built in-process previously had no wire form, so a Codex/Fabric runtime could only be evaluated locally. `jobs/runner_targets.py` adds the inverse of `_resolve_target`: it maps a live runner onto the target spec that reproduces it job-side. Both directions are platform concerns, so both live in the plugin — the SDK runtimes know nothing about job specs, target kinds, or the wire. They expose their own configuration as ordinary read-only attributes, and the plugin decides how that is spelled. Environment-supplied arguments (work roots, sandbox providers, process factories) never travel; the job supplies its own. State with no wire representation — injected skills, a custom `base_dir` or `codex_bin` — raises `UnsubmittableRunnerError` instead of being dropped, so a submitted run never silently differs from the local one it mirrors. Resolved secret values never enter a spec; only references do. Fabric container runner reaches the job path -------------------------------------------- `FabricRunnerTarget` gains `sandbox` (`FabricSandboxSpec`: provider, image, secrets). Setting it swaps the host `FabricAgentRuntime` for `FabricContainerRuntime`, whose evidence contract is a superset. Fields the sandboxed runtime cannot honor (`model`, non-default `timeout_s`, `capture_trajectory=False`) are rejected rather than ignored. The job compiler now surfaces a sandboxed runner's harness credentials as `from_secret` task environment, alongside metric and endpoint secrets. The plugin backend is remote ---------------------------- Both protocol methods now submit to the platform and read the completed result back from job artifacts; agent-eval jobs post to their own `agent-evaluate` collection, which required parameterizing the job-collection segment in the URL helpers and adding `AgentEvaluatorJob` (agent-eval payloads do not validate as `BaseJob[EvaluateSpec]`). Dataset results are read from the job's own result JSON because it is the one representation covering both the single-metric and benchmark shapes. The "use the plugin but run in-process" mode is removed rather than renamed — in-process evaluation is what `Evaluator()`'s default `LocalBackend` is for. That deletes `run_local` on both executors, their local spec resolvers, and `sdk/fs_utils.py`, whose only purpose was reading local job results off disk. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>

github-actions · 2026-08-04T17:11:18Z

🌿 Preview your docs: https://nvidia-preview-pr-736-unify-evaluator-entrypoints-schapman.docs.buildwithfern.com/nemo-platform

ngoncharenko reviewed Jul 16, 2026

View reviewed changes

github-actions Bot added the feat label Jul 17, 2026

SandyChapman force-pushed the unify-evaluator-entrypoints/schapman branch 2 times, most recently from ef32395 to e28965b Compare August 4, 2026 16:15

SandyChapman force-pushed the unify-evaluator-entrypoints/schapman branch from e28965b to 33a1eb9 Compare August 4, 2026 17:03

Navigation Menu

Uh oh!

feat(evaluator): unify dataset + task evaluation under one Evaluator - #736

feat(evaluator): unify dataset + task evaluation under one Evaluator#736
SandyChapman wants to merge 1 commit into
mainfrom
unify-evaluator-entrypoints/schapman

SandyChapman commented Jul 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 16, 2026 •

edited

Loading

Uh oh!

ngoncharenko Jul 16, 2026

Uh oh!

ngoncharenko Jul 16, 2026

Uh oh!

ngoncharenko Jul 16, 2026

Uh oh!

ngoncharenko Jul 16, 2026

Uh oh!

ngoncharenko Jul 16, 2026

Uh oh!

SandyChapman Jul 17, 2026

Uh oh!

ngoncharenko commented Jul 16, 2026

Uh oh!

github-actions Bot commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		tasks: Tasks to evaluate; each task carries its own metrics.
		trials: Precomputed trials to score. Mutually exclusive with ``target``.


		from nemo_evaluator_sdk.execution.evaluator import Evaluator

		evaluator = Evaluator(client.evaluator.as_backend())

Uh oh!

Conversation

SandyChapman commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this does

Submitting a live runner

Fabric container runner on the job path

The plugin backend is remote

Questions for reviewers

Known gaps (follow-ups)

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ngoncharenko Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

ngoncharenko Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

ngoncharenko Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

ngoncharenko Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

ngoncharenko Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

SandyChapman Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

ngoncharenko commented Jul 16, 2026

Uh oh!

github-actions Bot commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SandyChapman commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading