Make the metrics suite complete and usable from one object

[mkeeler43]

What this is

After #75 the evaluation layer on main registered eight metrics. There was no way to score several models at once without assembling an EvaluationContext by hand. The full suite lived on feat/dataset-sensitivity-fields as about forty separately maintained functions with prefixed names such as pca_mmd, lv_vendi and iog_median. This PR brings the suite to main on a contract that fits in one sentence, and adds the one object you use it through.

Nothing here touches the LVAE, the baselines, or the pool tooling. Those are separate PRs.

How you use it

from engiopt.evaluation import Board, METRICS

METRICS.explain()                                # every metric: the question it answers, direction, cost

board = Board(problem, reference=REF, conditions=CONDITIONS, train=TRAIN)   # saved designs
board.evaluate(models, space="pca", aggregation="median")
board.explain()                                  # what each column asks, which way is better, who it picks
board.rank("fog")

Board.load("beams2d", {"cgan_cnn_2d": "825831f6", "vqgan": None}, expensive=True)   # published checkpoints

Two of the four beams2d families were never trained at the script's default configuration. They load by the fingerprint of the configuration that was trained. Diffusion and VQGAN load by name.

example_metrics_suite.ipynb at the repo root walks all of this on beams2d. It loads four published checkpoints, builds three constructions from the dataset, and scores all seven under one spec with the physics columns included. It uses twelve conditions rather than the published fifty and medians rather than means. Both are declared in the notebook, and it runs in about seven minutes on a laptop.

The contract

A metric is a function of an EvaluationContext plus four declarations: name, family, cost and direction. The few metrics that need the actual designs rather than codes in some space say so with pixel_only. The first line of the docstring is the sentence explain() and --list-metrics print. higher_is_better=None marks a diagnostic, read beside the other columns and never ranked on.

Two things that used to be baked into metric names are arguments to Board.evaluate.

• space, where the question is asked. pixel and pca are built in. A learned latent space registers one projection function with register_space. This replaces the pca_*, lv_*, lvoff_* and pixel_* prefixes.
• aggregation, how one-value-per-design metrics collapse to a number. Mean or median, declared on the spec and recorded in every row. This replaces the *_median columns.

Metrics call ctx.reduce(values) and never hard-code a mean.

The metrics

Nineteen, down from about forty. Each name says what is measured.

Family	Metrics
distribution	mmd, coverage
diversity	vendi, dpp
conditions	cond_sens, per_condition_distance, volume_error
memorization	train_distance, train_distance_ratio (one metric, two outputs)
feasibility	viol
performance	iog, cog, fog, calls_to_near_optimum, gap_after_calls, reaches_reference_rate
cost	generation_seconds, n_parameters, train_minutes

Four definitions changed on the way. Each change has a reason a reviewer should know.

Memorization is a distance, not a rate. The old copy_rate counted designs within a per-element tolerance of a corpus that included the reference optima. No model can see the reference optima, and the tolerance was an arbitrary number in design units. train_distance reports the per-element distance from each design to the nearest design in the full training split. train_distance_ratio divides that by what the withheld reference designs score against the same split, so one reads as data-like and zero as retrieval. There is no tolerance anywhere and no automatic flag. The columns are read by people. The spec drops copy_tol, max_copy_rate and copy_corpus_size. The last of those was a 512-design subsample that had let a lookup table through half the time.

Settling is measured against the optimum, not the guess. calls_to_near_optimum counts optimizer calls until a design's gap stays within five percent of the reference optimum's objective for its conditions. A design that never gets there counts one more than the budget. An earlier form set the band from the design's own starting gap, which read a start of 1e9 as settled after one call. That form is gone, and so is first_call_gain, which had the same flaw. calls_to_parity is gone too, because a copier satisfies it in zero calls.

The kernel bandwidth comes from the data. mmd, vendi and dpp default to the median pairwise distance of the training designs. The value is resolved per evaluation and recorded in every row as kernel_sigma. The v2 specs pin no value. The old pinned 10.0 put every beams2d model near ten effective designs. The data-sized kernel separates the collapsed GAN at 1.03 from the rest at five to seven.

dpp is readable. It keeps its name and becomes the n-th root of the kernel determinant, bounded in (0, 1]. The raw determinant that v1 rows hold reads around 1e-20 on every real board and cannot be compared with this column. vendi is the diversity column to prefer, because a determinant still rises when a collapsed set is jittered.

The board from saved designs also scores one random half of the reference designs against the other half. That row is labeled reference (split-half). It is a scale reference at half the sample size, not a competitor. It shows where real designs land on each column, it is never ranked, and explain() reports it under split-half reference.

Specs

All four problems get a v2.json with the full metric list and the aggregation policy. v2 is the default everywhere a spec is resolved. v1.json stays committed so rows published under it can be read. Evaluating under v1 raises Unknown metric 'novelty' at metric selection, because that metric no longer exists. The leaderboard dataset holds no v1 rows.

Also in here

• OptimizationResults.trajectories and reference_objectives, one gap path and one reference objective per design. EvaluationContext.reduce, kernel_sigma, model_params, train_minutes and train_designs.
• Evaluator.context_for_designs, so constructions built from the dataset score under the same spec as loaded models. Board.from_evaluator scores both and keeps the sampled designs and the evaluator's metric registry.
• register_metric silently registered into the global registry when handed an empty custom one, because MetricRegistry is a Mapping and an empty mapping is falsy. Fixed.
• "brief" is replaced by "conditions" throughout, including three docstrings from Generator contract, shared evaluation layer, and HF-hosted leaderboard #75.
• LEADERBOARD.md, README.md and CONTRIBUTING_A_MODEL.md describe the suite that exists now.

Review

Soheyl's six inline comments on 59c198d were each reproduced from the code and fixed in cc5f16e. Replies are inline, each naming its test.

• One default spec across Evaluator.for_problem, the CLI and EvalSpec.load.
• Board takes the designs' conditions, and viol reports blank rather than raising without them.
• rank excludes the split-half row.
• The split-half row is documented as a scale reference.
• A board built from an evaluator keeps that evaluator's registry.
• The parameter count is no longer memoized per generator.

What was checked

• CI on the head commit: pytest 309 passed, 0 failed, 0 skipped. ruff and mypy pass. Read from the job log.
• Locally: 300 non-network tests pass, none skipped, on the fake problem. python -m engiopt.evaluate --list-metrics runs as documented.
• The notebook executes end to end on beams2d with no errors, physics included, in about seven minutes. It reproduces three results from the full boards. The lookup table beats every trained model but diffusion on final gap. The GAN initial gaps sit near 1e9 and 1e10. The permuted reference set scores an MMD of exactly zero with the worst final gap.
• Every pre-commit hook, pyright included, passed on each commit.

What was not checked

Nothing has been run at the published fifty conditions. The notebook's twelve is the only real-problem run. Board.load by bare name has been exercised only for the two families that have a canonical checkpoint. The trajectory metrics assume a gap path that falls. For the diffusion model it rises after the first call while the optimizer restores feasibility. The columns report that faithfully but do not explain it.

Follow-ups

• Uncertainty: bootstrap intervals for the physics columns from the per-design values the evaluator already computes, and disjoint folds over the full test split for the cheap metrics. Planned, not in this PR.
• Baselines: knn_retrieval, deconv_regression and the planted constructions, as a separate PR.
• space="lv", registered by the LVAE PR once the Lipschitz fixes land.

🤖 Generated with Claude Code

[SoheylM]

Thanks, Matthew. I rechecked the current head. The default-spec mismatch and the saved-design path look like issues to address before merging; I left the smaller API and comparison points inline as well.

[SoheylM]

Could we align the v2 default across the public entry points? EvalSpec.load("beams2d") selects v2 here, but Evaluator.for_problem() and the CLI still select v1 when no spec is given. That v1 spec includes the removed novelty metric, so the default Board.load() and CLI paths fail when they score. EvalSpec(problem_id=...) also defaults to v2 while retaining novelty in its metric list.

[mkeeler43]

Fixed in cc5f16e. Evaluator.for_problem, the CLI and EvalSpec.load now all resolve to v2 when given no spec, and the dataclass default metric list names only registered metrics. Test: test_the_default_spec_metrics_are_all_registered.

[SoheylM]

Thanks, the EvalSpec and Evaluator.for_problem defaults are fixed. The CLI still has spec = args.spec or f"{args.problem_id}/v1" in engiopt/evaluate.py:344, so running it without --spec still selects v1. I confirmed that selecting the metrics in beams2d/v1 raises KeyError: Unknown metric 'novelty'. Could the CLI use the same v2 default, with a regression test for its no---spec path?

[mkeeler43]

Reproduced first: the v1 spec still lists novelty, which the registry no longer serves, so the CLI's no-flag path raised the KeyError you saw. Fixed in 481fd39. The CLI no longer builds its own reference. --spec passes straight through and EvalSpec.load owns the only default, so the CLI cannot drift from the library again. tests/test_evaluate_default_spec.py pins both halves: main without --spec must hand spec=None to Evaluator.for_problem, and a bare problem reference must load the dataclass default version. The two docstrings that still said v1 are swept too.

[SoheylM]

Could this path take the conditions that belong to the saved designs, or skip metrics that require them? On beams2d, the default viol check calls check_constraints without the required volfrac and raises; volume_error has no target and returns NaN. As written, the documented Board(problem, reference=...).evaluate(...) path does not work with its default metrics on that problem.

[mkeeler43]

Fixed in cc5f16e. Board takes conditions= and passes them into the context for the model rows, so the documented path works on beams2d when the conditions are given. Without them, viol now returns blank instead of raising, and its docstring says so; volume_error already returned NaN. Test: test_viol_is_blank_without_the_designs_conditions.

[SoheylM]

Thanks, this prevents the viol crash. volume_error still cannot produce a value on the saved-design path: Board has no volume_condition field, and its new EvaluationContext(...) does not set one. I reproduced Board(..., conditions={"volfrac": ...}).evaluate(..., metrics=["volume_error"]) returning NaN even though the targets were supplied. Could Board pass an explicit volume-condition name, and carry it over from evaluator.spec in from_evaluator?

[mkeeler43]

Reproduced: your call returned NaN even with targets supplied, because the board never told its contexts which condition is the budget. Fixed in 481fd39. Board gains an explicit volume_condition field, declared rather than guessed, the same contract as EvalSpec, and passes it into every context it builds; from_evaluator carries it over from evaluator.spec, so re-scoring kept designs in another space still sees the budget. On the test problem, designs at their own budgets now score 0.0 and designs shifted by 0.05 score 0.048. test_volume_error_reaches_saved_designs pins both directions: a board told the budget scores it, and a board not told leaves the column blank rather than guessing a column name.

[SoheylM]

Could rank() exclude REFERENCE_ROW, as explain() already does? Right now the split-half comparison is sorted alongside the models and can appear as the winner, although it is not a model.

[mkeeler43]

Fixed in cc5f16e. rank drops the split-half row the same way explain does. Covered by test_the_reference_row_is_measured_and_never_picked.

[SoheylM]

Is this meant as a diagnostic baseline rather than a directly comparable score? Model rows use the full generated batch against the full reference set, while this row uses half against half. MMD and coverage depend on sample count, so I would avoid calling its values "what real designs score" without making that difference clear.

[mkeeler43]

Agreed. It is a scale reference at half the sample size, not a comparable row. The explain column is now named split-half reference, and the row's docstring and the notebook say it shows the order of magnitude a good model should reach, is not a number to beat, and is never ranked.

[SoheylM]

Could the returned board keep evaluator.registry? The evaluator can score a custom metric, but this Board defaults to the global registry. Its custom column is then omitted by explain(), and rank() cannot recognize it.

[mkeeler43]

Fixed in cc5f16e. from_evaluator builds the board with evaluator.registry, so a custom metric the evaluator scored is readable in explain and rankable. Test: test_from_evaluator_keeps_the_evaluators_registry.

[SoheylM]

Could we count parameters without caching by generator object? functools.cache keeps every scored generator, including its network, alive for the lifetime of the process. A sweep over many checkpoints could retain a lot of CPU or GPU memory for a calculation that is cheap to repeat.

[mkeeler43]

Fixed in cc5f16e. The functools.cache decorator is gone; the parameter count is recomputed per score and holds no reference to the generator.

[mkeeler43]

Thanks, Soheyl. All six are addressed in cc5f16e, each reproduced from the code first. CI on that commit: 309 passed, ruff and mypy clean. The notebook is being re-run with the renamed explain column and lands in the next push.

[SoheylM]

When train and sigma are both omitted, kernel_sigma is inferred separately from each context's ref_designs. Model rows use the full reference set, but the split-half row uses only the other half, so its MMD/DPP/Vendi values use a different bandwidth as well as a different sample size. In a simple two-cluster, eight-design example, the bandwidths are 14.14 and 1e-6. Could we resolve the bandwidth once from the full reference set (or training set) and pass that same value to every row?

[mkeeler43]

Reproduced: with no training split, a context whose reference half lands in one cluster infers the 1e-6 floor while the full set gives 6.3 on the test problem, the same divergence as your example. Fixed in 481fd39. Board.evaluate now resolves the bandwidth once, from the training designs when given and otherwise from the full reference set, in whatever space is being scored, and hands the same value to every context, the split-half row included. One scope note: when a training split is supplied, every row already sized its kernel from that split, so this changes only boards built without train; the notebook's PCA board is unaffected and its stored outputs stand. test_every_row_shares_one_inferred_bandwidth asserts the median heuristic runs exactly once per evaluate, on the full reference set.

[mkeeler43]

Thanks, Soheyl. All three are addressed in 481fd39, each reproduced from the code first. CI on that commit collected 313 tests and passed 313, with ruff, mypy and pre-commit clean. The four new tests pin the no-flag CLI path, the volume budget on the saved-design path, and the single shared bandwidth.

[SoheylM]

This column dictionary works for volume_error alone, but the same Board fails on a normal evaluate({"exact": ref.copy()}): default metrics include viol, and EvaluationContext.condition_at(0) calls conditions[0], raising KeyError: 0. I reproduced that path. Could we normalize column mappings to row-indexable conditions, or make the accepted conditions type explicit and use that type in this test?

[mkeeler43]

Reproduced: viol reads conditions row by row (conditions[0]), which a plain dict of columns answers with KeyError: 0, so the input my test blessed crashed a default evaluate. Fixed in 876e094, taking your first suggestion and one step past it. ConditionTable normalizes a dict of columns or a DataFrame at the board boundary so both access patterns work; test_dict_conditions_survive_a_full_evaluate is your reproduction as a test. A conditions object whose length disagrees with the reference is now refused with a message saying conditions are an answer key, not a filter, since a short one silently grades design i against the wrong brief. And because the briefs already sit beside the optima in the dataset, Board(reference=rows) now accepts the test rows directly and splits them itself, so the two cannot misalign; Evaluator.for_rows extends the same idea to the sampling path for a user-chosen slice, with the digest cleared since a custom draw is not the frozen contract. CI on 876e094: 319 passed, ruff, mypy and pre-commit clean.