Interactive Shapley estimator benchmark and private local evaluations #601

A reproducible benchmark for choosing Shapley and interaction estimators by game family, player count, query budget, and runtime. Researchers can evaluate a paper’s estimator privately against the same frozen games and baseline results.

Live benchmark · Implementation PR #602 · Runnable guide · Frozen reproduction bundle

Implemented

The five phases—local runner, static website, larger structured games, bounded Hopper campaigns, and statistical comparisons/history—are implemented with independent audits. The expanded suite covers all 22 public estimator classes and representatives of every shipped game family: 31 small recipes across six targets, plus 30-player tree SV/k-SII and 128-player KNN SV. Budgets are 2d, 8d, and 64d, with two seeds.

All 24,948 planned cells are recorded: 9,357 successful, 14,814 unsupported, and 777 failed; none remain pending. Failures are 754 SPEX minimum-budget errors, eight OddSHAP minimum-budget errors, and 15 ShaplEIG timeouts. Coverage means game-family representatives, not every dataset wrapper or independent model replication. Synthetic and causal examples stay in a separate diagnostics panel.

The redesigned website shows the nMSE/Elo table first, then per-game mean nMSE against relative or absolute budget and measured time, followed by publication history. Colors have redundant line styles/markers, with coordinated hover and keyboard focus. Plain HTML/CSS/JavaScript/SVG keeps deployment and review simple; no frontend build, database, or application server is required.

The sweep uses the same exclusive Hopper EPYC 9754 node, a fixed core per game panel, one native thread, fixed software, and common limits. With 64 concurrent workers, runtime remains diagnostic. A separate isolated single-core script is included for runtime qualification. Hardware and software provenance accompany the results.

Small-game truth is exact for frozen payoff tables; stochastic games explicitly represent one saved realization. Large-game truth uses existing structured solvers validated on exhaustive small counterparts. Existing estimator algorithms are unchanged. Public exports exclude private candidates and raw exception text; a local adapter/report never uploads automatically.

Scientific decisions for discussion

• Choose additional datasets, independent fitted models, and application weights. One recipe per family is not evidence of universal estimator quality.
• Expand budget grids and estimator configuration studies without silently tuning individual methods.
• Qualify isolated runtime repeatability before treating the time plots as a timing leaderboard.
• Review nMSE normalization, tie tolerance, complete-panel eligibility, and first-publication dates.
• Decide result ownership, refresh cadence, and whether hosting should move from the author’s fork to the main repository.

Confidence intervals are withheld because each current stratum has only one independent model cluster. More estimator seeds do not solve that limitation. The longer-term protocol remains in DESIGN.md, with verification in AUDITS.md.

@rtealwitter @mmschlk @Advueu963 — Teal, Max, and Santo: this issue remains the discussion home; the PR contains the implementation and reviewable changes.