Skip to content

docs(shipyard): define append-only recovery benchmark result ledger - #2746

Draft
Tristan-Stoltz-ERC wants to merge 2 commits into
research/shipyard-disturbance-tape-v0from
research/shipyard-result-ledger-v0
Draft

Tristan-Stoltz-ERC wants to merge 2 commits into
research/shipyard-disturbance-tape-v0from
research/shipyard-result-ledger-v0

Conversation

@Tristan-Stoltz-ERC

Copy link
Copy Markdown
Contributor

Summary

Adds a result-ledger contract and an explicitly not-run JSON fixture for the shipyard evaluation lab.

  • Defines terminal outcomes separately: feasible, infeasible_proven, budget_exceeded, timed_out, invalid_input, incomplete, invalid_candidate, evaluator_error, and not_run.
  • Requires every assigned episode to remain visible, including failures and non-started runs.
  • Specifies paired-comparison identity, independently recomputed metrics, uncertainty reporting, Pareto-frontier rules, proof labels, and publication/industry comparison gates.
  • Includes a synthetic example marked not_run; it is not a benchmark result.

Why this matters

A frozen disturbance corpus prevents input leakage; an append-only result ledger prevents outcome leakage and survivorship bias. Together, these make it harder to manufacture apparent superiority by dropping timeouts, treating heuristic failure as infeasibility, or comparing different test episodes.

Validation / claim boundary

Documentation and fixture only. No Rust implementation or CI pass is claimed. The fixture is illustrative, and no policy was executed. This contract improves evaluation integrity but does not establish optimizer quality, model validity, or industrial uplift.

Stacked on PR #2742; related protocol PR #2735 and implementation ticket issue #2743.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant