The Endless Exam

Mathematical constructions, from today's models toward superintelligence.

Website · Paper · PDF · Quick start · Results · Evaluate a model · Reproduce the paper · Specification

The Endless Exam is a benchmark of mathematical construction problems. A model produces an object, a verifier checks its defining property, and the score measures its quality relative to a published frontier or construction baseline. Better constructions earn higher scores, including after the reference is surpassed.

Why the Endless Exam

• Uncapped scores. A relative quality of 1 means parity. Ratios of 1.01, 1.5 and 2 preserve the magnitude of successive improvements.
• Expandable families. Changing dimensions, degrees, diameters or code lengths creates further instances and extends the challenge.
• Verifiable answers. Scores come from mathematical properties and explicit objectives. Compact constructions can describe large objects through checked products, algebraic representations and certificates.

Version 1 samples 69 distinct instances from 14 families, with one response per instance and configuration. The paper evaluates 9 models in 14 tool-free configurations, plus Astra high, Luna high and Opus 5.5 high with code and web access. The revised manuscript uses bench-v1.0, retaining its 69 instances, references and scoring rules. Thirty instances use published frontiers to compare models directly with existing mathematical results. The other 39 use verified construction baselines obtained from mathematical constructions and computational search. These are reproducible comparison points, not necessarily the strongest published constructions. Scores are not clipped at reference parity; accepted answer formats and verification budgets still constrain which constructions can be evaluated. Across the 30 published-frontier instances, every configuration has a 0% breakthrough rate, while continuous relative quality separates tool-free models from 0.07 to 0.76 and reaches 0.998 for Astra, 0.961 for Luna and 0.985 for Opus 5.5 with tools.

Quick start

Python 3.10+ on macOS or Linux:

git clone https://github.com/ricolalemon/endless-exam.git
cd endless-exam
python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

Verify a four-point cap set locally:

python bench/exam.py verify --family capset --params '{"d":2}' --answer examples/capset.json

The example returns valid: true and objective: 4. Export the benchmark's formal prompts and inspect one call:

python bench/exam.py export --output prompts.jsonl
python bench/exam.py prompt a3-capset-0

To evaluate a model through a Chat Completions-compatible API, set its key in OPENAI_API_KEY, then run:

python bench/exam.py run --model YOUR_MODEL --output output/your-model

This collects the 69 tool-free answers, verifies them and writes scores, token usage and attempt logs. Use --base-url for another provider or local server, --adapter responses for the Responses API, and --dry-run to inspect a request without making a model call. See the runner guide for effort settings, concurrency and resume.

Already using Codex or Claude Code? Sign in with the CLI, then use its adapter:

python bench/exam.py run --adapter codex --model gpt-6-astra --effort high --output output/codex-astra
python bench/exam.py run --adapter claude-code --model claude-fable-5-1 --effort high --output output/claude-fable

These are tool-free runs using each CLI's native system instructions. The adapters disable tools and customizations, retain native events and usage, and stop at the first completed answer or budget exhaustion. See CLI setup and supported versions.

You can also use your own harness. Export only the model inputs, collect answers in the standard format, then score them:

python bench/exam.py export --model-inputs-only --output prompts.jsonl
python bench/exam.py score answers.jsonl --output scores.json

Only run without --dry-run invokes a model or an external harness. Export, verification and scoring work offline. Tool-assisted harnesses can use the same task interface and a separate scoring mode.

Results

Score = 100 × mean relative quality over the 69 distinct instances. Each instance contributes one scored outcome; invalid submissions and tool-free generation-budget exhaustion receive zero. A Score of 100 represents average reference parity and can be exceeded. The Published frontiers column reports mean relative quality over those 30 instances.

Without tools

Model	Effort	Score	Published frontiers	Valid
GPT-6 Astra	high	91.90	0.76	100.0%
GPT-6 Astra	medium	75.34	0.66	98.6%
Claude Fable 5.1	medium	71.90	0.59	85.5%
Claude Opus 5.5	high	69.81	0.57	82.6%
Claude Opus 5.5	medium	66.70	0.58	88.4%
DeepSeek V4.1 Flash	high	44.90	0.38	75.4%
Claude Fable 5.1	high	43.73	0.54	52.2%
Qwen3.8-27B	high	38.07	0.31	62.3%
DeepSeek V4.1 Flash	low	37.11	0.39	78.3%
GPT-5.6 Luna	high	34.85	0.25	68.1%
GPT-5.6 Luna	medium	16.70	0.16	56.5%
Qwen3.5-27B	high	16.18	0.13	52.2%
Qwen3.5-9B	high	14.52	0.12	43.5%
Qwen3.5-4B	high	7.55	0.07	37.7%

All rows use the same 69-instance suite and a 128k output-token budget, including reasoning. Full results, uncertainty intervals and the historical Opus 5 subset are in the paper and machine-readable results. The per-instance index lists each published outcome and links it to the saved response, including failures.

With code and web access

Model	Effort	Score	Published frontiers	Valid
Claude Opus 5.5	high	180.06	0.985	100.0%
GPT-6 Astra	high	143.16	0.998	100.0%
GPT-5.6 Luna	high	119.35	0.961	95.7%

All three models receive the same mathematical prompts on all 69 instances, with four CPU threads, 16 GiB and two hours per instance, without a cumulative output-token cap. Astra matches 29 published frontiers, and Luna and Opus 5.5 each match 23; none exceeds one. They exceed 33, 25 and 33 construction baselines, respectively. The tool results include all outcomes, including Luna's two invalid answers and one memory-limit stop without an answer. Opus 5.5 has 69 valid submissions; its final covering construction remains valid at the two-hour deadline. Claude Code and Codex use different native tool interfaces; client versions and usage completeness are recorded with the released results.

Fourteen task families

Family	Construct	Improve
Cap sets	Progression-free points over F₃	Number of points
Spherical codes	Separated vectors	Number of vectors
Heilbronn triangles	Point configurations	Smallest triangle area
LABS	Binary sequences	Merit factor
Corner-free sets	Grid subsets avoiding corners	Set size
Matrix multiplication	Bilinear algorithms	Fewer scalar products
Degree–diameter graphs	Graphs with bounded degree and diameter	Number of vertices
Linear-equation-free sets	Integer sets avoiding a linear relation	Set size
Finite-field AP-free sets	Progression-free sets over Fq	Set size
MOLS	Mutually orthogonal Latin squares	Number of squares
Schur colourings	Sum-free colour classes	Coloured interval length
Covering designs	Blocks covering all required subsets	Fewer blocks
Shannon codes	Distinguishable codewords	Number of codewords
Trifference codes	Codes separating every triple	Number of codewords

Spherical codes and Heilbronn each have two task variants. The code also retains supplementary controls. The specification defines the formal suite and scoring, and the paper appendix gives illustrated examples and parameters.

Repository guide

Path	Contents
bench/exam.py	Public prompt export, verification and scoring CLI
bench/openceiling.py	Families, construction formats and primary verifiers
bench/candidate_families.py	Shannon and trifference implementations
bench/crosscheck.py	Independent checks
bench/refs/, bench/frontiers/	Frozen reference constructions and scoring values
bench/data/reference_witnesses/	Verified constructions attaining all 69 reference values
bench/results/	Released final responses and evaluation metadata
bench/results/published/index.csv	The 1,173 main-evaluation outcomes and their source records
bench/paper/	Paper, figures, tables and reproduction scripts
examples/	Small runnable construction examples

Reproduce and contribute

Reproduction instructions cover scores, figures and the paper build. Contributing explains verifier changes, new constructions and model submissions. Report bugs or proposed records through GitHub Issues. The data guide explains the result index, response selection and reference evidence.

Citation

@misc{zhang2026endless,
  author = {Muhan Zhang},
  title = {The Endless Exam: Mathematical Constructions from Today's Models toward Superintelligence},
  year = {2026},
  eprint = {2609.24555},
  archivePrefix = {arXiv},
  primaryClass = {cs.AI},
  doi = {10.48550/arXiv.2609.24555},
  url = {https://arxiv.org/abs/2609.24555}
}

License

Original code and documentation are released under the MIT license. See third-party notices for reference data, model responses and paper templates.