RFC 0013: the first evaluator error and precedence cases (closes #45)

[kikashy]

Closes #45.

The 0.2.0-draft evaluation suite carries an expectedErrorClass result shape and no case that uses it — 20 rows, all expecting a disposition. §8.4's four classes and the fixed order between them are specified, schema-supported, and entirely unexercised. The suite is frozen, so this proposes rows for the next version rather than rewriting published history.

Five cases; two of them are the point

Three are single-condition and would be caught by almost any implementation. The other two present inputs where more than one class genuinely applies, and the row fixes which one is reported.

That is the whole argument. §8.4 says the order exists so "two conforming implementations report the same class for the same inputs" — and an order that is never exercised is one two implementations can disagree about while both passing the suite.

Case 5 is why this is worth doing now. The normative order is not the order the classes are introduced in prose: unsupported-required-extension sits third, after malformed-input. So an implementation that checks extension support while reading the pack — before it ever looks at the evidence document — reports the wrong class. That is a plausible implementation, not a contrived one, and nothing in the current suite catches it.

Case 4 is §8.4's own worked example turned into a row.

What it declines to sketch

Rather than sketching them loosely: resource-exhaustion (needs a documented §10 limit reached while evaluating an admitted input); implementation-defined classes (a corpus row cannot require one without naming an implementation); error messages (§8.4 fixes the class identifier and explicitly leaves transport and wire format undefined, with §13 keeping the diagnostic contract open); and whether expectedErrorPhase belongs in the first batch, left as an open question rather than quietly decided.

Four open questions are listed for the reviewer, including whether five is the right size — a batch of just cases 4 and 5 would be defensible if the single-condition rows are judged obvious.

Verified, not recalled

• the quoted precedence order is the specification's own text, character for character;
• both quoted rationales likewise;
• the premise — 20 rows, 0 using expectedErrorClass — checked against the manifest;
• the sketched row shape checked against the case schema's oneOf (expectedDisposition or expectedErrorClass) and dependentRequired (expectedErrorPhase needs a class).

Registered in rfcs/README.md and web/build.py. Suite: 64 passed, 532 subtests. No normative or corpus change — this is a proposal.

🤖 Generated with Claude Code

[github-actions]

🎉 Thank you for your contribution to Judgment Pack!

If you’d like another task, choose a path that interests you:

• Go runtime
• Gateway and provenance
• Python and evaluator experiments
• Specification and documentation