[BANADDA]

Part of #155, pairs with microtensor-arenas#3.

Probe scoring by type

• exact: identifier mentioned, as before.
• word: first word, for yes/no and pass/fail.
• choice: the letter, for which issue.
• next_step: 1 when both the kind (read, edit, test, run, install, other) and the file match the agent's actual next command, 0.5 for either alone.

Continuation is the mean probe score. Answers longer than eight words still score 0.

Hallucination check. It counts only:

• paths with a /;
• lowercase file names with a code extension;
• exception types;
• name( calls;
• backticked tokens that look like code.

CamelCase words and capitalised names are no longer counted. An allow list covers common tool, brand and builtin names. A sentence naming GitHub, Node.js and README.md went from 0.5 invented to 0.

Other changes

• The grader enforces label.budget_tokens, the exact budget the instruction states.
• The probe agent prompt asks for the form each question names: identifier, word, letter, or kind plus file.

Local tests (not committed) cover word, choice and next step scoring and the allow list. The full suite and newest ruff are clean.

🤖 Generated with Claude Code