Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,39 @@
module the work's code can reach, and expected values never leave the
driver's memory. A pass also needs the driver's final `REFERENCE COMPARED
n/n` line: an exit 0 without it is a failure.
- **Drafted checks that pass before the work are redrafted, then dropped.**
With `--criteria auto` each drafted check is tried on a copy of the project
taken before the first step. One that already passes there cannot show the
task was done: it goes back to the drafter once, and is dropped if the
redraft still passes (journal: `check-lint-dropped`, rule
`P1-passes-before-work`). An always-on lint (L16) does the same for checks
that cannot fail by construction: `|| echo …`, `|| true`, `; exit 0`,
`…; echo $?`, `find … -exec … \;`, an `if … fi` with no failing branch,
and inline programs whose exit code does not depend on what they assert
(`python3 -c "print(x == y)"`, `node -e "console.log(a === b)"`, `jq`
without `-e`, `awk` with no `exit`, a printed PASS/FAIL).
- **A task check that prints its failure has failed.** A drafted or approved
check that exits 0 but whose last line of output is `False`, `FAIL`,
`FAILED` or `NO` (or whose `echo $?` tail printed non-zero) is read as a
failure at the bar and before the work. `print(rows == expected)` printing
False on wrong work used to pass and earn "verified".
- **`--require-discriminating`** (`MAAT_REQUIRE_DISCRIMINATING=1`, off by
default; `--review-advisory` implies it): "verified" needs an independent
value check that failed on the tree before the work and passes now.
Otherwise the new tier **`passed-untested`** ("passed checks that did not
test this work, not verified"): outcome `unverified`, exit code 3. Each
receipt row says how its check fared before the work.
- **Checks are tried before the work under the shell the bar runs them
with** (bash for drafted checks), so a bashism no longer "fails before the
work" under dash and passes at the bar. dash's exit 2 for a missing
`sh script` is read like bash's 127. A try that timed out counts as not
tried, not as failing.
- **The pre-work copy is checked before it is trusted.** Late checks (drafts
past the time cut, the reference check) are tried on a copy taken at turn
start. Its digest is taken when it is made and compared around every try;
a copy that changed during the work, or a check that names the project's
absolute path, counts as not tried. The copy's size and timing are
journalled (`pre-work-copy`).
- **Claims say who stood behind them** on the terminal, the window, job_end's
`claim`, the receipt index and the receipt: `verified (independent checks:
<judge>)`, `verified (your checks)`, `passed own checks (<worker>), not
Expand Down
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,8 @@ Exit codes: `0` the bar was met · `1` it was not · `2` usage · `3` finished
without a verdict (no bar, or `--only`/`--skip` left required checks unrun, which
is never reported as a pass), or passed without earning "verified": every check
that could prove the work was written by the worker model itself
(`passed-own-checks`), or nothing but builtins passed. "Verified" needs a
(`passed-own-checks`), with `--require-discriminating` none of them failed
before the work (`passed-untested`), or nothing but builtins passed. "Verified" needs a
passing check from another model (`--judge`) or a command check you wrote or
approved; builtins never carry it.

Expand Down Expand Up @@ -410,6 +411,12 @@ the task are drafted before the work, they are sealed, and the worker is held
to them. Passing checks the worker model drafted for itself are never
"verified": without `--judge <another model>` (or checks a person approved)
the best a run earns is `passed own checks (<model>), not verified`, exit 3.
Drafted checks are tried on a copy of the project before the work: one that
already passes there, or cannot fail by construction, is sent back to the
drafter once and dropped if the redraft is no better. `--require-discriminating`
goes further: "verified" then needs an independent value check that failed
before the work, and anything less is `passed checks that did not test this
work, not verified` (exit 3).

## Docs

Expand Down
7 changes: 6 additions & 1 deletion bench/harbor/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,12 @@ model, and since 2026-10-07 a run that passes only its own checks ends
`passed own checks (<model>), not verified` (job_end `tier:
passed-own-checks`, exit 3), never `verified`. Set `MAAT_JUDGE_MODEL` (and
`MAAT_JUDGE_URL` / `MAAT_JUDGE_KEY`) in the container to another model to make "verified"
reachable. The exit code never fails a trial (see below).
reachable. Drafted checks that pass on the untouched task, or cannot fail
by construction, are redrafted once and then dropped before they are
sealed. With `--require-discriminating` (or `MAAT_REQUIRE_DISCRIMINATING=1`)
a run whose independent checks never failed before the work ends
`passed-untested`, also exit 3. The exit code never fails a trial (see
below).

## What it measures

Expand Down
1 change: 1 addition & 0 deletions bench/harbor/summarize.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ def trial_row(d: Path) -> dict | None:
"unconfirmed" if (je or {}).get("outcome") == "verified" and ((je or {}).get("review") or {}).get("confirmed") is False
# Builds from 2026-10-07 say who wrote the checks in job_end's `claim`.
else "passed-own-checks" if (je or {}).get("tier") == "passed-own-checks"
else "passed-untested" if (je or {}).get("tier") == "passed-untested"
else "self-checked" if (je or {}).get("outcome") == "verified" and (je or {}).get("selfChecked") and not (je or {}).get("claim")
else (je or {}).get("outcome")
),
Expand Down
232 changes: 232 additions & 0 deletions bench/local/discriminating_replay.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,232 @@
"""Replay the "verified" tier with and without the discrimination rule.

python3 discriminating_replay.py [LANE_PREFIX ...] (default: 2026-10-07's lanes)

The rule (src/tiers.ts tierOf, `failedBefore`): "verified" needs a passing
independent runs+value check that FAILED on the tree before the work. One that
passed then too (receipt: `pass-vacuous`), was broken then ("sealed criteria
that did not run when tried before the work"), or joined after the work began
(late drafts, the reference check) with no pre-work try does not count.

Reads, offline, for every <task>-molt-<rep>-<arm> tree under
~/.cache/maat-bench/container-work/<lane>/: the session journal (.maat/log), the
last accepted full receipt, and the lane's grader row in container-results. No
model, no network. "Value" is src/tiers.ts assertsValue on the released
command, through the built dist/tiers.js; the critic's `surface` tag is not
journalled, so a value check the critic marked surface still counts here
(this can only overstate what the rule keeps).

Baselines:
recorded the run's own claim started with "verified" (what the build said)
pr32 recorded, and an independent check stood behind it: the claim names
independent checks, or (builds before 2026-10-07) the arm is not
`self`, i.e. a separate judge drafted the checks
The rule is applied on top of pr32.
"""
from __future__ import annotations

import json
import re
import subprocess
import sys
from collections import defaultdict
from pathlib import Path

CACHE = Path.home() / ".cache/maat-bench"
CW = CACHE / "container-work"
RESULTS = CACHE / "container-results"
REPO = Path(__file__).resolve().parents[2]
TODAY = ["q235", "mm3", "ml4b", "hk55", "dsv4", "gh", "ga3b", "gx"]
RUN_DIR = re.compile(r"^(?P<task>.+)-molt-(?P<rep>\d+)-(?P<arm>[a-z0-9]+)$")
RESULTS_LINE = re.compile(r"container-results/(results-[\w.-]+\.jsonl)")


def results_file(lane: Path) -> Path | None:
log = CACHE / f"{lane.name}.log"
if log.exists():
m = RESULTS_LINE.search(log.read_text(errors="replace")[:4000])
if m and (RESULTS / m.group(1)).exists():
return RESULTS / m.group(1)
return None


def grader_rows(path: Path) -> dict[tuple[str, str, int], dict]:
rows: dict[tuple[str, str, int], dict] = {}
for line in path.read_text().splitlines():
try:
r = json.loads(line)
except ValueError:
continue
if r.get("agent") == "molt":
rows[(r["task"], r["arm"], int(r["rep"]))] = r # the last row for a trial wins
return rows


def journal(tree: Path) -> list[dict]:
out: list[dict] = []
for f in sorted((tree / ".maat/log").glob("*.jsonl")):
for line in f.read_text(errors="replace").splitlines():
try:
out.append(json.loads(line))
except ValueError:
pass
return out


def task(n: str) -> str:
return n if n.startswith("task:") else f"task:{n}"


def receipt_results(tree: Path) -> dict[str, str]:
"""{check: result} from the last accepted full receipt: pass, pass-vacuous, fail, ..."""
# Builds that withhold hidden checks keep the full text under full/; older ones have only the one file.
full = sorted((tree / ".maat/receipts/full").glob("*-accepted.md")) or sorted((tree / ".maat/receipts").glob("*-accepted.md"))
if not full:
return {}
out: dict[str, str] = {}
name = None
for line in full[-1].read_text(errors="replace").splitlines():
if line.startswith("check: "):
name = line[len("check: "):].strip()
elif line.startswith("result: ") and name:
out[name] = line[len("result: "):].strip()
name = None
return out


def asserts_value(runs: list[str]) -> list[bool]:
js = (
f"import {{ assertsValue }} from {json.dumps(str(REPO / 'dist/tiers.js'))};"
"let s='';process.stdin.on('data',d=>s+=d).on('end',()=>console.log(JSON.stringify(JSON.parse(s).map(assertsValue))));"
)
p = subprocess.run(["node", "--input-type=module", "-e", js], input=json.dumps(runs), capture_output=True, text=True, check=True)
return json.loads(p.stdout)


def analyse(tree: Path, row: dict) -> dict:
ev = journal(tree)
notes = [e["data"] for e in ev if e.get("kind") == "note"]
upfront: set[str] = set()
late: set[str] = set()
broken: set[str] = set()
runs: dict[str, str] = {}
pre_failed: set[str] | None = None
for e in ev: # builds before hidden checks were withheld journalled the command with each run
if e.get("kind") == "bar_run":
for c in e["data"].get("checks", []):
if c.get("name", "").startswith("task:") and not str(c.get("detail", "")).startswith("[withheld"):
runs.setdefault(c["name"], c.get("detail", ""))
for d in notes:
text = d.get("text", "")
if text.startswith("task criteria sealed"):
upfront |= {task(c) for c in d.get("checks", [])}
elif d.get("kind") == "late-checks" and d.get("arrived") and "joined" in text and "NOT joined" not in text:
late |= {task(c) for c in d.get("checks", [])}
elif text.startswith("reference check joined"):
late.add("task:reference")
elif text.startswith("sealed criteria that did not run"):
broken |= {task(c) for c in d.get("checks", [])}
elif d.get("kind") == "checks-released":
for c in d.get("checks", []):
runs[task(c["name"])] = c.get("run", "") # the released command outranks a journalled detail
elif d.get("kind") == "pre-work-try": # builds with the rule journal it directly
pre_failed = (pre_failed or set()) | set(d.get("failed", []))
late -= upfront
res = receipt_results(tree)
passing = [n for n, r in res.items() if n.startswith("task:") and r in ("pass", "pass-vacuous")]
values = dict(zip(passing, asserts_value([runs.get(n, "") for n in passing]))) if passing else {}
strong = [n for n in passing if values.get(n)]
status = {}
for n in strong:
if pre_failed is not None:
status[n] = "failed" if n in pre_failed else ("passed" if res[n] == "pass-vacuous" else "untried")
elif res[n] == "pass-vacuous":
status[n] = "passed"
elif n in broken:
status[n] = "untried"
elif n in late:
status[n] = "late"
elif n in upfront:
status[n] = "failed"
else:
status[n] = "untried"
return {"strong": strong, "status": status}


def main(argv: list[str]) -> int:
prefixes = argv or TODAY
lanes = sorted(p for p in CW.iterdir() if p.is_dir() and any(re.fullmatch(rf"{re.escape(x)}-\d+", p.name) for x in prefixes))
tally: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int))
lost_right: list[str] = []
removed_wrong: list[str] = []
kept_wrong: list[str] = []
late_only: list[str] = []
for lane in lanes:
rf = results_file(lane)
if not rf:
print(f"# {lane.name}: no results file found, skipped", file=sys.stderr)
continue
rows = grader_rows(rf)
for tree in sorted(lane.iterdir()):
m = RUN_DIR.match(tree.name)
if not m or not tree.is_dir() or not (tree / ".maat/log").exists():
continue
row = rows.get((m["task"], m["arm"], int(m["rep"])))
if not row:
continue
claim = row.get("claim") or ""
if not claim.startswith("verified"):
continue
if claim.startswith(("verified (independent checks", "verified (your checks)")):
independent = True
else: # builds before 2026-10-07: a separate judge drafted unless the arm is self
independent = m["arm"] != "self"
right = bool(row.get("passed"))
key = f"{lane.name}/{tree.name}"
lane_key = lane.name.rsplit("-", 1)[0]
t = tally[lane_key]
t["recorded"] += 1
t["recorded_right"] += right
if not independent:
continue
t["pr32"] += 1
t["pr32_right"] += right
a = analyse(tree, row)
disc = [n for n in a["strong"] if a["status"][n] == "failed"]
if disc:
t["rule"] += 1
t["rule_right"] += right
if not right:
kept_wrong.append(f"{key} discriminating: {', '.join(disc)}")
else:
why = ", ".join(f"{n}={a['status'][n]}" for n in a["strong"]) or "no value check in the receipt"
if any(s == "late" for s in a["status"].values()):
late_only.append(f"{key} ({'right' if right else 'WRONG'}) {why}")
(lost_right if right else removed_wrong).append(f"{key} {why}")
tot: dict[str, int] = defaultdict(int)
print(f"{'lane':8} {'recorded':>12} {'PR #32':>12} {'+ rule':>12}")
for lane_key in sorted(tally):
t = tally[lane_key]
for k, v in t.items():
tot[k] += v
print(f"{lane_key:8} {t['recorded_right']:>5}/{t['recorded']:<6} {t['pr32_right']:>5}/{t['pr32']:<6} {t['rule_right']:>5}/{t['rule']:<6}")
print(f"{'all':8} {tot['recorded_right']:>5}/{tot['recorded']:<6} {tot['pr32_right']:>5}/{tot['pr32']:<6} {tot['rule_right']:>5}/{tot['rule']:<6} (right/verified)")
print(f"\nwrong verifieds the rule removes ({len(removed_wrong)}):")
for x in removed_wrong:
print(" " + x)
print(f"\nright verifieds the rule loses ({len(lost_right)}):")
for x in lost_right:
print(" " + x)
print(f"\nwrong verifieds the rule keeps ({len(kept_wrong)}):")
for x in kept_wrong:
print(" " + x)
if late_only:
print(f"\nof those denied, rested on a late-joined check with no pre-work try in this build ({len(late_only)}); "
"with the rule, Maat tries late checks on a copy taken before the work, so these may differ live:")
for x in late_only:
print(" " + x)
return 0


if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
Loading
Loading