Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,33 @@
a copy that changed during the work, or a check that names the project's
absolute path, counts as not tried. The copy's size and timing are
journalled (`pre-work-copy`).
- **Checks that pass before the work are kept as refuse-only guards.** A
drafted check the pre-work screen drops (P1) is still sealed, tagged
`guard`: it runs at the bar and a failure refuses the claim, but it never
counts toward "verified", and receipts label it "refuse-only guard". Up to
four, beside (not inside) the four-check cap. Not kept: a duplicate of a
sealed check, and one that pins today's output of the project's program on
the project's own data (`node summarize.js transactions.json | jq -e
'.zoe == "-3.50"'`, `sqlite3 app.db 'PRAGMA user_version' | grep -q '^1$'`),
which on the 2026-10-07 lanes failed 13 finished trees the grader accepted.
- **Fewer good drafted checks dropped before sealing** (measured with
`bench/local/drop_audit.py` over the 2026-10-07 lanes):
- the absolute-path rule reads what each program opens (a shell parser,
`src/shellwords.ts`), not the raw text: a sed program's `/g`, an awk
program, a grep pattern, a jq filter, printf's data and `sys.argv[1] +
"/old.py"` are not paths. Its four drops that day were all wrong.
- `./rotate.sh …` before rotate.sh exists is the deliverable missing:
tried before the work it now counts as failing there (it discriminates),
not as a command that could not run. A tool missing from PATH is still
broken.
- the mutation rule (L15) skips the value of `touch -d/-t/-r` and
`mkdir -m`, follows variables built on a mktemp directory
(`f="$d/x"; echo > "$f"`), and lets `> "$(mktemp -d)/out"` through.
- the cannot-fail rule (L16) no longer flags `…; exit 0` after an earlier
`exit 1` in the same shell, or `cond && exit 1 || exit 0`.
- a check that names the project by its absolute path is tried on the
pre-work copy, and run in the bar's throwaway copy, instead of reaching
the live folder.
- **Claims say who stood behind them** on the terminal, the window, job_end's
`claim`, the receipt index and the receipt: `verified (independent checks:
<judge>)`, `verified (your checks)`, `passed own checks (<worker>), not
Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -413,7 +413,9 @@ to them. Passing checks the worker model drafted for itself are never
the best a run earns is `passed own checks (<model>), not verified`, exit 3.
Drafted checks are tried on a copy of the project before the work: one that
already passes there, or cannot fail by construction, is sent back to the
drafter once and dropped if the redraft is no better. `--require-discriminating`
drafter once and dropped if the redraft is no better; one that already
passed is still sealed as a refuse-only guard, which can fail the claim but
never makes it verified. `--require-discriminating`
goes further: "verified" then needs an independent value check that failed
before the work, and anything less is `passed checks that did not test this
work, not verified` (exit 3).
Expand Down
3 changes: 2 additions & 1 deletion bench/harbor/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ passed-own-checks`, exit 3), never `verified`. Set `MAAT_JUDGE_MODEL` (and
`MAAT_JUDGE_URL` / `MAAT_JUDGE_KEY`) in the container to another model to make "verified"
reachable. Drafted checks that pass on the untouched task, or cannot fail
by construction, are redrafted once and then dropped before they are
sealed. With `--require-discriminating` (or `MAAT_REQUIRE_DISCRIMINATING=1`)
sealed; the ones that only passed before the work are kept as refuse-only
guards (they can fail a run, never verify it). With `--require-discriminating` (or `MAAT_REQUIRE_DISCRIMINATING=1`)
a run whose independent checks never failed before the work ends
`passed-untested`, also exit 3. The exit code never fails a trial (see
below).
Expand Down
358 changes: 358 additions & 0 deletions bench/local/drop_audit.py

Large diffs are not rendered by default.

7 changes: 6 additions & 1 deletion src/bar.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2201,7 +2201,11 @@ export async function runCheck(check: Check, ctx: BarContext): Promise<CheckResu
// Not execSync: a bar check is the longest thing molt runs (`npm test`,
// two minutes by default) and running it synchronously froze the terminal
// for its whole duration — including the ctrl+C that would have stopped it.
const r = await runCommand(check.run, {
// A check that names the project by its absolute path reads the copy
// too: otherwise `cd /app && git checkout …` reaches past the copy into
// the work, and the seal-time try (criteria.ts preflightCriteria), which
// points the same path at its own copy, judged a different tree.
const r = await runCommand(copy ? copy.map(check.run) : check.run, {
cwd: copy?.dir ?? ctx.cwd,
shell: draftedShell(check),
timeoutMs: check.timeoutMs,
Expand Down Expand Up @@ -2480,6 +2484,7 @@ export function formatBarFailure(result: BarResult, attempt: number, maxAttempts
if (failed.some((r) => r.hidden)) lines.push(DISPUTE_HINT, "");
for (const r of failed) {
lines.push(`--- FAILED: ${r.name} (${r.hidden ? "command withheld" : r.detail})`);
if (r.tags?.includes("guard")) lines.push("a guard: this held on the project before the work, and it does not now.");
if (r.exitCode !== undefined) lines.push(`exit code: ${r.exitCode}`);
lines.push(r.output.trim() || "(no output)");
lines.push("");
Expand Down
31 changes: 28 additions & 3 deletions src/checkwrites.ts
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,20 @@ export function maskQuoted(cmd: string): string {
function tempVars(cmd: string): Set<string> {
const out = new Set(["TMPDIR", "TMP", "TEMP", "TEMPDIR"]);
for (const m of cmd.matchAll(/\b([A-Za-z_]\w*)=["']?(?:\$\(|`)\s*mktemp\b/g)) out.add(m[1]!);
// And what is built on one: `f="$d/f $i.log"` names a file in the mktemp
// directory, so `echo x > "$f"` writes there. Followed to a fixed point:
// `g=$f.bak` is scratch once f is.
const assigns = [...cmd.matchAll(/(?<![\w$])([A-Za-z_]\w*)=("(?:\\.|[^"\\])*"|'[^']*'|[^\s;&|()]+)/g)];
for (let grew = true; grew; ) {
grew = false;
for (const m of assigns) {
if (out.has(m[1]!)) continue;
if (isScratchPath(shellWordValue(m[2]!), out)) {
out.add(m[1]!);
grew = true;
}
}
}
return out;
}

Expand Down Expand Up @@ -253,7 +267,9 @@ export function checkMutates(cmd: string, outer?: { temps: Set<string>; depth: n
if (!tm) continue;
const target = seg.raw.slice(start, start + tm[0].length);
for (let k = r.index ?? 0; k < start + tm[0].length; k++) consumed.add(k);
if (!isScratchPath(target, temps) && !inScratch[seg.at + start]) {
// `> "$(mktemp -d)/out"`: the segment stops at the `$(`, so read the target from the whole command.
const intoTemp = /^["']?(?:\$\(|`)\s*mktemp\b/.test(cmd.slice(seg.at + start));
if (!intoTemp && !isScratchPath(target, temps) && !inScratch[seg.at + start]) {
return `it writes to ${unquote(target)} with a redirect; a check only reads (write to a mktemp directory instead)`;
}
}
Expand Down Expand Up @@ -344,9 +360,18 @@ export function checkMutates(cmd: string, outer?: { temps: Set<string>; depth: n
}
case "touch":
case "mkdir":
case "mkfifo":
if (!allScratch(operands)) return `it creates ${quoteOne(operands.find((o) => !scratch(o)))} (\`${name}\`) in the project; a check only reads`;
case "mkfifo": {
// `touch -d "2020-01-01" f`, `touch -t 2001010000 f`, `touch -r ref f`, `mkdir -m 700 d`:
// the option's value is not a file it creates.
const valued = name === "touch" ? /^(-[dtr]|--date|--reference)$/ : /^(-m|--mode|-Z|--context)$/;
const files: Word[] = [];
for (let k = 0; k < args.length; k++) {
if (valued.test(args[k]!.text)) k++;
else if (!isFlag(args[k]!.text)) files.push(args[k]!);
}
if (!allScratch(files)) return `it creates ${quoteOne(files.find((o) => !scratch(o)))} (\`${name}\`) in the project; a check only reads`;
break;
}
case "chmod":
case "chown":
case "chgrp":
Expand Down
2 changes: 1 addition & 1 deletion src/cli.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -1388,7 +1388,7 @@ async function sealDraft(draft: Draft, args: Args, late = false): Promise<Return
const taskChecks = sealed.taskChecks.filter((c) => !drop.has(c.name));
if (!args.json) {
// Names only: the commands are printed when the job releases them (checks_released).
for (const c of taskChecks) process.stdout.write(`· criterion ${c.name} (command withheld until the job ends)\n`);
for (const c of taskChecks) process.stdout.write(`· ${c.tags.includes("guard") ? "guard" : "criterion"} ${c.name} (command withheld until the job ends)\n`);
for (const n of sealed.taskNotes) process.stdout.write(`· note ${n}\n`);
}
return { taskChecks, taskNotes: sealed.taskNotes, ...(sealed.requirements ? { requirements: sealed.requirements } : {}) };
Expand Down
Loading
Loading