Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,21 @@
- **`opencode://zen`** is the OpenCode endpoint (short name `opencode`). The old
`opencode://subscription` still works for this release, with a deprecation
notice.
- **Unattended runs stop when they stop advancing** (`MAAT_NO_PROGRESS_CALLS`,
default 30, `0` turns it off; on for every unattended run on an HTTP
backend, not on ACP backends yet). A call advances when it changes a file
in the project, reads something not read before, runs a command not run
before, or gets output from a command that differs from every earlier run
of it (timings aside), wherever the work happens: a build into `dist/`, a
venv, a config under `/etc`. After N calls in a row that advance nothing the
model is told once to finish or stop; after N more the turn ends, the work
on disk is judged, and the receipt says `ended: no progress` (never
verified: the model did not say it was done).
- **While an unattended job with hidden checks runs, the file tools stay in
the task.** `read_file`, `list_dir`, `grep` and `inspect` refuse paths
outside the project and under `.maat/` (except spilled output and
background logs, and paths the task names). bash is not blocked; this is a
budget control, not isolation.

## 0.2.2 — subscription backends removed

Expand Down
4 changes: 3 additions & 1 deletion bench/local/container/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,6 @@ ENV BENCH_AGENT_USER=agent
WORKDIR /opt
# Maat itself is installed per build by run-in-container.sh (FROM maat-bench + npm i -g the packed tarball).

ENV BENCH_WORK=/work RESULTS_DIR=/results
# run-in-container.sh overrides these: tasks run in a container-local /var/lib/bench-work and the
# host folders are mounted under /root, where the agent user cannot reach them.
ENV BENCH_WORK=/var/lib/bench-work BENCH_EXPORT=/root/bench-export RESULTS_DIR=/root/bench-results
35 changes: 28 additions & 7 deletions bench/local/container/run-in-container.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,10 @@
# Make the tarball with: npm run pack:cli && npm pack ./out-cli
# The image is maat-bench:<sha8 of the tarball>, built once and reused, so every run
# records exactly which Maat it tested. Env passed through: REFERENCE BENCH_MODEL
# BENCH_URL BENCH_LIMIT RESULTS (file name). Results land in
# ~/.cache/maat-bench/container-results/; each task's folder and Maat's log are kept
# in ~/.cache/maat-bench/container-work/<name>/ (the only host path it can write).
# BENCH_URL BENCH_LIMIT RESULTS (file name; run.py refuses a file holding another lane's rows). Results land in
# ~/.cache/maat-bench/container-results/; each task's folder and Maat's log are copied,
# once the task is graded, to ~/.cache/maat-bench/container-work/<name>/ (the only host
# path it can write; the agent never sees it, see "Task isolation" below).
# SUBSCRIPTION=grok: use the owner's Grok Build login (BENCH_URL=grok-build://subscription
# BENCH_MODEL=grok-4.7). Image is maat-bench-grok:<sha> (Dockerfile.grok: official grok
# installer, linux, pinned). ~/.grok/auth.json is mounted read-only OUTSIDE the container's
Expand Down Expand Up @@ -59,10 +60,27 @@ url=${url//127.0.0.1/host.docker.internal}; url=${url//localhost/host.docker.int
# run ends: on the command line (-e KEY=...) it showed in every `ps` listing.
envf=$(mktemp); chmod 600 "$envf"; trap 'rm -f "$envf"' EXIT
printf 'OPENROUTER_API_KEY=%s\n' "$key" > "$envf"
# Anthropic API lanes (BENCH_URL=https://api.anthropic.com/v1): the key comes from the
# Keychain item maat-bench-anthropic (paste it with container/set-anthropic-key.sh),
# never from the command line.
if [[ "$url" == *api.anthropic.com* || "${ARMS:-}" == *api.anthropic.com* ]]; then
akey=$(security find-generic-password -s maat-bench-anthropic -w 2>/dev/null) || { echo "no Keychain item maat-bench-anthropic: run bench/local/container/set-anthropic-key.sh" >&2; exit 1; }
printf 'ANTHROPIC_API_KEY=%s\n' "$akey" >> "$envf"
fi
# The graders and reference solutions are mounted where only root can reach (/root is 700),
# copied to a root-only /opt/bench, and run.py runs from there as root; it runs the agent
# as the unprivileged `agent` user (BENCH_AGENT_USER) and grades as root afterwards.
credmount=; startcmd='export MOLT_DIST_ABS=$(npm root -g)/@solvyx/molt/dist; mkdir -p /work && chmod 755 /work && rm -rf /opt/bench && cp -a /root/bench-src /opt/bench && chmod -R go-rwx /opt/bench && cd /opt/bench && python3 run.py "$@"'
#
# Task isolation, in every mode (plain, SUBSCRIPTION=grok, OPENCODE=1). A host bind mount
# ignores chown and does not enforce modes for other users (OrbStack, Docker Desktop), so the
# agent can read anything mounted where it can reach it: on 2026-10-07 a worker spent its whole
# budget reading the other tasks' folders and logs in a shared /work. So nothing from the host
# is mounted where the agent can reach it. The work folder and the results are mounted under
# /root (700, container-local, enforced). Each task runs in a private folder under the
# container-local /var/lib/bench-work (711, random per-task parent, removed after the task);
# Maat's logs are written to a root-only folder there; folder and logs are copied to the export
# only once the task is graded and every process of the agent user is gone (run.py finish_task).
credmount=; startcmd='export MOLT_DIST_ABS=$(npm root -g)/@solvyx/molt/dist BENCH_WORK=/var/lib/bench-work BENCH_EXPORT=/root/bench-export RESULTS_DIR=/root/bench-results; chmod 700 /root && mkdir -p $BENCH_WORK && chmod 711 $BENCH_WORK && rm -rf /opt/bench && cp -a /root/bench-src /opt/bench && chmod -R go-rwx /opt/bench && cd /opt/bench && python3 run.py "$@"'
if [ "$SUBSCRIPTION" = grok ]; then
credmount="-v $HOME/.grok/auth.json:/root/grok-cred/auth.json:ro"
# grok runs as `agent`, so the credential copy lives in ITS home, owned by it, mode 600.
Expand All @@ -81,9 +99,12 @@ if [ "$OPENCODE" = 1 ]; then
startcmd='install -d -o agent -g agent -m 700 /home/agent/.local /home/agent/.local/share /home/agent/.local/share/opencode && install -o agent -g agent -m 600 /root/oc-cred/auth.json /home/agent/.local/share/opencode/auth.json && '$startcmd
fi
echo "maat $img → $out/${RESULTS:-results.jsonl}"
docker run --rm --name "maat-bench-$name" \
-v "$local_dir":/root/bench-src:ro -v "$out":/results -v "$work":/work ${=credmount} \
# Pinned CPU and memory (BENCH_CPUS, BENCH_MEMORY): perf graders have wall-clock limits, and lanes
# often run side by side on the Mac. Recorded in every row's lane.
cpus=${BENCH_CPUS:-2}; mem=${BENCH_MEMORY:-4g}
docker run --rm --name "maat-bench-$name" --cpus "$cpus" --memory "$mem" \
-v "$local_dir":/root/bench-src:ro -v "$out":/root/bench-results -v "$work":/root/bench-export ${=credmount} \
--env-file "$envf" -e REFERENCE="${REFERENCE:-0}" -e BENCH_MODEL="${BENCH_MODEL:-}" \
-e ARMS="${ARMS:-}" -e BENCH_TASKS="${BENCH_TASKS:-}" -e BENCH_URL="$url" -e BENCH_LIMIT="${BENCH_LIMIT:-600}" -e BENCH_REASONING="${BENCH_REASONING:-}" -e BENCH_GATE="${BENCH_GATE:-}" -e RESULTS="${RESULTS:-results.jsonl}" \
-e MAAT_BUILD="$sha" -e PYTHONDONTWRITEBYTECODE=1 \
-e MAAT_BUILD="$sha" -e PYTHONDONTWRITEBYTECODE=1 -e BENCH_CPUS="$cpus" -e BENCH_MEMORY="$mem" \
"$img" sh -c "$startcmd" run "$@"
10 changes: 10 additions & 0 deletions bench/local/container/set-anthropic-key.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/bin/zsh
# Store an Anthropic API key for the bench in the macOS Keychain (item: maat-bench-anthropic).
# The key is read without echo and never appears on a command line or in shell history.
print -n "Paste your Anthropic API key (input hidden): "
read -rs key; print
[[ "$key" == sk-ant-* ]] || { print "That doesn't look like an Anthropic key (sk-ant-...). Nothing saved."; exit 1; }
# Fed to `security -i` on stdin, so the key is never in any process's argument list.
print -r -- "add-generic-password -U -s maat-bench-anthropic -a $USER -w $key" | security -i >/dev/null &&
security find-generic-password -s maat-bench-anthropic >/dev/null 2>&1 && print "Saved to Keychain as maat-bench-anthropic (${#key} chars, ends …${key[-4,-1]})."
unset key
7 changes: 4 additions & 3 deletions bench/local/crosscheck.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,8 @@ def fresh() -> Path:
d = fresh()
# a check names the folder of the run that drafted it (/work/<task>-molt-<n>); in another
# run's tree that path must mean "this tree", or every foreign tree fails it spuriously
run_cmd = re.sub(r"/work/[\w.-]+-molt-\d+", str(dst), cmd)
# (runs since 2026-10-07 have a private folder: /var/lib/bench-work/<random>/<task>-molt-<n>)
run_cmd = re.sub(r"(?:/var/lib/bench-work/[0-9a-f]+|/work)/[\w.-]+-molt-\d+", str(dst), cmd)
p = subprocess.Popen(run_cmd, shell=True, cwd=d, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
stdin=subprocess.DEVNULL, start_new_session=True)
try:
Expand Down Expand Up @@ -212,7 +213,7 @@ def exec_tree(run: dict, cmds: list[str]) -> str:
if set(cmds) <= set(old["checks"]):
return "cached"
else: # results from before the /work/<folder> rewrite: redo only commands that name such a path
cmds = [c for c in cmds if "/work/" in c]
cmds = [c for c in cmds if "/work/" in c or "/var/lib/bench-work/" in c]
if not cmds:
old["pathfix"] = True
out.write_text(json.dumps(old))
Expand Down Expand Up @@ -263,7 +264,7 @@ def execute(jobs: int) -> None:

# ------------------------------------------------------------------ analysis
def sig(rc: int, out: str, run_name: str) -> str:
o = re.sub(r"/work/[\w.-]+", "/work/T", out)
o = re.sub(r"(?:/var/lib/bench-work/[0-9a-f]+|/work)/[\w.-]+", "/work/T", out)
o = re.sub(r"0x[0-9a-f]+|\b[0-9a-f]{7,40}\b", "H", o)
o = re.sub(r"\d{4,}", "N", o)
o = re.sub(r"\s+", " ", o).strip()[:300]
Expand Down
200 changes: 200 additions & 0 deletions bench/local/grading.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,200 @@
"""
What every grading path shares (run.py, regrade_today.py, crosscheck.py, the validators).

- git_env / git_guard: grader git never refuses the worker's repository for its owner, and never
runs the worker's hooks or fsmonitor (B1 of the PR #33 audit).
- child_values / PY_EMIT / emit_js: a grader that has to run the worker's code in a process only
takes computed VALUES back from it, as one JSON record on a dedicated file descriptor; the
grader process compares them with the expected values, which never enter the worker's process
(B2). A last stdout line "OK" was forgeable by an atexit hook in the worker's module.
- safe_grade: a grader that raises records a failed row flagged grader_error instead of stopping
the lane (B4).
- links_into: a symlink in the task folder that resolves into the bench's own folders (graders,
reference solutions, results, export) fails the task before any grader follows it (N1).
"""

from __future__ import annotations

import contextlib
import json
import os
import subprocess
import tempfile
from pathlib import Path

# ---------------------------------------------------------------- B1: git as the grader sees it

# Applied in the command scope (GIT_CONFIG_COUNT), which outranks the repository's own config.
GIT_GRADE_CONFIG = (("core.hooksPath", os.devnull), ("core.fsmonitor", "false"))


def git_env(d: Path | str) -> dict[str, str]:
"""
Environment for every git command a grader runs on task folder `d`:

- safe.directory=<d> (both spellings, as given and resolved), so a repository owned by the
agent user is not refused as "dubious ownership" when root grades it. Git >= 2.35.2 printed
nothing for every grader call and four git tasks failed correct work.
- core.hooksPath=/dev/null and core.fsmonitor=false: the ownership refusal used to be the only
thing keeping root's git from running the worker's hooks or fsmonitor command.
- GIT_CONFIG_GLOBAL=/dev/null: the agent user's ~/.gitconfig is the worker's to write.
"""
p = Path(d)
dirs = list(dict.fromkeys([str(p), str(p.resolve())]))
pairs = [("safe.directory", x) for x in dirs] + list(GIT_GRADE_CONFIG)
env = {"GIT_CONFIG_GLOBAL": os.devnull, "GIT_CONFIG_COUNT": str(len(pairs)), "GIT_TERMINAL_PROMPT": "0"}
for i, (k, v) in enumerate(pairs):
env[f"GIT_CONFIG_KEY_{i}"] = k
env[f"GIT_CONFIG_VALUE_{i}"] = v
return env


@contextlib.contextmanager
def git_guard(d: Path | str):
"""git_env(d) in os.environ for the duration (graders' children inherit it), then restored."""
add = git_env(d)
stale = [k for k in os.environ if k.startswith(("GIT_CONFIG_KEY_", "GIT_CONFIG_VALUE_"))]
saved = {k: os.environ.get(k) for k in [*add, *stale]}
for k in stale:
del os.environ[k]
os.environ.update(add)
try:
yield
finally:
for k, v in saved.items():
if v is None:
os.environ.pop(k, None)
else:
os.environ[k] = v


# ---------------------------------------------------------------- B2: values out of process

VALUES_FD_ENV = "BENCH_VALUES_FD"

# Prepended to every Python harness: emit(v) writes v as the one record on the values channel and
# ends the process at once (os._exit, so an atexit hook in the worker's module cannot add to it).
PY_EMIT = (
"import os as _bo, json as _bj\n"
f"_BFD = int(_bo.environ.pop({VALUES_FD_ENV!r}))\n"
"def _btag(v):\n"
" if isinstance(v, tuple):\n"
" return {'$tuple': [_btag(x) for x in v]}\n"
" if isinstance(v, list):\n"
" return [_btag(x) for x in v]\n"
" if isinstance(v, dict):\n"
" return {k: _btag(x) for k, x in v.items()}\n"
" return v\n"
"def emit(v):\n"
" b = (_bj.dumps(_btag(v), default=repr) + '\\n').encode()\n"
" while b:\n"
" b = b[_bo.write(_BFD, b):]\n"
" _bo._exit(0)\n"
"def call(f, *a, **k):\n"
" try:\n"
" return ['ok', f(*a, **k)]\n"
" except Exception as e:\n"
" return ['err', type(e).__name__, str(e)[:300], isinstance(e, ValueError), isinstance(e, TypeError)]\n"
)

# The same for a Node harness: `emit(v)`.
JS_EMIT = (
"const __bfs = require('fs');\n"
f"const __bfd = Number(process.env.{VALUES_FD_ENV});\n"
"function emit(v) { __bfs.writeSync(__bfd, JSON.stringify(v) + '\\n'); process.exit(0); }\n"
)


def child_values(cmd: list[str], cwd: Path, timeout: float, env: dict | None = None) -> tuple[object | None, str]:
"""
Run `cmd` (a harness that loads the worker's code) and return (values, diagnostic).

The harness writes exactly one JSON record to the file descriptor named by $BENCH_VALUES_FD.
values is None when there is no record, more than one, or it is not JSON; stdout and stderr
are only ever a diagnostic, never a verdict.
"""
e = dict(os.environ)
e.update(env or {})
with tempfile.TemporaryFile() as ch:
e[VALUES_FD_ENV] = str(ch.fileno())
try:
r = subprocess.run(cmd, cwd=cwd, capture_output=True, timeout=timeout, env=e, pass_fds=(ch.fileno(),))
except subprocess.TimeoutExpired:
return None, f"timed out after {timeout}s"
ch.seek(0)
raw = ch.read()
diag = (r.stdout + b"\n" + r.stderr).decode(errors="replace").strip()
diag = diag.splitlines()[-1][:240] if diag else f"exit {r.returncode}"
if not raw:
return None, diag or "no values"
lines = raw.split(b"\n")
if len(lines) != 2 or lines[1] != b"":
return None, "the values channel held more than one record"
try:
return json.loads(lines[0]), diag
except ValueError:
return None, "the values channel held no valid JSON"


def py_values(d: Path, code: str, timeout: float = 60) -> tuple[object | None, str]:
"""Run PY_EMIT + code in a fresh python3 inside d (see child_values)."""
return child_values(["python3", "-c", PY_EMIT + code], d, timeout)


def tag(v):
"""Tuples as {"$tuple": [...]}, as emit() sends them: a tuple and a list stay unequal, as they
were when the comparison ran next to the worker's code."""
if isinstance(v, tuple):
return {"$tuple": [tag(x) for x in v]}
if isinstance(v, list):
return [tag(x) for x in v]
if isinstance(v, dict):
return {k: tag(x) for k, x in v.items()}
return v


def jsonish(v):
"""v as it looks after emit()'s JSON round trip, for comparing with child values."""
return json.loads(json.dumps(tag(v), default=repr))


# ---------------------------------------------------------------- B4: a grader that raises

def safe_grade(T, d: Path) -> tuple[bool, str, bool]:
"""
(passed, why, grader_error). A grader exception is a failed row with grader_error set, never
an abort: the lane went on to the next task's free retry of this sample. A TimeoutExpired is the
worker's program hanging, an ordinary fail.
"""
try:
with git_guard(d):
ok, why = T.grade(d)
return bool(ok), str(why), False
except subprocess.TimeoutExpired as e:
return False, f"timed out: {' '.join(map(str, e.cmd)) if isinstance(e.cmd, (list, tuple)) else e.cmd}"[:300], False
except (Exception, SystemExit) as e: # noqa: BLE001 - every failure is a verdict
return False, f"grader error: {e!r}"[:2000], True


# ---------------------------------------------------------------- N1: links into the bench

def links_into(d: Path, roots: list[Path]) -> list[str]:
"""Symlinks under d (not followed) whose target resolves into one of `roots`. A link to
something inside d itself is fine, wherever d lives."""
rs = [Path(r).resolve() for r in roots if r]
home = Path(d).resolve()
bad = []
for dirpath, dirnames, filenames in os.walk(d):
for name in dirnames + filenames:
p = Path(dirpath) / name
if not p.is_symlink():
continue
try:
t = p.resolve()
except (OSError, RuntimeError):
continue
if t == home or t.is_relative_to(home):
continue
if any(t == r or t.is_relative_to(r) for r in rs):
bad.append(os.path.relpath(p, d))
return bad
2 changes: 1 addition & 1 deletion bench/local/refs2.py
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ def solve_bash_rotate(d: Path):
i=$((i + 1))
if [ "$i" -gt "$n" ]; then echo "$f"; fi
done < <(cd "$dir" && ls -t -- *.log 2>/dev/null | while IFS= read -r x; do [ -f "$x" ] && echo "$x"; done) > "${TMPDIR:-/tmp}/rot.$$"
tail -r "${TMPDIR:-/tmp}/rot.$$" | while IFS= read -r f; do rm -f -- "$dir/$f"; echo "$f"; done
{ tac "${TMPDIR:-/tmp}/rot.$$" 2>/dev/null || tail -r "${TMPDIR:-/tmp}/rot.$$"; } | while IFS= read -r f; do rm -f -- "$dir/$f"; echo "$f"; done
rm -f "${TMPDIR:-/tmp}/rot.$$"
exit 0
''')
Expand Down
Loading
Loading