Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,7 @@ out-cli/
*.tgz
.molt/out/
bench/local/work/

# raw model output from bench probes stays local (xAI's AUP forbids using Grok
# output to develop ML models, so transcripts are never committed)
bench/local/ambiguity-results/
17 changes: 15 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,18 @@
# Changelog

## Unreleased

### Changed

- **OpenCode runs OpenCode Zen models only.** The OpenCode backend (worker and
judge) accepts `opencode/<model>` (e.g. `opencode/big-pickle`) and refuses
any other provider at parse time and at run time; the CLI is handed a config
that enables only the `opencode` provider, and no other provider's
credentials.
- **`opencode://zen`** is the OpenCode endpoint (short name `opencode`). The old
`opencode://subscription` still works for this release, with a deprecation
notice.

## 0.2.2 — subscription backends removed

### Removed
Expand All @@ -9,8 +22,8 @@
written. Stale `claude-code://`, `antigravity://`, `agy://`, and
`gemini-cli://` configs now fail with a clear removal message. Grok Build
(ACP) and the metered xAI API provider remain.
- **Droid benchmarking** from the harbor/local bench path, and **xAI rows**
from the committed fine-tune dataset (provider-terms alignment). See
- **xAI rows** from the committed fine-tune dataset, and third-party agent
adapters from the bench path (provider-terms alignment). See
`docs/provider-terms.md`.

### Changed
Expand Down
13 changes: 13 additions & 0 deletions bench/harbor/compare.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@
from datetime import datetime
from pathlib import Path

# Shared with bench/local/scoreboard.py (Wilson intervals, one definition).
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "local"))
from stats import rate # noqa: E402


def outcome_of(trial: Path) -> str | None:
"""
Expand Down Expand Up @@ -215,6 +219,15 @@ def prec(h):
return f"{100 * h['claimed_right'] / (h['claimed_right'] + h['claimed_wrong']):.0f}%"

print(f"{' done was true':16} {prec(ha):>{w}} {prec(hb):>{w}}")
def pr(h, kind):
if h is None:
return "—"
r, wr, un = h["claimed_right"], h["claimed_wrong"], h["passed_unclaimed"]
return rate(r, r + wr) if kind == "p" else rate(r, r + un)

# Wilson 95% intervals: at 89 tasks a 5-task gap is noise, so say how wide.
print(f"{'precision':16} {pr(ha, 'p')} | {pr(hb, 'p')} P(pass | said done)")
print(f"{'recall':16} {pr(ha, 'r')} | {pr(hb, 'r')} P(said done | pass)")
only_a, only_b = sorted(set(ta) - set(tb)), sorted(set(tb) - set(ta))
if only_a or only_b:
print(f"not compared — only in {na}: {len(only_a)} task(s); only in {nb}: {len(only_b)} task(s)")
Expand Down
13 changes: 11 additions & 2 deletions bench/harbor/molt_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ def __init__(
steps: int = 300,
batch: bool | str = False,
reveal_stuck: bool | str = False,
reference: bool | str = False,
reasoning_checks: str | None = None,
reasoning_retry: str | None = None,
review: int | str | None = None,
Expand Down Expand Up @@ -104,6 +105,8 @@ def __init__(
# Batch mode: one act call per reply carrying a list of actions.
self._batch = str(batch).lower() in ("1", "true", "yes", "on")
self._reveal_stuck = str(reveal_stuck).lower() in ("1", "true", "yes", "on")
# An independent reference check (src/reference.ts); needs python3 in the image.
self._reference = str(reference).lower() in ("1", "true", "yes", "on")
# Effort for drafting checks only, and for steps after a refusal.
self._reasoning_checks = reasoning_checks
self._reasoning_retry = reasoning_retry
Expand Down Expand Up @@ -230,10 +233,14 @@ async def run(
flags.append("--criteria auto")
if self._for:
flags.append(f"--for {shlex.quote(str(self._for))}")
elif self._pace:
else:
# Every run gets a deadline: a run the harness kills at its limit
# ends with no verdict, while one stopped by --for is judged on the
# tree it left. Paced runs keep a 15% margin; the rest stop a
# minute before the limit, enough to run the sealed checks.
limit = self._task_timeout_sec()
if limit:
flags.append(f"--for {max(60, int(limit * 0.85) - 60)}s")
flags.append(f"--for {max(60, int(limit * 0.85) - 60) if self._pace else max(60, int(limit) - 60)}s")
if self._map_tokens is not None:
flags.append("--no-map" if int(self._map_tokens) <= 0 else f"--map {int(self._map_tokens)}")
if self._max_tokens is not None:
Expand All @@ -247,6 +254,8 @@ async def run(
flags.append(f"--steps {self._steps}")
if self._batch:
flags.append("--batch")
if self._reference:
flags.append("--reference")
if self._reveal_stuck:
flags.append("--reveal-stuck")
if self._reasoning_checks:
Expand Down
87 changes: 87 additions & 0 deletions bench/local/ambiguity_probe.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
"""
Ambiguity probe experiment (frontier report, Bet 3).

python3 ambiguity_probe.py run <out.jsonl> [task,task|all] [reps] # Grok 4.7, inside the container only
python3 ambiguity_probe.py show <out.jsonl>

The model sees ONLY the task prompt and the pristine file listing (names + sizes, no contents,
no grader). `run` builds each task's starting folder on the host (setup only; no model runs on
the host), then runs ambiguity_probe_driver.mjs in a throwaway container named maat-bench-ab3-*
with the owner's Grok login mounted read-only outside HOME (as run-in-container.sh does).
Never run grok on the host: its config auto-approves its own tools.
Output files hold raw Grok text: keep them in ambiguity-results/ (gitignored), never commit them
(xAI's AUP forbids using its output to develop ML models).
"""
from __future__ import annotations
import json, os, subprocess, sys, tempfile
from pathlib import Path

HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE))
from tasks import TASKS # noqa: E402
from tasks2 import TASKS2 # noqa: E402

ALL = {T.name: T for T in TASKS + TASKS2}
IMG = os.environ.get("PROBE_IMAGE", "maat-bench-grok:6d91d673")
SYSTEM = (HERE / "ambiguity_probe_prompt.txt").read_text()


def listing(d: Path) -> str:
rows = []
for p in sorted(d.rglob("*")):
rel = p.relative_to(d)
if ".git" in rel.parts or not p.is_file():
continue
rows.append(f"{rel} ({p.stat().st_size} bytes)")
return "\n".join(rows) or "(empty folder)"


def user_prompt(T) -> str:
with tempfile.TemporaryDirectory() as t:
d = Path(t)
T.setup(d)
files = listing(d)
return f"TASK:\n{T.PROMPT}\n\nFILES PRESENT (name and size):\n{files}\n"


def run(out: Path, names: list[str], reps: int) -> None:
jobs = [{"id": f"{n}#{r}", "system": SYSTEM, "prompt": SYSTEM + "\n\n----\n\n" + user_prompt(ALL[n]) + "\nReply now with the JSON object only."} for n in names for r in range(reps)]
work = Path.home() / ".cache/maat-bench/container-work/ab3"
work.mkdir(parents=True, exist_ok=True)
(work / "jobs.json").write_text(json.dumps(jobs))
(work / "out.jsonl").unlink(missing_ok=True)
cmd = ["docker", "run", "--rm", "--name", f"maat-bench-ab3-{os.getpid()}",
"-e", f"PROBE_TIMEOUT_MS={os.environ.get('PROBE_TIMEOUT_MS', '240000')}", "-e", f"PROBE_CONC={os.environ.get('PROBE_CONC', '3')}", "-v", f"{HERE}:/bench:ro", "-v", f"{work}:/work",
"-v", f"{Path.home()}/.grok/auth.json:/grok-cred/auth.json:ro", IMG, "sh", "-c",
'mkdir -p $HOME/.grok /tmp/empty && install -m 600 /grok-cred/auth.json $HOME/.grok/auth.json && '
'cd /tmp/empty && '
'node /bench/ambiguity_probe_driver.mjs /work/jobs.json /work/out.jsonl']
subprocess.run(cmd, check=True)
with out.open("a") as f:
f.write((work / "out.jsonl").read_text())


def parse(text: str) -> list[dict]:
s, e = text.find("{"), text.rfind("}")
try:
return json.loads(text[s:e + 1]).get("probes", [])
except Exception:
return []


def show(out: Path) -> None:
for line in out.read_text().splitlines():
r = json.loads(line)
ps = parse(r.get("text", "")) if r.get("ok") else []
print(f"== {r['id']} ok={r.get('ok')} probes={len(ps)} {r.get('error','')[:100]}")
for p in ps:
print(f" - [{p.get('dimension')}] {p.get('input')!r}: A={p.get('outputA')!r} B={p.get('outputB')!r}")


if __name__ == "__main__":
cmd = sys.argv[1]
if cmd == "run":
names = sys.argv[3].split(",") if len(sys.argv) > 3 and sys.argv[3] != "all" else list(ALL)
run(Path(sys.argv[2]), names, int(sys.argv[4]) if len(sys.argv) > 4 else 1)
else:
show(Path(sys.argv[2]))
19 changes: 19 additions & 0 deletions bench/local/ambiguity_probe_driver.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
// Runs INSIDE the container: reads jobs.json [{id,system,prompt}], writes one line per ask to out.jsonl.
// Uses the installed Maat's askModel (the same call the criteria drafter makes), no tools.
import { readFileSync, appendFileSync } from "node:fs";
import { execSync } from "node:child_process";
const root = execSync("npm root -g").toString().trim();
const { askModel } = await import(`${root}/@solvyx/molt/dist/ask.js`);
const [jobsFile, outFile] = process.argv.slice(2);
const jobs = JSON.parse(readFileSync(jobsFile, "utf8"));
const conc = Number(process.env.PROBE_CONC || 3);
let i = 0;
async function worker() {
while (i < jobs.length) {
const j = jobs[i++];
const t0 = Date.now();
const r = await askModel({ baseUrl: "grok-build://subscription", model: "grok-4.7", system: j.system, prompt: j.prompt, maxTokens: 6000, reasoningEffort: process.env.PROBE_EFFORT || "low", cwd: "/tmp/empty", what: "probing ambiguity", timeoutMs: Number(process.env.PROBE_TIMEOUT_MS || 240000) });
appendFileSync(outFile, JSON.stringify({ id: j.id, secs: Math.round((Date.now() - t0) / 1000), ...r }) + "\n");
}
}
await Promise.all(Array.from({ length: conc }, worker));
14 changes: 14 additions & 0 deletions bench/local/ambiguity_probe_prompt.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
The files are NOT available to you (your working folder is empty): do not read, search or run anything. Everything you may use is in the message. Your FIRST and ONLY reply must be the JSON object, with no preamble.

You review a task given to a coding agent BEFORE any work starts. You see only the task text and the listing of the files that exist. Find the places where two reasonable readings of the task would make a correct-looking deliverable produce DIFFERENT OUTPUT on the same input.

List up to 8 probes. A probe is one concrete input or situation (a literal value, a line, a date, a number, an edge case), the two readings, and what each reading would output for that input. Prefer boundary cases: off-by-one in what counts, inclusive/exclusive ranges, rounding and formatting, whether a match must be a whole token or may be a substring, how two rules combine (and/or), empty or malformed input, ordering and ties, what counts as a status/category.

Rules:
- Only real divergence: if both readings give the same output on the input, it is not a probe.
- Do not list generic engineering concerns (error handling, performance, style, "should I add tests").
- Give literal inputs, not descriptions of inputs. Quote the task words each reading rests on.
- Fewer probes is better than padding. Return an empty list if the task is unambiguous.

Reply with JSON only:
{"probes":[{"input":"...","readingA":"...","outputA":"...","readingB":"...","outputB":"...","quote":"task words the ambiguity rests on","dimension":"one short phrase"}]}
Loading
Loading