[Eval] Score browser decision backends step by step on a public Mind2Web/WebLINX set in the jev-ultrafast request format #109

Why

Every browser number we publish today comes from a few live tasks (Allrecipes task 0, Google Flights), so comparing decision backends on the browser front is slow, noisy and depends on live sites changing. #13 asks for repeatable task suites; that is end-to-end. What is missing is the step level: "given this page state and goal, does the backend pick the right operation and element, and is its confidence any good?" That number is what #94 / #107's deferral threshold and #89's confidence semantics actually need, and it can be run offline in minutes.

Public step-level sets for exactly this shape now exist:

• Mlchemist/myjev-browser-decisions (updated 2026-10-18): browser steps from Mind2Web, WebLINX and others, converted into the request format that browser-use/jev-ultrafast sends to Jev (the tick our browser front follows, per docs/browser-front.md), with gold operation and element keys. It has website-disjoint calibration (652) and frozen benchmark (3,345, human labels only, audited) splits, and lists the 3,239 benchmark ids hosted Jev 1.13 answered. License CC BY-NC-SA 4.0, so it must be downloaded at run time, never vendored.
• JevBench VE-003 / VE-004: pre-registered Mind2Web next-step selection over all 9,378 test steps (Jev 1.13: 38.4% and 42.3%), a public reference point for hosted Jev.
• Mind2Web defines the metrics: element accuracy, operation F1 and step success, each step scored with the gold history.

Proposed scope (one PR, offline, no live sites)

1. A collector under benchmarks/ in the style of leg1_public231: it reads the dataset from a pinned Hugging Face revision with hash checks, sends each step through our DecisionModel layer (so any text backend works: jev, laya, laya-served, cua, clm, a local System One URL), saves every request and response, one request at a time, no retries.
2. Verify, don't assume, that the dataset's request body matches what BrowserDecisionModel builds. If they differ, convert into our shape and document the difference, so the benchmark measures the model on the questions our agent really asks.
3. A separate re-scoring command: operation accuracy, element accuracy, step success, Brier and ECE of the operation confidence, invalid-answer rate, p50/p95 latency; scored on the Jev-answered subset as well as the full split, with denominators shown.
4. Use the calibration split to emit the per-backend threshold table that [Feat] Add confidence-gated browser deferral #107's scripts/calibrate_browser_deferral.py would otherwise need live runs for.
5. One results file with hosted Jev plus at least one open backend, recording backend revision, hardware and Agents SHA.

Acceptance criteria

• No dataset rows are committed; the license and the Mind2Web authors' no-republish request are stated in the README of the benchmark.
• Backends that fail or time out on a step count as wrong, not skipped.
• The doc says plainly that step success is not task success and does not replace [Eval] Add repeatable browser and desktop task suites #13.
• Tests run on a tiny synthetic fixture in the same format, with no network.

Out of scope

• Screenshot backends (omnijev, onejev, jev-vl-served): the set has no images. A multimodal arm can follow.
• Training or fine-tuning on the train split.

Related

#13 (task suites), #54 (roadmap), #89 / #90 (confidence semantics and option order), #94 / #107 (deferral threshold), and #108 (Kev qualification).