[Eval] Add a small Online-Mind2Web live slice for browser agents (distinct from offline myjev steps) #116

Motivation

Browser numbers in this repo come from a handful of live tasks (WebVoyager Allrecipes task 0, Google Flights) plus the offline step-level collector in #109 / PR #111 on Mlchemist/myjev-browser-decisions. That offline leg answers "given this frozen state, does the backend pick the right operation/element?". It does not answer "does the full agent finish a realistic live-web task?".

#13 asks for repeatable browser/desktop suites in general; #104 is one external Anthropic browser_toolset baseline on Allrecipes only. What is still missing is a small, fixed live-web slice whose tasks and sites are already curated by the community for exactly that end-to-end question.

Online-Mind2Web (OSU NLP; paper An Illusion of Progress? Assessing the Current State of Web Agents) is 300 tasks across 136 real websites, with human / WebJudge-style judging and an explicit easy/medium/hard split by step count. Industry write-ups still treat it as the live-web counterpart to self-hosted WebArena clones (Steel WebArena leaderboard context, Notte glossary).

A 10–20 task seeded subset (prefer easy, public-domain-safe sites the agent already has recipes for) would give Agents a second live denominator next to Allrecipes without standing up WebArena infrastructure.

Proposed work

1. Pick a fixed, seeded list of ≤20 Online-Mind2Web task ids (document the seed and the easy/medium mix). Prefer sites that do not require authenticated sessions; skip or mark tasks that need login/payment.
2. Add a thin runner (script or benchmarks/legs/ entry) that: starts from a clean browser profile, runs s1a run (or the browser agent entrypoint) with a pinned --model and budget, saves the job folder, and records success under the same independent check style used for Allrecipes (or WebJudge if we adopt their auto-judge — say which).
3. First backend: hosted Jev (text). Optional second row: OmniJev / OneJev once images are on. Count successes, failures, timeouts, BLOCKED, wall time, decision p50/p95, cost if any.
4. Write docs/results/online-mind2web/ with the task id list, Agents SHA, date, and raw traces (redact keys). Point from docs/benchmarks.md. Do not vendor the full 300-task set if the upstream license forbids redistribution; download at run time and pin the revision.

Acceptance criteria

• Task id list and seed are committed; every run names Agents SHA, model/checkpoint, budget and date.
• Denominators include failures and timeouts (no silent drops).
• Clearly labelled as a live suite, separate from [Eval] Score browser decision backends step by step on a public Mind2Web/WebLINX set in the jev-ultrafast request format #109 offline step accuracy and from [Eval] Add an Anthropic browser_toolset_20260801 baseline on WebVoyager Allrecipes task 0 #104's Anthropic baseline.
• License / attribution for Online-Mind2Web and any judge model are in NOTICE / the results README.

References

• https://github.com/osu-nlp-group/online-mind2web
• https://arxiv.org/abs/2504.01382
• https://leaderboard.steel.dev/leaderboards/webarena/
• https://www.notte.cc/glossary/ai-browser-agents/what-are-browser-agent-benchmarks
• Related: [Eval] Add repeatable browser and desktop task suites #13, [Eval] Score browser decision backends step by step on a public Mind2Web/WebLINX set in the jev-ultrafast request format #109, [Eval] Add an Anthropic browser_toolset_20260801 baseline on WebVoyager Allrecipes task 0 #104, [Feat] Score browser decision backends step by step on myjev-browser-decisions (offline, any text backend) #111