Skip to content

Fast mode: Jev picks browser actions, the LLM only writes text - #35

Merged
ahamsel merged 1 commit into
mainfrom
claude/jev-fast-mode
Sep 28, 2026
Merged

ahamsel merged 1 commit into
mainfrom
claude/jev-fast-mode

Conversation

@ahamsel

@ahamsel ahamsel commented Sep 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

Builds on #33 (OpenRouter) and #34 (Playwright 1.63), both merged.

Adds fast mode: Jev, TypeSafe's "System One" decision model, picks each browser action through OpenRouter with the same API key. Jev doesn't generate text. It answers typed questions (choose one of N, yes/no) with calibrated probabilities in ~0.3s, so it replaces the LLM wherever the question is really a choice. The LLM still writes anything textual (search queries, the final answer), and the regular LLM planner takes over whenever Jev can't finish. Off by default; Fast mode (Jev) toggle in Settings (OpenRouter only).

Step loop (lib/jev/fast-mode.ts): one Jev call per step asks a flat action choice (click_<ref>, type_<ref>, scroll, back, done, blocked) plus goal_done, stuck and submit_after_typing in the same call. Hands off to the LLM planner (with a summary of the steps taken) when Jev reports blocked or stuck, repeated actions change nothing, the next click looks irreversible (buy, pay, delete, send...), there's no starting page, or the step/time budget runs out. Actions that errored or changed nothing are removed from the options, per URL.

Page state (lib/jev/page.ts): built from the accessibility snapshot alone, because each Playwright MCP browser_evaluate costs ~0.5-1s. Elements carry landmark info (main content vs site navigation); unnamed links take their child text (docs sites like MDN wrap link text in <code>); pages with more than ~250 actionable elements are trimmed to task-relevant main content.

Jev gates in front of LLM yes/no calls (lib/jev/gates.ts):

  • Preflight (one call): "needs the browser?" skips the LLM router when confident (first turn of a thread only, since follow-ups need the router to expand them) and "independent sites in parallel?" skips the LLM split analysis.
  • Answer check: accepts a final answer supported by the page without the LLM reviewer. Jev never rejects on its own; an unconvinced Jev defers to the LLM reviewer.

Observability: Jev decisions are logged (jev_decision) and saved as stepN_jev.json artifacts with probabilities for tuning thresholds; steps are recorded with source: "jev"; the performance summary counts Jev calls and time.

Benchmark (npm run bench:fast-mode): runs fixed public-web tasks headless in both modes and checks answers against expected patterns.

LLM-only Fast mode
Passed 8/8 8/8
Median time 19.0s 4.3s
Total 200.6s 68.8s

One run per task per mode with google/gemini-3.8-flash; OpenRouter latency varies run to run, so treat these as indicative. In that run the MDN task handed off to the LLM because the map() link had no accessible name and was dropped by the parser, and a failing click was retried; both are fixed here, and MDN now completes in fast mode.

Privacy: in fast mode the page text excerpt, element labels and URLs of every step are sent to TypeSafe through OpenRouter. Documented in README, ARCHITECTURE and next to the Settings toggle.

Testing

  • npm run build
  • npm run health
  • Electron syntax checks if electron/ changed (node --check electron/main.cjs)
  • UI smoke test if visible behavior changed: Settings shows the toggle for OpenRouter only; it saves and survives reload
  • npm test (28 pass; tests/jev.test.ts covers the parser, candidate selection, Jev client, step loop against a fake browser, and the gates), npm run lint, npx tsc --noEmit
  • Live runs through OpenRouter + Jev: the 8-task benchmark above, a chat message ("hey, what can you do?") correctly answered without the browser, and a no-URL search task

Checklist

  • No API keys, cookies, browser profile data, or local run artifacts are included
  • Documentation is updated when behavior or setup changed
  • Screenshots or video are attached for meaningful UI changes

🤖 Generated with Claude Code

@ahamsel
ahamsel force-pushed the claude/jev-fast-mode branch from f71e9f8 to 2410a14 Compare September 28, 2026 17:20
@ahamsel
ahamsel changed the base branch from claude/playwright-upgrade to main September 28, 2026 17:20
Fast mode lets TypeSafe's Jev decision model (through OpenRouter, same
key) pick each browser action, while the LLM only writes typed text and
the final answer. Off by default; toggle in Settings (OpenRouter only).

- lib/jev/client.ts: System One decision API (choice/noul questions)
- lib/jev/page.ts: page state from the accessibility snapshot alone
  (every browser_evaluate costs ~0.5-1s); unnamed links take their
  child text; large pages are trimmed to task-relevant main content
- lib/jev/fast-mode.ts: one Jev call per step with a flat action
  choice plus goal_done/stuck/submit; failed or no-op actions are
  excluded per URL; handoff to the LLM planner when blocked, stuck,
  repeating, over budget, or before irreversible-looking clicks
- lib/jev/gates.ts: Jev preflight skips the LLM router (first turn
  only) and split analysis when confident; answer check accepts a
  supported answer without the LLM reviewer (never rejects alone)
- Jev decisions logged and saved as step artifacts; performance
  summary counts Jev calls
- scripts/bench-fast-mode.ts (npm run bench:fast-mode) compares
  fast mode with the LLM-only agent on fixed public-web tasks

Benchmark (8 tasks, 1 run each): LLM-only 8/8, median 19.0s;
fast mode 8/8, median 4.3s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ahamsel
ahamsel force-pushed the claude/jev-fast-mode branch from 2410a14 to 214fe93 Compare September 28, 2026 17:20
@ahamsel
ahamsel merged commit cc19c40 into main Sep 28, 2026
1 check passed
@ahamsel
ahamsel deleted the claude/jev-fast-mode branch September 29, 2026 00:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant