diff --git a/.env.example b/.env.example index ecaab8f..1546a8d 100644 --- a/.env.example +++ b/.env.example @@ -4,6 +4,8 @@ # MODEL_SYNTH_MODEL=anthropic/claude-sonnet-5 # Strong synth model # MODEL_REVIEW_MODEL=google/gemini-3.8-flash # Finish-review model # MODEL_SYNTH_ENABLED=1 # Enable synth pass on finish +# WEBPILOT_FAST_MODE=1 # Jev picks browser actions (OpenRouter only) +# JEV_MODEL=typesafe/jev-1.13 # Jev decision model # MODEL_TIMEOUT_MS=120000 # Per-call timeout (ms) # OpenRouter credentials (https://openrouter.ai/settings/keys) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index baf8d12..0ba8b04 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -61,6 +61,23 @@ Dependabot opens a weekly grouped PR for `playwright` and `playwright-core` (sta 3. Run one real task (`npm run agent:cli -- "Go to https://example.com and tell me the page heading."`) to confirm the snapshot format still parses. 4. For releases, run the `Package Desktop` workflow to package and smoke-test the desktop app on every platform. +## Fast Mode (Jev) + +Fast mode (`fastMode` in settings, `WEBPILOT_FAST_MODE=1`) puts TypeSafe's Jev decision model in charge of each browser step. Jev returns typed decisions with calibrated probabilities in ~0.3s instead of generating text, so it replaces the LLM wherever the question is really a choice. It is reached through OpenRouter with the same API key (`/api/v1/systemone`, model `typesafe/jev-1.13`, override with `JEV_MODEL`); it is not available with Ollama. + +Modules under `lib/jev/`: + +- `client.ts`: the decision API (`choice` and `noul` questions, retries, answer parsing). +- `page.ts`: builds the page state from the accessibility snapshot alone (a numbered list of actionable elements with landmark info, plus a text excerpt). Every Playwright MCP `browser_evaluate` costs ~0.5-1s, so fast mode avoids them. Pages with more than ~250 elements are trimmed to the ones in main content that share words with the task. +- `fast-mode.ts`: the step loop. One Jev call per step asks a flat `action` choice (`click_`, `type_`, scroll, back, done, blocked) plus `goal_done`, `stuck` and `submit_after_typing`. The LLM only writes text for inputs and phrases the final answer. +- `gates.ts`: Jev yes/no gates in front of LLM calls. A preflight asks "needs the browser?" and "independent sites in parallel?" in one call, skipping the LLM router and split analysis when Jev is confident. An answer check accepts a supported final answer without the LLM reviewer; Jev never rejects on its own. + +Control returns to the LLM planner (with a summary of the steps taken) when Jev reports the task blocked or stuck, several actions in a row make no visible change, the next click looks irreversible (buy, pay, delete, send...), there is no starting page, or the step/time budget runs out. Actions that made no change are removed from the options before Jev decides again. + +Each Jev step is recorded with `source: "jev"`, and `stepN_jev.json` artifacts keep Jev's probabilities for tuning thresholds. `npm run bench:fast-mode` compares fast mode with the LLM-only agent on the same tasks. + +Privacy: in fast mode the page text excerpt, element labels and URLs of every step are sent to TypeSafe (through OpenRouter). + ## Tool Schema Versioning The agent exposes a stable WebPilot browser tool schema from `lib/tool-schema.ts` instead of scattering raw provider payloads through the runtime. The current schema version is `webpilot.browser-tools.v1`. diff --git a/README.md b/README.md index 278a22c..09dfefd 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,7 @@ WebPilot is meant to be a hackable, local-first alternative to closed agentic br - Desktop app built with Electron and Next.js. - Browser automation through Playwright MCP, running on the latest stable Playwright. +- Fast mode: TypeSafe's Jev decision model (through OpenRouter) picks each browser action in ~0.3s while the LLM only writes text, with automatic fallback to the LLM planner. - Model providers: OpenRouter (one key for Gemini, Claude, GPT, Qwen, DeepSeek and hundreds more) and local Ollama. - Runtime settings UI for provider, model, browser source, profile strategy, headless mode, and isolation. - Local run recording with logs, step traces, artifacts, timing, and final results. @@ -79,6 +80,18 @@ MODEL_BASE_URL=http://127.0.0.1:11434/v1 The settings UI can discover local Ollama models and hides known embedding-only models when Ollama reports enough metadata to identify them. +## Fast Mode + +Turn on **Fast mode (Jev)** in Settings (OpenRouter only). Instead of asking the LLM to plan every click, WebPilot asks [Jev](https://openrouter.ai/typesafe/jev-1.13), a decision model that returns a choice with probabilities in about 0.3 seconds. The LLM still writes anything that needs text (search queries, the final answer), and the regular planner takes over whenever Jev is stuck, blocked, or about to click something irreversible. + +Measure it on your own tasks: + +```bash +OPENROUTER_API_KEY=sk-or-... npm run bench:fast-mode +``` + +In fast mode the page text and element labels of each step are sent to TypeSafe through OpenRouter. + ## Browser Modes The app supports these runtime browser modes: @@ -103,6 +116,7 @@ npm run browsers:install npm run browser:smoke npm run health npm run agent:cli -- "Go to https://example.com and summarize the page" +npm run bench:fast-mode npm run mcp:stdio npm run desktop:build ``` diff --git a/RUNNING.md b/RUNNING.md index 9bc576a..cc26eed 100644 --- a/RUNNING.md +++ b/RUNNING.md @@ -64,6 +64,8 @@ Optional: - `MODEL_NAV_MODEL=...` - `MODEL_SYNTH_MODEL=...` - `MODEL_SYNTH_ENABLED=1` +- `WEBPILOT_FAST_MODE=1` (Jev picks browser actions; OpenRouter only) +- `JEV_MODEL=typesafe/jev-1.13` - `RUN_STORE_DIR=./agent_runs` - `THREAD_STORE_DIR=./agent_threads` - `MAX_STEPS=160` diff --git a/TESTING.md b/TESTING.md index 8a833bf..598ee8e 100644 --- a/TESTING.md +++ b/TESTING.md @@ -22,6 +22,18 @@ This verifies the versioned browser tool schema adapter against current WebPilot It also covers the OpenRouter/Ollama model client with a mocked `fetch`: request URLs and headers, tool-call round trips including `reasoning_details`, retries on rate limits, error messages, migration of legacy Gemini/OpenAI/Claude settings, and parsing of the OpenRouter models catalog. No API key or network access is needed. +`tests/jev.test.ts` covers fast mode without network access: accessibility-snapshot parsing, candidate selection, the Jev client, the step loop against a fake browser (done, no-progress handoff, irreversible-click guard, typing with submit), and the Jev gates. + +## Fast Mode Benchmark + +```bash +npm run browsers:install +OPENROUTER_API_KEY=sk-or-... npm run bench:fast-mode +npm run bench:fast-mode -- --only mosaic,voyager --modes fast --repeat 3 +``` + +Runs a fixed set of public-web tasks (Wikipedia, MDN, example.com) headless in both the LLM-only mode and fast mode, checks each final answer against expected patterns, and writes JSON and Markdown reports with time, steps, LLM calls, Jev calls and fast-mode handoffs to `e2e_reports/`. Model latency on OpenRouter varies run to run, so compare medians over `--repeat 3` or more before drawing conclusions. + ## Deterministic Browser Smoke ```bash diff --git a/app/activity/[runId]/page.tsx b/app/activity/[runId]/page.tsx index 95f3d83..e87ca11 100644 --- a/app/activity/[runId]/page.tsx +++ b/app/activity/[runId]/page.tsx @@ -135,6 +135,8 @@ const PERFORMANCE_LABELS: Record = { synthDurationMs: "Synth time", coordinatorCallCount: "Coordinator calls", coordinatorDurationMs: "Coordinator time", + jevCallCount: "Jev decisions", + jevDurationMs: "Jev time", }; function formatSize(sizeBytes?: number) { diff --git a/components/settings/ProviderForm.tsx b/components/settings/ProviderForm.tsx index 9505560..3a73952 100644 --- a/components/settings/ProviderForm.tsx +++ b/components/settings/ProviderForm.tsx @@ -502,6 +502,20 @@ export function ProviderForm() { label="Synthesis enabled" /> + {provider === "openrouter" && ( +
+ update({ fastMode: v })} + label="Fast mode (Jev)" + /> +

+ Jev, a fast decision model on OpenRouter, picks each browser action in about 0.3s and the navigation model only writes text. + It falls back to the normal planner when stuck. Page text and element labels are sent to TypeSafe for each step. +

+
+ )} +