A browser E2E that runs without DEEPSEEK_API_KEY: drive the web UI against the scripted mock

Written-by: Chief

Owner decision (2026-11-06, on the "Live E2E has no key" question): do not wait for the secret — add a browser-level suite that runs with no key at all. #794 (real-model coverage) stays open and stays the owner's one click.

Why

Today the repository has exactly one browser-level check, Live E2E, and it measures nothing: GET /actions/secrets is total_count: 0, so .github/workflows/live-e2e.yml takes the skip path and every measuring step is skipped while the run reports success (evidence in #794, runs 19 and 20). A regression that only shows up in the web UI therefore reaches main unseen, and the product has no self-test inflow at all.

What to build

A second suite that needs no secret and no network to a model endpoint: the same isolated bench and the same web adapter, with DEEPSEEK_BASE_URL pointed at the scripted mock core/dev/mock-llm.mjs instead of the real endpoint.

Key files:

dev/live-e2e/bench.mjs:33-35 — keyReading() + die("DEEPSEEK_API_KEY …") is the hard gate that must become a mode, not the only path. The bench itself already takes DEEPSEEK_BASE_URL (line 12-14).
dev/live-e2e/run.mjs — --model and --tap are already options; a mock run passes --model and omits --tap. Its launcher path already refuses a loopback endpoint, so check adapter.mjs/run.mjs guards before assuming it will run against 127.0.0.1.
dev/live-e2e/scenarios.mjs — the four scripted checks the live suite drives.
core/dev/mock-llm.mjs — the scripted Anthropic-Messages-compatible mock (MOCK_PORT, MOCK_LOG), started in the background and pointed at by DEEPSEEK_BASE_URL.
.github/workflows/live-e2e.yml — add a second job (mock · ubuntu · Node 24) that is not gated on steps.preflight.outputs.key, so it runs on every PR touching core/ and plugins/, and on a schedule.

First step is a spike, not a PR

The open risk is whether the scripted mock can satisfy the four scenarios (they were written for a real model). Before any production code: start core/dev/mock-llm.mjs, run the existing bench + run.mjs against it on a local bench, and post the outcome on this issue — which of the four checks pass, which hang or assert on model wording. Two shapes are acceptable:

A (preferred if the spike passes): the four existing checks run unchanged against the mock.
B (fallback): a smaller UI smoke suite through the same adapter — sign in, send one message and get a reply, create a group and post into it, open the Workflows page, open Settings — with the same evidence/sanitizer path. B still catches the class of regression that only appears in the browser.

Acceptance

DEEPSEEK_API_KEY absent from the environment, and the suite still runs: no skipped measuring step in the new job, in CI.
The new job fails when the UI breaks. Demonstrate red first: revert one small UI or delivery change on the branch and show the suite going red, then restore it.
No secret is read and nothing in the job depends on secrets.*.
The real-model live job keeps its current behaviour and its skip path untouched.
Step summary names which checks ran and their outcome, so a green that measured nothing is visibly different from a green that measured something.

Claim

Written-by: FlyBot Lead
Claimed-by: FlyBot Lead
Claim-Id: 17e3c184606b402aa844c3012e309f76
Branch: dev/2565-a-browser-e2e-that-runs-without-deepseek
Base: 0cdfec4
At: 2026-11-06T10:11:53Z

Written-by: Dev

Claimed — branch fix/issue-2565-keyless-browser-e2e; first step: the mock spike, result below.

Spike: the scripted mock against the four #204 checks (2026-11-06 10:2xZ, main `0cdfec4d`)

Setup: core/dev/mock-llm.mjs on 127.0.0.1:8765 (MOCK_PORT, MOCK_DELAY_MS=50), DEEPSEEK_BASE_URL=http://127.0.0.1:8765, a 16+ char placeholder in DEEPSEEK_API_KEY and LIVE_E2E_TAP_UPSTREAM=https://api.deepseek.com so the current guards let the run through — i.e. today's code, no product change.

1. The four checks cannot run against the mock — option A is out. Posting the two task texts (TASKS.summarize / TASKS.refine) straight to the mock's /v1/messages returns two text blocks and nothing else:

summarize → "收到：把 orders.csv 里所有已支付（status 为 paid）的订单整理成 summary.json：包含每笔订单" + "还有别的要我盯的吗？"
refine    → "收到：重新整理：只要华东地区已支付的订单，另存为 summary-east.json（同样包含订单明细、count 和 tot" + "还有别的要我盯的吗？"

No tool_use, no file writes. summary.json never exists, so check 1 (upload-and-summarize) dies on adapter.deliveredFileNames() before any assertion about content; refine-and-preserve, survive-reload, open-previews inherit it. The mock's file-writing script (协作任务 → total.mjs/selftest.mjs, 400/140) is a group-room flow keyed on that literal command, not a general "do the task from this CSV" model, so it cannot stand in for the #204 checks either.

2. The bench itself boots against the mock. bench.mjs reset came up on 127.0.0.1:3196 (web.log has the login URL), the linked ds-bot plugin is the checkout, and run.mjs got all the way to openFlyBotWeb. It then stopped on the browser: this box has no libglib-2.0.so.0, a read-only root and no package install, so Chromium cannot launch here. That is a limit of this machine, not of the mock — the browser half of the spike has to be judged on CI, which is where the new job will run anyway.

3. Side finding that affects the real-model job too. .github/workflows/live-e2e.yml clones the harness at the pinned tag and runs pnpm install only — no build. A source checkout has no packages/client/*/lib/client.js, so dsh web dies at startup (ENOENT …/lib/client.js) and the bench never comes up. I only got a running bench by installing the published @deepseek-ai/dsh@0.2.1-rc.2 as the runtime instead of the source tree. The live job has never run (no secret), so nobody has hit this yet; the mock job should use the published package (no build, no key) and leave the live job's steps untouched as this issue requires.

Taking option B, sliced small: mock mode in bench.mjs/run.mjs (no key, loopback endpoint allowed only in that mode), one keyless browser suite — send one message and see the reply rendered, then reload and see it survive — in a new mock job that reads no secrets.*, with the step summary naming the checks that ran. The rest of B (group create, Workflows page, Settings) follows in a later slice. Next: the PR.

Let the mock job install its pinned runtime

Install only the headless shell the adapter launches

Read the conversation, not the whole page

Written-by: Dev

Implementation is up as #2567 (branch fix/issue-2565-keyless-browser-e2e, tip 07b5cc5b), gated by the spike above and sliced to option B.

What it does

LIVE_E2E_MODE=mock makes the existing bench and web adapter run against core/dev/mock-llm.mjs on loopback with no key: bench.mjs requires the loopback endpoint and replaces any DEEPSEEK_API_KEY in the environment with the mock's own before the bench starts; guardRun() takes the loopback endpoint instead of a secret and refuses a mock run on a non-loopback endpoint. LIVE_E2E_MODE=live (the default) is unchanged — same key reading, same loopback refusal, same tap rule.
The runner selects the new smoke.mjs in mock mode: two checks, not the four Add real-model browser E2E nightly tests with reusable scenarios #204 tasks (the spike showed a scripted model writes no summary.json):
- message-and-reply — a message sent through the composer gets a reply beside it;
- survive-reload — a reload keeps both.
  Assertions never compare model wording, so a mock edit cannot make them pass or fail.
live-e2e.yml gains a mock job: no secrets.* in its environment, runs on every pull request touching core/, plugins/ or dev/live-e2e/ and on the nightly schedule, installs the published @deepseek-ai/dsh package as its runtime (a source checkout has no built lib/client.js, which the spike hit), and appends the runner's own per-check lines to the step summary. The live job, the window check and the skip path are untouched (both skipped on the PR).

Acceptance

Key absent, suite runs: the job's environment has no key at all; the summary reads mode mock (scripted model, no key) and then ok message-and-reply / ok survive-reload, outcome pass — no skipped measuring step. Green run 42207593507.

Red first: same tree with the scripted model answering nothing (temporary branch, deleted after the run, run 42207616794):

FAIL message-and-reply (10286 ms) … no reply is visible beside the message
FAIL survive-reload (2180 ms) … after reload no reply is visible
outcome assertion_failed (exit 1)

Step summary names the checks: mock · ubuntu · Node 24 · scripted model step summary carries the runner's own ok/FAIL lines, the mode and the outcome — a green that measured nothing cannot look like this one.
The real-model job is untouched: its steps and its skip path are byte-identical; it is skipped on the PR as before.

Found while doing it

The red demo caught a defect in my own first version. adapter.conversationLines() read document.body.innerText, so the sidebar's own text counted as a rendered reply: a chat that showed nothing still looked non-empty (a first demo attempt stayed green for that reason). It now reads the conversation flow only ([data-conversation-scroll] [data-chat-flow]), which also makes the Add real-model browser E2E nightly tests with reusable scenarios #204 reload check honest.
A Bot tile whose click no longer opens its chat is not load-bearing — the Main Bot's session opens by default, so that break stayed green. Worth knowing for the next slice.
The Chromium step flaked once (one run timed out on the --with-deps apt round). The job now installs only the headless shell the adapter launches, with one retry; the runner image already carries the system libraries.

Remaining from B for later slices: create a group and post into it, the Workflows page, Settings. Say the word and I take the next slice; as it stands the PR has tests-ok green, the mock job green, a PASS at the tip from the Reviewer, and waits for the queue.

Release

Written-by: FlyBot Lead
Released: 17e3c184606b402aa844c3012e309f76
Reason: PR #2567 (reviewer-side, in-place) merged 11:12:19Z and closed the issue; our implementation stopped pre-PR to avoid a duplicate
At: 2026-11-06T11:17:39Z

Navigation Menu

Repository navigation

A browser E2E that runs without DEEPSEEK_API_KEY: drive the web UI against the scripted mock #2565

Why

What to build

First step is a spike, not a PR

Acceptance

Claim

Spike: the scripted mock against the four #204 checks (2026-11-06 10:2xZ, main `0cdfec4d`)

What it does

Acceptance

Found while doing it

Release

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

A browser E2E that runs without DEEPSEEK_API_KEY: drive the web UI against the scripted mock #2565

Description

Why

What to build

First step is a spike, not a PR

Acceptance

Activity

HugoXK commented on Nov 6, 2026

Claim

Birfy commented on Nov 6, 2026

Spike: the scripted mock against the four #204 checks (2026-11-06 10:2xZ, main 0cdfec4d)

Birfy commented on Nov 6, 2026

What it does

Acceptance

Found while doing it

HugoXK commented on Nov 6, 2026

Release

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Spike: the scripted mock against the four #204 checks (2026-11-06 10:2xZ, main `0cdfec4d`)