Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/scripts/test-workflow.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -318,7 +318,7 @@ const liveSteps = stepsOf(live)
const KEYED = [
'npm ci (core)',
'Install the pinned DSH runtime',
'Clone DeepSeek Harness at the pinned tag',
'Chromium for the browser',
'Start the usage tap',
'Start the real-model bench',
'Run the four checks',
Expand Down
45 changes: 33 additions & 12 deletions .github/workflows/live-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,9 @@ name: Live E2E
# path, with DEEPSEEK_BASE_URL pointed at the scripted mock (core/dev/mock-llm.mjs). It
# reads no secret, so it runs on every pull request that touches core/ or plugins/, not only
# on main — and its step summary names each check it ran, so its green is visibly different
# from the live job's skip. The `live` job is unchanged.
# from the live job's skip. The `live` job runs the same published runtime as the mock job;
# it differs in the key, the usage tap and the four real-model checks (it used to clone the
# harness at the pinned tag and never build it, so `dsh web` died at startup with ENOENT).

on:
schedule:
Expand Down Expand Up @@ -109,21 +111,40 @@ jobs:
working-directory: core
run: npm ci --no-fund --no-audit
timeout-minutes: 5
# The runtime is the published package, not a source checkout: a checkout has no built
# client artifacts (dsh web exits with ENOENT on packages/client/*/lib/client.js), and
# this job must not spend ten minutes building DSH to run four checks. Same runtime,
# playwright-core and node_modules/.pnpm layout as the mock job below: `pnpm dsh`
# starts the bench and core/dev/desktop-linux/lib.mjs resolves playwright-core from
# .pnpm/node_modules.
- name: Install the pinned DSH runtime
if: steps.preflight.outputs.key == 'true'
working-directory: core
run: |
DSH=$(node -p "require('./dev/harness/package.json').dependencies['@deepseek-ai/dsh']")
npm run harness -- --no-save "@deepseek-ai/dsh@$DSH"
echo "DSH=$DSH" >> "$GITHUB_ENV"
timeout-minutes: 5
- name: Clone DeepSeek Harness at the pinned tag
DSH=$(node -p "require('./core/dev/harness/package.json').dependencies['@deepseek-ai/dsh']")
RUNTIME="$RUNNER_TEMP/dsh-runtime"
mkdir -p "$RUNTIME"
cd "$RUNTIME"
printf '{"name":"dsh-live-e2e-runtime","private":true,"packageManager":"pnpm@12.16.1"}\n' > package.json
# pnpm 12 refuses dependency build scripts by default; the pinned runtime's native
# helpers (subprocess spawn, pty, protobuf) are the same packages the mock job runs.
printf 'dangerouslyAllowAllBuilds: true\n' > pnpm-workspace.yaml
corepack enable
corepack pnpm add "@deepseek-ai/dsh@$DSH" playwright-core@1.61.1
echo "DSH_HARNESS=$RUNTIME" >> "$GITHUB_ENV"
timeout-minutes: 8
# The headless shell is what the adapter launches. No --with-deps: the runner image
# already carries Chrome's system libraries, so no apt-get round; one retry covers the
# download itself.
- name: Chromium for the browser
if: steps.preflight.outputs.key == 'true'
run: |
git clone --depth 1 --branch "dsh-v$DSH" https://github.com/deepseek-ai/deepseek-harness.git deepseek-harness
corepack enable
pnpm install --dir deepseek-harness
timeout-minutes: 10
for attempt in 1 2; do
node "$DSH_HARNESS/node_modules/playwright-core/cli.js" install chromium --only-shell && exit 0
echo "::warning::chromium install attempt $attempt failed; retrying"
sleep 5
done
exit 1
timeout-minutes: 8
- name: Start the usage tap
if: steps.preflight.outputs.key == 'true'
run: |
Expand Down Expand Up @@ -157,7 +178,7 @@ jobs:
LIVE_E2E_TAP_UPSTREAM: https://api.deepseek.com
run: |
node dev/live-e2e/run.mjs "$LIVE_E2E_HOME" "$WEB_PORT" \
--model "$MODEL" --harness deepseek-harness \
--model "$MODEL" --harness "$DSH_HARNESS" \
--tap "$LIVE_E2E_HOME/tap.jsonl" --evidence evidence --tz Asia/Shanghai
- name: Stop the bench and the tap
if: always() && steps.preflight.outputs.key == 'true'
Expand Down
5 changes: 3 additions & 2 deletions dev/live-e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,9 @@ adapter `playwright-core` from `node_modules/.pnpm/node_modules`.

`.github/workflows/live-e2e.yml` runs the same flow nightly at 00:30 Asia/Shanghai (and
on dispatch) on `main` only, with a window gate so a quiet `main` spends no tokens: it
clones the harness at the tag pinned in `core/dev/harness/package.json`, starts the tap,
resets the bench, runs the four checks and uploads the sanitizer-cleared allow-list.
installs the published `@deepseek-ai/dsh` the pin in `core/dev/harness/package.json` names,
starts the tap, resets the bench, runs the four checks and uploads the sanitizer-cleared
allow-list.
The keyless `mock` job runs the two checks above on every pull request that touches `core/`
or `plugins/` and on the same nightly schedule, with no `secrets.*` in its environment and
the runner's summary appended to the job's step summary ([#2565](https://github.com/ThinkFlowLab/FlyBot/issues/2565)).
Expand Down
Loading