Skip to content

ci(cli): run E2E tests in parallel #1726

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
betegon wants to merge 12 commits into
base: main
Choose a base branch
from
Open

ci(cli): run E2E tests in parallel #1726

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
0af6b36
ci(cli): run E2E tests in parallel
betegon Oct 29, 2026
2c1cde7
ci(cli): remove unused E2E environment setup
betegon Oct 29, 2026
756b806
ci(cli): prepare E2E and unit inputs once
betegon Oct 30, 2026
1b42251
ci(cli): reuse npm build setup and share Node pins
betegon Oct 30, 2026
c2e83c0
ci(cli): shard unit tests and merge coverage (#1727)
betegon Nov 2, 2026
0eb2b50
ci(cli): cap the E2E job at 10 minutes
betegon Nov 2, 2026
920ad14
Merge branch 'main' into bt/cli-e2e-parallel
betegon Nov 2, 2026
4dbe4f1
ci(cli): run E2E in Build against the nightly build artifacts
betegon Nov 2, 2026
029c6db
ci(cli): give the Build status job a timeout
betegon Nov 2, 2026
dafb78c
ci(test): drop the CLI binary path from the shared E2E step
betegon Nov 3, 2026
7f804fb
ci(cli): validate the npm package once, from the Build artifact
betegon Nov 3, 2026
d3ded9c
ci(cli): keep pull request builds from uploading sourcemaps
betegon Nov 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/node-versions/20
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
20.20.2
1 change: 1 addition & 0 deletions .github/node-versions/22
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
22.23.4
1 change: 1 addition & 0 deletions .github/node-versions/24
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
24.22.1
174 changes: 116 additions & 58 deletions .github/workflows/cli-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,18 @@ on:
branches:
- main
- release/cli/**
pull_request:
Comment thread
betegon marked this conversation as resolved.
Resolved
workflow_dispatch:

concurrency:
group: cli-build-${{ github.ref }}
cancel-in-progress: false
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
Comment on lines 12 to +13

@BYK BYK Nov 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this makes sense now:

  1. We should cancel in progress for the same group
  2. For pushes to a branch/main github.ref makes sense, for PRs I think it should be the PR number so we don't keep building multiple times for the same PR if there are new commits in rapid succession.


permissions:
contents: read

env:
SENTRY_CLIENT_ID: ${{ vars.SENTRY_CLIENT_ID }}
NODE_VERSION_20: "20.20.2"
NODE_VERSION_22: "22.23.4"
NODE_VERSION_24: "24.22.1"

jobs:
nightly-version:
Expand All @@ -28,7 +26,7 @@ jobs:
outputs:
version: ${{ steps.version.outputs.version }}
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@v7
- name: Compute version from the commit timestamp
id: version
run: node scripts/cli-nightly-version.mjs
Expand All @@ -42,23 +40,14 @@ jobs:
strategy:
fail-fast: false
matrix:
include:
- target: darwin-arm64
os: macos-latest
- target: darwin-x64
os: ubuntu-latest
- target: linux-x64
os: ubuntu-latest
- target: linux-arm64
os: ubuntu-latest
- target: windows-x64
os: ubuntu-latest
# Pull requests build the Linux binary the E2E tests run against.
include: ${{ fromJSON(github.event_name == 'pull_request' && '[{"target":"linux-x64","os":"ubuntu-latest"}]' || '[{"target":"darwin-arm64","os":"macos-latest"},{"target":"darwin-x64","os":"ubuntu-latest"},{"target":"linux-x64","os":"ubuntu-latest"},{"target":"linux-arm64","os":"ubuntu-latest"},{"target":"windows-x64","os":"ubuntu-latest"}]') }}

@BYK BYK Nov 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very hard to reason about, can you find a simpler way to express it?

steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@v7
- uses: pnpm/action-setup@a7487c7e89a18df4991f7f222e4898a00d66ddda # v4
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
- uses: actions/setup-node@v7
with:
node-version: ${{ env.NODE_VERSION_22 }}
node-version-file: .github/node-versions/22

@BYK BYK Nov 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is the Nightly build with SEA, we should be using Node 24

cache: pnpm
- name: Install dependencies
run: pnpm install --frozen-lockfile
Expand Down Expand Up @@ -114,14 +103,15 @@ jobs:
fi
- name: Build
env:
SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
RELEASE_BUILD: "1"
# Pull request builds skip the sourcemap upload and the .gz copies.
SENTRY_AUTH_TOKEN: ${{ github.event_name != 'pull_request' && secrets.SENTRY_AUTH_TOKEN || '' }}
RELEASE_BUILD: ${{ github.event_name != 'pull_request' && '1' || '' }}
FOSSILIZE_SIGN: ${{ (github.ref_name == 'main' || startsWith(github.ref_name, 'release/cli/')) && 'y' || 'n' }}
APPLE_CERT_PASSWORD: ${{ secrets.APPLE_CERT_PASSWORD }}
APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
run: pnpm --filter sentry run build -- --target ${{ matrix.target }}
- name: Smoke test

@BYK BYK Nov 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not smoke test, this is signature verification

if: matrix.target == 'linux-x64' || matrix.target == 'darwin-arm64'
if: matrix.target == 'darwin-arm64'
env:
SENTRY_AUTH_TOKEN: ""
SENTRY_TOKEN: ""
Expand All @@ -143,14 +133,26 @@ jobs:
shell: bash
run: rcodesign verify packages/cli/dist-bin/sentry-darwin-arm64
- name: Upload binary artifact
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
uses: actions/upload-artifact@v7
with:
name: sentry-${{ matrix.target }}
path: |
packages/cli/dist-bin/sentry-*
!packages/cli/dist-bin/*.gz
- name: Upload generated CLI sources
if: matrix.target == 'linux-x64'
uses: actions/upload-artifact@v7
with:
name: cli-generated
path: |
packages/cli/src/generated/
packages/cli/src/sdk.generated.ts
packages/cli/src/sdk.generated.d.cts
if-no-files-found: error
retention-days: 7
- name: Upload compressed artifact
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
if: github.event_name != 'pull_request'

@BYK BYK Nov 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry, shouldn't we be uploading these for pull requests too for E2E testing?

uses: actions/upload-artifact@v7
with:
name: sentry-${{ matrix.target }}-gz
path: packages/cli/dist-bin/*.gz
Expand All @@ -169,7 +171,7 @@ jobs:
from-version: ${{ steps.generate.outputs.from-version }}
steps:
- name: Download binaries and compressed artifacts
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
uses: actions/download-artifact@v8
with:
pattern: sentry-*-*
path: new-binaries
Expand All @@ -188,7 +190,7 @@ jobs:
patches-dir: patches
- name: Upload patches
if: steps.generate.outputs.has-patches == 'true'
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
uses: actions/upload-artifact@v7
with:
name: sentry-patches
path: patches/*.patch
Expand All @@ -207,21 +209,21 @@ jobs:
packages: write
steps:
- name: Download compressed binaries
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
uses: actions/download-artifact@v8
with:
pattern: sentry-*-gz
path: artifacts
merge-multiple: true
- name: Download binaries for patch integrity annotations
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
uses: actions/download-artifact@v8
with:
pattern: sentry-*-*
path: binaries
merge-multiple: true
- name: Download optional delta patches
id: patches
continue-on-error: true
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
uses: actions/download-artifact@v8
with:
name: sentry-patches
path: patches
Expand Down Expand Up @@ -263,29 +265,23 @@ jobs:
timeout-minutes: 20
environment: ${{ github.ref == 'refs/heads/main' && 'production' || startsWith(github.ref, 'refs/heads/release/cli/') && 'cli-release' || '' }}
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@v7
- uses: pnpm/action-setup@a7487c7e89a18df4991f7f222e4898a00d66ddda # v4
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
- uses: actions/setup-node@v7
with:
node-version: ${{ env.NODE_VERSION_22 }}
node-version-file: .github/node-versions/22
cache: pnpm
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Bundle
env:
SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
SENTRY_AUTH_TOKEN: ${{ github.event_name != 'pull_request' && secrets.SENTRY_AUTH_TOKEN || '' }}

@BYK BYK Nov 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this should be a workflow-level definition as it seems to be repeated?

run: pnpm --filter sentry run bundle
- name: Smoke test
env:
SENTRY_AUTH_TOKEN: ""
SENTRY_TOKEN: ""
SENTRY_CONFIG_DIR: ${{ runner.temp }}/.sentry-smoke
run: node packages/cli/dist/bin.cjs --help
- name: Pack
working-directory: packages/cli
run: npm pack
- name: Upload npm artifact
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
uses: actions/upload-artifact@v7
with:
name: npm-package
path: packages/cli/*.tgz
Expand All @@ -298,7 +294,7 @@ jobs:
working-directory: packages/mcp-server
run: npm pack
- name: Upload mcp-server npm artifact
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
uses: actions/upload-artifact@v7
with:
name: mcp-server-npm-package
path: packages/mcp-server/*.tgz
Expand All @@ -313,28 +309,71 @@ jobs:
matrix:
node: [20, 22, 24]
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: ${{ matrix.node == 24 && env.NODE_VERSION_24 || matrix.node == 20 && env.NODE_VERSION_20 || env.NODE_VERSION_22 }}
node-version-file: .github/node-versions/${{ matrix.node }}
- name: Download npm artifact
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
uses: actions/download-artifact@v8
with:
name: npm-package
path: packages/cli
path: ${{ runner.temp }}/npm-package
- name: Test packaged CLI
env:
SENTRY_AUTH_TOKEN: ""
SENTRY_TOKEN: ""
SENTRY_CONFIG_DIR: ${{ runner.temp }}/.sentry-smoke
RUNTIME_BIN: sentry
RUNTIME_PACKAGE: sentry
run: |
package_file=$(find packages/cli -maxdepth 1 -name 'sentry-*.tgz' -print -quit)
test -n "$package_file"
npm install --ignore-scripts --prefix "$RUNNER_TEMP/npm-smoke" "$GITHUB_WORKSPACE/$package_file"
node "$RUNNER_TEMP/npm-smoke/node_modules/sentry/dist/bin.cjs" --help
packages=("$RUNNER_TEMP"/npm-package/sentry-*.tgz)
[[ "${#packages[@]}" == 1 ]]
RUNTIME_ARCHIVE="${packages[0]}" bash scripts/validate-npm-runtime.sh

test-e2e:
name: E2E Tests (Node 24)
needs: [build-binary, build-npm]
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: pnpm/action-setup@a7487c7e89a18df4991f7f222e4898a00d66ddda # v4
- uses: actions/setup-node@v7
with:
node-version-file: .github/node-versions/24
cache: pnpm
- run: pnpm --filter 'sentry...' install --frozen-lockfile
# The tests import CLI sources that read the generated files.
- name: Download generated CLI sources
uses: actions/download-artifact@v8
with:
name: cli-generated
path: packages/cli/src
- name: Download Linux binary
uses: actions/download-artifact@v8
with:
name: sentry-linux-x64
path: packages/cli/dist-bin
- name: Download npm package
uses: actions/download-artifact@v8
with:
name: npm-package
path: ${{ runner.temp }}/npm-package
- name: Unpack the binary and npm bundle
run: |
chmod +x packages/cli/dist-bin/sentry-linux-x64
tar -xzf "$RUNNER_TEMP"/npm-package/sentry-*.tgz -C packages/cli --strip-components=1 package/dist
- name: Run E2E tests
env:
SENTRY_CLI_BINARY: ${{ github.workspace }}/packages/cli/dist-bin/sentry-linux-x64
run: |
test -x "$SENTRY_CLI_BINARY"
test -s packages/cli/dist/bin.cjs
test -s packages/cli/dist/index.cjs
pnpm --filter sentry exec vitest run test/e2e

build-docs:
name: Build Docs
if: github.event_name != 'pull_request'
needs: build-binary
runs-on: ubuntu-latest
timeout-minutes: 30
Expand All @@ -343,11 +382,11 @@ jobs:
SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
PUBLIC_SENTRY_ENVIRONMENT: ${{ (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/heads/release/cli/')) && 'production' || 'development' }}
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/checkout@v7
- uses: pnpm/action-setup@a7487c7e89a18df4991f7f222e4898a00d66ddda # v4
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
- uses: actions/setup-node@v7
with:
node-version: ${{ env.NODE_VERSION_24 }}
node-version-file: .github/node-versions/24
cache: pnpm
- name: Install dependencies
run: pnpm install --frozen-lockfile
Expand All @@ -363,7 +402,7 @@ jobs:
- name: Build docs
run: pnpm --filter sentry-cli-docs run build
- name: Download Linux binary
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
uses: actions/download-artifact@v8
with:
name: sentry-linux-x64
path: dist-bin
Expand All @@ -390,7 +429,26 @@ jobs:
printf '%s\n' '{"version":3}' > "$output_dir/.vercel/output/config.json"
(cd "$output_dir" && zip -qr "$GITHUB_WORKSPACE/vercel.zip" .vercel)
- name: Upload docs artifact
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
uses: actions/upload-artifact@v7
with:
name: vercel
path: vercel.zip

build-status:
name: Build status
if: always()
needs: [build-binary, build-npm, test-npm, test-e2e]
runs-on: ubuntu-latest
timeout-minutes: 5
permissions: {}
steps:
- name: Require the CLI build and E2E tests
env:
RESULTS: ${{ join(needs.*.result, ' ') }}
run: |
for result in $RESULTS; do
if [[ "$result" != success ]]; then
echo "::error::A required Build job finished with: $result"
exit 1
fi
done
Loading
Loading