Skip to content

Add an experimental Dawn backend (generated Cython bindings) #839

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hmaarrfk wants to merge 8 commits into from
Closed

Add an experimental Dawn backend (generated Cython bindings) #839

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1b7b80d
Generate Cython declarations of Dawn's webgpu.h in the codegen
hmaarrfk Oct 1, 2026
5c90510
Add an experimental Dawn backend implemented in Cython
hmaarrfk Oct 1, 2026
7ebd4b0
Add Dawn CI workflow and call-overhead benchmarks
hmaarrfk Oct 1, 2026
80b2b37
Run the Cython Dawn backend in Pyodide via Emdawnwebgpu
hmaarrfk Oct 1, 2026
62a43fe
Enumerate all Dawn adapters and match wgpu-native's adapter info
hmaarrfk Oct 1, 2026
077226f
Add Pyodide jobs to the Dawn CI workflow
hmaarrfk Oct 1, 2026
1d92d70
Port backend-agnostic fixes and tests from the cffi Dawn backend
hmaarrfk Oct 1, 2026
39595ca
Deploy the Pyodide demo to the fork's Pages; benchmark vs wgpu-native
hmaarrfk Oct 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
394 changes: 394 additions & 0 deletions .github/workflows/dawn.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,394 @@
name: Dawn backend

# CI for the (optional, experimental) Dawn backend. This is separate from the
# main CI, so that it does not affect the wgpu-native jobs. The same Cython
# extension is built for two targets:
#
# * Native, against Dawn from conda (Dawn, Cython and the compilers are
# obtained with micromamba). Tested on lavapipe (software Vulkan).
# * Pyodide, against Emdawnwebgpu (Dawn's webgpu.h on top of the browser's
# WebGPU). Tested in Node.js (navigator.gpu from the `webgpu` npm package,
# which is Dawn, on lavapipe) and in headless Chrome (SwiftShader).
#
# The per-call overhead benchmark (tools/bench_backends.py) runs natively (Dawn
# and wgpu-native), in Node.js and in Chrome, as informational steps.
#
# On pushes to the `dawn` branch of the fork, the demo page is deployed to
# GitHub Pages, with the demo of the cffi variant (#840, branch dawn-pyodide)
# built from that branch and put under /cffi/. Note: as long as the workflow of
# dawn-pyodide also deploys (to the root), the last deployment wins.
#
# See tools/build_dawn.py and tools/dawn_pyodide.

on:
workflow_dispatch:
push:
branches:
- main
- dawn
pull_request:
branches:
- main

permissions:
contents: read

env:
# Pyodide and its build tools. Emdawnwebgpu (pinned in tools/build_dawn.py)
# must be from the same Dawn release as the vendored headers. The JS glue is
# generated with Pyodide's Emscripten at build time.
PYODIDE_VERSION: "314.0.7"
PYODIDE_BUILD_VERSION: "0.39.1"
# The tests that apply to the Dawn backend in Pyodide. Not included: tests that
# need wgpu-native, subprocesses, trio or rendercanvas.
PYODIDE_TESTS: >-
tests/test_dawn_backend.py tests/test_not_finite.py tests/test_util_compute.py
tests/test_util_core.py tests/test_wgpu_native_buffer.py
tests/test_wgpu_native_compute_tex.py tests/test_wgpu_native_render.py
tests/test_wgpu_native_render_tex.py tests/test_wgpu_native_texture.py
tests/test_wgpu_occlusion_query.py tests/test_diagnostics.py
# Tests specific to wgpu-native (repr, diagnostics of all backends)
PYODIDE_DESELECT: >-
tests/test_wgpu_native_buffer.py::test_buffer_init1
tests/test_diagnostics.py::test_diagnostics_with_backends
# In the browser, validation errors arrive asynchronously (uncapturederror
# event), so they are logged instead of raised by the failing call.
CHROME_DESELECT: >-
tests/test_dawn_backend.py::test_validation_error_raises_at_call_site
tests/test_dawn_backend.py::test_shader_error_raises
tests/test_dawn_backend.py::test_encoder_errors_raise_at_finish
tests/test_wgpu_native_buffer.py::test_buffer_init3
tests/test_wgpu_native_texture.py::test_view_formats

jobs:
test-dawn:
name: Dawn ${{ matrix.os }} py${{ matrix.python-version }}
timeout-minutes: 30
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-latest
python-version: '3.12'
- os: ubuntu-latest
python-version: '3.14'
defaults:
run:
shell: bash -el {0}
steps:
- uses: actions/checkout@v4

- name: Install lavapipe (software Vulkan driver)
run: |
sudo apt-get update -y -qq
sudo apt install -y mesa-vulkan-drivers

- name: Install Dawn and build tools with conda
uses: mamba-org/setup-micromamba@v2
with:
environment-name: wgpu-dawn
condarc: |
channel_priority: strict
channels:
- mark.harfouche
- conda-forge
create-args: >-
python=${{ matrix.python-version }}
dawn
cython>=3.1
setuptools
c-compiler
cxx-compiler
cffi
numpy
pytest
psutil
anyio
trio
rendercanvas
imageio
pypng
requests
cache-environment: false

- name: Show versions
run: |
micromamba list | grep -E "dawn|cython|python |vulkan"
ls /usr/share/vulkan/icd.d/

- name: Build the Dawn backend
run: |
python tools/build_dawn.py
ls -la wgpu/backends/dawn/

- name: Test the Dawn backend
env:
WGPUPY_BACKEND: dawn
PYTHONPATH: ${{ github.workspace }}
run: |
# Use lavapipe as the Vulkan driver
export VK_ICD_FILENAMES=$(ls /usr/share/vulkan/icd.d/lvp_icd*.json | head -1)
python -c "import wgpu; print([a.summary for a in wgpu.gpu.enumerate_adapters_sync()])"
# Dawn-specific tests, plus the generic tests that also apply to Dawn.
pytest -v tests/test_dawn_backend.py tests/test_api.py tests/test_async.py \
tests/test_canvas.py tests/test_not_finite.py tests/test_util_compute.py \
tests/test_util_core.py tests/test_wgpu_native_buffer.py \
tests/test_wgpu_native_compute_tex.py tests/test_wgpu_native_render.py \
tests/test_wgpu_native_render_tex.py tests/test_wgpu_native_texture.py \
tests/test_wgpu_occlusion_query.py \
--deselect tests/test_wgpu_native_buffer.py::test_buffer_init1

- name: Run the scripts that also run in Pyodide
env:
WGPUPY_BACKEND: dawn
PYTHONPATH: ${{ github.workspace }}
run: |
export VK_ICD_FILENAMES=$(ls /usr/share/vulkan/icd.d/lvp_icd*.json | head -1)
cd tools/dawn_pyodide
python compute_async.py
python compute_sync.py
python triangle.py

- name: Report on the full test suite (informational)
continue-on-error: true
env:
WGPUPY_BACKEND: dawn
PYTHONPATH: ${{ github.workspace }}
run: |
export VK_ICD_FILENAMES=$(ls /usr/share/vulkan/icd.d/lvp_icd*.json | head -1)
pytest tests -q -rfE --continue-on-collection-errors || true

- name: Benchmark the per-call overhead, Dawn vs wgpu-native (informational)
continue-on-error: true
env:
PYTHONPATH: ${{ github.workspace }}
run: |
export VK_ICD_FILENAMES=$(ls /usr/share/vulkan/icd.d/lvp_icd*.json | head -1)
python tools/download_wgpu_native.py
python tools/bench_backends.py --all

codegen-dawn:
name: Dawn codegen up to date
timeout-minutes: 10
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -U -e .[codegen]
- name: Test codegen (includes the Dawn pxd and mappings)
run: |
pytest -v codegen

pyodide-build:
name: Build the Pyodide wheel
runs-on: ubuntu-latest
timeout-minutes: 30
env:
PYODIDE_XBUILDENV_PATH: /home/runner/xbuildenv
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.14'
- name: Install pyodide-build
run: |
python -m pip install --upgrade pip
pip install pyodide-build==${PYODIDE_BUILD_VERSION}
- name: Install the Pyodide cross-build environment and Emscripten
run: |
pyodide xbuildenv install ${PYODIDE_VERSION}
pyodide xbuildenv install-emscripten
- name: Build the wgpu wheel with the Dawn backend for Pyodide
run: |
source ${PYODIDE_XBUILDENV_PATH}/${PYODIDE_VERSION}/emsdk/emsdk_env.sh
emcc --version | head -1
# No wgpu-native; the Dawn extension is compiled against Emdawnwebgpu
WGPU_PY_BUILD_NOARCH=1 WGPU_PY_BUILD_DAWN=1 pyodide build --exports pyinit -o dist
- name: Show the wheel
run: |
ls -l dist
python -m zipfile -l dist/*.whl | grep -E "\.so|\.js"
- uses: actions/upload-artifact@v4
with:
name: pyodide-wheel
path: dist/*.whl

pyodide-node:
name: Pyodide in Node (webgpu npm, lavapipe)
needs: pyodide-build
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- uses: actions/download-artifact@v4
with:
name: pyodide-wheel
path: dist
- uses: actions/setup-node@v4
with:
node-version: '26'
- name: Install lavapipe
run: |
sudo apt-get update -y -qq
sudo apt-get install -y mesa-vulkan-drivers libvulkan1
- name: Install npm packages
run: cd tools/dawn_pyodide && npm install --no-audit --no-fund
- name: Run scripts
run: |
WHEELS=$(ls $PWD/dist/*.whl | tr '\n' ',')
cd tools/dawn_pyodide
node run_node.mjs $WHEELS compute_async.py
node run_node.mjs $WHEELS compute_sync.py
node run_node.mjs $WHEELS triangle.py
- name: Run wgpu-py tests
run: |
WHEELS=$(ls $PWD/dist/*.whl | tr '\n' ',')
DESELECT=$(for t in ${PYODIDE_DESELECT}; do echo -n " --deselect $t"; done)
cd tools/dawn_pyodide
node run_node.mjs $WHEELS --pytest ../../tests ${PYODIDE_TESTS} -k "not spirv" ${DESELECT}
- name: Benchmark the per-call overhead (informational)
continue-on-error: true
run: |
WHEELS=$(ls $PWD/dist/*.whl | tr '\n' ',')
cd tools/dawn_pyodide
node run_node.mjs $WHEELS ../bench_backends.py

pyodide-chrome:
name: Pyodide in headless Chrome (SwiftShader)
needs: pyodide-build
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- uses: actions/download-artifact@v4
with:
name: pyodide-wheel
path: dist
- uses: actions/setup-node@v4
with:
node-version: '26'
- name: Install npm packages
run: cd tools/dawn_pyodide && npm install --no-audit --no-fund
- name: Prepare site
run: |
mkdir site
cp tools/dawn_pyodide/runner.html tools/dawn_pyodide/*.py tools/bench_backends.py dist/*.whl site/
zip -qr site/tests.zip tests/*.py conftest.py
echo "WHEELS=$(cd dist && ls *.whl | tr '\n' ',')" >> $GITHUB_ENV
# Pyodide itself (and its packages) come from the CDN
echo "PYODIDE_URL=https://cdn.jsdelivr.net/pyodide/v${PYODIDE_VERSION}/full/pyodide.mjs" >> $GITHUB_ENV
google-chrome --version
- name: Run scripts
run: |
cd tools/dawn_pyodide
node run_chrome.mjs ../../site "runner.html?pyodide=${PYODIDE_URL}&wheels=${WHEELS}&script=compute_async.py"
node run_chrome.mjs ../../site "runner.html?pyodide=${PYODIDE_URL}&wheels=${WHEELS}&script=compute_sync.py"
SCREENSHOT=../../triangle.png node run_chrome.mjs ../../site "runner.html?pyodide=${PYODIDE_URL}&wheels=${WHEELS}&script=triangle.py&canvas=1"
- name: Run wgpu-py tests
run: |
ARGS=$(echo ${PYODIDE_TESTS} | tr ' ' ',')
DESELECT=$(for t in ${PYODIDE_DESELECT} ${CHROME_DESELECT}; do echo -n ",--deselect,$t"; done)
cd tools/dawn_pyodide
node run_chrome.mjs ../../site "runner.html?pyodide=${PYODIDE_URL}&wheels=${WHEELS}&tests=tests.zip&pytest=${ARGS},-k,not spirv${DESELECT}"
- name: Benchmark the per-call overhead (informational)
continue-on-error: true
run: |
cd tools/dawn_pyodide
node run_chrome.mjs ../../site "runner.html?pyodide=${PYODIDE_URL}&wheels=${WHEELS}&script=bench_backends.py"
- name: Upload the triangle screenshot and the demo page
if: always()
uses: actions/upload-artifact@v4
with:
# Serve the site directory (e.g. python -m http.server) and open
# runner.html?script=triangle.py&canvas=1 in a browser with WebGPU.
name: pyodide-demo
path: |
triangle.png
site

pages:
name: Deploy the demo to GitHub Pages (fork only)
if: github.event_name != 'pull_request' && github.repository == 'hmaarrfk/wgpu-py' && github.ref == 'refs/heads/dawn'
needs: [pyodide-build, pyodide-node, pyodide-chrome]
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
pages: write
id-token: write
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
concurrency:
group: github-pages
cancel-in-progress: false
env:
PYODIDE_XBUILDENV_PATH: /home/runner/xbuildenv
steps:
- uses: actions/checkout@v4
- uses: actions/download-artifact@v4
with:
name: pyodide-wheel
path: site/wheels
- name: Assemble the demo (this branch, Cython)
run: |
cp tools/dawn_pyodide/demo/index.html tools/dawn_pyodide/demo/demo.py site/
python3 - site "${GITHUB_SHA}" <<'EOF'
import json, os, sys
d, sha = sys.argv[1:]
json.dump({
"pyodide": os.environ["PYODIDE_VERSION"],
"dawn": "v20260929.200159",
"commit": sha[:10],
"wheels": sorted(os.listdir(os.path.join(d, "wheels"))),
}, open(os.path.join(d, "wheels.json"), "w"))
EOF
# The demo of the cffi variant (#840) is built from its branch, so that one
# workflow deploys the whole site. If that fails, the main demo still deploys.
- uses: actions/checkout@v4
with:
ref: dawn-pyodide
path: cffi-src
- uses: actions/setup-python@v5
with:
python-version: '3.14'
- name: Build the demo of the cffi variant (#840) for /cffi/
id: cffi
continue-on-error: true
run: |
pip install pyodide-build==${PYODIDE_BUILD_VERSION} build hatchling
pyodide xbuildenv install ${PYODIDE_VERSION}
pyodide xbuildenv install-emscripten
source ${PYODIDE_XBUILDENV_PATH}/${PYODIDE_VERSION}/emsdk/emsdk_env.sh
mkdir -p site/cffi/wheels
(cd cffi-src/dawn_bindings && pyodide build --exports pyinit -o ../../site/cffi/wheels)
(cd cffi-src && WGPU_PY_BUILD_NOARCH=1 python -m build --wheel -o ../site/cffi/wheels)
cp cffi-src/dawn_bindings/demo/index.html cffi-src/dawn_bindings/demo/demo.py site/cffi/
python3 - site/cffi "$(git -C cffi-src rev-parse HEAD)" <<'EOF'
import json, os, sys
d, sha = sys.argv[1:]
json.dump({
"pyodide": os.environ["PYODIDE_VERSION"],
"dawn": "v20260929.200159",
"commit": sha[:10],
"wheels": sorted(os.listdir(os.path.join(d, "wheels"))),
}, open(os.path.join(d, "wheels.json"), "w"))
EOF
- name: Placeholder for /cffi/ if its build failed
if: steps.cffi.outcome != 'success'
run: |
rm -rf site/cffi && mkdir -p site/cffi
echo '<!doctype html><meta charset="utf-8"><title>cffi demo</title><p>The demo of the cffi variant (<a href="https://github.com/pygfx/wgpu-py/pull/840">#840</a>) failed to build in the last deploy. <a href="../">The Cython demo</a>.</p>' > site/cffi/index.html
- run: ls -R site
- uses: actions/configure-pages@v5
- uses: actions/upload-pages-artifact@v3
with:
path: site
- id: deployment
uses: actions/deploy-pages@v4
Loading
Loading