⚡ kernel_zoo

English | 简体中文

Dynamic leaderboard and benchmark platform for LLM GPU kernels.

A git-repo-as-database system: every kernel implementation is a directory in a kernel_data git repository, bench_result.yaml is the scoreboard, and git log is the full attempt history. No external database.

---

✨ Features

• Git repository as the database — kernels, scores, and history all live in a plain git repo; the server never touches a DBMS.
• HTTP server (FastAPI) — kernel listing/detail, benchmark submission, long-poll status, runner registration/claim/heartbeat, edit rebase, health and live index.json.
• Benchmark runner CLI — claims tasks, builds the test environment, runs benchmark.py, and reports structured results back, with a pluggable container backend (Docker or a stub for tests).
• WebUI single-page app — vanilla ES modules, zero build step, served by the server at /webui/; list + detail views driven by index.json.
• Schema-driven contracts — every artifact (desc.yaml, bench_result.yaml, benchmark output, submission envelope) is validated against JSON Schemas; the HTTP API contract lives in api/openapi.yaml.
• Trust-the-benchmark accept path — benchmark.py decides accept/reject; the server applies accepted results to the repo via a two-phase git commit (source commit → bench-result commit) with atomic rollback.
• Layered CI — unit/integration tests with per-module coverage gates (core ≥90%, total ≥80%), a separate end-to-end job, WebUI JS tests, and strict Ruff + mypy checks.

🧩 Architecture

┌──────────────┐  POST /api/submissions   ┌────────────────────────────────┐
│  Submitter   │ ───────────────────────▶ │                                │
│ (agent / CI) │   multipart tar.gz       │          FastAPI server        │
└──────────────┘                          │  ┌──────────────────────────┐  │
                                          │  │ queue_manager            │  │
                                          │  │  pending / running / done│  │
┌──────────────┐  GET /api/runner/claim   │  │  + heartbeat sweep + LRU │  │
│    Runner    │ ◀─────────────────────── │  └──────────────────────────┘  │
│  (container) │                          │  ┌──────────────────────────┐  │
│              │  POST .../result (JSON)  │  │ indexer (debounced +     │  │
└──────────────┘ ───────────────────────▶ │  │  periodic) → index.json  │  │
                                          │  └──────────────────────────┘  │
                                          │  ┌──────────────────────────┐  │
                                          │  │ repo_writer              │  │
                                          │  │  two-phase git commit    │  │
                                          │  │  + atomic rollback       │  │
                                          │  └──────────────────────────┘  │
                                          └──────────────┬─────────────────┘
                                                         │ git
                                          ┌──────────────▼─────────────────┐
                                          │   kernel_data git repo          │
                                          │   desc.yaml · bench_result.yaml │
                                          │   · build_test_env.py · src/     │
                                          └─────────────────────────────────┘

A kernel package lives in the kernel repo as <arch>/<op_class>/<sub_class>/<quant>/<name>/ containing a .kernel_package/ directory with four files — desc.yaml, bench_result.yaml, build_test_env.py, benchmark.py — plus a free-form src/ area.

🚀 Quickstart

# 1. Install (dev extra brings test/lint tooling)
python -m pip install -e '.[dev]'

# 2. Point the server at a kernel repo (default: ../kernel_data)
export KERNEL_ZOO_KERNEL_REPO_PATH=/path/to/kernel_data

# 3. Start the server
kernel_zoo-server
# → uvicorn on http://127.0.0.1:8000

# 4. Open the WebUI and check health
open http://127.0.0.1:8000/webui/
curl http://127.0.0.1:8000/api/health

Submit a benchmark:

# Package a kernel directory as <kernel_id>.tar.gz (see the authoring guide),
# then:
curl -X POST http://127.0.0.1:8000/api/submissions \
  -H "X-Submitter-Id: alice" \
  -F "kernel_id=gfx928/Attention/MHA/fp16/reference_impl" \
  -F "submitter_id=alice" \
  -F "package=@package.tar.gz;type=application/gzip"

# Long-poll the result (default timeout 30 s):
curl -H "X-Submitter-Id: alice" \
  http://127.0.0.1:8000/api/submissions/<uuid>/status

⚙️ Configuration

All configuration is via environment variables (or an optional YAML file passed with kernel_zoo-server --config). Env always wins.

Server

Variable	Default	Meaning
KERNEL_ZOO_PLATFORM_REPO_PATH	cwd	This platform repository
KERNEL_ZOO_KERNEL_REPO_PATH	<platform>/../kernel_data	The git repo holding kernels
KERNEL_ZOO_QUEUE_DIR	<platform>/.queue	Benchmark task queue (pending/running/done)
KERNEL_ZOO_INDEX_PATH	<platform>/webui/index.json	Built index output
KERNEL_ZOO_HOST / KERNEL_ZOO_PORT	127.0.0.1 / 8000	Bind address
KERNEL_ZOO_RUNNER_KEYS_JSON	{}	Runner key → arch whitelist ({"r1": ["gfx928"]})
KERNEL_ZOO_LONGPOLL_TIMEOUT_SECONDS	30.0	Submission/claim long-poll deadline
KERNEL_ZOO_GIT_AUTHOR_NAME / _EMAIL	kernel_zoo	Author on accepted-result commits

Runner

Variable	Default	Meaning
KERNEL_ZOO_SERVER_URL	—	Server base URL to poll
KERNEL_ZOO_RUNNER_KEY	—	Pre-shared runner key
KERNEL_ZOO_RUNNER_ARCH	—	Machine arch, e.g. gfx928
KERNEL_ZOO_CONTAINER_BACKEND	docker	docker or stub (tests)
KERNEL_ZOO_CONTAINER_IMAGE	kernel_zoo-runner:latest	Image for the task
KERNEL_ZOO_CONTAINER_TIMEOUT_S	600.0	Per-task timeout
KERNEL_ZOO_RUNNER_WORKDIR	temp	Scratch dir for a task

🔧 CLI tools

Command	Module	Purpose
kernel_zoo-server	kernel_zoo.cli.server	Run the HTTP server
kernel_zoo-validate-package	kernel_zoo.cli.validate_package	Validate a kernel_package dir/tarball against the schemas
kernel_zoo-indexer-now	kernel_zoo.cli.indexer_now	Build index.json immediately
kernel_zoo-runner	kernel_zoo.runner.cli	Claim + run benchmark tasks

Every tool is also runnable as python -m kernel_zoo.<module> ....

📡 HTTP API

Method	Path	Auth	Summary
GET	/api/kernels	—	List all kernel_packages
GET	/api/kernels/{kernel_id}	—	Kernel detail
GET	/api/kernels/{kernel_id}/bench-history	—	Best/current metric summary
POST	/api/kernels/{kernel_id}/claim	submitter	Claim a package for optimization
POST	/api/submissions	submitter	Submit a package for benchmarking
GET	/api/submissions/{uuid}/status	submitter	Long-poll submission status
POST	/api/submissions/{uuid}/result	runner	Report a benchmark result
POST	/api/edits/rebase	submitter	Rebase a local branch onto main
POST	/api/runner/register / claim / heartbeat	runner	Runner lifecycle
GET	/api/health	—	Health + queue depth
GET	/api/index.json	—	Pre-built WebUI index
GET	/webui/	—	WebUI SPA

Auth: submitter via X-Submitter-Id (any non-empty string), runner via X-Runner-Key (pre-shared). Full reference: docs/api_reference.md.

📚 Documentation

Document	Language	Contents
docs/api_reference.md	English	HTTP API, roles, error envelope, kernel_id convention
docs/kernel_package_format.md	English	On-disk format & every schema
docs/kernel_package_guide.md	简体中文	Step-by-step authoring guide for a new kernel_package
docs/kernel_package_guide_en.md	English	Rewrite an existing GPU kernel (code + tests) into a kernel_package
CONTRIBUTING.md	English	Contribution guide for the kernel_zoo platform itself
api/openapi.yaml	—	Normative OpenAPI contract
schemas/	—	JSON Schemas for every artifact
docs/examples/	—	Example kernel_package

🧪 Development

# Unit + integration (non-e2e) with the layered coverage gate
pytest -m "not e2e" --cov=kernel_zoo --cov-report=json:coverage.json
python tools/check_coverage.py --cov-json coverage.json   # core ≥90%, total ≥80%

# End-to-end (real server + runner subprocesses)
pytest -m e2e

# WebUI JS tests (Node 24)
node --test 'tests/webui/**/*.test.mjs'

# Lint / type-check
ruff check kernel_zoo tests tools
ruff format --check kernel_zoo tests tools
mypy kernel_zoo tests tools

CI (GitHub Actions) runs the non-e2e suite with the coverage gate plus the WebUI JS tests on 3.11/3.12, and end-to-end tests in a separate job.

📄 License

MIT — see [project].license in pyproject.toml.