Skip to content

Commit f0bb768

Browse files
committed
docs: publish a documentation site on GitHub Pages
## Summary ### Why? SubmitQueue's docs exist only as Markdown in the repo. A public site with a landing page, navigation and search (like cadenceworkflow.io) makes the project easier to discover and the RFCs easier to browse. ### What? - `tool/docsite/`: MkDocs Material config that renders `doc/` in place, so docs are not copied and the nav picks up new RFCs automatically. Versions are pinned. - `hooks/repo_links.py` points links that leave `doc/` (code, package READMEs, `AGENTS.md`) at GitHub, so `--strict` stays on and still fails on broken links between pages. - `hooks/nav_titles.py` titles the sections Guides and Design (RFCs) and puts Quickstart first. - `doc/index.md` + `overrides/home.html`: landing page with a hero, feature cards and a quickstart snippet; green brand theme with light and dark modes. - `.github/workflows/docs.yml`: builds the site on PRs that touch `doc/**` or `tool/docsite/**`, and deploys to https://uber.github.io/submitqueue/ from `main`. It is not a required check. - `make docs-serve` / `make docs-build`; `tool/docsite` is in `.bazelignore`, and its build output and venv are ignored by git and yamlfmt. Pages is already enabled on the repo (source: GitHub Actions, deploys restricted to `main`), so merging publishes the site. ## Test Plan ✅ `make docs-build` (strict) passes ✅ `make docs-serve`: checked landing page, tabs, sidebar order, links into code, light and dark modes, and an 845px-wide window ✅ `make fmt` clean
1 parent d5752e6 commit f0bb768

15 files changed

Lines changed: 465 additions & 1 deletion

File tree

‎.bazelignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,4 @@ pkg
22
bin
33
.git
44
.idea
5+
tool/docsite

‎.github/workflows/docs.yml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
name: Docs
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
paths:
8+
- doc/**
9+
- tool/docsite/**
10+
- .github/workflows/docs.yml
11+
pull_request:
12+
paths:
13+
- doc/**
14+
- tool/docsite/**
15+
- .github/workflows/docs.yml
16+
workflow_dispatch:
17+
18+
permissions:
19+
contents: read
20+
21+
jobs:
22+
build:
23+
name: Build site
24+
runs-on: ubuntu-latest
25+
steps:
26+
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
27+
with:
28+
persist-credentials: false
29+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
30+
with:
31+
python-version: "3.13"
32+
33+
- name: Install MkDocs
34+
run: pip install -r tool/docsite/requirements.txt
35+
36+
- name: Build site
37+
run: mkdocs build --strict -f tool/docsite/mkdocs.yml
38+
39+
- uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
40+
with:
41+
path: tool/docsite/build
42+
43+
deploy:
44+
name: Deploy to GitHub Pages
45+
if: ${{ github.event_name != 'pull_request' && github.ref == 'refs/heads/main' }}
46+
needs: build
47+
runs-on: ubuntu-latest
48+
permissions:
49+
pages: write
50+
id-token: write
51+
environment:
52+
name: github-pages
53+
url: ${{ steps.deployment.outputs.page_url }}
54+
# Never cancel a deploy in progress; a newer run queues behind it.
55+
concurrency:
56+
group: pages
57+
cancel-in-progress: false
58+
steps:
59+
- id: deployment
60+
uses: actions/deploy-pages@368f82528645a54fb793d4d04e342629a3f51346 # v5.0.1

‎.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,8 @@ bin/
2323

2424
# Make completion cache
2525
.make_targets_cache
26+
27+
# Documentation site build output and virtualenv
28+
/tool/docsite/build/
29+
/tool/docsite/.venv/
30+
/tool/docsite/hooks/__pycache__/

‎.yamlfmt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,6 @@ formatter:
33
indent: 2
44
retain_line_breaks_single: true
55
include_document_start: false
6+
exclude:
7+
- tool/docsite/.venv
8+
- tool/docsite/build

‎Makefile‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,9 @@ RUNWAY_COMPOSE_FILE = service/runway/server/docker-compose.yml
2525
# Fixed project name for local manual testing (tests use unique random names)
2626
RUNWAY_LOCAL_PROJECT = runway
2727

28+
# Virtualenv holding the documentation site toolchain
29+
DOCS_VENV = tool/docsite/.venv
30+
2831
# yamlfmt version for YAML formatting (override with: make fmt YAMLFMT_VERSION=v0.16.0)
2932
YAMLFMT_VERSION ?= v0.16.0
3033

@@ -148,7 +151,7 @@ define assert_clean
148151
fi
149152
endef
150153

151-
.PHONY: build build-all-linux build-runway-linux build-submitqueue-gateway-client build-submitqueue-gateway-linux build-submitqueue-gateway-server build-submitqueue-orchestrator-linux build-stovepipe-linux build-stovepipe-linux-debug check-gazelle check-mocks check-tidy clean clean-proto demo-requests deps e2e-test fmt gazelle integration-test integration-test-submitqueue-consumer integration-test-extensions integration-test-submitqueue-gateway integration-test-submitqueue-orchestrator license-fix lint lint-binary lint-fmt lint-license local-init-runway-queue-schema local-init-stovepipe-schemas local-runway-start local-runway-stop local-submitqueue-stop local-submitqueue-clean local-submitqueue-gateway-start local-submitqueue-gateway-stop local-init-submitqueue-schemas local-submitqueue-logs local-submitqueue-orchestrator-start local-submitqueue-orchestrator-stop local-submitqueue-ps local-submitqueue-restart local-submitqueue-start local-stop local-stovepipe-debug-start local-stovepipe-logs local-stovepipe-start local-stovepipe-stop mocks proto query-deps query-targets run-client-runway run-client-submitqueue-gateway run-client-submitqueue-orchestrator run-client-stovepipe run-queue-admin test test-no-cache test-race tidy tidy-bazel tidy-go help
154+
.PHONY: build build-all-linux build-runway-linux build-submitqueue-gateway-client build-submitqueue-gateway-linux build-submitqueue-gateway-server build-submitqueue-orchestrator-linux build-stovepipe-linux build-stovepipe-linux-debug check-gazelle check-mocks check-tidy clean clean-proto demo-requests deps docs-build docs-serve docs-venv e2e-test fmt gazelle integration-test integration-test-submitqueue-consumer integration-test-extensions integration-test-submitqueue-gateway integration-test-submitqueue-orchestrator license-fix lint lint-binary lint-fmt lint-license local-init-runway-queue-schema local-init-stovepipe-schemas local-runway-start local-runway-stop local-submitqueue-stop local-submitqueue-clean local-submitqueue-gateway-start local-submitqueue-gateway-stop local-init-submitqueue-schemas local-submitqueue-logs local-submitqueue-orchestrator-start local-submitqueue-orchestrator-stop local-submitqueue-ps local-submitqueue-restart local-submitqueue-start local-stop local-stovepipe-debug-start local-stovepipe-logs local-stovepipe-start local-stovepipe-stop mocks proto query-deps query-targets run-client-runway run-client-submitqueue-gateway run-client-submitqueue-orchestrator run-client-stovepipe run-queue-admin test test-no-cache test-race tidy tidy-bazel tidy-go help
152155

153156

154157
build: ## Build all services and examples
@@ -263,6 +266,15 @@ demo-requests: ## Create N changes, enqueue each as it is created, and watch (PR
263266
deps: tidy-go ## Download and tidy Go dependencies
264267
@echo "Dependencies installed!"
265268

269+
docs-build: docs-venv ## Build the documentation site into tool/docsite/build (fails on broken links)
270+
@$(DOCS_VENV)/bin/mkdocs build --strict -f tool/docsite/mkdocs.yml
271+
272+
docs-serve: docs-venv ## Serve the documentation site at http://127.0.0.1:8000 with live reload
273+
@$(DOCS_VENV)/bin/mkdocs serve -f tool/docsite/mkdocs.yml
274+
275+
docs-venv: ## Create the documentation virtualenv in tool/docsite/.venv
276+
@test -x $(DOCS_VENV)/bin/mkdocs || { python3 -m venv $(DOCS_VENV) && $(DOCS_VENV)/bin/pip install -q -r tool/docsite/requirements.txt; }
277+
266278
e2e-git-test: ## Run the hermetic git E2E (real merger against a bare repo; no credentials)
267279
@echo "Running hermetic git end-to-end tests..."
268280
@$(BAZEL) test //test/e2e/submitqueue:go_default_test --test_output=errors \

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# SubmitQueue
22

33
[![CI](https://github.com/uber/submitqueue/actions/workflows/ci.yml/badge.svg)](https://github.com/uber/submitqueue/actions/workflows/ci.yml)
4+
[![Docs](https://img.shields.io/badge/docs-uber.github.io%2Fsubmitqueue-blue)](https://uber.github.io/submitqueue/)
45
[![Go Version](https://img.shields.io/github/go-mod/go-version/uber/submitqueue)](go.mod)
56
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
67
[![Slack](https://img.shields.io/badge/Slack-join%20the%20community-4A154B?logo=slack&logoColor=white)](https://join.slack.com/t/submitqueue/shared_invite/zt-46gkqj682-7zcQphxm2pYqkjDo9lbmYA)

‎doc/index.md‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
---
2+
title: Home
3+
template: home.html
4+
hide:
5+
- navigation
6+
- toc
7+
---
8+
9+
<div class="sq-github-only" markdown>
10+
11+
# SubmitQueue
12+
13+
A high-performance speculative submission queue that keeps your trunk consistently green at scale.
14+
15+
</div>
16+
17+
SubmitQueue does not validate changes one at a time. It speculatively rebases and validates many changes in parallel against predicted future states of HEAD. Changes whose validations pass land automatically. When a validation fails, SubmitQueue isolates the offending change and retries the rest, with no human involved. It is designed for large monorepos and fast-moving teams, where concurrent changes can introduce subtle conflicts and destabilize builds.
18+
19+
## Why SubmitQueue
20+
21+
<div class="grid cards sq-features" markdown>
22+
23+
- :material-source-branch-check:{ .lg .middle } **Speculative validation**
24+
25+
---
26+
27+
SubmitQueue builds a tree of possible future HEADs and validates the paths most likely to land, in parallel.
28+
29+
[:octicons-arrow-right-24: Speculation](rfc/submitqueue/speculation.md)
30+
31+
- :material-shield-check-outline:{ .lg .middle } **Isolates failures**
32+
33+
---
34+
35+
A failing change is isolated and rejected while the rest of its batch carries on to land.
36+
37+
[:octicons-arrow-right-24: Orchestrator workflow](rfc/submitqueue/workflow.md)
38+
39+
- :material-puzzle-outline:{ .lg .middle } **Pluggable extensions**
40+
41+
---
42+
43+
Build runners, change providers, storage, queues and scorers are vendor-agnostic interfaces with swappable implementations.
44+
45+
[:octicons-arrow-right-24: Extension contract](rfc/submitqueue/extension-contract.md)
46+
47+
- :material-tray-full:{ .lg .middle } **Durable, queue-driven pipeline**
48+
49+
---
50+
51+
Every stage is an idempotent consumer on an at-least-once message queue, with optimistic locking and no distributed transactions.
52+
53+
[:octicons-arrow-right-24: SQL-based queue](rfc/sql-queue-rfc.md)
54+
55+
</div>
56+
57+
## Components
58+
59+
<div class="grid cards" markdown>
60+
61+
- :material-call-merge:{ .lg .middle } **SubmitQueue**
62+
63+
---
64+
65+
The gateway and orchestrator that accept, batch, speculate on, build and land changes.
66+
67+
[:octicons-arrow-right-24: Workflow](rfc/submitqueue/workflow.md)
68+
69+
- :material-airplane-landing:{ .lg .middle } **Runway**
70+
71+
---
72+
73+
The landing service. It owns VCS operations, conflict checks and merges, on SubmitQueue's behalf.
74+
75+
[:octicons-arrow-right-24: Workflow](rfc/runway/workflow.md)
76+
77+
- :material-check-decagram-outline:{ .lg .middle } **Stovepipe**
78+
79+
---
80+
81+
The post-land pipeline. It validates landed commits and tracks the last green revision.
82+
83+
[:octicons-arrow-right-24: Workflow](rfc/stovepipe/workflow.md)
84+
85+
</div>
86+
87+
## Try it in a minute
88+
89+
You need only Docker. No repository, account or token is required.
90+
91+
```bash
92+
make local-submitqueue-start # Gateway + Orchestrator + Runway + MySQL
93+
make demo-requests # create changes, enqueue them, watch them land
94+
make local-submitqueue-stop
95+
```
96+
97+
The [Quickstart](howto/QUICKSTART.md) goes from a fake provider to a local git repository to real GitHub pull requests. Questions? Join the [Slack community](https://join.slack.com/t/submitqueue/shared_invite/zt-46gkqj682-7zcQphxm2pYqkjDo9lbmYA).

‎tool/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,10 @@ bazel build //...
2323

2424
The Bazel version is controlled by `.bazelversion` at the repository root. Update that file to change the Bazel version used by the wrapper.
2525

26+
## Documentation site
27+
28+
`tool/docsite` holds the MkDocs Material configuration that renders `doc/` as the site published at https://uber.github.io/submitqueue/. The site reads `doc/` in place; `doc/index.md` is its landing page. Links that leave `doc/` (code, package READMEs, `AGENTS.md`) are rewritten to GitHub by `hooks/repo_links.py`, so the strict build still fails on a broken link between pages. The nav is generated from the directory tree; `hooks/nav_titles.py` maps directory names to section titles (`howto` → Guides, `rfc` → Design (RFCs)), and `overrides/home.html` renders the landing-page hero. `make docs-serve` previews it locally and `make docs-build` runs the strict build. `.github/workflows/docs.yml` builds it on pull requests and deploys it from `main`. The directory is listed in `.bazelignore`, since it contains no Bazel targets.
29+
2630
## Git sandbox
2731

2832
`tool/gitsandbox` creates the bare repository that `make local-submitqueue-start PROVIDER=git` merges into. It runs before the stack starts, because Runway clones that repository at boot and fails if the target does not already exist. The result is one seed commit on the target branch. Running it again leaves an existing repository unchanged, so a restart keeps commits that earlier runs landed. The Makefile invokes it; `bazel run //tool/gitsandbox -- -sandbox-dir <dir>` is the direct form.

‎tool/docsite/hooks/nav_titles.py‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
"""Give navigation sections readable titles.
2+
3+
The nav is generated from the directory tree so new docs appear without config
4+
changes; MkDocs titles a section by capitalizing its directory name, which
5+
yields "Howto" and "Rfc". This maps directory names to display titles instead.
6+
"""
7+
8+
from mkdocs.structure.nav import Section
9+
from mkdocs.structure.pages import Page
10+
11+
SECTION_TITLES = {
12+
"howto": "Guides",
13+
"rfc": "Design (RFCs)",
14+
"runway": "Runway",
15+
"steps": "Stages",
16+
"stovepipe": "Stovepipe",
17+
"submitqueue": "SubmitQueue",
18+
}
19+
20+
# Pages listed here sort first within their section, in this order.
21+
LEADING_PAGES = ["howto/QUICKSTART.md"]
22+
23+
24+
def on_nav(nav, config, files):
25+
_retitle_sections(nav.items)
26+
_relink_previous_and_next_pages(nav)
27+
return nav
28+
29+
30+
def _retitle_sections(items):
31+
for item in items:
32+
if not isinstance(item, Section):
33+
continue
34+
item.title = SECTION_TITLES.get(item.title.lower(), item.title)
35+
item.children.sort(key=_leading_page_rank)
36+
_retitle_sections(item.children)
37+
38+
39+
def _relink_previous_and_next_pages(nav):
40+
# MkDocs links previous/next pages before on_nav runs, so a reorder must redo them.
41+
nav.pages = list(_pages_in_order(nav.items))
42+
for i, page in enumerate(nav.pages):
43+
page.previous_page = nav.pages[i - 1] if i > 0 else None
44+
page.next_page = nav.pages[i + 1] if i + 1 < len(nav.pages) else None
45+
46+
47+
def _pages_in_order(items):
48+
for item in items:
49+
if isinstance(item, Section):
50+
yield from _pages_in_order(item.children)
51+
elif isinstance(item, Page):
52+
yield item
53+
54+
55+
def _leading_page_rank(item):
56+
src = getattr(getattr(item, "file", None), "src_uri", None)
57+
return LEADING_PAGES.index(src) if src in LEADING_PAGES else len(LEADING_PAGES)

‎tool/docsite/hooks/repo_links.py‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
"""Rewrite doc links that leave docs_dir into links to the GitHub source tree.
2+
3+
The docs link to code and package READMEs elsewhere in the repository. Those
4+
targets are not part of the site, so strict mode would reject them; pointing
5+
them at GitHub keeps strict mode on for links between pages.
6+
"""
7+
8+
import os
9+
import re
10+
11+
REPO_BLOB_URL = "https://github.com/uber/submitqueue/blob/main/"
12+
REPO_TREE_URL = "https://github.com/uber/submitqueue/tree/main/"
13+
14+
INLINE_LINK = re.compile(r"(\]\()([^)\s]+)(\))")
15+
REFERENCE_LINK = re.compile(r"^(\s*\[[^\]]+\]:\s*)(\S+)(.*)$", re.MULTILINE)
16+
17+
18+
def on_page_markdown(markdown, page, config, files):
19+
docs_dir = os.path.abspath(config["docs_dir"])
20+
repo_root = os.path.dirname(docs_dir)
21+
page_dir = os.path.dirname(os.path.join(docs_dir, page.file.src_path))
22+
23+
def rewrite_target(target):
24+
if re.match(r"^[a-z][a-z0-9+.-]*:", target) or target.startswith(("#", "/")):
25+
return target
26+
path, sep, anchor = target.partition("#")
27+
resolved = os.path.normpath(os.path.join(page_dir, path))
28+
if resolved == docs_dir or resolved.startswith(docs_dir + os.sep):
29+
return target
30+
if not resolved.startswith(repo_root + os.sep):
31+
return target
32+
rel = os.path.relpath(resolved, repo_root).replace(os.sep, "/")
33+
base = REPO_TREE_URL if os.path.isdir(resolved) else REPO_BLOB_URL
34+
return base + rel + sep + anchor
35+
36+
def rewrite_prose(text):
37+
text = INLINE_LINK.sub(lambda m: m.group(1) + rewrite_target(m.group(2)) + m.group(3), text)
38+
return REFERENCE_LINK.sub(lambda m: m.group(1) + rewrite_target(m.group(2)) + m.group(3), text)
39+
40+
# Even-indexed chunks are prose, odd-indexed chunks are fenced code.
41+
chunks = re.split(r"(^\s*(?:```|~~~).*?^\s*(?:```|~~~)[^\n]*$)", markdown, flags=re.MULTILINE | re.DOTALL)
42+
return "".join(rewrite_prose(c) if i % 2 == 0 else c for i, c in enumerate(chunks))

0 commit comments

Comments
 (0)