Skip to content

ci: tiered fail-fast pipeline — stop waiting 20min for already-failed CI - #61

Merged
ezutfen merged 1 commit into
mainfrom
ci/tiered-workflow-orchestrator
Jul 12, 2026
Merged

ci: tiered fail-fast pipeline — stop waiting 20min for already-failed CI#61
ezutfen merged 1 commit into
mainfrom
ci/tiered-workflow-orchestrator

Conversation

@ezutfen

@ezutfen ezutfen commented Jul 12, 2026

Copy link
Copy Markdown

Problem

When a PR fails CI in the first 3 minutes (e.g. larql-models test failure), the other ~15 jobs keep running for 20 minutes because the 13 per-crate workflows are independent — none knows about the others failures.

Solution

A single ci.yml orchestrator calls the 13 per-crate workflows as reusable workflows (workflow_call), arranged in dependency tiers:

Tier 0 (leaf crates):     models, core, boundary
Tier 1 (substrate):       compute, compute-cuda, compute-vulkan
Tier 2 (engine):          inference, vindex
Tier 3 (API layer):       kv, lql
Tier 4 (applications):    server, cli, shannon-verify

A failure in any tier immediately cancels all downstream tiers. Within each tier, all jobs run in parallel across the 4 self-hosted runners — no parallelism lost within a tier.

Changes

  • New: ci.yml — single CI entry point (triggers on pull_request + push to main). Calls per-crate workflows via uses: with needs: chains.
  • Modified: 13 per-crate workflows — added workflow_call trigger, removed push/pull_request triggers (orchestrator handles those now). Each retains workflow_dispatch for manual/standalone runs.
  • Unchanged: bench-regress.yml, larql-compute-metal.yml (macOS-only, manual), ci-image.yml (image builder, push-only).

Impact

Scenario Before After
All green ~20 min ~20 min (same)
Tier 0 failure (leaf crate) ~20 min ~3-5 min
Tier 2 failure (inference) ~20 min ~8-10 min
Duplicate runs on PRs Yes (13 independent triggers) None (single orchestrator)

Path-filter optimization sacrificed (every PR runs full pipeline), but with sccache + persistent cargo volume most crates build in 2-4 min, and tiered gating means fast failures without waiting for slow jobs.

Verification

  • All 17 workflow files pass YAML validation
  • DAG verified: no cycles, 5-tier critical path
  • No duplicate triggers: ci.yml is the only workflow with pull_request/push

Restructure CI to eliminate the '16 jobs run for 20min when 1 failed in
3min' problem. A single ci.yml orchestrator calls the 13 per-crate
workflows in dependency tiers:

  Tier 0: models, core, boundary (leaf crates)
  Tier 1: compute, compute-cuda, compute-vulkan
  Tier 2: inference, vindex
  Tier 3: kv, lql
  Tier 4: server, cli, shannon-verify

A failure in any tier cancels all downstream tiers immediately. Within
each tier, jobs run in parallel across the 4 self-hosted runners.

Changes:
- Create ci.yml as the single entry point (pull_request + push triggers)
- Add workflow_call trigger to all 13 per-crate workflows
- Remove push/pull_request triggers from per-crate workflows (no duplicate runs)
- Per-crate workflows retain workflow_dispatch for manual/standalone runs
- Standalone workflows unchanged (bench-regress, compute-metal, ci-image)

The dependency tiers follow the Cargo workspace dep graph:
  models → compute → inference → kv → cli → shannon-verify
  core → vindex → lql → server
  boundary
@ezutfen
ezutfen merged commit eb40807 into main Jul 12, 2026
35 of 36 checks passed
@ezutfen
ezutfen deleted the ci/tiered-workflow-orchestrator branch July 12, 2026 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant