ci: tiered fail-fast pipeline — stop waiting 20min for already-failed CI - #61
Merged
Conversation
Restructure CI to eliminate the '16 jobs run for 20min when 1 failed in 3min' problem. A single ci.yml orchestrator calls the 13 per-crate workflows in dependency tiers: Tier 0: models, core, boundary (leaf crates) Tier 1: compute, compute-cuda, compute-vulkan Tier 2: inference, vindex Tier 3: kv, lql Tier 4: server, cli, shannon-verify A failure in any tier cancels all downstream tiers immediately. Within each tier, jobs run in parallel across the 4 self-hosted runners. Changes: - Create ci.yml as the single entry point (pull_request + push triggers) - Add workflow_call trigger to all 13 per-crate workflows - Remove push/pull_request triggers from per-crate workflows (no duplicate runs) - Per-crate workflows retain workflow_dispatch for manual/standalone runs - Standalone workflows unchanged (bench-regress, compute-metal, ci-image) The dependency tiers follow the Cargo workspace dep graph: models → compute → inference → kv → cli → shannon-verify core → vindex → lql → server boundary
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a PR fails CI in the first 3 minutes (e.g. larql-models test failure), the other ~15 jobs keep running for 20 minutes because the 13 per-crate workflows are independent — none knows about the others failures.
Solution
A single
ci.ymlorchestrator calls the 13 per-crate workflows as reusable workflows (workflow_call), arranged in dependency tiers:A failure in any tier immediately cancels all downstream tiers. Within each tier, all jobs run in parallel across the 4 self-hosted runners — no parallelism lost within a tier.
Changes
ci.yml— single CI entry point (triggers onpull_request+pushto main). Calls per-crate workflows viauses:withneeds:chains.workflow_calltrigger, removedpush/pull_requesttriggers (orchestrator handles those now). Each retainsworkflow_dispatchfor manual/standalone runs.bench-regress.yml,larql-compute-metal.yml(macOS-only, manual),ci-image.yml(image builder, push-only).Impact
Path-filter optimization sacrificed (every PR runs full pipeline), but with sccache + persistent cargo volume most crates build in 2-4 min, and tiered gating means fast failures without waiting for slow jobs.
Verification
ci.ymlis the only workflow withpull_request/push