[crybo-rybo]

Problem

On an M3 Pro (J516S, T6030, G15 backend), a sustained Vulkan compute client such as llama.cpp under Ollama delays every other GPU client by 40-200 ms, the compositor included, so the whole desktop stutters while a model generates (#387). Context priority doesn't help: file.rs refuses HIGH on anything but G17, and on G15 priority only orders the host pick anyway.

The G15 engine runs one VM at a time. Firmware::publish() stops at the first picked job whose VM differs from the newest flight's, so a job of another VM waits until every published flight has retired. A compute client keeps up to MAX_INFLIGHT (2) flights published, and the compute horizon folds up to COMPUTE_HORIZON (16) of its ready jobs into one prefix. That is bounded by the free compute slots, not by time. With llama.cpp, one SUBMIT is one compute command of ~100 dispatches and ~6-8 ms of GPU time (gemma4:12b), ~11-13 per token, with no CPU wait between them (estimates from the graph size, the eval rate and the stats counters). So the compositor waits for the rest of a token, and asahi-g15/stats shows horizons of 2-16 jobs and a longest compute prefix interval of 244 ms.

ASAHI_MESA_DEBUG=sync confirms this: it keeps one llama.cpp job in the kernel at a time, so no horizon forms and no second flight is published. That restores the idle p50, but it costs ~70% of the throughput because the CPU records the next command buffer while the GPU idles. So the fix belongs in the engine's publication policy.

Change

Policy only, in drivers/gpu/drm/asahi/g15/. No new firmware interface is used: the encoding, barriers, stamps, slots, the one-VM rule and recovery are untouched. While contended, the firmware sees more, smaller compute prefixes and more VM switches.

• Contention signal (engine.rs:666-683): when publish() holds a job of another VM behind published flights, it records contended_ns. The engine counts as contended for g15_share_window_ms after that (is_contended(), Engine::share_budget(), engine.rs:489, :546). Testing for "a job of another VM is pending" doesn't work: that job would have won the pick. The moment it was held is the reliable signal.
• While contended:
  • Nothing is published behind a flight that holds computes, so one such prefix is in flight instead of MAX_INFLIGHT (share_inflight(), gated on tail.computes != 0 in publish()). Render-only flights are not budgeted and keep their second flight, so publication among render-only clients is exactly as before. A client whose jobs mix compute commands into its frames (geometry or tessellation lowered to compute, WebGPU) is budgeted like a compute client while contended. A VM's own budgeted compute prefix may still follow its render-only flight; the waiter then sees one render prefix more than the budget.
  • The computes of a prefix, horizon included, are capped at g15_share_budget_us divided by the queue's mean compute command time, and never below one while a compute slot is free (share_cap(), engine.rs:567; applied in publish_prefix_into() at engine.rs:943).
  • The member loop and the horizon stop at the cap. The horizon reports a new stop reason 4, "budget". The submit-ahead estimate (horizon_ready()) gets the same cap.
  • Since at least one command is always admitted, the E2BIG/EAGAIN path is unchanged.
  • The cap never reorders anything within a queue: a capped job keeps its place with sent advanced.
• The mean is an EWMA (weight 1/8) per logical queue: QueueState.avg_compute_ns (queue.rs:56). stat_retire() feeds it at retirement from the CDM stage start/end ticks (24 MHz) it already reads for each compute slot, counting only the valid intervals (poll.rs:538-588, ewma_update()/ticks_ns() at engine.rs:578/:589). A queue with no sample yet gets one command per prefix while contended.
• Uncontended, or with g15_share_budget_us=0: publication is exactly as before. A lone compute client keeps two flights and the full horizon, so its publication decisions are unchanged; a matched headless throughput comparison was not run.

The module doc at the top of engine.rs describes the budget. Commit 2 adds tools/asahi/g15-share-budget-tests.py and registers it in tools/aurora-ci/host-tests.json (Rust 1.85.0 pin).

Testing

J516S (MacBook Pro 16" M3 Pro, T6030 G15S), Omarchy 4.5.0, Hyprland, AC power, kernel 7.2.11-14-1-ARCH (linux-aurora 7.2.11.aurora14-1). The load is gemma4:12b generating throughout under Ollama (Vulkan). The probe is a GLES clear plus glFinish, 300 samples, sleeping 16 ms after each; its source is in #387. The numbers are the probe's GPU completion latency, not displayed-frame latency, and "over 8.3 ms" is the share of probe samples, not of frames. In ms:

Condition	p50	p95	max	over 8.3 ms	eval rate
idle, no load	0.84	1.03	3.47	0%
stock module	54.28	197.43	202.83	62%	12-14 tok/s
stock + ASAHI_MESA_DEBUG=sync (control, not a fix)	0.81	11.28	22.70	8%	3.65 tok/s
v1, superseded (one flight behind every flight)	8.83	26.86	28.36	62%	13.99 tok/s
v1, superseded, + GGML_VK_MAX_NODES_PER_SUBMIT=20 in Ollama	4.65	10.00	15.61	6.7%	13.61 tok/s
this PR (budget 4000 µs, window 500 ms)	9.52	26.33	33.80	77%	14.31 tok/s
this PR + GGML_VK_MAX_NODES_PER_SUBMIT=20 in Ollama	1.01	11.03	14.44	10%	13.48 tok/s

• The patch alone: p50 drops from 54 to 9.5 ms and the maximum from 203 to 34 ms, with no throughput loss observed (14.31 tok/s; stock 12-14 across runs, so differences of a few percent are within that spread). The cap is one command, so the p50 is about one llama.cpp command plus the switch. That sits at the 8.3 ms threshold, so the over-8.3 ms share swings between runs (62% v1, 77% here) without the distribution moving.
• With 20 nodes per submit: llama.cpp ends a command buffer every ~20 graph nodes instead of ~100, so commands are ~5x shorter and the budget has finer pieces to cut at. p50 1.0 ms, p95 11 ms, max 14 ms, for 3-6% of the eval rate (one run each).
• v1 vs this PR: llama.cpp's publication is identical under both (its flights always hold computes); the probe's render side is back on stock pipelining here, which is most likely where the lower p50 in the 20-nodes row comes from. The hardware runs used the current code; the final revision differs from the measured module by comments only.
• The sync control is the diagnosis: one job in flight gives the idle p50, which points at the horizon and second flight rather than at the engine's turnaround. On that boot the gaps averaged ~14 µs each.
• In normal desktop use during generation, the patch alone is a clear improvement; with 20 nodes per submit I did not notice stutter.

Counters from asahi-g15/stats in the v1 boot (+ marks a delta over the run; v1 counted render-only prefixes in share: flights too, this PR counts compute prefixes only):

• share: flights +20708 capped 8071 hz_budget 8071 hold_vm +162563
• compute count 20614 prefixes 18810 (1.1 commands per prefix; the stock boot's cumulative stats were count 111291 prefixes 58839)
• horizons: 17156 of one job and 1543 of two. The stock boot showed horizons of 2-16 jobs.
• compute max_us 462055: one prefix took 462 ms from publication to retirement. That interval includes any wait behind its predecessor, and the counters alone don't isolate a command; with the cap at one it was most likely a single prompt-processing command, but that attribution is inferred (see the limits below).
• journalctl -k: no asahi faults, timeouts, halts or recoveries in either patched boot.

How it was built and loaded; host tests; checkpatch

• Build: asahi.ko as an out-of-tree module against the installed 7.2.11-14-1-ARCH at 4fdd0fc5c931, with the recipe's toolchain: clang 18.1.8, rustc 1.85.0, bindgen 0.73.2 (tools/aurora-bindgen), pahole 1.32. The installed .config was used unchanged.
• Module checks: vermagic, dependencies and device aliases match stock, and all 512 undefined symbols are exported by the installed kernel. .BTF and .BTF.base are present.
• Warnings: the same 30 rustc warnings as an unmodified build of the base commit, so none are new.
• Load: asahi loads from the initramfs, so the module was loaded from a second Limine UKI entry whose initramfs carries it. The default entry stayed stock as the fallback. The build-id was checked after boot.
• Host tests (host-tests.py --only, the six g15 suites that read the changed files): input-error, pb-growth, stats-drain, queue-credit, probe-init and the new share-budget suite pass. In the new suite (--mutations), all 15 runtime mutants of the policy bodies die at their intended assertions, and 6 source-level wiring mutants (the share_cap budget argument zeroed, the share_inflight condition forced false, the compute gate dropped, the hold count using the ungated limit, the share stats counting render prefixes, the horizon's capped mark dropped) fail the source checks they target. The CI self-test (148 tests) is OK.
• checkpatch: checkpatch.pl --strict reports nothing in the code apart from two module parameter description strings over 100 columns, which match the existing g15_* descriptions. Its only trailer remark is the Co-Authored-By casing, which other commits in this tree use too.

Parameters

Both parameters are load-time only: module! registers every parameter with permissions 0, as with the other g15_* parameters. Set them with asahi.g15_share_budget_us=… on the kernel command line, or with options asahi … in modprobe.d, which needs a regenerated initramfs because asahi loads from it. The share: line prints both, so a capture records which policy ran.

• g15_share_budget_us (u32, default 4000): the compute GPU time of one VM's flight (one published prefix) while another VM waits. 0 turns the policy off; other values are clamped to 100..1 000 000 µs.
  • 4000 µs is under half a 120 Hz frame (8.3 ms). It is the same value as the T8122 runtime's m3_render_batch_budget_us default.
  • With 7 ms commands it gives one command per prefix; with 0.5 ms commands, eight.
  • Other values were not measured.
• g15_share_window_ms (u32, default 500, clamped to 1..10 000): how long after a held job the budget applies.
  • The window is refreshed only when a job of another VM is held. Hyprland redraws on damage, so typing gives a frame about every 100 ms; a 100 ms window would lapse between keystrokes and let a keystroke's frame land behind an unbudgeted flight. 500 ms covers typing and scrolling pauses.
  • The cost is one VM-switch drain per compute command (~30-150 µs, about 2%) for half a second after the desktop goes quiet.

asahi-g15/stats gets a new last line; every existing line keeps its format:

share: flights N capped N hz_budget N hold_vm N budget_us N window_ms N

The fields count prefixes with computes published under the budget, those it cut short, horizons it ended, and holds of a job of another VM behind published flights (also hold[3]), followed by the two parameters. hold[0] on the horizon line now also counts holds at the budget's one-flight limit.

Limits and follow-ups

• Prefill is one command. The kernel can't split a command, so a prompt-processing command (most likely the 462 ms prefix above) still holds every other client for its full length. Bounding that needs Honeykrisp to end the CDM control stream every K dispatches on G15 instead of merging a whole command buffer into one stream (hk_cmd_buffer.c); a single long dispatch stays indivisible even then. I'll propose that separately in omacom/mesa.
• The budget is a count, not a timer. It admits as many commands as fit the queue's mean, so a sudden change in command cost overshoots until the EWMA catches up, one command is always admitted, and a mixed prefix's renders are outside the estimate. It is not preemption and not a hard 4 ms bound.
• The first competing job after a quiet spell still waits for whatever was published before its hold recorded the contention: up to two unbudgeted flights, or a full horizon. The window only shapes publication after that. The probe keeps the contention warm, so this first-interaction hitch is not in the table and was not measured separately. With several VMs or priorities competing, a waiter can also see more than one budgeted prefix.
• GGML_VK_MAX_NODES_PER_SUBMIT=20 is a complementary user-space knob: it makes llama.cpp's commands shorter so the budget can cut finer. It goes in an Ollama service drop-in (Environment=GGML_VK_MAX_NODES_PER_SUBMIT=20). It gave the best result above for 3-6% of the eval rate, but it does not guarantee short prefill commands: the node threshold applies to graph submission generally, and a single dispatch can dominate. I measured it only with this patch, not on the stock module.
• M3 Air (T8122) is not changed by this PR. From reading m3/m3_runtime.rs and m3/m3_batch_policy.rs, not from testing:
  • The T8122 runtime already avoids this stall by construction. A compute batch takes commands of one packet only (may_join() admits later packets into render batches only), and pipeline_slot() never publishes compute behind an active batch.
  • So another client waits for at most one llama.cpp job there.
  • m3_render_batch_budget_us bounds render batches only.
  • A long prefill command would stall it the same way. T6032 also binds the M3 runtime; only T6030 and T6031 run the G15 engine.

Not tested:

• T6031 (M3 Max, G15C), T6032 and T8122 boards
• compute clients other than Ollama's llama.cpp, and models other than gemma4:12b
• render clients that submit computes (games with geometry or tessellation lowered to compute, WebGPU)
• external displays
• g15_share_budget_us=0 on hardware (the host tests cover the policy-off path)
• budget or window values other than the defaults
• a matched headless throughput comparison (stock vs. patched, no other GPU client)
• the first interaction after the window has lapsed
• runs longer than about an hour

Merge Danger

Door: two-way. The change is policy only, and asahi.g15_share_budget_us=0 restores today's publication without a rebuild.

Blast Radius: small to medium. It affects the G15 backend only, so T6030 and T6031, and only while a job of another VM has recently been held. Then a client whose flights hold computes publishes one prefix at a time, which costs one VM-switch drain per prefix. Render-only flights are not budgeted. Render clients whose jobs include compute commands are budgeted like compute clients while contended.

Revisions

• v1: the one-flight limit applied behind every flight. Superseded; see the v2 comment.
• v2: the limit applies only behind a flight that holds computes; wiring assertions and mutants in the host test; wording corrected. Patch-alone row added in the follow-up comment.
• v2.1: comments, docs and the test file polished after a second local review; code unchanged. Body corrected where it still quoted v1 numbers.

Links: #387 (report, probe source), the source-inspection comment with the horizon/flight diagnosis.

🤖 Generated with Claude Code

[crybo-rybo]

v2 pushed (6beaf9d7a3dc + 79498706cd66, same base), after a local review of v1.

• Scope narrowed: in v1 the one-flight rule applied to every flight while contended, so two render clients that contended (compositor + browser/game, no compute running) lost their second flight for 500 ms, which the measurements never covered. v2 applies it only behind a flight that holds computes (tail.computes != 0), so render-only publication is exactly as before and llama.cpp's path is unchanged. Residual, documented: a VM's own budgeted compute prefix may follow its render-only flight, so a waiter can see one render prefix more. share: flights/capped now count compute prefixes only.
• Tests: the host suite now asserts both call sites and the gate; --mutations adds 5 source-level wiring mutants (both call sites disconnected, the gate dropped, the hold count ungated, the stats condition widened), all of which fail their check. 15 runtime mutants as before. Six G15 host suites, the 148 CI self-tests, check-commits and git diff --check pass.
• Wording: the probe measures GPU completion latency, not frames; max_us 462055 is a prefix publish-to-retire interval and the single-command attribution is inferred; "no throughput cost" is now "no loss observed in this run"; the limits say the budget is a count estimate, not a timer, and that the first interaction after a quiet spell still waits for the flights already published.
• Re-measured on the same M3 Pro, v2 module (build-id c450ad98…, same build recipe and checks) with GGML_VK_MAX_NODES_PER_SUBMIT=20: p50 1.01 / p95 11.03 / max 14.44 ms, 10% of samples over 8.3 ms, 13.48 tok/s, no asahi fault in the kernel log. v1 in that configuration: 4.65 / 10.00 / 15.61, 6.7%, 13.61 tok/s. One run each; the lower p50 is most likely the probe's render side back on stock pipelining. The patch-alone row was not repeated.

The PR description carries the full table and a revision history.

[crybo-rybo]

Patch-alone row for v2 added to the table (message-only re-commit, 70ce42884450 + af8b3f8f13f4, same trees): p50 9.52 / p95 26.33 / max 33.80 ms, 77% of samples over 8.3 ms, 14.31 tok/s, same boot and module, Ollama drop-in removed. That matches v1's patch-alone row (8.83 / 26.86 / 28.36, 13.99 tok/s); with ~7 ms commands the p50 sits at the 8.3 ms threshold, so the over-8.3 share swings between runs (62% vs 77%) without the distribution moving.

[crybo-rybo]

v2.1 (6aeb9169fcd2 + bed2d45390f9): no code change (git diff 70ce42884450 6aeb9169fcd2 -- drivers touches comments only), after a second local review of v2.

• Docs: share_inflight's doc described the wrong quantity; fixed. Stats.hold[0] now says it also counts holds at the budget's one-flight limit, and the share: stats comment says compute prefixes. Module doc shortened.
• Claims qualified: render clients whose jobs include compute commands (geometry/tessellation lowered to compute, WebGPU) are budgeted like compute clients while contended; only render-only flights are untouched. "Nothing firmware-visible changes" became "no new firmware interface; more, smaller compute prefixes and more VM switches while contended".
• Commit message rewritten: results rows labelled by revision, the quoted counters labelled as the earlier revision's (it counted render-only prefixes in share flights, which is why +20708 exceeds the 18810 compute prefixes), wrap artefacts removed.
• Test: source checks tolerate whitespace, two self-referential checks removed, the stats-line check allows later additions, a 6th wiring mutant (horizon's capped mark), mutant tables at module level.
• Body: the results bullets and the eval-rate cost now use this revision's numbers (max 34 ms, 3-6%), v1 rows marked superseded, build details folded.