ggml : reuse compute buffers for MTP (#27282)

[mushang0]

Overview

Fixes #27282
Native MTP creates a target context and an MTP context, each requiring independent scheduler state. Both currently reserve separate GPU compute backing storage, though in the validated native MTP path they never use the compute workspace concurrently. Near the VRAM limit this duplicate reservation causes OOM on the second allocation.

This PR keeps schedulers and allocation plans independent but shares the physical compute buffers via reference counting. A scheduler detaches and allocates independently if it later needs a larger reservation. Automatic enablement is restricted to the validated single-sequence, single-CUDA-device native MTP path; all other cases fall back to independent buffers.

A narrow scheduler-level API is added because no existing interface can express shared backing storage with independent scheduler state.

Additional information

Tested on: RTX 4090 · Qwen3.8-27B · native MTP

Memory

Metric	Master	Patch	Delta
Peak GPU memory	22,723 MiB	21,681 MiB	−1,042 MiB (~1.02 GiB)

OOM regression

With 1,024 MiB external GPU pressure:

• master: startup OOM
• patch: starts successfully, /health returns HTTP 200

Correctness / Performance

512/2048 token runs: output hash and accepted/generated counts identical between master and patch. No meaningful performance regression observed (512 tok: 74.63 → 75.28 t/s; 2048 tok: 73.27 → 73.49 t/s).

Tests

• test-alloc — PASS
• test-backend-ops — 13,348/13,348 PASS
• ASAN / UBSAN — PASS
• llama-bench, llama-perplexity — PASS
• CPU local CI — PASS

Default CUDA local CI is blocked by a pre-existing -Werror=stringop-overflow in unmodified ggml-cuda.cu; reproduces identically on clean master.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES — AI was used to explore the codebase, analyze design options, assist with implementation, and verify tests. This PR description was written by the author.

[ggml-gh-bot]

Hi @mushang0, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[mushang0]

I verified the implementation, design, testing, and PR description per guidelines. AI was used solely for exploration, debugging, and testing.