feat(core): ryzenai-corelib 0.53.0, Qwen3.8-27B, Q4 Qwen3.6, paged context to each model's maximum

docs/core-setup-linux.md

@@ -9,7 +9,7 @@ Written against the working install on this machine. Paths use `$P` for the
 dependency prefix (here `/home/alfxu/ddprefix`); substitute your own.
 
 > **Status:** the core backend has been run on hardware on Windows only. The
-> steps below are updated for corelib 0.33.1, but they have not been run end
+> steps below are updated for corelib 0.53.0, but they have not been run end
 > to end on Linux at this revision. The [Known issues](#known-issues) are from
 > an earlier Linux bring-up.
 
@@ -52,7 +52,7 @@ loaded from, so those libraries must sit beside it; nothing else is needed at
 run time, no kernel store and no environment variable.
 
 **Version must match exactly.** `src/include/core/corelib_api.hpp` has a
-`#error` for anything other than corelib **0.33.1**:
+`#error` for anything other than corelib **0.53.0**:
 
 ```bash
 grep -E 'CORELIB_VERSION_(MAJOR|MINOR|PATCH)' \
@@ -235,10 +235,10 @@ shape is a dead end.
 |---|---|
 | `RYZENAI_CORELIB_TRACE=xrt` | trace XRT calls (`kernels,xrt` for both logs, `all` for every category) |
 | `RYZENAI_CORELIB_SAFE_MODE` | force staging buffers, no host-pointer import |
-| `RYZENAI_CORELIB_NO_RUN_POOL` | disable run-object reuse |
 | `RYZENAI_CORELIB_NO_CRASH_HANDLER` | leave crashes to the debugger |
-| `RYZENAI_CORELIB_REPORT_FALLBACKS=1` | when a stream is released, report host (CPU fallback) time per artifact, and every dispatch that ran on a later PDI than its first choice; no output means nothing fell back |
-| `RYZENAI_CORELIB_MAX_RUNS_IN_FLIGHT` | how many runs of one kernel a stream lets run ahead (64 unset, `0` for no cap) |
+| `RYZENAI_CORELIB_REPORT_FALLBACKS=1` | when a stream is released, report host and GPU time per artifact, and every dispatch that ran on a later PDI than its first choice; no output means nothing fell back |
+| `RYZENAI_CORELIB_DISABLE_GPU=1` | keep streams created while it is set off the APU's GPU, so whatever no NPU kernel takes runs on the host |
+| `RYZENAI_CORELIB_PROFILE` | `1` prints a per-phase device-time report when a stream is released; a path also writes it as JSON |
 | `RYZENAI_CORELIB_KERNELS_ROOT` | a kernel store to load from, for corelib development only; every model library already carries its kernels, so a normal run sets nothing |
 
 ### Other rough edges

docs/docs/models/phi.md

@@ -27,11 +27,11 @@ flm run phi4-mini-it:4b
 
 ## 🧪 Model Card: Phi-4-mini-instruct on the core backend (developer preview)
 
-- **Tag:** `phi4-mini-it-core:4b` — its own tag, beside `phi4-mini-it:4b`, which stays the NPU2/Q4NX build. They are two entries in the one catalog, `model_list.json`, describing two different packages, and nothing about one is a patch on the other. Two things decide whether an install offers this one, and **the first is the machine**: `supported_platforms` says `dev04` here and `aie2p` there, checked against the generation FastFlowLM reads from the device, so on shipping silicon this tag is not listed even by a build that has corelib compiled in. Only then does the second apply — whether corelib's kernels were linked at all, which the `-core` suffix on the family name is what asks for. `utils::get_device()` is still a stand-in and currently answers `dev04` for every build, so a corelib developer build offers this tag and not the NPU2 one; a build without corelib on the same setting offers no models at all, and says so.
+- **Tag:** `phi4-mini-it-core:4b` — its own tag, beside `phi4-mini-it:4b`, which stays the NPU2/Q4NX build. They are two entries in the one catalog, `model_list.json`, describing two different packages, and nothing about one is a patch on the other. Two things decide whether an install offers this one, and **the first is the machine**: `supported_platforms` says `dev04` here and `aie2p` there, checked against the generation FastFlowLM reads from the device, so on shipping silicon this tag is not listed even by a build that has corelib compiled in. Only then does the second apply — whether corelib's kernels were linked at all, which the `-core` suffix on the family name is what asks for. A corelib build asks corelib which NPU the machine has, so on a dev04 NPU it offers this tag and not the NPU2 one, and on an aie2p NPU the reverse.
 - **Backend:** `core` — the backend names the kernel provider; FastFlowLM reaches these kernels through AMD's `ryzenai_corelib`
 - **Source format:** GGUF, read directly by corelib's own `phi4mini` model, the way Qwen3, Qwen3.5, Qwen3.6 and Gemma 4 run. No ONNX model, no tensor manifest, and no converted or packed weight file is produced or shipped.
 - **Quantization:** whichever GGUF `flm pull` fetched. AMD's **`Q4_G128`** file, from Phi-4-mini's ONNX export, is packed exactly as it is, apart from its FP16 scales narrowing to BF16. The fallback, unsloth's **`Q8_0`**, is requantized to **group 64** on every load, which is a **lossy** second quantization step, so its output differs from the Q8_0 source.
-- **Usable generation window:** 4095 tokens — the rendered prompt plus the requested output together, so the largest admissible prompt is 4094. An over-capacity request is rejected with HTTP 400 *before* any work is submitted to the device. Note this is far below the model's 128k context; see below for why.
+- **Context:** up to 131072 tokens, and 131072 by default, the most corelib runs Phi-4-mini at. The usable generation window is one less than the context: the rendered prompt plus the requested output together. An over-capacity request is rejected with HTTP 400 *before* any work is submitted to the device.
 - **Availability:** Windows and Linux, and this is a **developer build**. The core runtime is not packaged by the MSI, Inno or snap installer; you build against corelib yourself.
 
 On core this tag pulls from pinned repositories, because the GGUF publishers do not ship the tokenizer files FastFlowLM's tokenizer frontend consumes. The GGUF comes from AMD's repository when `flm pull` can reach it: it is private for now, so that takes a Hugging Face token with access to it (`HF_TOKEN`, or `hf auth login`). Otherwise the pull falls back to unsloth's:
@@ -76,23 +76,20 @@ The `src/test/phi4_core` suite is still Windows-only and is not configured on Li
 
 ### Pointing FastFlowLM at the runtime
 
-A core build (`-DFLM_ENABLE_CORE=ON`) **links corelib in**, because the NPU device the whole process shares is the one behind corelib's hardware context rather than a device FastFlowLM opens itself. Point the build at the library with `RYZENAI_CORELIB_INCLUDE_DIR` and `RYZENAI_CORELIB_LIBRARY`. `FLM_CORE_CORELIB_PATH` selects a shared library (`.dll` on Windows, `.so` elsewhere) only in the older dynamically loading configuration; in a statically linked core build it is ignored, and `flm` says so if it is set.
+A core build (`-DFLM_ENABLE_CORE=ON`) **links corelib in**. corelib keeps its NPU device to itself and opens a hardware context per stream on it; FastFlowLM opens its own device for every other family, and a core model shares no buffer with those. The core backend is offered only where corelib reports a usable NPU (`ryzenai_corelib_npu_usable`): elsewhere corelib would run a session on the GPU or the host. Point the build at the library with `RYZENAI_CORELIB_INCLUDE_DIR` and `RYZENAI_CORELIB_LIBRARY`. `FLM_CORE_CORELIB_PATH` selects a shared library (`.dll` on Windows, `.so` elsewhere) only in the older dynamically loading configuration; in a statically linked core build it is ignored, and `flm` says so if it is set.
 
-The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.33.1`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build installs `ryzenai_corelib` plus one `ryzenai_corelib_model_<name>` library per model, each carrying that model's kernels; corelib loads them only from its own directory, so they must sit beside it (a Windows core build copies them next to `flm.exe`). Each engine enables its model's kernel bundle on the stream it opens; no kernel store or environment variable is involved. Corelib's own dependency directory must be reachable on `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux).
+The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.53.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. corelib's default build installs `ryzenai_corelib` plus one `ryzenai_corelib_model_<name>` library per model, each carrying that model's kernels; corelib loads them only from its own directory, so they must sit beside it (a Windows core build copies them next to `flm.exe`). Each engine enables its model's kernel bundle on the stream it opens; no kernel store or environment variable is involved. Corelib's own dependency directory must be reachable on `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux).
 
 ```powershell
 flm pull phi4-mini-it-core:4b
 flm run  phi4-mini-it-core:4b
 ```
 
-### Why the context is 4096, and why the usable window is one less
+### Context, and why the usable window is one less
 
-Phi-4-mini itself supports 128k, and the existing `phi4-mini-it:4b` tag defaults to 32k. This backend gives you 4095. That is a real functional regression and it has two separate causes, which are worth keeping apart.
-
-**The 4096 ceiling is a correctness boundary, not a buffer size.** 4096 is exactly Phi-4-mini's `rope.scaling.original_context_length`. LongRoPE selects its factors by *sequence length*, not per position: at or below the original length the short factors apply, above it the long ones do. corelib's `phi4mini` reads only the short factors, so 4096 is the point past which the rope tables would silently be wrong. It is enforced rather than assumed — corelib refuses a GGUF whose `phi3.rope.scaling.original_context_length` is not exactly 4096. Raising this ceiling means deriving the long factors, not enlarging an array.
+corelib runs Phi-4-mini to its own 131072 tokens on paged attention, growing the KV cache a page at a time. LongRoPE selects its factors by *sequence length*, not per position: corelib ropes every position with the short factors at a capacity of one 4096-token page and with the long factors past it, as llama.cpp does, so a context set to 4096 or less takes the short ones and anything larger the long ones. A GGUF without `rope_factors_long` runs 4096 only.
 
 **The extra −1 is the core backend's.** It decodes at most one token fewer than its context, so that any request the server admits has room to finish rather than failing partway. It costs exactly one token.
-
 ### No fallback
 
 Backend selection follows the tag, never a filename or a quantization level. The `-core` suffix *is* the request for corelib's kernels, and the entry's `supported_platforms` decides whether this machine is offered it at all. There is nothing to fall back to and nothing to resolve between: `phi4-mini-it:4b` and `phi4-mini-it-core:4b` are separate tags, so asking for one never gets you the other. Once this tag is selected, there is no fallback: if corelib is missing, unloadable, or the wrong version, the tag **fails to load with a diagnostic** rather than quietly running on CPU or on the NPU2/Q4NX backend.

docs/docs/models/qwen.md

@@ -109,25 +109,36 @@ flm run qwen3:8b
 
 - **Tags:** `qwen3-core:0.6b`, `qwen3-core:1.7b`, `qwen3-core:4b`, `qwen3-core:8b` — their own tags, beside the NPU2 `qwen3:*` entries. `supported_platforms` is `dev04`, and the `-core` suffix is what asks for corelib's kernels.
 - **Backend:** `core` — one engine for all four sizes. The GGUF's own metadata selects the size and is cross-checked against it.
-- **Source format:** GGUF `Q8_0`, read directly. Every weight is requantized to **group 64** while it is packed, except the 0.6B lm_head, which packs at **group 32**.
-- **Usable generation window:** 4095 tokens, prompt plus output. Over-capacity requests are rejected with HTTP 400 before any work reaches the device.
-- **Availability:** developer build against corelib 0.33.1 — see [Phi-4 on the core backend](phi.md).
-
-The same corelib path also runs `gemma4-it-core:e2b` and `gemma4-it-core:e4b` (text only), `qwen3.5-core:9b` and `qwen3.6-moe-core:35b-a3b` (text only). Qwen3 stays at 4096; the other three go further.
-
-FastFlowLM sends a long prompt to these models in chunks of `max_prefill_len` (4096) tokens that end on multiples of 4096 counted from the start of the conversation, not from the start of the turn. A later turn's prompt therefore starts with at most one chunk inside a 4096-token page, and every chunk after that starts on a page boundary, which is where corelib runs a whole page as one pass.
-
-- **`qwen3.5-core:9b` — up to 32768 tokens.** The default is 4096; raise it with `--ctx-len` (`-c`). Past 4096 the context is rounded up to whole 4096-token pages. Long contexts have three costs:
-    - **Memory, up front.** The KV cache is allocated when the model is created, at the full capacity: 32 KiB per token (8 attention layers × 4 KV heads × 256 × BF16, for K and V), so 32768 tokens is 1 GiB on top of the weights.
-    - **Prompts run at decode speed.** corelib's Qwen3.5 has no batched prefill: every prompt token is one decode step, so a long prompt takes as long as generating that many tokens.
-    - **Accuracy past the first page.** Attention past 4096 tokens runs on a paged decode kernel that drifts a little further from the reference with each page it reads (corelib's `PAGED_DECODE_PAGE_DRIFT`). corelib reports Qwen3.5-4B and 9B still recalling a fact from position 0 at 31,983 tokens.
-- **`qwen3.6-moe-core:35b-a3b` — up to 131072 tokens**, the most corelib has verified on the NPU (the model's own limit is 262144; corelib refuses more). The default is 4096; raise it with `--ctx-len` (`-c`). Long contexts have three costs:
-    - **Memory, up front.** The KV cache is allocated when the model is created, at the full capacity you ask for (rounded up to whole 4096-token pages), whether or not a conversation ever fills it: about 20 KiB per token (10 attention layers × 2 KV heads × 256 × BF16, for K and V), so 131072 tokens is about 2.5 GiB on top of the weights.
-    - **Long prompts run in passes of up to a page.** The first 4096 tokens from position 0 are one pass, and so is each later chunk that starts on a page boundary, padded to the page; past the second page these passes are slightly less accurate (corelib's `PAGED_PREFILL_DEEP_PAGES`, which corelib measured at perplexity 13.17 against 13.18 exact at 12,500 tokens). A chunk starting inside a page runs as passes up to the next page boundary, padded to at least 256 rows where that fits, also on the NPU; only a pass in a page's last 255 positions is padded to 128 rows and runs its attention on the CPU. corelib steps a few short remainders one token at a time instead: a pass of fewer than 5 tokens, a 128-row pass with only a few live tokens, and a pass starting in the last 127 positions of a page.
-    - **Checkpoints take host memory.** A checkpoint (what lets a conversation rewind without replaying) copies the KV cache into host memory, in whole 4096-token pages up to the current position: about 80 MiB per filled page, so up to about 2.5 GiB at a full 131072-token context.
-- **`gemma4-it-core:e2b` / `gemma4-it-core:e4b` — up to 16384 tokens**, the length corelib has validated paged Gemma 4 to, and never past the GGUF's own `gemma4.context_length`. The default is 16384. Up to 4096 any length works; above 4096 the context must be a multiple of 4096. The KV cache for the whole context is allocated up front, when the model loads.
-
-Qwen3, Qwen3.5, Qwen3.6, Gemma 4 and Phi-4 run as corelib's own models, which pack their weights on every load: there is no packed-weight cache beside the GGUF. A `<gguf name>.core-weights.bin` / `.json` pair left there by an earlier build is no longer read and can be deleted.
+- **Source format:** GGUF, read directly by corelib. With a Hugging Face token that can read AMD's private repositories (`HF_TOKEN`, or `hf auth login`), `flm pull` fetches AMD's 4-bit file, which packs exactly as it is; without one it falls back to the public GGUF. A public `Q8_0` is requantized to 4-bit groups on every load, which is lossy.
+- **Context:** each tag's limit is the most corelib runs that model at; the default is lower where corelib or our own runs found answers going wrong past it. Over-capacity requests are rejected with HTTP 400 before any work reaches the device.
+- **Availability:** developer build against corelib 0.53.0 — see [Phi-4 on the core backend](phi.md).
+
+The same corelib path runs every `-core` tag:
+
+| Tag | Default file | Public fallback | Context limit | Default context |
+|---|---|---|---|---|
+| `qwen3-core:0.6b` | unsloth `Q8_0`, refit to group 64 (lm_head 32) | — | 32768 | 32768 |
+| `qwen3-core:1.7b` / `4b` / `8b` | AMD `Q4_G128` | unsloth `Q8_0` | 32768 | 32768 |
+| `qwen3.5-core:4b` / `9b` | AMD `Q4_G128` | unsloth `Q8_0` | 262144 | 32768 |
+| `qwen3.6-moe-core:35b-a3b` | AMD `Q4_G32` | unsloth `Q8_0` | 262144 | 32768 |
+| `qwen3.8-core:27b` | AMD `Q4_G128`, on the NPU | unsloth `Q4_0`, on the GPU | 262144 | 32768 |
+| `gemma4-it-core:e2b` / `e4b` | AMD `Q4_G32` | unsloth `Q8_0` | 131072 | 65536 |
+| `phi4-mini-it-core:4b` | AMD `Q4_G128` | unsloth `Q8_0` | 131072 | 131072 |
+| `llama3.2-core:1b` / `3b`, `llama3.1-core:8b` | `Q8_0`, refit to group 64 | — | 131072 | 131072 |
+
+Every model's attention is paged: corelib keeps the KV cache in 4096-token pages and grows it a page at a time, so a large context costs memory only as a conversation fills it. Raise the context with `--ctx-len` (`-c`), up to the limit.
+
+FastFlowLM sends a long prompt in chunks of `max_prefill_len` (4096) tokens that end on multiples of 4096 counted from the start of the conversation, not from the start of the turn, so every chunk after a turn's first starts on a page boundary, where corelib runs a whole page as one pass. Llama, Qwen3, Phi-4-mini and Gemma 4 run a later turn together with the earlier tokens of the page it starts in. Qwen3.5, Qwen3.6 and Qwen3.8 carry recurrent state and continue from where they are; going back to the last checkpoint restores a snapshot, which copies the filled KV pages into host memory, and going back anywhere else replays the conversation.
+
+Known limits, each corelib's (its `docs/known-problems.md` names them):
+
+- **Qwen3.6 past 32768.** Paged attention can return garbage: a passkey asked for after about 48k tokens came back as `!` repeated (on an AIE4 test machine). A context past 32768 still loads, and says so.
+- **Qwen3.8 on long prompts.** A long prompt can come back as `!` repeated, run to run and from corelib's own runner too: on an AIE4 test machine a passkey was answered at 6k and 14k tokens and garbled at 10k and 28k. Prefill passes shorter than a page past position 0 run their attention on the CPU (`PAGED_PREFILL_SHORT_CHUNK`).
+- **Qwen3.5 and Qwen3.8 decode** drift from the reference the more pages it reads (`PAGED_DECODE_PAGE_DRIFT`).
+- **Gemma 4 past position 65535** (`PAGED_ATTENTION_POSITION_WRAP`).
+- **Llama on a very long, repetitive prompt.** A passkey in about 96k tokens of one repeated sentence came back as `!` repeated, from corelib's own runner too; Phi-4-mini answered the same prompt.
+- **A 4096-token prefill pass can fail to complete** (`PAGED_PREFILL_PREEMPTION_TIMEOUT`), more often while another process uses the NPU. corelib fails only that call, so FastFlowLM resets the session and runs the conversation again from the start, once; a second failure needs an unload and reload.
+Every core model runs as corelib's own model, which packs its weights on every load: there is no packed-weight cache beside the GGUF. A `<gguf name>.core-weights.bin` / `.json` pair left there by an earlier build is no longer read and can be deleted.
 
 ```powershell
 flm pull qwen3-core:4b