feat(pflash): Qwen3.5-0.8B PFlash runtime: scoring head, segment and request probes, strict budget selection, chat views, cache-aware admission, fallback

[Graffioh]

PFlash makes long-prompt prefill cheaper. A small drafter (Qwen3.5-0.8B) reads the whole prompt, keeps the parts that matter for the request, and the target model prefills only those. This PR is the complete C++ runtime, in 5 commits on main (5f402c61f).

How it works

1. Admission. With --prefill-compression auto, a prompt is compressed once it brings at least --prefill-threshold tokens that no cache can restore (PFLASH_ADMISSION=cache), so a warm prefix cache is never thrown away. --max-ctx is the longest raw prompt PFlash takes; a longer one gets HTTP 400 in OpenAI's prompt-overflow wording. A prompt that fits but leaves no room for its reply is compressed.
2. Scoring. The drafter runs blocks 0–14. A trained scoring head ranks the prompt against the request, a segment probe cuts it into whole units (paragraphs, functions, records), and a request probe finds the request in the human messages (one relative query candidate plus a 512-token verbatim pin). Each part loads from GGUF and fails closed.
3. Selection. The question, instructions, system prompt, tool definitions and role headers are always kept. Units fill the rest of the budget by relevance per token, in document order. The budget follows a length curve.
4. Chats. The server keeps the view it served for each conversation. A small new turn is appended verbatim and reuses the cache; a large one is compressed on its own (PFLASH_ARRIVE); recall brings back passages a new question needs; a drifted view is rebuilt. Drafter scoring sessions resume exactly, and survive a drafter release in host RAM, so a follow-up scores only its new tokens.
5. Tool sessions (a request with tools, tool calls or tool results) are served raw: the agent's history and tool outputs are never compressed. PFLASH_AGENT_PASTE=1 opts into compressing a big newest user paste; it is off because follow-ups after it are slower than full prefill.
6. Fallbacks. If scoring fails, the raw prompt is served with full prefill when it fits (PFLASH_FAIL_CLOSED=1 returns the error instead). When the relevance over a whole prompt is too flat for the budget to choose (PFLASH_FALLBACK_SPREAD, e.g. "list every X"), the request also gets full prefill. Every fallback is recorded in usage.timings.pflash and /props.

Code lives in src/pflash/; the old Qwen3-0.6B drafter scorer is removed. The drafter IPC gains compress2/compress3 for a remote drafter.

Defaults

With a trained head (PFLASH_SCORING_HEAD_GGUF) and compression on, every setting left unset takes the release profile, the configuration the results below were measured with:

setting	release profile	without a head
--prefill-threshold	16384	32000
keep ratio	curve 16384:0.315 32768:0.270 65536:0.270 120000:0.225	flat 0.05
PFLASH_SELECT_MODE / _SEGMENTS / _SCORE	budget_only / probe / density	legacy selection
PFLASH_SELECT_CHUNK_SIZE / _QUERY_TOKENS / _QUERY_HEAD	256 / 128 / 1	by input size / 8 / 0
PFLASH_SELECT_QUERY_PROBE, PFLASH_QUERY_CANDIDATES, PFLASH_QUERY_PIN	1, 1, 512	off
PFLASH_SELECT_PARAGRAPH_JOIN	0	1
PFLASH_ADMISSION, PFLASH_ARRIVE, PFLASH_CHAT_REBUILD_MIN_TOKENS	cache, 1, 65536	length, 0, 0
PFLASH_FALLBACK_SPREAD	0.11	off

• The values are in server/launch_profiles.h and install like a launch profile: a variable that is already set keeps its value, and an explicit --prefill-threshold, --prefill-keep-ratio or --prefill-curve wins. The startup log says how many were applied.
• The probe parts need their GGUFs. The query defaults step aside for experiment arms they cannot combine with (another scorer, the identifier query, request stage 2, PFLASH_SELECT_QUERY_PROBE=0).
• --prefill-skip-park stays opt-in (the target and drafter must fit together; the results used a 32 GB card), and so does PFLASH_AGENT_PASTE.
• The Docker entrypoint forwards LUCE_PREFILL_KEEP / LUCE_PREFILL_THRESHOLD only when set, so the profile applies there too.

Every flag, its range and the experiments that remain (fold, request stage 2, query grid and oracle replay, identifier query) are in PFLASH_CONFIGURATION.md. Settings are validated at startup; an invalid one stops the server and names the variable. PFLASH_COLD_HISTORY, PFLASH_AGENT_HISTORY and LUCE_PC_SINGLE_TURN_DEEP are removed (#835 snapshots every turn).

How to run

PFLASH_SCORING_HEAD_GGUF=/path/seg-scoring-head.gguf \
PFLASH_SEGMENT_PROBE_GGUF=/path/segment-probe.gguf \
PFLASH_REQUEST_PROBE_GGUF=/path/request-probe.gguf \
luce_server Qwen3.8-27B-IQ4_XS-pure.gguf --cache-type-k q4_0 --cache-type-v q4_0 --max-ctx 204800 \
  --prefill-compression auto --prefill-drafter Qwen3.5-0.8B-BF16.gguf --prefill-skip-park

Results

Modal L40S, Qwen3.8-27B IQ4_XS with Q4_0 KV, release profile against full prefill (--prefill-compression off). About 1 answer in 10 flips between near-identical runs, so ±3 on a small set is noise.

set	full prefill	PFlash	fell back (spread)
RULER (15)	13/15	13/15	2
NIAH (100, 16K–120K)	100/100	100/100	9
NoLiMa (24, 16K/32K)	14/24	14/24	11
LongBench v2 (12)	6/12	7/12	3
BRIGHT (28, 32K)	23/28	23/28	2
InfiniteBench CodeDebug (32, ~98K)	26/32	25/32	0
LongCodeQA (114, 32K/64K/128K)	57/114	62/114	0

Median time to first token, seconds:

workload	full prefill	PFlash	speed-up
NIAH 32K / 64K / 120K	15.2 / 38.4 / 106.7	6.3 / 11.9 / 20.0	2.4x / 3.2x / 5.3x
NoLiMa 32K	15.1	6.6	2.3x
LongBench v2	12.1	5.2	2.3x
BRIGHT 32K	15.5	6.1	2.5x
CodeDebug ~98K	60.3	15.0	4.0x
LongCodeQA 32K / 64K / 128K	10.6 / 27.5 / 68.0	5.3 / 9.8 / 16.6	2.0x / 2.8x / 4.1x

Prompts under 16K tokens are served raw. Answers that fell back are byte-identical to full prefill.

On an R9700 (same configuration): tool sessions are served raw with answers byte-identical to full prefill and time to first token within 0.6 s; at --max-ctx 204800 a 204,650-token prompt compresses to 46,327 tokens with a 30.6 GiB peak of 31.9 GiB and a 206K one gets the 400, and a resumed drafter session cuts a follow-up's scoring from ~30 s to ~2 s.

Tests

• test_server_unit (HIP gfx1201): 938 passed, 0 failed, 4 skipped. It covers selection, fold, request probe and stage 2, query candidates, identifier query, session resume and host-RAM offload, admission against --max-ctx, tool sessions with the paste switch on and off, the raw-fallback view cleanup, the fix(server): each turn reuses the previous one, long prompts share time, late system messages work #835 snapshot of a tool-session view, and the release profile (installed only with a head, explicit settings win, experiment arms still start).
• test_entrypoint.sh: PASS, including the PFlash ratio and threshold forwarding.
• Release defaults on an R9700: one binary, the explicit release env and flags against only the three GGUFs and --prefill-compression auto, on five public requests (LongBench v2, Document QA, CodeDebug, BRIGHT, a two-turn chat). Served prompts, cache reuse, answers and selection records are identical on all 6 requests (timings aside), including the spread fallback on the LongBench item and the cached follow-up; the server log shows the 14 defaults applied, threshold 16384 and the curve.
• test_cpy_strided_dst passes on the GPU. pflash_fold_replay and pflash_daemon replay selections offline; the replay matches all 80 request-probe parity cases.

Commits

1. fix(ggml-cuda): take the transpose copy only into a contiguous destination (backport of ggml-cuda : fix cpy transposed path corrupting non-contiguous dst ggml-org/llama.cpp#27663). Without it, a prompt of 8192k+1 tokens gave NaN scorer logits on HIP.
2. feat(pflash): the drafter scorer, strict selection, chat views, request probe and resumable scoring sessions.
3. feat(server): tool sessions raw, fallbacks and the context cap.
4. feat(pflash): the release profile as the default with a trained head.
5. fix(pflash): --max-ctx bounds the raw prompt PFlash takes, instead of a fixed cap.

Limitations

• PFlash runs on the single-request engine; the batched launcher does not use it yet.
• The drafter scores the whole prompt, history included, so --max-ctx must leave room for its scoring memory beside the target: 204800 on a 32 GB R9700 with the 27B IQ4_XS target. If the drafter runs out of memory, the request falls back to full prefill.
• The request-probe message mapping fails on some raw agent traces (trimmed contents, tool-call-only turns); those pastes fall back to raw.

Supersedes #773 (the in-graph drafter mask is in commit 2).

🤖 Generated with Claude Code

[Graffioh]

Record of the unpushed selection work, 30 Sep 2026 (worktree cleanup). None of this is in this PR. Each item is now on a branch of the fork Graffioh/lucebox-hub, based on this PR's head 555ee2a97.

Query-head fix: PFLASH_SELECT_QUERY_HEAD=1 (d0c750100, on archive/pflash-fold-select-20260930)

• The adopted scoring head's served configuration uses it (luce_box feat(dflash): daemon scripts improvements (GPU split, Windows, defaults) #139, recipe revision 7), so this switch is part of the configuration behind the current numbers.
• The problem: with latest_user and a fixed 128-token tail, a user who asks first and pastes afterwards gets a query window of pure document text. On the pflash-bench pilot, question-first items scored 0/5 against 15/19 for question-after.
• The fix: the server splits the latest user message into blocks (pflash_query_blocks).
  • The head is the leading prose block before the first paste-like line.
  • The tail is the last paragraph after the last paste-like line, and it replaces the fixed tail window.
  • Each block is capped at 256 tokens (PFLASH_SELECT_QUERY_HEAD_TOKENS / _TAIL_TOKENS) and is sent to the drafter as extra_query_spans at the query's own weight. It is pinned like an instruction span.
  • A message without paste structure keeps today's window. The switch is off by default and needs strict selection with latest_user.
• It sits on top of the fold commits below and uses their block parser, so porting it here means taking pflash_fold.* as well, or splitting the parser out.

Fold selection: PFLASH_SELECT_STRATEGY=fold (14a4aec42, with the tests 599217b39 and e7624d638 and the pflash_fold_replay tool; also on archive/pflash-fold-select-20260930)

Adaptive top-k selection: PFLASH_SELECT_STRATEGY=adaptive (6468f55ba, on archive/pflash-adaptive-select-20260930, which contains the fold branch)

• Not adopted. Recipe revision 7 records the result: 85 / 102 against 94 at the fixed budget on the private suite. At 64K–128K it kept a quarter to two thirds fewer tokens.

Other variants, now pushed instead of kept as local backups

• archive/pflash-container-headers-20260930: container headers and cut markers, request-structure headers (PFLASH_SELECT_STRUCT_HEADERS), and the early recall variants. It also has the first copy of PFLASH_DRAFTER_PROFILE=1.
• The "Tried and not adopted" rows of the description:
  • archive/pr746-recall-missing-topic-switch-20260930 (9c52b0599)
  • archive/pr746-rebuild-on-topic-switch-20260930 (5063f3c46)
  • archive/pr746-restate-strongest-segments-20260930 (d10462a77)
  • archive/pr746-recall-any-size-20260930 (db4ce1817)
• archive/pr746-pre-fold-20260919: the branch before the 19 Sep fold (two-scorer split budget, sub-unit probe head, oversize split guard).
• archive/pflash-trackA-head-weights-20260930: optional per-head weights for the scoring head (Track A, not adopted).

The local worktrees (pflash-qwen35-hybrid, pflash-qwen35-drafter-fixes) were removed. This PR's branch is unchanged.

[davide221]

CI failed on c2de8a5e

Windows build fails: new PFlash file uses a Linux-only stat field (st_mtim)

The Windows (MSVC) build of the pull request fails to compile one of the new PFlash source files. The code reads the nanosecond modification time through stat::st_mtim, which does not exist in the Windows C runtime's stat struct. This is a portability bug in the change under test; Linux and HIP builds are unaffected.

How to fix: Use st_mtime on Windows (keep st_mtim on POSIX) in pflash_file_identity

1. Edit server/src/pflash/pflash_session_offload.cpp in pflash_file_identity.
2. Compute mtime_ns via #ifdef _WIN32 from s.st_mtime (seconds), else from s.st_mtim as before.
3. Print mtime_ns in the identity string instead of the inline st_mtim expression.
4. Push and re-run the Windows CI job.

_{windows compile error: non-portable stat member · high confidence · run}

Details and evidence

Failed: Build Windows (MSVC + CUDA, library + server targets) › Build Windows targets

Job Build Windows (MSVC + CUDA, library + server targets), step Build Windows targets, fails compiling src/pflash/pflash_session_offload.cpp:

pflash_session_offload.cpp(294): error C2039: 'st_mtim': is not a member of 'stat'
C:\...\ucrt\sys/stat.h(87): note: see declaration of 'stat'

pflash_file_identity() (server/src/pflash/pflash_session_offload.cpp:288-296) builds a file identity string from stat and reads s.st_mtim.tv_sec / s.st_mtim.tv_nsec at line 294. st_mtim is a POSIX extension; the Windows UCRT struct stat only has st_mtime (whole seconds), so MSVC errors out. The function was added by this PR (new file, used from server/src/pflash/qwen35_drafter.cpp:877,884 to key stored drafter sessions to the drafter file). The unit test for it (server/test/unit/test_pflash_session_offload.cpp:397-408) only requires the identity to change when the file's size/content changes, so second resolution on Windows is acceptable.

Other things to try

• Reproduce locally on Windows or with MinGW: compile server/src/pflash/pflash_session_offload.cpp and confirm st_mtim is the only non-portable member.
• Alternative: use std::filesystem::last_write_time (C++17, portable) instead of stat, but that is a larger change than the #ifdef fix.

Patch (1 file)

Not applied anywhere. To try it, save the diff as fix.patch and run git apply fix.patch.

diff --git a/server/src/pflash/pflash_session_offload.cpp b/server/src/pflash/pflash_session_offload.cpp
index 5d15b523..79d762c5 100644
--- a/server/src/pflash/pflash_session_offload.cpp
+++ b/server/src/pflash/pflash_session_offload.cpp
@@ -290,8 +290,14 @@ std::string pflash_file_identity(const std::string & path) {
     struct stat s {};
     if (::stat(path.c_str(), &s) != 0) return path + "|missing";
     std::ostringstream out;
+#ifdef _WIN32
+    // The Windows CRT struct stat has no st_mtim; st_mtime is whole seconds.
+    const long long mtime_ns = (long long) s.st_mtime * 1000000000LL;
+#else
+    const long long mtime_ns = (long long) s.st_mtim.tv_sec * 1000000000LL + (long long) s.st_mtim.tv_nsec;
+#endif
     out << path << '|' << (unsigned long long) s.st_dev << ':' << (unsigned long long) s.st_ino << '|'
-        << (long long) s.st_size << '|' << (long long) s.st_mtim.tv_sec * 1000000000LL + (long long) s.st_mtim.tv_nsec;
+        << (long long) s.st_size << '|' << mtime_ns;
     return out.str();
 }

Prompt for a coding agent

A GitHub Actions run failed in Luce-Org/lucebox. Please fix it.

- Pull request #746 (branch `codex/pflash-segment-probe-experiment`), commit c2de8a5e7c206a8bce9992d0f00d1454409d845f
- Workflow: CI (https://github.com/Luce-Org/lucebox/actions/runs/38249665121)
- Failed: Build Windows (MSVC + CUDA, library + server targets) › Build Windows targets

Diagnosis:
Job `Build Windows (MSVC + CUDA, library + server targets)`, step `Build Windows targets`, fails compiling `src/pflash/pflash_session_offload.cpp`:

```
pflash_session_offload.cpp(294): error C2039: 'st_mtim': is not a member of 'stat'
C:\...\ucrt\sys/stat.h(87): note: see declaration of 'stat'
```

`pflash_file_identity()` (server/src/pflash/pflash_session_offload.cpp:288-296) builds a file identity string from `stat` and reads `s.st_mtim.tv_sec` / `s.st_mtim.tv_nsec` at line 294. `st_mtim` is a POSIX extension; the Windows UCRT `struct stat` only has `st_mtime` (whole seconds), so MSVC errors out. The function was added by this PR (new file, used from server/src/pflash/qwen35_drafter.cpp:877,884 to key stored drafter sessions to the drafter file). The unit test for it (server/test/unit/test_pflash_session_offload.cpp:397-408) only requires the identity to change when the file's size/content changes, so second resolution on Windows is acceptable.

Proposed fix: Use st_mtime on Windows (keep st_mtim on POSIX) in pflash_file_identity
1. Edit `server/src/pflash/pflash_session_offload.cpp` in `pflash_file_identity`.
2. Compute `mtime_ns` via `#ifdef _WIN32` from `s.st_mtime` (seconds), else from `s.st_mtim` as before.
3. Print `mtime_ns` in the identity string instead of the inline `st_mtim` expression.
4. Push and re-run the Windows CI job.

Suggested patch (a starting point):
```diff
diff --git a/server/src/pflash/pflash_session_offload.cpp b/server/src/pflash/pflash_session_offload.cpp
index 5d15b523..79d762c5 100644
--- a/server/src/pflash/pflash_session_offload.cpp
+++ b/server/src/pflash/pflash_session_offload.cpp
@@ -290,8 +290,14 @@ std::string pflash_file_identity(const std::string & path) {
     struct stat s {};
     if (::stat(path.c_str(), &s) != 0) return path + "|missing";
     std::ostringstream out;
+#ifdef _WIN32
+    // The Windows CRT struct stat has no st_mtim; st_mtime is whole seconds.
+    const long long mtime_ns = (long long) s.st_mtime * 1000000000LL;
+#else
+    const long long mtime_ns = (long long) s.st_mtim.tv_sec * 1000000000LL + (long long) s.st_mtim.tv_nsec;
+#endif
     out << path << '|' << (unsigned long long) s.st_dev << ':' << (unsigned long long) s.st_ino << '|'
-        << (long long) s.st_size << '|' << (long long) s.st_mtim.tv_sec * 1000000000LL + (long long) s.st_mtim.tv_nsec;
+        << (long long) s.st_size << '|' << mtime_ns;
     return out.str();
 }
```

Check out that commit, confirm the diagnosis, make the fix (adjust it if the diagnosis is wrong), and re-run the failing step locally before pushing.

_{Reply with /ci-fix and what to change to get a revised diagnosis.}

[davide221]

Merged main into the branch (eca7265) to clear the conflict with #842. validate_request_context keeps main's routed flag and its reply trim, and still sends your PFlash maximum-context error when compression would run. test_admission keeps both sets of tests. No other changes.

The merged tree (main with #839, plus this PR) builds on HIP gfx1201, and test_server_unit gives 939 passed, 0 failed, 4 skipped.

One small follow-up for later: the raw fallback in pflash_fallback_decision still refuses when raw prompt + max_tokens is over max_ctx, while a direct request now gets its reply trimmed (fit_reply_to_context). It only matters with PFlash on.