Repository navigation
model : DeepSeek-V4.1-Flash-1017 full stack pin (all DeepSeek-V4.1 PRs except #292) - #298
Closed
danielhanchen wants to merge 71 commits into
Closed
danielhanchen wants to merge 71 commits into
danielhanchen wants to merge 71 commits into
Conversation
Three sub-IQ1_S data types: IQ1_S's algorithm with the codebook the mapping step may choose from cut to a subset of iq1s_grid, and the index stored in fewer bits. IQ1_XS id 64 46 B / 256 1024-entry grid 10-bit index 1.4375 bpw IQ1_XXS id 65 42 B / 256 512-entry grid 9-bit index 1.3125 bpw IQ1_XXXS id 66 38 B / 256 256-entry grid 8-bit index 1.1875 bpw Dequantization is IQ1_S's rule throughout, y[j] = d*(2*ls + 1)*(grid[idx][j] + delta). GGML_TYPE_COUNT is 67; 42..63 are left for upstream, which already uses 42 for Q2_0, so a stock build rejects these files rather than misreading them. CPU, CUDA dequant, MMVQ and MMQ. The MMQ path targets the current ggml_cuda_mmq_util_funcs / ggml_cuda_mmq_config API and registers the types in every per-architecture config table; Blackwell falls through to Ampere and needs none. scripts/iqxs_gen_tables.py regenerates all six codebook tables in ggml-common.h byte-identically from the three committed .npy grids, so the tables are reproducible rather than pasted. Tested on B200, CUDA 13, sm_100. test-backend-ops -o MUL_MAT passes 2/2 backends both default and with GGML_CUDA_FORCE_MMQ=1, 11 cases per type matching IQ1_S. Note the default run never enters the new MMQ kernels, since small n routes to MMVQ.
# Conflicts: # ggml/src/ggml-common.h
# Conflicts: # ggml/src/ggml-cuda/mmq.cuh
ggml-org#30168 made the src1 precision an explicit template argument of mul_mat_q_case and removed its default.
…onverters Mirror the IQ1_S CASE rows (same QK_K block, QR 8, Q8_0 SRAM layout) for IQ1_XS, IQ1_XXS and IQ1_XXXS in mmq-config-gcn.cuh and mmq-config-pascal-older.cuh, so MMQ selection on those arches no longer finds zero configs and trips GGML_ASSERT(J_best > 0). Add the three dequantizers to ggml_get_to_bf16_cuda to match the FP16/FP32 tables (cuBLAS BF16 compute path for quantized src0). Map the three types in the loader's guessed-ftype switch.
FP8 32x32 linears are dequantized to BF16 (exact), routed experts are copied bit-exact as MXFP4, Engram tables are precomputed as BF16 (e4m3 * 2^e8m0, exact) and streamed in bounded chunks, router and Engram gate are kept in F32. --outtype f32 keeps every dense tensor and the Engram tables in F32. The Engram tables of all layers are written as one per_layer_token_embd (rows concatenated in layer order) with the PLE keys of qwen4exp: ple.layers, ple.ngram_size, ple.heads_per_ngram, ple.layer_multipliers, ple.head_offsets (absolute rows), ple.head_vocab_sizes and embedding_length_per_layer_input. The compressed token map and the pad token id stay deepseek41.engram keys.
…upport - arch, KV keys and Engram tensors for deepseek41; the Engram tables of all layers are one per_layer_token_embd described by the qwen4exp PLE keys, the compressed token map and the pad token id are the only new Engram keys - hparams for the cross-layer KV / indexer sharing and the two-level candidate selection - DSV4 KV cache: compress ratios are taken from the slot state instead of the V4 constants; for V4.1 the CSA slot is ratio 2, the HCA slot ratio 1, without overlap and without the lightning-indexer cache, the index K is stored next to the compressed KV and only the KV source layers own rows - the window cells keep the token ids (saved and restored with the state) for the Engram n-gram history
Engram (host-side n-gram hash, rows gathered on the CPU from per_layer_token_embd, which is read lazily from the mapping like the qwen4exp n-gram table, its rows prefetched per ubatch), single-pass mHC, ratio 1/2 compressors with cross-layer KV / index-K / top-k sharing and the two-level candidate block selection. Every compressed layer attends densely over [window | all compressed rows] with the top-k applied as a mask. The indexer score, candidate blocks and top-k are computed per chunk of query tokens, so the [index heads, tokens, rows] intermediate stays near 2^26 elements at long context (a single chunk, the plain graph, whenever it fits). Tokens shared by several sequences are supported; embedding input skips Engram, like the masked positions of the reference. The model reuses the DeepSeek-V4 loader and graph helpers; the HC mix computation is split out of build_hc_pre (unchanged for V4). The F32 router, HC and indexer matmuls request F32 precision, and the router weights are normalized by sum + 1e-20 as in the reference. hash_layer_count is optional for V4.1: it has no hash layers, and V4.1 GGUFs converted before the key was written lack it.
…d DSML tool calls - test-llama-archs: a 6-layer config with window-only, Engram, ratio-2 and ratio-1 KV sources, top-k reuse and a candidate-masked index layer; the model saver writes the deepseek41 sharing and Engram keys (round trip) - the official chat template (ported from encoding.py), template detection, and the DSML tool call parser for the V4.1 tag names (leading space: <｜DSML｜ calls>, <｜DSML｜ invoke>, <｜DSML｜ parameter>) - a V4.1 content continuation in thinking mode consumes the bare </think> of the generation prompt
- remove the V4.1 chat template file, its template detection, the DSML parser change and their test-chat cases; the converter drops the inherited DeepSeek-V4 default template (a template shipped with the model is still added by set_vocab) - hash_layer_count is required again (the converter writes 0 for V4.1) - the converter no longer imports sympy - engram_index moves out of models.h; comments that restate the code are removed
For x < -16.6, 1.0f + expf(x) rounds to 1.0f in F32 and logf() returns 0, although softplus(x) ~= exp(x) is a normal float down to x ~= -87. log1pf(expf(x)) keeps it. DeepSeek-V4/V4.1 route with sqrt(softplus(logits)): when every selected router logit is about -23 all expert weights became 0 and the token's routed experts were dropped (the reference uses torch softplus). Fix the CPU and CUDA SOFTPLUS ops and the fused CUDA sqrt-softplus top-k MoE path. Add SOFTPLUS_RANGE to test-backend-ops.
The AMX buffer packs a single ne[0] x ne[1] slice, yet supports_op accepted batched weights and ggml_backend_amx_mul_mat indexed src0 and dst per batch, adding byte offsets to typed pointers. - A 3D quantized weight (DeepSeek V4 attn_output_a, Q8_0) landed in the AMX buffer, AMX refused the permuted activation, the CPU backend cannot read AMX-packed data, and the scheduler aborted with GGML_ASSERT(*cur_backend_id != -1). - A 2D weight times a batched src1 (DeepSeek V4 indexer attn_k) read past the packed weight and wrote dst at the wrong offset, crashing. supports_op now requires a 2D src0, and the kernel treats all rows of the contiguous src1/dst as one gemm. Unbatched multiplies are unchanged.
MXFP8 is the OCP microscaling FP8 format: 32 E4M3 values sharing one E8M0 scale (33 bytes per block). DeepSeek-V4.1 stores its Engram tables this way; keeping them as MXFP8 halves the table size against the exact BF16 lookup (203 GB vs 393 GB) with identical dequantized values. The type is gather-only: CPU and CUDA support GET_ROWS (source 0) and refuse every other op in supports_op, so misuse fails at scheduling. Row (de)quantizers use 64-bit block loops since the tables exceed 2^31 elements. gguf-py gains the type and a reference (de)quantizer; test-backend-ops covers GET_ROWS. MXFP8 takes the next free type id, 43.
The checkpoint's Engram tables are e4m3 rows with one e8m0 scale per 32 elements, i.e. MXFP8 block for block: copy them bit-exact as [e8m0 | 32 x e4m3] blocks into per_layer_token_embd instead of precomputing the BF16 lookup. NaN scale or code bytes are rejected. DSV41_ENGRAM_TYPE=bf16 keeps the exact BF16 table (F32 with --outtype f32).
… size F32 x F32 MUL_MAT ran in exact F32 only for small batches; medium batches used TF32 tensor cores (MMF) and large ones cuBLAS tensor-op math, so results depended on the batch size even at GGML_PREC_F32. When src1 is requested at GGML_PREC_F32 ([TAG_GGML_PREC]) and the weights are F32, use the F32 matrix-vector kernel up to MMVF_MAX_BATCH_SIZE columns, else cuBLAS SGEMM with CUBLAS_DEFAULT_MATH (restored to CUBLAS_TF32_TENSOR_OP_MATH afterwards, as in solve_tri). Other paths are unchanged. test-backend-ops: MUL_MAT prec-F32 cases at NMSE 1e-10 across the MMVF / MMF / cuBLAS ranges.
…put) The radix top-k gather placed elements with atomicAdd slot counters, so both the set of tied elements at the k-th key and the order of the selected indices changed from run to run. Exact ties are common (masked or ReLU-zeroed rows), and consumers that gather rows in top-k order then accumulate in a different order every run. Each block now owns a contiguous column chunk. A count pass records the elements above and equal to the cut per chunk; the gather pass writes them at the sum of the earlier chunks' counts plus an ordered in-block prefix count. Each row holds the elements above the cut, then the ties with the lowest column indices, both in ascending column order. The shared atomic counter is gone. test-backend-ops: TOP_K cases with fewer than k positive values and the rest exactly 0.0 across the radix path.
llama-server --embeddings --pooling none toggles llama_set_embeddings per request. With pooling NONE the toggle did not trigger a re-reserve, so a model whose graph depends on cparams.embeddings (DeepSeek-V4.1 returns the flat HC state as t_embd) could run on a stale reservation, and ggml_gallocr segfaulted on the first /completion. Re-reserve on every toggle.
…ngram host placement - Tokens shared by several sequences (llama-batched decodes the prompt once with n_seq_id = n_parallel) read the n-gram history of their first sequence. llama_kv_cache_dsv4::init_batch now checks that all of the token's sequences have the same predecessors (batch first, then the cache via the new llama_kv_cache::seq_pos_tok_le) and refuses the batch otherwise (decode returns 1, caches untouched). - Mixed token/embd ubatches: embedding (image) positions hash as DEAD, get Engram gate 0, and their KV cells record LLAMA_TOKEN_NULL. - The Engram table (per_layer_token_embd) stays in host memory under tensor overrides and split modes when it is not read lazily.
… engram_gate - The compressed-attention path did not apply the Hadamard rotation that build_raw_attention applies to a quantized K cache. It now rotates q, the window K and the output like the window-only path, and rotates the compressed rows in F32 before storing them in the cache type for the concat. - Quantized compressed K / index K views are dequantized directly to F32 (CUDA has no same-type copy for strided quantized views); with LLAMA_ATTN_ROT_DISABLE=1 they go through F32 as well. - engram_gate is an elementwise weight: dequantize a quantized copy in the graph, and keep llama-quantize from quantizing it. F16/BF16/F32 caches build the same graph as before.
…calls Add the V4.1 chat template and detect it in common_chat. V4.1 keeps the V4 DSML tool call format but its tag names carry a leading space, so the DeepSeek parser handles both. The converter embeds the V4.1 template in place of the V4 one. Add V4.1 parser cases to test-chat.
…parators The V4.1 template reads keys the OpenAI message struct drops (task, per-message tools, tool call namespace) and joins content blocks with blank lines around images. Keep unknown message keys on common_chat_msg and restore them, with media markers as separate text blocks, only for templates using the V4.1 DSML tags. Fixes encoding.py golden cases 3-5 on the common_chat apply path.
…em-message tools The V4.1 template only accepts low/high/max or 1-100 and raises otherwise, so map minimal -> low, xhigh -> max and medium -> unset (template default) instead of failing the request. Tools declared only on a system message now also feed the tool call parser and grammar. Both are V4.1 only.
…er layer split Layers read the compressed KV (and index K) of their kv source layer. With -sm layer a source and its consumers can sit on different devices, and the scheduler then copied the whole compressed-cache view across devices once per consumer every ubatch (740 MB per decoded token at 64k depth on 2 cards). The first consumer on a device other than its source's now owns a mirror cache layer in the same llama_kv_cache (shared cell bookkeeping, seq ops and clears). The source's new compressed rows are appended to it in that consumer's graph and every consumer on the device reads the local copy. llama_dsv41_kv_read_layer() is the single definition used by the cache and the graph. Compressor state stays with the source. No change on one device or without KV offload; LLAMA_DSV41_KV_MIRROR=0 disables it. The state format does not depend on the split: only kv source layers are serialized and every mirror is refilled from its source on read, so state saved under one split loads under any other and by a build without mirrors.
…r, bounded FA, decode gather) The reference graph attends over [window | every compressed row] with the top-k as a mask, scores the indexer with an unfused chain over an [n_idx_head, n_tokens, n_comp] F32 intermediate and passes n_kv_max = 0 to flash attention. With flash attention (F16 masks): - the indexer uses ggml_lightning_indexer on the strided index-K view, scoring the ubatch as one chunk with the same top-k selection - FA gets n_kv_max = window + top-k, so the CUDA sparse FA path runs - ubatches with at most 32 queries per stream gather only each query's top-k compressed rows and attend over [window | selected] with one FA stream per query; keys and mask values match the dense path, only the key order changes. Gathered rows and masks are shared by all layers using the same top-k. A quantized K cache (-ctk q8_0 / q4_0) takes the same paths. Without flash attention nothing changes. LLAMA_DSV41_SPARSE=0 restores the reference path (logits bitwise equal to it).
…PU graph splits) The token embedding and the Engram table live in host memory, so each GET_ROWS on them ran as its own CPU split between device splits (7 splits per decode token on 2 cards). The rows are now gathered and converted to F32 on the host in set_input (same to_float / memcpy as the CPU GET_ROWS, so bitwise identical) and enter the graph as F32 inputs, leaving one split per device. build_inp_embd() gets an opt-in host_gather flag that falls back to GET_ROWS when it cannot apply (device-resident table, LoRA on the embedding, Hadamard-latent table, mixed batches). LLAMA_DSV41_HOST_GATHER=0 restores the in-graph GET_ROWS.
…ariant) DeepSeek-V4.1 ships a 3-stage DSpark block drafter (mtp.0..2). It is exported as a separate DFLASH GGUF with convert_hf_to_gguf.py --dspark and used with --spec-type draft-dspark, like the V4 DSpark draft. - DFLASH graph_dsv41: single-pass mHC (each sublayer collapses with the previous sublayer's pre-mix, output with the last FFN pre-mix, no hc_head), no per-head q norm, plain rope; markov and confidence heads as V4. Flagged by dflash.hyper_connection.single_pass. - Block queries see the main-KV window up to the anchor plus the whole block; n_swa grows by one so the first query's mask row reaches window_size positions before the anchor. - deepseek41 exports the target features (mean over the hc copies of the layer input) via embeddings_layer_inp. - Converter: DeepseekV41DSparkModel (MXFP4 experts bit-exact, dense BF16, conf_proj F32; embeddings and lm head come from the target).
Reuse the DeepSeek-V4-Flash-Vision ViT + 3x3 aligner graph with the V4.1 token block: [START] ([IMAGE]*w [NEWLINE])*h [END] in reading order, no lead pads, no pad row, no image_pad tensor, causal decode. Port the V4.1 resize plan (min_pixels upscale, single solve, 1024-token cap) and PIL ImageOps.pad (round half to even), and round the normalized pixels to bf16 as load_image does. Converter: DeepseekV41VisionModel writes the mmproj from the HF-style vision_config.
im2col only reads the kernel shape. The zero-stride view was non-contiguous, so CUDA rejected the op and the 3x3 unfold fell back to CPU. A contiguous view of the same shape keeps it on the GPU; outputs are byte-identical. Add the aligner shape to test-backend-ops.
Member
Author
|
Pushed 9930bb5, which merges three fixes from the leaf branches:
Net diff against the previous head is 3 files, +6/-2. CPU repros for each pass on this pin and on the full published stack; no-adapter logits are bit-identical. The pin-set repin from d1781c3 follows in a separate pr-set change. |
Member
Author
|
Pushed 29b3b4b: namespaced DeepSeek V4.1 tool calls now parse. The template writes the tool name qualified with its namespace, but the parser built the expected tool-name literal from the bare |
This was referenced Oct 22, 2026
danielhanchen
changed the base branch from
dsv41-pin-base61b
to
dsv41-pin-base61c
October 25, 2026 21:51
This was referenced Oct 27, 2026
Member
|
Closing in favor of #332. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Single pin for DeepSeek-V4.1-Flash-1017: the full DeepSeek-V4.1 stack (#289 and its follow-ups) minus #292, so one entry in
pr-set.jsoncarries the whole model and the nightly merge stays additive.Base is
dsv41-pin-base61b= the pinned #61 commit (05a6693, IQ1_XS / IQ1_XXS / IQ1_XXXS as fork-local types 64..66) with upstream797de63b2merged in. Both #61 and this stack add aggml_typeand editGGML_TYPE_COUNT, which no additive merge can resolve, so this pin carries the combined enum (MXFP8 = 43,IQ1_XS..IQ1_XXXS = 64..66,GGML_TYPE_COUNT = 67) and must sit after #61 inpr-set.json. Against that base theggml.handgguf-py/gguf/constants.pychanges are pure additions. MXFP8 stays at 43 because the published DeepSeek-V4.1 GGUFs store the n-gram table as type 43.Included:
Not included:
ggml-cuda.cu,top-k.cuand itstest-backend-opscases) are reverted in one commit, so the CUDA top-k and F32 matmul paths are unchanged from upstream.Built from the published PR heads with no other code changes: #297 head (which already carries #289 to #296 merged down the stack), one commit reverting the #292 diff, then #290, #291 and the #61 base merged (only conflict: the two type enums, resolved as their union).
Testing:
LLAMA_FATAL_WARNINGS=ON,ctest -L mainllama-server,llama-cli,test-backend-opsllama_pinset.py sim, aged to newest tags) with this pin appended to the currentpr-set.json