model : fix SWA not being enabled for EXAONE 4.5

[junmo-kim]

Overview

I ran into two problems trying to run EXAONE 4.5 locally with llama.cpp.

1. The KV cache uses far more memory than expected, so I cannot use the model's full training context.
2. Even with a reduced context size, I don't get a meaningful answer. The output is just nonsense (example below).

Digging in: the model is supposed to use sliding window attention, but the code never takes that path. load_arch_hparams() in src/models/exaone4.cpp checks hparams.n_layer() == 64 before LLM_KV_NEXTN_PREDICT_LAYERS has been read. n_layer() returns n_layer_all - n_layer_nextn and n_layer_nextn defaults to 0, so for a GGUF carrying the MTP head (block_count=65, nextn=1) it evaluates to 65 at that point and the whole SWA block is skipped.

The fix moves that read above the check; n_layer() then already accounts for the MTP head when the condition runs. The assert that validates the key moves with it, matching how the other nextn architectures keep the two adjacent.

Additional information

This affects the official LGAI-EXAONE release as well as third-party conversions. EXAONE 4.0 is unaffected: no MTP head, so block_count is 64 and the check matches.

It is easy to miss because n_swa looks fine either way. The ml.get_key(LLM_KV_ATTENTION_SLIDING_WINDOW, ...) call sits outside the block, so it reads 4096 even when SWA is off, and llama_model_n_swa() reports that too. Only swa_type stays LLAMA_SWA_TYPE_NONE.

The loader commit

The new test in tests/test-llama-archs.cpp needs this commit. The metadata-only path of create_tensor asserts on a null buffer type, but buft_for_tensor returns nullptr by design for TENSOR_SKIP tensors, which is how architectures with nextn/MTP layers mark theirs. The file-backed path a few lines below already returns nullptr for the same tensors, so this is not a behaviour change from a caller's point of view. It also makes llama_model_init_from_user work for the other MTP architectures, not only exaone4. I kept it separate since it is in the loader rather than the model, and can split it out if preferred.

Before / after

Same GGUF (official Q4_K_M), same prompt (11,271 tokens), llama-completion:

before

print_info: n_layer_all           = 65
print_info: n_swa                 = 4096
print_info: is_swa_any            = 0
llama_kv_cache: size = 4160.00 MiB ( 16384 cells, 65 layers, 1/1 seqs)

Answer: ok.
What is the entry 0 status ok.
What is the question:     ok.
(24 tokens exhausted, no EOS)

after

print_info: n_layer_all           = 65
print_info: n_swa                 = 4096
print_info: is_swa_any            = 1
llama_kv_cache_iswa: creating non-SWA KV cache, size = 16384 cells
llama_kv_cache: size = 1088.00 MiB ( 16384 cells, 17 layers, 1/1 seqs)
llama_kv_cache_iswa: creating     SWA KV cache, size =  4608 cells
llama_kv_cache: size =  864.00 MiB (  4608 cells, 48 layers, 1/1 seqs)

Answer: 7748 [end of text]

The prompt plants a fact near the start and asks for it after ~11k tokens, which is past n_swa.

@nuxlear (author of #21733): the config lists sliding_window: 4096 with an LLLG pattern, and the split after the fix comes out 16 global + 48 local + 1 MTP. Is that what the model expects?

Tests

Added three cases to tests/test-llama-archs.cpp, one for each side of the branch. A 32B model with an MTP head (block_count 65) has to come out with SWA enabled, the same model without one (block_count 64) has to as well, and the 1.2B shape (block_count 31) has to not. The first fails before the fix; the other two pin the condition against being widened or dropped, and each of the three is the only one that catches its own case.

The full arch walk that ctest -L main runs passes: exit 0, no failures.

The change is backend-agnostic: it only touches hparams ordering and the loader's metadata-only path. The before/after on a real model was measured on Windows with Vulkan.

Left alone: sliding_window_pattern is arr[bool] in the GGUF while the code reads it into a uint32_t via get_key_or_arr with required=false. It is silently ignored today, and the hardcoded period of 4 happens to match. I can file that separately.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - codebase navigation, builds and measurements, the regression test, and an independent review of this description. The find and the investigation are mine.

[CISC]

Good catch, thank you for the fix.

Please remove the test though, we would like test-llama-archs to be as agnostic as possible and this should not be necessary.