[Bug]: GenericMTMDChatHandler does not load chat template from model with chat_format=None

[FennelFetish]

Hello. The GenericMTMDChatHandler skips loading of embedded chat templates from a model when constructed with chat_format=None.
Since the chat_format parameter for the __init__ method is optional, I would expect it to load the chat template from the model in this case.

I've investigated and found a workaround, but I'm not sure I get the big picture so I didn't make a PR.

Example Case

MODEL  = "/models/Qwen3.5-9B-GGUF/Qwen3.5-9B-Q8_0.gguf"
MMPROJ = "/models/Qwen3.5-9B-GGUF/mmproj-BF16.gguf"

def create_llama_with_chathandler():
    chat_handler = GenericMTMDChatHandler(
        chat_format         = None,
        mmproj_path         = MMPROJ,
        verbose             = True,
    )

    return Llama(
        model_path          = MODEL,
        #mmproj_path         = MMPROJ,
        n_gpu_layers        = "all",
        n_ctx               = 8192,
        n_batch             = 512,
        n_threads           = 11,
        ctx_checkpoints     = 0,
        verbose             = True,
        chat_handler        = chat_handler
    )

With these arguments, the MTMDChatHandler.__init__() method will fallback to setting self.chat_format = self.CHAT_FORMAT:

llama-cpp-python/llama_cpp/llama_multimodal.py

Line 165 in 1762647

if (not hasattr(self, "chat_format") or self.chat_format is None) and chat_template_override is None:

Later in GenericMTMDChatHandler.__call__(), it runs _resolve_chat_format()
which is supposed to load the template once when the model has become available.
However, the guard at the top exits the function early, because self.chat_format is not None:

llama-cpp-python/llama_cpp/llama_multimodal.py

Line 1611 in 1762647

if self.chat_format is not None:

But even if it would load the template, it is not applied:
MTMDChatHandler._render_mtmd_prompt() uses self.chat_template to render the text,
but self.chat_template is not updated in GenericMTMDChatHandler._resolve_chat_format().

Workaround

The following is a simple workaround, but it has to call and modify protected members which is brittle.
But it's nice to even have the possibility for this easy hack due to the separated functions you've introduced! (I've used those for other things too)

def create_llama_with_chathandler_workaround():
    chat_handler = GenericMTMDChatHandler(
        chat_format         = None,
        mmproj_path         = MMPROJ,
        verbose             = True,
    )

    llm = Llama(
        model_path          = MODEL,
        #mmproj_path         = MMPROJ,
        n_gpu_layers        = "all",
        n_ctx               = 8192,
        n_batch             = 512,
        n_threads           = 11,
        ctx_checkpoints     = 0,
        verbose             = True,
        chat_handler        = chat_handler
    )

    # APPLY WORKAROUND: Load the model's embedded chat template manually

    # Set to None so _resolve_chat_format() loads the template
    chat_handler.chat_format = None
    chat_handler._resolve_chat_format(llm)

    # Apply the loaded template
    chat_handler._change_chat_template(chat_handler.chat_format)

    return llm

My Usecase:

I have an image captioning tool that allows to setup models with different backends (Gemma4, Qwen3.5, etc.).
For GGUF, each backend entry is associated with a llama-cpp-python chat handler class.
Another "Generic GGUF" entry uses GenericMTMDChatHandler to load the template from the model.
To have a single code path, I always construct the Llama instance with a chat_handler argument.
Additionally, the user can provide a path to a jinja template file to override the template. The text is loaded from the file and always passed as chat_template_override.
(code)

Sidenote

I have noticed that MTMDChatHandler._change_chat_template() processes the jinja template differently from llama_chat_format.Jinja2ChatFormatter. It would be nice to have the loopcontrols extension available.

llama-cpp-python/llama_cpp/llama_chat_format.py

Line 307 in 1762647

environment = ImmutableSandboxedEnvironment(

Full Test Code (with rendererd template text)

This contains 3 different functions for loading the model.
The comments above the functions show the rendered text.

from llama_cpp import Llama
from llama_cpp.llama_multimodal import GenericMTMDChatHandler


MODEL  = "/mnt/ai/Models/MM-LLM/Qwen3.5-9B-GGUF/Qwen3.5-9B-Q8_0.gguf"
MMPROJ = "/mnt/ai/Models/MM-LLM/Qwen3.5-9B-GGUF/mmproj-BF16.gguf"

FILES = ["/home/user/Pictures/red-tree-with-eyes.jpeg"]


# GenericMTMDChatHandler(_process_mtmd_prompt): Rendered prompt length: 271 chars, Media count: 1.

# Rendered prompt: <|im_start|>system
# You are a professional photographer that describes scenes in concise but precise English prose.<|im_end|>
# <|im_start|>user
# Describe this image in one sentence.<|vision_start|><__media__><|vision_end|><|im_end|>
# <|im_start|>assistant
# <think>
#
# </think>
def create_llama():
    return Llama(
        model_path          = MODEL,
        mmproj_path         = MMPROJ,
        n_gpu_layers        = "all",
        n_ctx               = 8192,
        n_batch             = 512,
        n_threads           = 11,
        ctx_checkpoints     = 0,
        verbose             = True,
        #chat_handler        = chat_handler
    )


# GenericMTMDChatHandler(_process_mtmd_prompt): Rendered prompt length: 162 chars, Media count: 1.

# Rendered prompt: ,You are a professional photographer that describes scenes in concise but precise English prose.
# USER: Describe this image in one sentence.<__media__>
# ASSISTANT:
def create_llama_with_chathandler():
    chat_handler = GenericMTMDChatHandler(
        chat_format         = None,
        mmproj_path         = MMPROJ,
        verbose             = True,
    )

    return Llama(
        model_path          = MODEL,
        #mmproj_path         = MMPROJ,
        n_gpu_layers        = "all",
        n_ctx               = 8192,
        n_batch             = 512,
        n_threads           = 11,
        ctx_checkpoints     = 0,
        verbose             = True,
        chat_handler        = chat_handler
    )


# GenericMTMDChatHandler(_process_mtmd_prompt): Rendered prompt length: 271 chars, Media count: 1.

# Rendered prompt: <|im_start|>system
# You are a professional photographer that describes scenes in concise but precise English prose.<|im_end|>
# <|im_start|>user
# Describe this image in one sentence.<|vision_start|><__media__><|vision_end|><|im_end|>
# <|im_start|>assistant
# <think>
#
# </think>
def create_llama_with_chathandler_workaround():
    chat_handler = GenericMTMDChatHandler(
        chat_format         = None,
        mmproj_path         = MMPROJ,
        verbose             = True,
    )

    llm = Llama(
        model_path          = MODEL,
        #mmproj_path         = MMPROJ,
        n_gpu_layers        = "all",
        n_ctx               = 8192,
        n_batch             = 512,
        n_threads           = 11,
        ctx_checkpoints     = 0,
        verbose             = True,
        chat_handler        = chat_handler
    )

    # Apply workaround: Load the model's embedded chat template manually

    # Set to None so _resolve_chat_format() loads the template
    chat_handler.chat_format = None
    chat_handler._resolve_chat_format(llm)

    # Apply the loaded template
    chat_handler._change_chat_template(chat_handler.chat_format)

    return llm



def run(llm: Llama, file: str) -> str:
    prompt = [
        {"type": "text", "text": "Describe this image in one sentence."},
        {"type": "image_url", "image_url": f"file://{file}"},
    ]

    messages = [
        {"role": "system", "content": "You are a professional photographer that describes scenes in concise but precise English prose."},
        {"role": "user", "content": prompt},
    ]

    completion = llm.create_chat_completion(messages, add_generation_prompt=True)

    msg = completion["choices"][0]["message"]
    answer = msg["content"].strip()

    print(f"=== {file} ===")
    print(answer)
    return answer


def main(files: list[str]):
    # Loading variants: create_llama(),  create_llama_with_chathandler(),  create_llama_with_chathandler_workaround()
    llm = create_llama_with_chathandler()  # <<< Change loading variant here

    answers = {}
    for file in files:
        answers[file] = run(llm, file)

    print()
    print()
    for file, answer in answers.items():
        print(f"=== {file} ===")
        print(answer)
        print()

    print()
    print()


if __name__ == "__main__":
    main(FILES)

Issue Template

Details

Prerequisites

• I am running the latest code from the JamePeng/llama-cpp-python branch.
• I carefully followed the README.md.
• I have verified the issue is not a duplicate.
• I have tested it using the official binary llama-cli or llama-server provided by llama.cpp, and the problem (exists/does not exist) still exists.
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Environment & Hardware Configuration

RTX 4090 on Linux Kubuntu 26.04 - should not matter

Toolchain Versions

Provide the exact versions or commit hashes:

• Python: Python 3.12.3
• Python Library: pip list
• Compiler: g++ (Ubuntu 15.2.0-16ubuntu1) 15.2.0
• llama-cpp-python Commit: patch(Llama): Increase chunk preview limit to 128 in Llama.eval exception
• vendor/llama.cpp Commit: -

Model & Logic Context

• Model Source: https://huggingface.co/unsloth/Qwen3.5-9B-GGUF
• Model Path: Qwen3.5-9B-Q8_0.gguf
• Multimodal Path: mmproj-BF16.gguf

[JamePeng]

There is no need to import GenericMTMDChatHandler separately; simply pass the mmproj_path during Llama initialization. The initialization process will automatically load the built-in chat template.

If you need to pass chat_handler parameters:

# Initialize the Llama model with vision support
llm = Llama(
    model_path=MODEL_PATH,
    mmproj_path=MMPROJ_PATH,
    n_gpu_layers=-1,    # Use all available GPU layers
    n_ctx=8192,         # Context window size
    ctx_checkpoints=0,
    verbose=True,
    verbosity=2,
    chat_handler_kwargs={
        "extra_template_arguments": {
            "enable_thinking": True,
        },
        "verbose": True,
    },
)