Skip to content

WebGPU backend roadmap for Q1_0 kernels? #24

Closed

Description

@pmenic

Hi, thanks for the great work on Bonsai-8B and the Q1_0 quantization.

I'd like to ask about plans for running Q1_0 models in the browser via WebGPU.

Current state (as far as I can see):

  • The Q1_0 custom kernels exist for CUDA (.cu), Metal (.metal), and Vulkan (GLSL/SPIR-V), plus CPU.
  • Upstream ggml-org/llama.cpp has an active WebGPU backend in development (multiple PRs merged in recent weeks, including Intel subgroup matrix support).
  • However, the WebGPU backend upstream only targets standard quant types; there is no WGSL implementation of the Q1_0 kernel yet.

Questions:

  1. Is a WGSL port of the Q1_0 kernel on your roadmap, or under consideration?
  2. If not planned internally, would you be open to a community PR once the upstream WebGPU backend stabilizes?
  3. Any preference between waiting for upstream WebGPU to mature vs. an earlier experimental port?

Running Bonsai-8B purely in-browser (no server, no install) would be a great showcase of the 1-bit quantization story — curious where this sits in your plans.

Thanks!

Activity

  1. khosravipasha commented on Apr 16, 2026

    @khosravipasha
    Collaborator

    Thanks :)

    Yeah WebGPU backend in llama.cpp for Q1_0 would be amazing. Currently we are not actively working on it.
    Of course would love a PR (can do here, or main llama.cpp or both :) )
    I have not played around with the webgpu backend so don't know how good it is

    One a related note Huggingface team made a WebGPU demo you can try here:
    I believe its using their transformer.js library.
    https://huggingface.co/spaces/webml-community/bonsai-webgpu

  2. khosravipasha commented on Apr 19, 2026

    @khosravipasha
    Collaborator

    Closing this now, happy to check out a webgpu backend for 1-bit and ternary models in llama.cpp.
    Won't have the bandwith near future to work on ourselves, but PRs are welcome (probably best to go to main llama.cpp)

    For now can check these two demos:

  3. khosravipasha commented on May 5, 2026

    @khosravipasha
    Collaborator

    Just noticed a webgpu kernel was merged in main llama.cpp here: ggml-org#22374

    Have not picked up the changes in our fork yet, will do sometime this week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions