Skip to content

feat(int4): add nibble_pack/unpack for true 0.5 B/elem int4 storage - #329

Open
lokic233 wants to merge 1 commit into
google:mainfrom
lokic233:fix/int4-nibble-packing
Open

lokic233 wants to merge 1 commit into
google:mainfrom
lokic233:fix/int4-nibble-packing

Conversation

@lokic233

@lokic233 lokic233 commented Jul 8, 2026

Copy link
Copy Markdown

Summary

Addresses #328. qwix int4 QArrays currently store at 1.0 bytes/element (JAX's jnp.int4 byte-pads). This PR adds nibble_pack() / nibble_unpack() utilities that store two 4-bit values per uint8 byte — achieving the true 0.5 B/elem (4× vs fp32, 2× vs int8) memory reduction that int4 is meant to provide.

What's included

  • qwix/_src/core/nibble_pack.py: nibble_pack(q) [N,K]→[N,K/2] uint8 and nibble_unpack(packed) [N,K/2]→[N,K] int8 in [-8,7]
  • tests/nibble_pack_test.py: roundtrip exactness, memory halving, boundary values

Measured

  • Verified on NVIDIA GB200 + Google TPU v7x (jax 0.10.2)
  • nibble_pack achieves exactly 0.5 B/elem (vs 1.0 for raw jnp.int4)
  • Roundtrip is bit-exact across the full [-8,7] range
  • Used in production for a fused W4A16 matmul kernel on TPU (0.83× bf16, rel 0.0026)

Note

We acknowledge the repo's policy on external contributions. This is offered as a concrete, tested implementation of the fix described in #328, ready to adapt/integrate however the maintainers prefer.

— via Navi on behalf of @lokic233

Currently, qwix int4 QArrays store values at 1.0 bytes/element because
JAX's jnp.int4 byte-pads each value. This means int4 quantization pays
the quality cost of 4-bit with zero memory benefit over int8.

This adds nibble_pack() and nibble_unpack() utilities that store two
4-bit values per uint8 byte, achieving the true 0.5 B/elem (4x vs fp32,
2x vs int8) memory reduction that int4 is meant to provide.

Measured: nibble_pack gives exactly 0.5 B/elem on both GPU (GB200) and
TPU (v7x), roundtrip-exact, with the standard GPTQ/AWQ packing layout.

Addresses: google#328
@jshin1394

Copy link
Copy Markdown
Collaborator

Thanks for submitting this PR and sharing your benchmarks, @lokic233! Storing int4 at true 0.5 B/elem is a huge win for memory footprint, and it's great to see your production results for the fused W4A16 TPU kernel.

  1. Before we merge this as a standalone utility, we'd love to learn more about your follow-up plan for integrating this into Qwix. Do you have a follow-up plan to integrate nibble_pack / unpack natively into QArray / qarray.py or checkpoint_util.py? Is the goal for Issue int4 QArray stores at 1.0 bytes/element (no nibble-packing) — zero memory benefit #328 is for qwix.quantize() and QArray to automatically store int4 at 0.5 B/elem out of the box?

  2. Minor comment regarding file location: we'd suggest placing these helpers in qwix/_src/utils/packing.py (as pack_int4 / unpack_int4) so _src/core stays focused on core tensor engine primitives and we have a clean home for future bit-packing formats.

Looking forward to hearing your thoughts and roadmap for this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants