Conversation
Currently, qwix int4 QArrays store values at 1.0 bytes/element because JAX's jnp.int4 byte-pads each value. This means int4 quantization pays the quality cost of 4-bit with zero memory benefit over int8. This adds nibble_pack() and nibble_unpack() utilities that store two 4-bit values per uint8 byte, achieving the true 0.5 B/elem (4x vs fp32, 2x vs int8) memory reduction that int4 is meant to provide. Measured: nibble_pack gives exactly 0.5 B/elem on both GPU (GB200) and TPU (v7x), roundtrip-exact, with the standard GPTQ/AWQ packing layout. Addresses: google#328
|
Thanks for submitting this PR and sharing your benchmarks, @lokic233! Storing int4 at true 0.5 B/elem is a huge win for memory footprint, and it's great to see your production results for the fused W4A16 TPU kernel.
Looking forward to hearing your thoughts and roadmap for this! |
Summary
Addresses #328. qwix int4 QArrays currently store at 1.0 bytes/element (JAX's
jnp.int4byte-pads). This PR addsnibble_pack()/nibble_unpack()utilities that store two 4-bit values per uint8 byte — achieving the true 0.5 B/elem (4× vs fp32, 2× vs int8) memory reduction that int4 is meant to provide.What's included
qwix/_src/core/nibble_pack.py:nibble_pack(q)[N,K]→[N,K/2] uint8 andnibble_unpack(packed)[N,K/2]→[N,K] int8 in [-8,7]tests/nibble_pack_test.py: roundtrip exactness, memory halving, boundary valuesMeasured
nibble_packachieves exactly 0.5 B/elem (vs 1.0 for rawjnp.int4)Note
We acknowledge the repo's policy on external contributions. This is offered as a concrete, tested implementation of the fix described in #328, ready to adapt/integrate however the maintainers prefer.
— via Navi on behalf of @lokic233