Repository navigation
Adding LZ4ASDF format to ChunkCodecLibLz4? #72
Description
This package implements its own custom LZ4 encoding and decoding. It would be nice to have this in ChunkCodecLibLz4 too. The current supported LZ4 formats in ChunkCodecLibLz4 are Numcodecs (for Zarr), HDF5, Frame, and Block, but it looks like ASDF has its own slightly different format.
IIUC this is defined in https://github.com/asdf-format/asdf/blob/14c40c83fc3cca7285b59be3976bd29e2ba35543/asdf/_compression.py#L65
Based on this, it looks like Lz4 frame should be using "lz4f" as the 4-byte key not "lz4\0" to be readable by python (but I haven't set up tests for this yet).
Activity
icweaver commentedon Aug 27, 2026
Looks like it's time for future me to revisit this again. Will take a look tomorrow
icweaver commentedon Aug 28, 2026
Nice, I really like how much this simplifies things. Thanks for the suggestion!
I've started a PR here to get the lz4(f) split-out groundwork in place here first so that we can start discussing an upstream design for an
LZ4ASDFCodec. Looking forward to your thoughts!#73
nhz2 commentedon Sep 25, 2026
I've added compat tests for asdf to ChunkCodecs.jl in JuliaIO/ChunkCodecs.jl#96