Skip to content

fold whole blocks in the any-length codec - #2

Open
crypt0miester wants to merge 20 commits into
mainfrom
block-fold
Open

fold whole blocks in the any-length codec#2
crypt0miester wants to merge 20 commits into
mainfrom
block-fold

Conversation

@crypt0miester

@crypt0miester crypt0miester commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

any-length codec is a block fold

a whole 64-byte block enters the value at once, value = value * 2^512 + block against a compile-time 2^512, so the value is reduced once per limb per block instead of once per limb per word. decode reads ten characters a pass into 64-bit words instead of five into 32-bit words. the fold runs in place against a sliding window, and on x86 sums its columns eight to a register.

at 1232 bytes, one solana packet:

encode decode
m4 max, before 41.1 us 27.2 us
m4 max, after 7.31 us 7.54 us
zen 5, before 45.9 us 21.8 us
zen 5, after 11.5 us 8.00 us

stack frame for encode drops from 4344 bytes to 1900, which matters because sbf's is 4096. the previous fold was over it.

removals

drops the variable feature, src/place_values.rs and src/variable_simd.rs. 1068 insertions against 13012 deletions. the crate now has one path per direction and its whole constant set is about 1.3 kib.

that feature reached 8.10 us encoding a packet on zen 5 with hand-written avx-512, which the fold does not match there yet (11.5). it beats it comfortably on aarch64, and beats the default it replaced on both. four hand-written x86 kernels were measured; the one kept is worth 1.27x, and the note in src/fold.rs records why the other three lost.

two bugs, both found by fuzzing

both pre-existing, both on the leading-ones edge that a width sweep misses because it varies length rather than content.

  • decoded_len under-sized its result. a leading 1 is a whole byte, so 88 ones decode to 88 bytes where the 0.733 ratio said 66. callers sizing buffers as documented got spurious errors.
  • decode accepted up to 1702 characters, so a long run of ones decoded to more bytes than encode would take back. the codec could produce values it could not re-encode.

new: ci and fuzzing

there was no ci at all. adds fmt, clippy, tests in debug and release across x86-64 and aarch64, a thumbv6m-none-eabi no_std build, an msrv check, and a fuzz smoke run. also typechecks the benchmarks, which had been silently broken by a signature change.

two libfuzzer targets, both differential against bs58. TAPE_PATH pins the dispatch path, without which only the widest kernel a machine has is ever reached.

alloc feature

default = ["alloc"], crate stays #![no_std]. above MAX_VARIABLE_LEN the scratch comes from the heap and there is no length limit. programs build with default-features = false and keep the stack path. no api change either way. the cost past a packet is quadratic and measured: four times the input is about fifteen times the work.

breaking, 0.1.0 to 0.2.0

  • encode_32 and encode_64 now return u8 rather than usize. call sites index with as usize.
  • the variable feature is gone.
  • alloc is on by default, so encode no longer refuses input past MAX_VARIABLE_LEN. programs that want the old behaviour build with default-features = false.

testing

variable_lengths sweeps every width 0 to 1232 against bs58 in both directions, up from a 300-byte sample. variable_paths runs the same comparison with portable, avx2 and avx-512 each pinned in turn. fuzzed on zen 5 across all three paths under addresssanitizer, 4.84m runs, no failures.

follow-ups, not in this pr

  • ci fuzzes only the runner's native path. three lines in ci.yml would give it the same matrix run by hand here.
  • the x86 encode gap is open: 11.5 us against the 8.10 the deleted tables reached, on the sendTransaction path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant