fold whole blocks in the any-length codec - #2
Open
crypt0miester wants to merge 20 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
any-length codec is a block fold
a whole 64-byte block enters the value at once,
value = value * 2^512 + blockagainst a compile-time2^512, so the value is reduced once per limb per block instead of once per limb per word. decode reads ten characters a pass into 64-bit words instead of five into 32-bit words. the fold runs in place against a sliding window, and on x86 sums its columns eight to a register.at 1232 bytes, one solana packet:
stack frame for encode drops from 4344 bytes to 1900, which matters because sbf's is 4096. the previous fold was over it.
removals
drops the
variablefeature,src/place_values.rsandsrc/variable_simd.rs. 1068 insertions against 13012 deletions. the crate now has one path per direction and its whole constant set is about 1.3 kib.that feature reached 8.10 us encoding a packet on zen 5 with hand-written avx-512, which the fold does not match there yet (11.5). it beats it comfortably on aarch64, and beats the default it replaced on both. four hand-written x86 kernels were measured; the one kept is worth 1.27x, and the note in
src/fold.rsrecords why the other three lost.two bugs, both found by fuzzing
both pre-existing, both on the leading-ones edge that a width sweep misses because it varies length rather than content.
decoded_lenunder-sized its result. a leading1is a whole byte, so 88 ones decode to 88 bytes where the 0.733 ratio said 66. callers sizing buffers as documented got spurious errors.decodeaccepted up to 1702 characters, so a long run of ones decoded to more bytes thanencodewould take back. the codec could produce values it could not re-encode.new: ci and fuzzing
there was no ci at all. adds fmt, clippy, tests in debug and release across x86-64 and aarch64, a
thumbv6m-none-eabino_std build, an msrv check, and a fuzz smoke run. also typechecks the benchmarks, which had been silently broken by a signature change.two libfuzzer targets, both differential against bs58.
TAPE_PATHpins the dispatch path, without which only the widest kernel a machine has is ever reached.alloc feature
default = ["alloc"], crate stays#![no_std]. aboveMAX_VARIABLE_LENthe scratch comes from the heap and there is no length limit. programs build withdefault-features = falseand keep the stack path. no api change either way. the cost past a packet is quadratic and measured: four times the input is about fifteen times the work.breaking, 0.1.0 to 0.2.0
encode_32andencode_64now returnu8rather thanusize. call sites index withas usize.variablefeature is gone.allocis on by default, soencodeno longer refuses input pastMAX_VARIABLE_LEN. programs that want the old behaviour build withdefault-features = false.testing
variable_lengthssweeps every width 0 to 1232 against bs58 in both directions, up from a 300-byte sample.variable_pathsruns the same comparison with portable, avx2 and avx-512 each pinned in turn. fuzzed on zen 5 across all three paths under addresssanitizer, 4.84m runs, no failures.follow-ups, not in this pr
ci.ymlwould give it the same matrix run by hand here.sendTransactionpath.