Skip to content

Fix review findings and make the fountain decoder allocation-free - #17

Merged
odudex merged 16 commits into
mainfrom
perf/in-place-fountain-and-fixes
Sep 27, 2026
Merged

odudex merged 16 commits into
mainfrom
perf/in-place-fountain-and-fixes

Conversation

@odudex

@odudex odudex commented Sep 27, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes from a review of cUR, plus a rework of the multipart decoder that makes lossy scans of large URs 11–13× faster on ESP32-P4. Each fix is its own commit and does not depend on the rework.

Performance

Mixed frames were allocation-bound: each one copied every recovered fragment and stored equation it was checked against. Equations are now reduced in place, recovered fragments are indexed by fragment number, and each frame is parsed with a single allocation.

Decoder time on an ESP32-P4 (360 MHz, PSRAM 200 MHz):

scan before after worst frame
250 fragments, 60% missed 1.77 s 0.14 s 43 → 8.5 ms
400 fragments, 60% missed 5.79 s 0.47 s 94 → 20 ms
1000 fragments, 30% missed 22.2 s 1.65 s 117 → 31 ms

The two perf commits do not change decoding: after every frame the decoder state matches the preceding commit, across 1,900 randomized scans and 300,000 mutated frames.

Fixes

  • Fountain: an allocation failure during reassembly stalled the scan forever; a PRNG draw of exactly 1.0 read past the index list; alias-table rounding now matches bc-ur and foundation-ur-py; seq_num 0 and inconsistent fragment sizes are rejected.
  • hd-key / keypath: malformed chain codes, origins and children paths were silently dropped, yielding a different xpub, a missing key origin or a non-ranged descriptor; private keys were rendered as xpubs; child ranges became *; indexes above 32 bits were truncated.
  • CBOR: string lengths are checked against the input before allocating; repeated map keys and trailing bytes are rejected.
  • Parsing: paths and sequence components split like the reference decoders; UR characters are matched as ASCII regardless of locale.
  • Bindings:
    • MicroPython UR objects never freed their memory.
    • C allocations leaked when a MicroPython allocation raised.
    • Frames with an embedded NUL were accepted by their prefix.
    • CPython raised on lone surrogates.
    • FountainEncoder views could form uncollectable cycles.
    • first_seq_num = 2^32−1 failed only at the first part.
    • The MicroPython unix port did not build under its float warnings.

Stricter parsing now rejects input that used to be accepted: hd-keys without a chain code, repeated CBOR keys, trailing bytes and malformed sequence components.

Every fragment was already held, so no later part re-triggered the join and the scan never completed.
prng_next_int() returns max + 1 when prng_next_double() rounds to 1.0, which read one element past the remaining-index list.
CPython sets LC_CTYPE from the environment, where islower() and tolower() also accept Latin-1 letters.
ur_attr never fell through to locals_dict, so the GC found no finaliser and every UR object leaked its native copy.
A MemoryError, or the UnicodeError a non-UTF-8 bip39 word raises, long-jumped past the free.
…ts by number

Mixed frames were allocation-bound: each copied every recovered fragment and stored equation it was checked against.
Frames with an embedded NUL or lone surrogate are rejected instead of truncated or raised, MicroPython UR types must be NUL-free str, empty payloads get a real buffer, the CPython FountainEncoder view joins cyclic GC, first_seq_num stops at 2^32 - 2, and the unix port builds under -Werror=float-conversion.
@odudex
odudex merged commit 4dc346e into main Sep 27, 2026
4 checks passed
@odudex
odudex deleted the perf/in-place-fountain-and-fixes branch October 30, 2026 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant