Skip to content

perf: add direct binary encoding for collection models - #20

Merged
arjanz merged 2 commits into
mainfrom
codex/direct-binary-encoding
Nov 9, 2026
Merged

arjanz merged 2 commits into
mainfrom
codex/direct-binary-encoding

Conversation

@arjanz

@arjanz arjanz commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Encoding collection models builds intermediate codec wrappers before producing bytes. This PR adds a direct path that writes supported models into a single byte buffer.

  • Enable the path automatically through Serializable.to_jam_bytes() for models whose top-level fields are arrays, vectors, bounded vectors or maps.
  • Optimize primitive writes, integer/hash sequences, byte payloads, empty collections and short length prefixes.
  • Preserve custom codec hooks, overridden to_codec_type() methods and runtime schema changes through compatibility checks and fallback.
  • Preserve validation order and map sorting, including duplicate-key value comparisons.
  • Keep mixed records on the existing encoding path.

No Pyjamaz call-site changes are required; the optimization becomes available when the updated codec is installed.

Validation

  • 240 codec tests and 110 subtests passed on Python 3.12 and 3.13.
  • 45 Pyjamaz codec and serialization integration tests passed.
  • Replayed 1,000 traces across eight suites with matching state roots.
  • Compared 24,360 encoded values against the existing encoder, with identical binary output.
  • Added coverage for invalid inputs, custom hooks, mutable definitions, empty collections and input isolation.

Performance

Local median wall times from three alternating runs against the existing encoder:

Trace suite Change
fallback 2.8% faster
safrole 3.7% faster
storage 2.2% faster
storage_light 2.6% faster
preimages 0.9% slower
preimages_light 7.9% faster
fuzzy 0.9% faster
fuzzy_light 0.6% slower

End-to-end gains are modest and workload-dependent. Measurements showed noticeable run-to-run variation, so small differences should be treated cautiously.

Encode supported collection models into a single byte buffer without
building intermediate codec wrappers.

Preserve the existing path for mixed records, custom codecs and
unsupported inputs, including validation order and map sorting.

Add coverage for binary parity, custom hooks, mutable schemas,
empty collections and input isolation.
Encode supported collection models into a single byte buffer without
building intermediate codec wrappers.

Preserve the existing path for mixed records, custom codecs and
unsupported inputs, including validation order and map sorting.

Add coverage for binary parity, custom hooks, mutable schemas,
empty collections and input isolation.
@arjanz
arjanz merged commit 3b729f6 into main Nov 9, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant