Skip to content

feat: share canonical Arrow ingestion schema fingerprints - #386

Merged
beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-arrow-contract
Oct 19, 2026
Merged

beinan merged 1 commit into
lance-format:mainfrom
beinan:codex/wal-arrow-contract

Conversation

@beinan

@beinan beinan commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Ingress and materializers need the same schema identity after metadata is reloaded in a different map order. Add an optional arrow-contract helper that canonicalizes Arrow 58 schema JSON before applying the existing ingestion-contract hash. It sorts nested object keys while preserving field order, types, nullability, metadata values and dictionary attributes.

The helper needs schema metadata only. Replicas can keep using the Arrow-free contract feature. CI covers both JSON map-order configurations; fixed canonical bytes and an independently checked digest detect representation changes during dependency upgrades.

Refs #378.

Testing

  • Four new schema tests passed, including fixed bytes/hash, nested metadata/reload stability, meaningful schema changes and dictionary attributes.
  • Optional-feature release tests passed with both JSON map-order configurations; release Clippy passed.
  • Full repository check script passed: formatting, workspace Clippy and all-target Rust tests (594 passed); Python tests (229 passed, 2 skipped, 1 xfailed), Ruff formatting/lint and Pyright.

@beinan
beinan merged commit 4c4e936 into lance-format:main Oct 19, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant