Skip to content

postcard-dyn: support non-string map keys - #303

Open
teddytennant wants to merge 2 commits into
jamesmunns:mainfrom
teddytennant:postcard-dyn-non-string-map-keys
Open

teddytennant wants to merge 2 commits into
jamesmunns:mainfrom
teddytennant:postcard-dyn-non-string-map-keys

Conversation

@teddytennant

@teddytennant teddytennant commented Aug 8, 2026 •

Copy link
Copy Markdown

postcard-dyn rejects any schema whose map key is not a String, in both directions:

// source/postcard-dyn/src/ser.rs
OwnedDataModelType::Map { key, val } => {
    // TODO: impling blind because we can't test this, oops
    //
    // TODO: There's also a mismatch here because serde_json::Value requires
    // keys to be strings, when postcard doesn't.
    if key.ty != OwnedDataModelType::String {
        return Err(Error::ShouldSupportButDont);
    }

serde_json::Value does require object keys to be strings, but serde_json
itself does not refuse the same types postcard-dyn does — it stores primitive
keys as their string representation. So a HashMap<u32, u32> round trips
through serde_json but not through postcard-dyn.

Reproducer from #275:

#[derive(Serialize, Deserialize, Schema)]
struct Foobar {
    map: HashMap<u32, u32>,
}

let foobar = Foobar { map: HashMap::from([(1, 1)]) };

// works
let _json_encoded = serde_json::to_vec(&foobar).unwrap();

let postcard_encoded = postcard::to_stdvec(&foobar).unwrap();
let schema = OwnedNamedType::from(Foobar::SCHEMA);

// panics with ShouldSupportButDont
let _json_dyn = postcard_dyn::from_slice_dyn(&schema, &postcard_encoded).unwrap();

The change

de_map_key / ser_map_key handle keys of the types serde_json can store as
a string — bool, all the integer types, String, and unit variants of an
enum — by going through that string form, and only that string form. The key
still goes down the ordinary de_named_type / ser_named_type path for its
schema type, so the wire encoding is unchanged and there is only one place that
knows how each type is encoded.

This is the right layer for it because the mismatch is purely between the
postcard data model and serde_json::Value, which is exactly what these two
functions bridge; postcard itself already handles these keys fine.

Types serde_json also cannot turn into a string key still return
ShouldSupportButDont rather than being silently mangled, and that is asserted
in the test.

With this, the reproducer above returns {"map":{"1":1}}, equal to what
serde_json::to_value produces for the same struct.

Map entry order

serde_json::Map keeps its entries in the lexicographic order of the stringified
key, so a Value cannot carry the order the entries were decoded in. "10" sorts
before "2", and a unit variant sorts by name rather than by declaration, so both
integer and enum keys came back out of to_stdvec_dyn reordered.

Rather than try to preserve the decoded order, to_stdvec_dyn now recovers each
key and writes the entries in the key type's own order instead of the Value's.
That is the order a BTreeMap<K, V> writes them in, so the byte round trip is
exact for any map that was in key order to begin with. A HashMap never had an
order to preserve, before or after this.

Deliberately not changed

  • char and float keys. serde_json does store those as strings too, but
    de_named_type has OwnedDataModelType::Char => todo!(), so accepting char
    keys would give a serializer the deserializer can't undo, and round tripping a
    float through a decimal string is not something I wanted to do silently. Both
    are noted in a TODO next to the new code.
  • The wire format, and everything outside the map-key path.

The same change is applied to postcard-dyn-ng, matching how other fixes have
been mirrored into the -ng crates. The two hunks are identical apart from the
-ng schema type shapes.

Verification

New test de::test::maps round trips a map through postcard → Value →
postcard for string, bool, u32, i64 and unit-variant-enum keys, asserts
the intermediate Value matches what serde_json::to_value produces for the
same map, and asserts a (u8, u8) key is still rejected by both from_slice_dyn
and to_stdvec_dyn.

It also covers the cases where the key order and the string order disagree: keys
of differing digit width, negative keys of the same width, mixed sign, an enum
whose variant names sort against their declaration order, and a nested map. Each
of those fails without the ser.rs half of the change.

Before the fix (test present, source change reverted):

running 2 tests
test ser::test::maps ... ok
test de::test::maps ... FAILED

---- de::test::maps stdout ----
thread 'de::test::maps' panicked at source/postcard-dyn/src/de.rs:540:50:
called `Result::unwrap()` on an `Err` value: ShouldSupportButDont

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out

Note the string-keyed case passes there — that half is the control, and it is
unaffected by this change.

After:

running 11 tests
test ser::test::enums ... ok
test ser::test::ints ... ok
test ser::test::maps ... ok
test de::test::smoke ... ok
test ser::test::opts ... ok
test ser::test::seqs ... ok
test de::test::maps ... ok
test ser::test::serde_j ... ok
test ser::test::strs ... ok
test ser::test::structs ... ok
test ser::test::tups ... ok

test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Reverting only the ser.rs half and leaving de.rs fixed also fails the test,
at the re-serialize step — the round trip catches an asymmetric ser/de rather
than just one direction.

Full ./ci.sh feature set passes: cargo test --all is green across the
workspace (0 failures), cargo clippy --all --all-targets -- --deny=warnings
is clean, and cargo fmt --all -- --check is clean.

Note on #194

#194 replaces postcard-dyn's de.rs wholesale with a reserialize/ module and
drops the ShouldSupportButDont path along the way. It has been idle since
2025-06, and this is a scoped fix for the specific bug in #275 rather than a
claim on that work — but if #194 lands first, this becomes redundant and should
just be dropped.

Closes #275.

@netlify

netlify Bot commented Aug 8, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for cute-starship-2d9c9b canceled.

Name Link
🔨 Latest commit 1542a1b
🔍 Latest deploy log https://app.netlify.com/projects/cute-starship-2d9c9b/deploys/6ab0c8f7c68bff000830a715

@fallenmi fallenmi left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new numeric-key path does not preserve the map-entry order across the byte round trip claimed here. from_slice_dyn inserts the stringified keys into the default serde_json::Map, which orders them lexicographically, and to_stdvec_dyn then serializes in that string order. Numeric order diverges as soon as digit widths differ.

On the exact head 512b6801d67e2ec7674bd8e4c65b9543a040ceca, I added the same map_round_trip case with BTreeMap::from([(2u32, 20u8), (10, 100)]). Both postcard-dyn and postcard-dyn-ng fail identically: the original bytes are [2, 2, 20, 10, 100], while the reserialized bytes are [2, 10, 100, 2, 20]. The existing 22 focused tests pass because their numeric keys (1, 2) happen to have the same numeric and lexicographic order.

Please cover a differing-order case and preserve the decoded entry order (or explicitly change the round-trip contract); otherwise this feature silently changes postcard wire order for valid numeric-key maps.

Disclosure: this review was prepared with Codex assistance; I independently verified the diff, the exact-head test oracle in both mirrored crates, and the original focused suites.

@teddytennant

Copy link
Copy Markdown
Author

Good catch, the existing cases only passed because 1 and 2 sort the same either way. The same thing hits enum keys, where postcard orders by variant index and the string form orders by name.

A Value can't carry the decoded order, so I sorted it back on the way out instead. to_stdvec_dyn now recovers each key and writes the entries in the key type's own order, which is the order a BTreeMap<K, V> writes them in, so the round trip is exact for any map that was in key order to begin with. A HashMap never had an order to preserve either way. I've written that down in the PR body since it is a real contract statement, not just a bug fix.

Added your case, a negative one, a mixed sign one, the enum, and a nested map. Each of those fails without the ser.rs half. cargo test --all with the ci.sh feature set is green, and clippy with --deny=warnings and fmt --check are clean.

@teddytennant
teddytennant force-pushed the postcard-dyn-non-string-map-keys branch from 8fd8345 to 1542a1b Compare September 21, 2026 06:04
serde_json::Value requires object keys to be strings, so postcard-dyn
rejected any schema whose map key was not a String. serde_json itself
does not have that restriction on the types it will accept as a map
key: it stores primitive keys as their string representation instead,
so a HashMap<u32, u32> round trips through serde_json but not through
postcard-dyn.

Do the same thing here for bool, the integer types, and unit variants
of an enum, in both directions. Key types serde_json also cannot store
as a string still return ShouldSupportButDont.
serde_json::Map keeps its entries in the lexicographic order of the
stringified key, which is only the order postcard wrote them in when the
keys really are strings. "10" sorts before "2", and a unit variant sorts
by name rather than by declaration, so a map with either kind of key came
back out of to_stdvec_dyn with its entries in a different order than they
went in.

Value can't carry the decoded order, so recover the real key on the way
out and sort the entries by it before writing. That is the order a
BTreeMap<K, V> was written in, which makes the byte round trip stable.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

postcard-schema: maps with non-string keys don't work in cases that are supported by serde_json

2 participants