postcard-dyn: support non-string map keys - #303
teddytennant wants to merge 2 commits into
Conversation
✅ Deploy Preview for cute-starship-2d9c9b canceled.
|
fallenmi
left a comment
There was a problem hiding this comment.
The new numeric-key path does not preserve the map-entry order across the byte round trip claimed here. from_slice_dyn inserts the stringified keys into the default serde_json::Map, which orders them lexicographically, and to_stdvec_dyn then serializes in that string order. Numeric order diverges as soon as digit widths differ.
On the exact head 512b6801d67e2ec7674bd8e4c65b9543a040ceca, I added the same map_round_trip case with BTreeMap::from([(2u32, 20u8), (10, 100)]). Both postcard-dyn and postcard-dyn-ng fail identically: the original bytes are [2, 2, 20, 10, 100], while the reserialized bytes are [2, 10, 100, 2, 20]. The existing 22 focused tests pass because their numeric keys (1, 2) happen to have the same numeric and lexicographic order.
Please cover a differing-order case and preserve the decoded entry order (or explicitly change the round-trip contract); otherwise this feature silently changes postcard wire order for valid numeric-key maps.
Disclosure: this review was prepared with Codex assistance; I independently verified the diff, the exact-head test oracle in both mirrored crates, and the original focused suites.
|
Good catch, the existing cases only passed because 1 and 2 sort the same either way. The same thing hits enum keys, where postcard orders by variant index and the string form orders by name. A Added your case, a negative one, a mixed sign one, the enum, and a nested map. Each of those fails without the |
8fd8345 to
1542a1b
Compare
serde_json::Value requires object keys to be strings, so postcard-dyn rejected any schema whose map key was not a String. serde_json itself does not have that restriction on the types it will accept as a map key: it stores primitive keys as their string representation instead, so a HashMap<u32, u32> round trips through serde_json but not through postcard-dyn. Do the same thing here for bool, the integer types, and unit variants of an enum, in both directions. Key types serde_json also cannot store as a string still return ShouldSupportButDont.
serde_json::Map keeps its entries in the lexicographic order of the stringified key, which is only the order postcard wrote them in when the keys really are strings. "10" sorts before "2", and a unit variant sorts by name rather than by declaration, so a map with either kind of key came back out of to_stdvec_dyn with its entries in a different order than they went in. Value can't carry the decoded order, so recover the real key on the way out and sort the entries by it before writing. That is the order a BTreeMap<K, V> was written in, which makes the byte round trip stable.
postcard-dynrejects any schema whose map key is not aString, in both directions:serde_json::Valuedoes require object keys to be strings, butserde_jsonitself does not refuse the same types postcard-dyn does — it stores primitive
keys as their string representation. So a
HashMap<u32, u32>round tripsthrough
serde_jsonbut not throughpostcard-dyn.Reproducer from #275:
The change
de_map_key/ser_map_keyhandle keys of the typesserde_jsoncan store asa string —
bool, all the integer types,String, and unit variants of anenum — by going through that string form, and only that string form. The key
still goes down the ordinary
de_named_type/ser_named_typepath for itsschema type, so the wire encoding is unchanged and there is only one place that
knows how each type is encoded.
This is the right layer for it because the mismatch is purely between the
postcard data model and
serde_json::Value, which is exactly what these twofunctions bridge; postcard itself already handles these keys fine.
Types
serde_jsonalso cannot turn into a string key still returnShouldSupportButDontrather than being silently mangled, and that is assertedin the test.
With this, the reproducer above returns
{"map":{"1":1}}, equal to whatserde_json::to_valueproduces for the same struct.Map entry order
serde_json::Mapkeeps its entries in the lexicographic order of the stringifiedkey, so a
Valuecannot carry the order the entries were decoded in."10"sortsbefore
"2", and a unit variant sorts by name rather than by declaration, so bothinteger and enum keys came back out of
to_stdvec_dynreordered.Rather than try to preserve the decoded order,
to_stdvec_dynnow recovers eachkey and writes the entries in the key type's own order instead of the
Value's.That is the order a
BTreeMap<K, V>writes them in, so the byte round trip isexact for any map that was in key order to begin with. A
HashMapnever had anorder to preserve, before or after this.
Deliberately not changed
charand float keys.serde_jsondoes store those as strings too, butde_named_typehasOwnedDataModelType::Char => todo!(), so acceptingcharkeys would give a serializer the deserializer can't undo, and round tripping a
float through a decimal string is not something I wanted to do silently. Both
are noted in a TODO next to the new code.
The same change is applied to
postcard-dyn-ng, matching how other fixes havebeen mirrored into the
-ngcrates. The two hunks are identical apart from the-ngschema type shapes.Verification
New test
de::test::mapsround trips a map through postcard →Value→postcard for string,
bool,u32,i64and unit-variant-enum keys, assertsthe intermediate
Valuematches whatserde_json::to_valueproduces for thesame map, and asserts a
(u8, u8)key is still rejected by bothfrom_slice_dynand
to_stdvec_dyn.It also covers the cases where the key order and the string order disagree: keys
of differing digit width, negative keys of the same width, mixed sign, an enum
whose variant names sort against their declaration order, and a nested map. Each
of those fails without the
ser.rshalf of the change.Before the fix (test present, source change reverted):
Note the string-keyed case passes there — that half is the control, and it is
unaffected by this change.
After:
Reverting only the
ser.rshalf and leavingde.rsfixed also fails the test,at the re-serialize step — the round trip catches an asymmetric ser/de rather
than just one direction.
Full
./ci.shfeature set passes:cargo test --allis green across theworkspace (0 failures),
cargo clippy --all --all-targets -- --deny=warningsis clean, and
cargo fmt --all -- --checkis clean.Note on #194
#194 replaces
postcard-dyn'sde.rswholesale with areserialize/module anddrops the
ShouldSupportButDontpath along the way. It has been idle since2025-06, and this is a scoped fix for the specific bug in #275 rather than a
claim on that work — but if #194 lands first, this becomes redundant and should
just be dropped.
Closes #275.