fix: flatten models in tuples and nested mapping values as lists already are

[shcheklein]

Models stored in a tuple are written double-encoded, unlike the same models in a list.

class Box(dc.DataModel):
    label: str
    score: float

class Frame(dc.DataModel):
    boxes: tuple[Box, ...]

dc.read_values(frame=[Frame(boxes=(Box(label="cat", score=0.9),))]).save("frames")

What lands in the column:

before   ["{\"label\":\"cat\",\"score\":0.9}"]     a string holding JSON
after    [{"label":"cat","score":0.9}]             an object, same as list[Box]

Why

_flatten_fields_values decided what to convert from the runtime type of the value and knew only list and dict. A tuple matched neither, so it was passed through with its Box objects intact and the warehouse serialized each one separately. The dict branch had the same gap one level down: it converted a value that was a model, but not one that was a list of them.

It now decides from the declared type instead, and converts elements one at a time.

This changes the stored format

tuple[Model, ...] columns written before this will not compare equal to ones written after — union/distinct, subtract and merge between an old and a new dataset will disagree. Reads are unaffected: both spellings still load as the declared type, verified by writing on main and reading here.

That break is deliberate. The current spelling is a bug, the shape is rare — tuple[Model, ...] as a stored signal appears once in this repository and nowhere in src/ or examples/ — and leaving it costs a permanent gap in #1943 (below).

Deciding from the annotation, not the value

Two things follow from that, both of which an earlier revision got wrong:

• A collection that cannot hold a model is never walked. list[int] and tuple[int, int] pass through untouched, so embedding vectors cost nothing to write and keep their identity for content hashing.
• Elements are converted individually rather than from whatever the first one happens to be. A fixed-length tuple needs this: tuple[Item, int] would otherwise call model_dump() on the int.

Not fixed here

tuple[int, Model] is still written as an array of strings, byte-identical to main. The warehouse chooses the array conversion from the first element, so a leading non-model sends the whole array down the per-element path. Independent of this change.

Side effect

This closes the two shapes #1943 carries a strict xfail for. A dict arriving inside a live model has already been through model_dump, which merges keys that serialize to the same JSON name; the same dict inside a list arrives intact. Flattening them alike lets the existing check see both keys, with no new code there.

[cloudflare-workers-and-pages]

Deploying datachain with    Cloudflare Pages

Latest commit:	eb04914
Status:	✅  Deploy successful!
Preview URL:	https://3e34f83a.datachain-2g6.pages.dev
Branch Preview URL:	https://fix-flatten-tuple-of-models.datachain-2g6.pages.dev

View logs

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Copilot]

Pull request overview

Aligns tuple and nested mapping model serialization with existing list behavior.

Changes:

• Flattens models inside tuples and nested mapping values.
• Adds flattening and persistence round-trip tests.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
src/datachain/lib/convert/flatten.py	Adds tuple and recursive mapping flattening.
tests/unit/lib/test_signal_schema.py	Tests collection flattening shapes.
tests/unit/lib/test_datachain.py	Tests save/read round trips.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[shcheklein]

Closing this. The review found that changing the physical Array(JSON) representation makes existing datasets stop comparing equal to newly written ones — union/distinct, subtract and merge between a legacy and a modern dataset all disagree, with no migration path.

That is the same failure I documented in #1943 as the storage-convergence dead end, and I checked the wrong half of it here: I verified old data still reads, not that old and new rows still compare.

Two further findings, both confirmed:

• Flattening rebuilds every collection, so a list[int] of one million goes from ~0 ms to 231 ms per flatten. That lands on every embedding vector on every write.
• tuple[int, Model] stays double-encoded regardless, because the warehouse's array path dispatches on the first element — so the change does not fully achieve its goal without also touching that fast path.

The one piece worth keeping is annotation-guided traversal: deciding what to descend into from the declared type rather than isinstance of the value. That fixes the performance problem, removes the first-element guessing, and needs no representation change. Worth doing on its own if anyone picks it up.

The xfail in #1943 stays; it is cheaper than either route around it.

[shcheklein]

Superseded. The tuple work is being reconsidered on top of #1943 rather than as a format change in its own right, and the diagnosis moved: the double-encoding of tuple[Model, ...] comes from the column type, not from flattening — see #1963.

Findings from this PR worth keeping for whatever replaces it: unwrap Optional before reading collection arguments, decide what to walk from the declared type rather than the runtime value, and normalize UDF outputs through flatten_value/_flatten_row as well as _flatten_fields_values.