fix: keep None in an array wherever it sits

[shcheklein]

list[int | None] fails to save when the first element is None:

class Scores(dc.DataModel):
    vals: list[int | None]

chain.map(s=lambda: Scores(vals=[1, None]), output=Scores)   # ok
chain.map(s=lambda: Scores(vals=[None, 2]), output=Scores)   # UdfError

Three places decided how to treat a whole array by looking at one element of it.

convert_type read val[0]: a leading 1 matched the item type, so the array
was stored untouched and the None rode along; a leading None did not match,
so every element went to a converter — and None has nothing to convert.

flatten read value[0] the same way: a leading model meant model_dump() on
every element including the Nones, a leading None meant the models were left
raw, and neither could answer for a list of mixed shapes.

TypeReadConverter.float maps None to nan, which is right for a bare float
column and wrong for a nullable element — without that, these arrays would newly
save and then read back [1.0, nan]. Scalar Optional[float] already returns
None; arrays were the outlier.

What that changes

list[int | None]	main	here
[1, None]	[1, None]	[1, None]
[None, 2]	UdfError	[None, 2]
[None, None]	UdfError	[None, None]

list[float | None]	main	here
[1.0, None]	[1.0, nan]	[1.0, None]
[None, 2.0]	UdfError	[None, 2.0]

A model nested inside a dict item was converted in one order and not the other:

list[dict[str, Item] | None]	main	here
[{"a": Item(n=1)}, None]	TypeError: Object of type Item is not JSON serializable	[{'a': Item(n=1)}, None]
[None, {"a": Item(n=1)}]	works	works

And a None reached a non-nullable column unchecked, again depending on
where it sat:

dc.read_values(x=[[1, None]], output={"x": list[int]}).to_values("x")
# main: [[1, None]]   -- stored into Array(Int64)
# here: UdfError

On disk

Only arrays that contain a None change, and of those only the order that was
already inconsistent with itself:

list[dict | None]	before	after
[{"k": 1}, None]	[{"k":1},null]	unchanged
[None, {"k": 1}]	["null","{\"k\":1}"]	[null,{"k":1}]

Arrays without a None take exactly the path they took before — list[int],
list[str], list[dict], list[Model], list[list[int]], tuple[int, ...],
tuple[str, ...], tuple[Model, Model], tuple[int, int], and mixed arrays
like ({"k": 1}, 2) are byte-identical, checked by writing each with both
versions and diffing the stored column.

Left for #1968

How main behaves today, untouched here, because fixing them changes the stored
shape of arrays that already round-trip:

• A model beside a plain value is left raw. ({"a": Item(n=2)}, 1) raises;
  reverse the order and it writes.
• Element order picks a mixed array's layout. ({"k": 1}, 2) stores
  [{'k': 1}, 2], (2, {"k": 1}) stores ['2', '{"k":1}'].
• A tuple stores unlike the list of the same values. list[int] stores
  [1,2]; tuple[int, ...] stores ["1","2"] — what
  #1963 runs into.
• Non-finite floats become null in dict or Any fields.
• A nested list cannot hold a None at the outer level.
  list[list[int] | None] writes [[1, 2], [3]] but not [None, [1, 2]].

Stack

1. this PR — None anywhere in an array
2. fix: recognize every spelling of a nullable array element #1976 — the spellings of a nullable element type
3. fix: store a list of models that admits a None #1977 — a list of models that admits a None

Replaces #1970, which carried all three at once.

[codecov]

Codecov Report

❌ Patch coverage is 92.00000% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/datachain/lib/convert/flatten.py	71.42%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!