Repository navigation
[doc] Document the contract of storage-side callables (data_parser, merge_fn) with a tutorial #195
Copy link
Copy link
Open
Description
Background
Three KV APIs run a user callable on the SimpleStorage units:
| API | Callable | What it receives |
|---|---|---|
kv_put / kv_batch_put |
data_parser(data) -> data |
The unit's share of the new batch, before it is stored. A regular tensor column arrives as one batched tensor; nested-tensor and NonTensorStack columns arrive as lists. |
kv_batch_get (#191) |
data_parser(data, args) -> data |
The stored values, as one list per column with one value per sample (tensor columns included), plus the matching parser_args entries. |
kv_update / kv_batch_update (#182) |
merge_fn(old, new) -> stored |
One sample's stored value and its new value, per field. |
They are powerful but easy to misuse, and their contracts are documented unevenly, in places inaccurately.
Gaps in the current docs
- put
data_parser: "must modify in-place" is inaccurate. The docstrings ofkv_putandkv_batch_putsay the parser must modify values in place, but the unit requires it to return a dict and raisesTypeErrorotherwise (_apply_data_parser). Modifying in place is allowed, not required. - The get and put
data_parserlook interchangeable but are not. The get docstring calls itself the get-side counterpart of put's parser, but the two differ in arity ((data, args)vs(data)) and in input structure (per-sample lists vs a batched tensor for regular tensor columns). Passing a put parser tokv_batch_getfails on the unit withtakes 1 positional argument but 2 were given. merge_fndoes not say thatoldis the stored object.apply_updatepasses the stored value itself (L255). If amerge_fnmodifiesoldin place, the guarantee that a failed merge leaves storage unchanged no longer holds: samples merged before the failure have already changed.- Runtime constraints are not stated in one place. All three run synchronously on the unit's single worker thread, so a slow one delays every other request to that unit. A call that uses one of them is not retried on a storage-unit timeout, and a timeout has an unknown outcome. Callables travel through cloudpickle, so module-level functions must be importable on the units, and tuples inside
parser_argsarrive as lists.
Expected work
- Docs. One shared contract for storage-side callables in the README, referenced from the docstrings of
kv_put,kv_batch_put,kv_batch_get,kv_update,kv_batch_updateand their async variants: where it runs, the input structure per API, no in-place modification of stored values, no side effects, keep it light, no retry, serialization, and how errors reach the caller. Fix gaps 1-3 above. - Dedicated tutorial. A new
tutorial/07_*.pythat walks through putdata_parser(resolving references into content), getdata_parserwithparser_args(trimming a tensor field and picking items from a list field in one get), andkv_updatewithmerge_fn(appending to a field). Include one in-place example and show why it corrupts stored data. - Notebook demo. Extend section 11 of
tutorial/basic.ipynb("Lazy Data Parsing withdata_parser") with a get-sidedata_parserdemo, and add akv_update/merge_fndemo.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity