[Python][Hosted ML] Import standard datasets into immutable TinyChain dataset services

[code-tc]

Parent: TinyChain-Inc/tcv2#79

Objective

Provide a Python-client utility that imports common in-memory, file, streaming, and hosted dataset representations into immutable, versioned TinyChain dataset services suitable for hosted ML training.

Initial sources

• pandas DataFrame and Series
• NumPy arrays and structured arrays
• CSV and compressed CSV
• PyArrow Table, record batches, and Parquet
• Hugging Face Dataset and IterableDataset
• Python iterators/generators yielding bounded batches

Server-side acquisition from remote URLs, Hugging Face Hub, or object storage is a later connector path. The MVP may import data already accessible to the authorized Python process.

Proposed contract

Provide a documented API equivalent to:

dataset = tc.dataset.import_data(
    source,
    name="training-data",
    target="label",
    schema=schema,
    batch_size=batch_size,
    metadata=metadata,
)

Return an immutable dataset revision/handle usable by the training service, not a fully materialized client copy.

Procedure

1. Define canonical mappings from supported Python/Arrow types to TinyChain table/tensor/stream types.
2. Infer schema where unambiguous and require an explicit override/confirmation for ambiguous or lossy cases.
3. Preserve nullability, dimensions, precision, categorical encoding, timestamps/time zones, feature/target roles, and supported metadata.
4. Stream bounded chunks with backpressure; never require whole-source materialization.
5. Support resumable upload with stable source/import identity and idempotent chunk handling.
6. Compute integrity digests and publish an immutable dataset manifest/revision.
7. Record source kind, import time, schema, transformations, fidelity warnings, license/use metadata supplied by the user, and client/library versions.
8. Represent train/validation/test splits and derived views as manifests/references where possible rather than physical copies.
9. Reject corrupt, partial, unauthorized, incompatible, or unsupported data explicitly.
10. Keep credentials and unrestricted local paths out of manifests, logs, and error payloads.
11. Add local fixtures plus HTTP and PyO3 integration coverage where supported.

Acceptance criteria

• Every initial source type imports through one documented API.
• Large and iterable sources are uploaded in bounded chunks with backpressure.
• Interrupted upload resumes without duplicating an accepted dataset revision or chunk.
• Round-trip fixtures report and satisfy documented schema/type/precision fidelity.
• Ambiguous or lossy conversions require explicit user direction.
• The result is immutable, integrity-verifiable, independently addressable, and authorized.
• Derived splits preserve lineage to the source revision.
• No source credential, unrestricted filesystem path, or unauthorized sample leaks into metadata or diagnostics.
• The resulting handle is consumed by the hosted-training fixture without backend topology configuration.
• Unit, optional-dependency, HTTP, PyO3, interruption, and hostile-input tests pass.

Dependencies and concurrency

• Consumes the dataset/artifact manifest and resumable-transfer contracts from TinyChain-Inc/tcv2#67.
• Uses canonical Python Table/Tensor/service behavior; it must not introduce a second serialization path.
• Can be implemented concurrently with the training-service contract.
• Test fixtures may begin before the final server ingestion route; live HTTP/PyO3 acceptance waits for it.

Non-goals

• Ambient server access to arbitrary local files or remote URLs
• A credentialed Hugging Face Hub backend connector
• Silent lossy coercion
• Loading an entire streaming dataset into memory
• Prescribing a tenant-specific feature schema