Replace AssayState with durable PipelineRun

[parashardhapola]

The central change is not merely replacing AssayState. It separates three concepts that were previously mixed together: immutable results, workflow execution history, and live metadata.

Summary

This branch removes the assumption that a datastore has one mutable “current analysis.”

Previously, analysis artifacts existed, but AssayState, encoded paths, live I columns, and published result columns determined which graph, embedding, or clustering was current. Running another analysis could replace that state. Later operations could silently resolve a different upstream result, or interpret an existing result against changed cell or feature selections.

The new model makes analysis identity explicit:

• ArtifactRef identifies one exact persisted result.
• PipelineRun records one complete workflow invocation.
• Live metadata remains an annotation and input surface, not a registry for analytical outputs.

run = ds.pipeline.run(assay="RNA", label="baseline")

pca = run["pca"]
clusters = run["clusters"]

ds.plots.embedding(run=run, layout="umap", color_by="clusters")
markers = ds.get_markers(marker=run["markers"])
adata = ds.to_anndata(run=run)

Reopening baseline reconstructs the same analysis even if live I, metadata, or later analysis branches have changed.

Explicit artifact contracts

Granular APIs across feature selection, graph construction, embeddings, clustering, quality control, mapping, trajectory, metrics, statistical testing, and marker search now require exact upstream ArtifactRef values.

This removes:

• implicit current-result lookup
• AssayState and get_assay_state
• encoded graph paths and latest_* pointers
• feature-selection aliases such as "hvgs"
• update_state behavior
• result-producing label= arguments
• automatic publication of UMAP, clustering, scores, markers, or selections into live metadata

Producers now return artifacts. Separate loaders reconstruct richer results where needed, including mapping, trajectory, PARIS, enrichment, LISI, and marker outputs.

This allows several parameter branches to coexist without one becoming globally current, and prevents downstream operations from accidentally combining incompatible cells, features, coordinates, or graphs.

Durable pipeline runs

DataStore.pipeline.run() now returns a durable PipelineRun instead of a dictionary of results accompanied by metadata mutations.

Each run persists:

• the validated recipe configuration
• an ordered stage ledger
• exact artifact outputs
• whether each artifact was created or reused
• stage timing and process-tree memory observations
• completed, skipped, failed, or interrupted stage status
• frozen cell and feature fields
• failure and interruption details

Configuration is validated before a run record is created. A completed run exposes its artifact mapping and frozen views. Failed or interrupted runs remain inspectable through report(), but do not expose partial outputs as if they were a successful analysis.

Artifact reuse remains provenance-based. Artifact IDs are not content hashes. Scarf searches complete artifacts with matching operation, parameters, and inputs, validates their payload, and records whether each stage created or reused its output.

Frozen cells and features

A run captures its input selection, analysis selection, feature universe, identities, and requested metadata at the start of the workflow.

run.cells and run.features therefore represent the axes used by that invocation:

• later changes to live I do not alter the run
• compact result arrays remain aligned to the stored selection
• full-axis reads fill rows that were not selected
• ordered row identities are checked again when fields are read
• replacing or reordering cells or features fails closed
• highly variable features remain a distinct selection rather than redefining feature I

Plotting and AnnData/H5AD export can consume these frozen views directly, so they do not reconstruct an analysis from whichever columns happen to be live at export time.

Pipeline recipe and clustering decision

The default RNA recipe now records filtering, cell-cycle scoring, feature selection, normalization, PCA, optional Harmony, graph construction, UMAP, Leiden candidates, PARIS, doublet scoring, and marker search as explicit sequential stages.

Automatic clustering selection is itself persisted:

• Leiden resolutions are compared using one deterministic shared sample
• pairwise work is bounded to at most 10,000 cells
• rare clusters receive reserved sample capacity
• invalid candidates and tie order are recorded
• run["clusters"] points to the exact winning Leiden artifact
• the selected labels are not copied into another result
• PARIS remains available as a diagnostic and is never treated as the automatic winner

When Harmony is enabled, clustering is evaluated in the corrected coordinates. Doublet scoring deliberately retains its separate uncorrected PCA graph branch.

Failure, interruption, and labels

Run and stage documents use strict persisted shapes and set their completion marker last. Unknown, incomplete, or inconsistent records fail closed.

Cooperative interruption records terminal state before propagating the signal. A hard process termination can leave an incomplete record, which is retained rather than guessed at or repaired.

There is intentionally no on-disk resume or same-run retry protocol. A new invocation receives a new run ID and may reuse any complete artifacts produced before the interruption.

Optional run labels are immutable names for successful runs:

• labels are claimed atomically during finalization
• failed and interrupted runs do not own the label
• backends without atomic conditional creation reject labeled runs before computation
• a torn finalization blocks label reuse until the operator explicitly abandons the exact claim
• Scarf does not infer abandonment from elapsed time

Downstream integration

The artifact contract is applied consistently across the repository:

• filtering and HVG selection return immutable selections instead of rewriting I
• mapping projections and references validate exact query and reference lineage
• trajectory and fate-mapping results freeze their source selections and feature identities
• statistical results remain retrievable by their original ref after matrix or grouping changes
• agent parameter tuning can evaluate multiple branches without publishing transient state
• plotting rejects result artifacts associated with incompatible selections
• H5AD and Seurat imports expose imported embeddings and clusterings as artifacts
• run-based export uses the frozen analysis rather than live metadata

The merged agent and statistical-testing work is not a separate feature introduced here. Those workflows were ported onto the same explicit artifact and durable-run contracts.

Compatibility policy

This is an intentional breaking change with no silent compatibility layer.

Stores containing {assay}/state are rejected on open. Scarf does not inspect or migrate that document. Repacking can preserve counts and the current physical layout, but it does not translate legacy encoded analysis results into artifacts.

Repacking a current store preserves pipeline records because its axes are unchanged. Subsetting and merging do not copy source runs, and an overwriting merge clears destination run records whose row identity is no longer valid.

[codecov]

Codecov Report

❌ Patch coverage is 85.10638% with 28 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
scarf/agent/characterize_covariates.py	85.18%	16 Missing ⚠️
scarf/agent/biological_interpretation.py	85.00%	12 Missing ⚠️

📢 Thoughts on this report? Let us know!