Use managed Compute clusters for lc materialize

[EiffL]

lc materialize now uses the managed Compute clusters introduced in jupyterlab-lightcone#42, so a cluster can serve successive runs and multiple projects. An existing SLURM allocation retains priority; otherwise the engine selects exactly one compatible live cluster, or uses the host with its existing login-node guard.

The engine announces lightcone.cluster/1 support and exposes venue.attached_cluster(root) for the sidebar to reuse. Registry discovery and attachment live in one module; execution continues through the existing scheduler and worker interfaces.

• Read backend state for local, SLURM and Gateway records, with bounded connection/startup waits and actionable refusals. Selected clusters never silently fall back to local execution.
• Use mutual TLS or Gateway's native report credentials; check every worker's Lightcone, Distributed and Python versions and project access before preparing recipes. Gateway is an optional install extra and uses its public read-only API without creating cluster lifecycle handles.
• Give each run unique scheduler keys and its own recipe environment, reusing the existing environment scrub. Worker host/job settings remain local, and container environment policy is preserved.
• Reuse one shared-image-store guard for allocations and attached clusters, including SLURM workers that have not joined yet.
• Announce the selected target before work and include it in the JSON report. Closing a run closes only its client. Interrupted shared runs preserve unfinished outputs and explain recovery, since disconnecting cannot stop an executing recipe safely.

No cluster creation, scaling or shutdown moves into the CLI. Existing allocation execution remains supported; containerized Gateway projects remain outside this contract.

Validation: all 679 tests passed across the full run and a targeted retry of one transient RO-Crate network failure; Ruff, mypy and Python compilation pass. Coverage includes a real external TLS scheduler/worker, concurrent same-name clones sharing a scheduler, surviving unrelated work, native Gateway credentials, backend failures and interruption cleanup. CI installs the Gateway extra to exercise its native credential contract. Live SLURM and Gateway deployment validation still requires those environments.

[github-actions]

❌ Eval

Metric	Value
Outputs check	skipped
Agent run	failure
Turns	1
Tool calls	0
Cost	$0.00
Agent wall time	0m00s
Model	claude-sonnet-5

lc status

unavailable

Confusion & pain points (Claude analysis)

Trace analysis unavailable.

Full trace: agent-trace artifact on this run.