Use managed Compute clusters for lc materialize

.github/workflows/tests.yml

@@ -52,7 +52,7 @@ jobs:
           activate-environment: true
 
       - name: Install dependencies
-        run: uv sync --group dev
+        run: uv sync --group dev --extra gateway
 
       - name: Report the sandbox this runner can build
         # Printed before the suite so a failure downstream can be read

README.md

@@ -44,11 +44,43 @@ drafted with any AI coding assistant.
 - **Multiverse analysis** — declare methodological decisions with multiple defensible options; `lc` materializes your analysis across every universe you define
 - **Provenance by construction** — every output is committed to git together with a content-addressed manifest and a re-runnable run record; git-annex carries the bytes, so results travel with the repository
 - **Locked, isolated execution** — a project's environment is `pyproject.toml` + `uv.lock`; recipes run in it under a sandbox (Landlock on Linux, Seatbelt on macOS) that keeps undeclared files out and stray writes contained
-- **Containers and HPC** — declare `[tool.lightcone.image]` and recipes run in a content-addressed image archived in the repository itself; a SLURM allocation is detected and used automatically, every node included
+- **Containers and HPC** — declare `[tool.lightcone.image]` and recipes run in a content-addressed image archived in the repository itself; use every node of a SLURM allocation or attach to a cluster from JupyterLab's Lightcone sidebar
 - **Publication view** — declare a license and `lc materialize` maintains an [RO-Crate](https://www.researchobject.org/ro-crate/) of the project and its provenance, ready to archive or deposit
 
 → [Full documentation](https://docs.lightconeresearch.org)
 
+## Compute clusters
+
+Start a cluster in JupyterLab's **Lightcone sidebar › Compute**, then run
+`lc materialize` as usual. The CLI chooses an execution target in this order:
+
+1. An existing SLURM allocation, when `SLURM_JOB_ID` is set.
+2. The one compatible live cluster registered by the sidebar.
+3. This machine, subject to the existing HPC login-node guard.
+
+The run names its target, and `--json` includes a `venue` object. If several
+clusters can serve the project, stop all but one. A selected cluster that is
+queued, unreachable, or incompatible with the installed engine produces an
+actionable refusal; the run does not silently switch targets. Workers need
+the same Lightcone, Dask Distributed and Python versions as the CLI, and must
+see the project at the same filesystem path.
+
+Clusters remain available after each run. The CLI closes its connection;
+cluster creation, scaling and shutdown remain with the sidebar and its
+backend. Interrupting an attached run leaves unfinished outputs uncommitted:
+stop the cluster and inspect them before restoring or rerunning. Direct-mode
+recipes receive the invoking shell's environment while retaining the workers'
+own host and job settings; containers retain their existing environment policy.
+
+Local and SLURM clusters also support containerized projects. Using several
+worker hosts requires a shared image store such as `podman-hpc`. Dask Gateway
+supports direct-mode projects in the same JupyterHub image; install its
+optional client and use the hub's Gateway configuration:
+
+```bash
+uv tool install 'lightcone-cli[gateway]'
+```
+
 ## License
 
 BSD 3-Clause — see [LICENSE](LICENSE) for details.

pyproject.toml

@@ -29,9 +29,13 @@ dependencies = [
     "rich>=13.0",
     "git-annex>=10.2026",
     "distributed>=2026.7",
+    "psutil>=5.8",
     "rocrate>=0.15",
 ]
 
+[project.optional-dependencies]
+gateway = ["dask-gateway>=2025.4"]
+
 [dependency-groups]
 dev = [
     "pytest>=8.0",
@@ -92,6 +96,10 @@ namespace_packages = true
 explicit_package_bases = true
 mypy_path = "src"
 
+[[tool.mypy.overrides]]
+module = ["dask_gateway.*"]
+ignore_missing_imports = true
+
 [[tool.mypy.overrides]]
 module = ["astra.*", "rocrate.*"]
 ignore_missing_imports = true

src/lightcone/cli/commands.py

@@ -292,27 +292,37 @@ def materialize(
     records the environment and the commit that produced it. Pass
     --refresh to remake those too.
     """
+    from rich.markup import escape
+
     from lightcone.engine import container as engine_container
     from lightcone.engine import materialize as engine
     from lightcone.engine.project import current_project
 
     root = current_project()
-    if not check_only and not as_json:
-        # The engine never prints, and the build it may be about to run
-        # can take minutes — so the one place that owns the console says
-        # so before handing over. Conditional mood, deliberately: the
-        # engine's own refusals (a dirty tree, an invalid spec) come
-        # first and cost no build, so this must promise nothing.
+
+    def announce(selected: dict[str, str | int]) -> None:
+        if selected["kind"] == "cluster":
+            target = f"{selected['label']} ({selected['backend']}, {selected['id']})"
+        elif selected["kind"] == "allocation":
+            target = f"SLURM allocation ({selected['nodes']} nodes)"
+        else:
+            target = "this host"
+        _console().print(f"Running on {escape(target)}")
+        # Announce the build before it starts; engine refusals still come
+        # first, so this promises nothing until preparation succeeds.
         state, tag, _ = engine_container.image_state(root)
         if state == "absent":
             _console().print(
                 f"image absent — the run rebuilds [bold]{tag}[/bold] first "
                 "(this can take minutes)"
             )
+
     if check_only:
         report = engine.check(root, targets, refresh=refresh)
     else:
-        report = engine.materialize(root, targets, refresh=refresh)
+        report = engine.materialize(
+            root, targets, refresh=refresh, on_venue=None if as_json else announce
+        )
 
     if as_json:
         click.echo(json.dumps(report.as_dict(), indent=2))