fix: honor cuda_devices in Singularity backend and support multiple GPUs in Docker

[MarcelRosier]

Summary

Fixes #164.

• Singularity: cuda_devices was silently ignored — the --nv flag exposes all host GPUs. The Singularity backend now sets SINGULARITYENV_CUDA_VISIBLE_DEVICES around the container run (Singularity passes it into the container as CUDA_VISIBLE_DEVICES), so only the requested host GPUs are visible. Any previously set value is restored after the run, and a warning is logged if it was already set.
• Docker: cuda_devices now accepts a comma-separated list of device IDs (e.g. "0,1"); entries are split on commas and whitespace is stripped.
• Documented that cuda_devices refers to host GPU IDs and takes precedence over a host-level CUDA_VISIBLE_DEVICES (e.g. set by SLURM) inside the container; updated AGENTS.md and docs/guides/backends.md accordingly.

Testing

• uv run pytest — 93 passed (tests mock container execution; no Docker daemon/GPU required)
• uv run ruff check . and uv run ruff format --check . — clean

[Copilot]

Pull request overview

This PR fixes GPU device selection inconsistencies across container backends in the BraTS orchestrator, ensuring cuda_devices is actually enforced in Singularity and correctly supports multi-GPU selection in Docker.

Changes:

• Singularity now restricts visible GPUs by setting SINGULARITYENV_CUDA_VISIBLE_DEVICES during the container run (and restores any prior value).
• Docker now supports comma-separated multi-GPU selection by splitting cuda_devices into multiple device IDs.
• Documentation and architecture notes were updated to clarify that cuda_devices refers to host GPU IDs and how it interacts with host-level CUDA environment variables.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
brats/core/singularity.py	Adds a context-managed environment override so Singularity honors cuda_devices while running with --nv.
brats/core/docker.py	Updates Docker device request construction to support multiple GPU IDs via comma-splitting.
tests/core/test_singularity.py	Adds tests validating Singularity CUDA env override/restore behavior and CPU-path behavior.
tests/core/test_docker.py	Adds a test ensuring Docker splits multi-GPU cuda_devices correctly (including whitespace tolerance).
docs/guides/backends.md	Documents multi-GPU selection and clarifies Singularity GPU restriction behavior.
AGENTS.md	Updates backend dispatch documentation to reflect cuda_devices being honored by both backends.

Suppressed comments (1)

tests/core/test_singularity.py:417

• This test asserts the CUDA env var is absent after the run, but it may already be set in the caller environment. To verify the CPU path truly doesn't touch the env, explicitly set a sentinel value and assert it remains unchanged (with cleanup).

        self.assertIsNone(captured_env["value"])
        options = mock_client.run.call_args.kwargs["options"]
        self.assertNotIn("--nv", options)
        self.assertNotIn("SINGULARITYENV_CUDA_VISIBLE_DEVICES", os.environ)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Process-global environment mutation can select incorrect GPUs during concurrent Singularity runs.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 8/8 changed files
• Comments generated: 2
• Review effort level: Balanced

[neuronflow]

lgtm
(but no time for an in-depth review :( )

[neuronflow]

in general we should find an agreement with singularity vs. apptainer #153