feat: per-sandbox cgroup v2 CPU/memory/pids limits with level-tied rlimit fallback

.github/workflows/ci.yml

@@ -57,7 +57,8 @@ jobs:
       - name: Test adapters against real servers
         run: |
           uv run pytest tests/integration \
-            --ignore=tests/integration/test_bubblewrap_isolation.py -q
+            --ignore=tests/integration/test_bubblewrap_isolation.py \
+            --ignore=tests/integration/test_resource_limits.py -q
       - name: Dump middleware logs on failure
         if: failure()
         run: docker compose -f compose.integration.yaml logs --tail 100
@@ -99,7 +100,8 @@ jobs:
             agent-sandbox-test:latest \
             uv run --no-sync pytest \
               tests/integration/test_bubblewrap_isolation.py \
-              tests/integration/test_template_isolation.py -v
+              tests/integration/test_template_isolation.py \
+              tests/integration/test_resource_limits.py -v -rs
 
   fleet:
     name: Fleet (two replicas)

docs/CONFIGURATION.md

@@ -69,6 +69,21 @@ profile identity.
 | `SANDBOX_TMPFS_BYTES` | `536870912` (512 MiB) | Size of the sandbox's private tmpfs. |
 | `SANDBOX_CPU_SECONDS` | `300` | CPU seconds per process (`RLIMIT_CPU`), not a shared sandbox CPU quota or wall-clock deadline. |
 | `SANDBOX_MAX_PROCESSES` | `64` | `RLIMIT_NPROC` counts processes/threads for the real UID, subject to OS privilege semantics; not an execution-scoped cgroup PID budget. See `SANDBOX_TOOLCHAINS` for Go's compiler limits. |
+| `SANDBOX_CGROUP_MODE` | `auto` | Per-sandbox cgroup v2 limits: `auto` uses them when the worker can manage a delegated cgroup and otherwise falls back to rlimits, `required` refuses to start without them, `off` never uses them. The `strict` level is only selectable with cgroups active. See [ISOLATION.md](ISOLATION.md#resource-limits-by-level). |
+| `SANDBOX_CPU_QUOTA` | `2.0` | CPU cores per sandbox (`cpu.max`), shared by every command in it. `0` leaves CPU unlimited. cgroup scheme only. |
+| `SANDBOX_MEMORY_LIMIT_BYTES` | `2147483648` | Memory per sandbox (`memory.max`, with `memory.swap.max=0`); a command over it is OOM-killed as a group and returns `OOM_KILLED` / `SANDBOX_MEMORY_LIMIT_EXCEEDED`. `0` leaves it unlimited. cgroup scheme only. |
+| `SANDBOX_PIDS_LIMIT` | `512` | Processes and threads per sandbox (`pids.max`). `0` leaves it unlimited. cgroup scheme only. |
+| `SANDBOX_WORKER_MEMORY_RESERVE_BYTES` | `268435456` | Memory kept for the worker's control process: set as `memory.min` on its leaf and subtracted from the sandboxes' aggregate `memory.max`. cgroup scheme only. |
+| `SANDBOX_ADDRESS_SPACE_LIMIT_BYTES` | `8589934592` | Fallback per-process `RLIMIT_AS`, applied only when cgroups are not active. Larger than the memory limit because Node and the JVM reserve address space they never use. `0` disables it. |
+| `SANDBOX_JVM_MAX_RAM_FROM_LIMIT` | `true` | Size JVMs from `SANDBOX_MEMORY_LIMIT_BYTES`: the polyglot image's Java launchers prepend `-XX:MaxRAM=<limit>` (`-J-XX:MaxRAM` for javac, jar, jshell, …), so the default heap is a quarter of the sandbox limit instead of the host's RAM. User flags (`-Xmx`, `-XX:MaxRAM`, `-XX:MaxRAMPercentage`) come later and win. No stderr note, unlike `JAVA_TOOL_OPTIONS`. Applies to every level and scheme; JDKs without these launchers are not affected. |
+| `SANDBOX_SUPERVISOR_INTERVAL_SECONDS` | `0.1` | Without cgroups, how often the supervisor samples `/proc` to enforce `SANDBOX_MEMORY_LIMIT_BYTES` (sum of PSS) and `SANDBOX_CPU_QUOTA` (SIGSTOP/SIGCONT duty cycle) over all processes of the sandbox UID. Shorter is tighter and costs more worker CPU. |
+| `SANDBOX_DISK_LIMIT_BYTES` | `10737418240` | Allocated bytes per sandbox (workspace, home, cache, envs), every scheme. Growth past it kills running executions with `SANDBOX_DISK_QUOTA_EXCEEDED`; File API writes over it are refused (HTTP 507). Deleting is always allowed. `0` disables it. |
+| `SANDBOX_DISK_CHECK_INTERVAL_SECONDS` | `2.0` | How often the disk usage of a sandbox with a running execution is measured. |
+| `SANDBOX_IO_READ_BPS` | `268435456` | `io.max rbps` per sandbox on the device holding `SANDBOX_LOCAL_ROOT` (cgroup scheme). `0` is unlimited. |
+| `SANDBOX_IO_WRITE_BPS` | `268435456` | `io.max wbps`, as above. |
+| `SANDBOX_IO_READ_IOPS` | `0` | `io.max riops`, as above. |
+| `SANDBOX_IO_WRITE_IOPS` | `0` | `io.max wiops`, as above. |
+| `SANDBOX_IO_DEVICE` | `""` | `major:minor` for `io.max` when it cannot be derived from `SANDBOX_LOCAL_ROOT` (overlay, network filesystems). |
 | `SANDBOX_MAX_OPEN_FILES` | `512` | Open descriptors per process (`RLIMIT_NOFILE`), not per sandbox. |
 | `SANDBOX_MAX_FILE_SIZE_BYTES` | `1073741824` (1 GiB) | Largest individual file (`RLIMIT_FSIZE`), not an aggregate disk quota. |
 | `SANDBOX_READONLY_MOUNTS` | `/usr,/bin,/sbin,/lib,/lib64,/etc` | Host trees mounted read-only into every sandbox. |

docs/ISOLATION.md

@@ -151,12 +151,113 @@ Choose the Level and other policies for the workload:
 | Agents needing process privacy or reliable PID-namespace teardown | Require `standard` or above; keep network and resource policy choices explicit. |
 | Hostile public tenants | A namespace Level alone, including `strict`, is insufficient. Require suitable enforced aggregate budgets/process supervision and a network boundary (no network or mandatory selective enforcement), or a stronger execution backend such as a VM. |
 
-`/healthz` reports `resource_control` separately from `cgroup_namespace`:
-the built-in backend advertises configured per-process/UID/file limits, but
-`sandbox_cpu_quota`, `sandbox_memory_quota`, `sandbox_pid_quota`,
-`sandbox_disk_quota`, and `execution_cgroup_supervision` are all `false`.
-An enforcing plugin must report its actual controls, not infer them from
-the Level's name. See [Hardening](HARDENING.md) for conditional enhancements.
+`/healthz` reports `resource_control` separately from `cgroup_namespace`.
+It states the scheme actually enforced (`cgroup_v2` or `rlimit`), and
+`sandbox_cpu_quota`, `sandbox_memory_quota`, `sandbox_pid_quota` and
+`execution_cgroup_supervision` are `true` only when the worker created and
+manages the cgroups. `sandbox_disk_quota` is always `false`. An enforcing
+plugin must report its actual controls, not infer them from the Level's name.
+See [Hardening](HARDENING.md) for conditional enhancements.
+
+## Resource limits by level
+
+Every sandbox is limited in aggregate, over all of its processes. How a limit
+is enforced depends on the isolation level and on what the outer runtime
+permits. The worker negotiates this at startup, the same way it negotiates the
+level, and `/healthz` reports the result under `resource_control`. `scheme`
+is either `cgroup_v2` or `rlimit+supervisor`. `enforcement` says how each
+dimension is enforced: by the kernel, by the supervisor (with its precision),
+or not at all.
+
+| Level | cgroup v2 manageable (default `SANDBOX_CGROUP_MODE=auto`) | No cgroup v2: cgroup v1 host, no user namespaces, or `SANDBOX_CGROUP_MODE=off` |
+| --- | --- | --- |
+| `basic` | **cgroup scheme** | **rlimit+supervisor scheme** |
+| `standard` | cgroup scheme | rlimit+supervisor scheme. The PID namespace also ends every process of an execution when the execution ends. |
+| `strict` | cgroup scheme, plus the cgroup namespace | **Not available.** `strict` promises kernel-enforced limits. `auto` selects `standard` instead; an explicit `SANDBOX_ISOLATION_LEVEL=strict` refuses to start (`resource limits: …` in `probe_failures`). |
+
+Each dimension under each scheme:
+
+| Dimension (setting, default) | cgroup scheme | rlimit+supervisor scheme |
+| --- | --- | --- |
+| Memory (`SANDBOX_MEMORY_LIMIT_BYTES`, 2 GiB) | Kernel: `memory.max` per sandbox, `memory.swap.max=0`, `memory.oom.group=1` per execution. | Supervisor: the sum of PSS over the sandbox UID's processes is sampled every `SANDBOX_SUPERVISOR_INTERVAL_SECONDS` (0.1 s). Above the limit, every process of the UID is SIGKILLed. A fast allocation can overshoot for up to one interval, so per-process `RLIMIT_AS` (`SANDBOX_ADDRESS_SPACE_LIMIT_BYTES`, 8 GiB) stays as a hard backstop. |
+| CPU (`SANDBOX_CPU_QUOTA`, 2 cores) | Kernel: `cpu.max`. | Supervisor: a token bucket over the UID's CPU time (including reaped children), with up to one CPU-second of burst. When the bucket is empty the processes are SIGSTOPped, then SIGCONTed once it refills. Measured: about 1.2 cores over a 4 s spin at a 1-core quota, because of the burst. |
+| Processes (`SANDBOX_PIDS_LIMIT` 512 / `SANDBOX_MAX_PROCESSES` 64) | Kernel: `pids.max`. | Kernel: `RLIMIT_NPROC`. Every sandbox has its own UID, so this already counts the whole sandbox. |
+| Disk (`SANDBOX_DISK_LIMIT_BYTES`, 10 GiB) | Supervisor, every `SANDBOX_DISK_CHECK_INTERVAL_SECONDS` (2 s): when allocated blocks under the sandbox root are over the quota and still growing, the running executions are killed. File API writes are refused with HTTP 507. Deleting files is always allowed. `RLIMIT_FSIZE` caps any single file and `/tmp` is a sized tmpfs. | Same. |
+| I/O (`SANDBOX_IO_READ_BPS`/`WRITE_BPS`, 256 MiB/s; `…_IOPS`, off) | Kernel: `io.max` on the device holding `SANDBOX_LOCAL_ROOT`, or on `SANDBOX_IO_DEVICE`. Needs a real block device; overlay and tmpfs have none, and `enforcement.io` says so. | Supervisor: storage bytes from `/proc/<pid>/io`, duty-cycled like CPU. Needs `CAP_SYS_PTRACE` on the worker. Without it, only the lowest I/O priority (`ionice`) is applied, and `enforcement.io` says so. |
+| Wall clock (`SANDBOX_DEFAULT_TIMEOUT_SECONDS`, 300) | Per execution. | Per execution. |
+
+What a runaway command gets back:
+
+| Event | Response |
+| --- | --- |
+| Memory over the limit | `status=OOM_KILLED`, `error_code=SANDBOX_MEMORY_LIMIT_EXCEEDED`. Under the rlimit+supervisor scheme, a single process can instead hit `RLIMIT_AS` first and get `ENOMEM`. |
+| Process limit reached | Forks fail with `EAGAIN`. Under cgroups the response carries `error_code=SANDBOX_PIDS_LIMIT_REACHED`. |
+| Disk over the quota | `error_code=SANDBOX_DISK_QUOTA_EXCEEDED`. |
+| CPU or I/O over the quota | Throttled, not killed. |
+| Leftover background process | Killed when the execution ends (cgroup) or when the sandbox's last execution ends (UID kill). |
+
+The worker keeps memory for itself: its own leaf has `memory.min` set to
+`SANDBOX_WORKER_MEMORY_RESERVE_BYTES`, and the sandboxes' aggregate
+`memory.max` is the container limit minus that reserve. Release and suspend
+remove the sandbox's cgroup and stop supervising it.
+
+Some limits cannot be made exact without the kernel:
+
+- A supervisor samples; it does not intercept. Overshoot is bounded by
+  allocation rate × interval. Use `SANDBOX_CGROUP_MODE=required` or `strict`
+  where that bound is not acceptable.
+- Disk usage is measured, not reserved. A burst can pass the quota by
+  whatever it writes within one check interval. True `ENOSPC` per sandbox
+  needs project quotas (XFS `prjquota`, ext4 `project`) or a per-sandbox
+  loop filesystem, and both need `CAP_SYS_ADMIN`, which the worker
+  deliberately does not have.
+- Supervised I/O cannot charge a process that starts and exits between two
+  samples, because Linux keeps no cumulative I/O counters for reaped
+  children.
+
+### Outer runtime requirements for the cgroup scheme
+
+The worker needs a cgroup v2 hierarchy, the `cpu`, `memory` and `pids`
+controllers delegated to its own cgroup, and a private cgroup namespace:
+
+- **Docker / Compose**: cgroup v2 hosts give containers a private cgroup
+  namespace by default (`cgroupns: private`). Docker mounts it read-only; the
+  worker then mounts a fresh cgroup2 from a user namespace it creates
+  (`unshare --user --map-root-user --mount --cgroup`), which the kernel makes
+  read-write and rooted at the container's own cgroup. This needs nothing
+  beyond the `seccomp=unconfined` the Compose files already set: no
+  capabilities, no privileged mode, no host cgroup mount. Set a container
+  memory limit (`mem_limit` / `--memory`) so the reserve has a ceiling to
+  subtract from.
+- **Hosts that restrict unprivileged user namespaces** (Ubuntu 23.10+ AppArmor,
+  `kernel.apparmor_restrict_unprivileged_userns=1`) block that mount as they
+  block Bubblewrap; the same override from the AppArmor section above
+  applies. Otherwise `auto` falls back to the rlimit scheme and says why in
+  `resource_control.cgroup.detail`.
+- **cgroup v1 hosts, `cgroupns: host`** without a writable delegated subtree,
+  and runtimes that forbid user namespaces: rlimit scheme only.
+- **Kubernetes**: the same mechanism works where the pod may create user
+  namespaces (the seccomp profile allows `unshare`/`mount`); give the pod a
+  memory limit. Verify with `/healthz` on the target cluster.
+  Verified on kind (Kubernetes 1.31) with this container spec; `emptyDir` sits
+  on the node's overlay, so `io.max` needs a block-backed volume or
+  `SANDBOX_IO_DEVICE`:
+
+  ```yaml
+  securityContext:
+    runAsUser: 0
+    seccompProfile: {type: Unconfined}
+    appArmorProfile: {type: Unconfined}
+  resources:
+    limits: {memory: 4Gi}
+  ```
+- **Supervised I/O without cgroups** additionally needs `cap_add: [SYS_PTRACE]`
+  on the worker, to read other UIDs' `/proc/<pid>/io`. It also lets the
+  supervisor read PSS instead of RSS.
+
+The sandbox UIDs cannot change their own limits: cgroup files are owned by the
+worker's root, and the helper mount is not visible inside the Bubblewrap mount
+namespace.
 
 ## Selection
 

docs/TOOLCHAINS.md

@@ -190,3 +190,19 @@ notice.
 
 Every enabled language it cannot check is reported as SKIP with the reason, and
 `--strict` turns a SKIP into a failure so a partial run cannot pass quietly.
+
+## JVM heap follows the sandbox memory limit
+
+Below `strict` the JVM cannot see a cgroup of its own, so it sizes its default
+heap from the host's RAM: a quarter of it, which can be far larger than the
+sandbox memory limit. The image's Java launchers therefore start every JVM
+with `-XX:MaxRAM=<SANDBOX_MEMORY_LIMIT_BYTES>`, and every JDK tool with
+`-J-XX:MaxRAM=…`. This applies at every level and under both resource
+schemes. The flag goes first on the command line, so anything the user
+passes (`-Xmx`, `-XX:MaxRAM`, `-XX:MaxRAMPercentage`) still wins. Maven and
+Gradle reach the same launchers through `JAVA_HOME`.
+
+The launchers do not use `JAVA_TOOL_OPTIONS` or `JDK_JAVA_OPTIONS`, because
+both print a "Picked up …" note to stderr and that breaks tools that parse
+stderr. Set `SANDBOX_JVM_MAX_RAM_FROM_LIMIT=false` to turn this off. A
+system JDK without these launchers is not adjusted.

runtime/src/agent_sandbox_runtime/bubblewrap.py

@@ -141,6 +141,9 @@ def build(self, command: SandboxCommand) -> list[str]:
             args.append(f"--cpu={profile.cpu_seconds}:{profile.cpu_seconds}")
         if profile.max_processes is not None:
             args.append(f"--nproc={profile.max_processes}:{profile.max_processes}")
+        if profile.max_address_space_bytes is not None:
+            limit = profile.max_address_space_bytes
+            args.append(f"--as={limit}:{limit}")
         args.extend(
             [
                 f"--nofile={profile.max_open_files}:{profile.max_open_files}",

runtime/src/agent_sandbox_runtime/profile.py

@@ -28,6 +28,7 @@ class SandboxRuntimeProfile:
     max_processes: int | None = 64
     max_open_files: int = 512
     max_file_size_bytes: int = 1024 * 1024 * 1024
+    max_address_space_bytes: int | None = None
     readonly_mounts: tuple[str, ...] = (
         "/usr",
         "/bin",
@@ -56,6 +57,8 @@ def validate(self) -> None:
                 raise ValueError(f"Sandbox {name} must be positive")
         if self.cpu_seconds is not None and self.cpu_seconds < 1:
             raise ValueError("Sandbox cpu_seconds must be positive")
+        if self.max_address_space_bytes is not None and self.max_address_space_bytes < 1:
+            raise ValueError("Sandbox max_address_space_bytes must be positive")
         if self.max_processes is not None and self.max_processes < 1:
             raise ValueError("Sandbox max_processes must be positive")
 

runtime/src/agent_sandbox_runtime/toolchains.py

@@ -235,7 +235,10 @@ def rust_toolchain(
 
 
 def java_toolchain(
-    *, maven_repository_url: str | None = None, java_home: str | None = None
+    *,
+    maven_repository_url: str | None = None,
+    java_home: str | None = None,
+    jvm_max_ram_bytes: int | None = None,
 ) -> Toolchain:
     """JVM languages with Maven and Gradle.
 
@@ -252,14 +255,23 @@ def java_toolchain(
         environment["MAVEN_MIRROR_URL"] = maven_repository_url
     if java_home:
         environment["JAVA_HOME"] = java_home
+    if jvm_max_ram_bytes:
+        # Read by the image-owned launchers, which pass -XX:MaxRAM.
+        environment["SANDBOX_JVM_MAX_RAM"] = str(jvm_max_ram_bytes)
     return Toolchain(
         name="java",
         environment=environment,
         path_entries=(
             f"{ENVS_ROOT}/java/bin",
             *((f"{java_home}/bin",) if java_home else ()),
         ),
-        reserved_names=("MAVEN_OPTS", "GRADLE_USER_HOME", "SBT_OPTS", "MAVEN_MIRROR_URL"),
+        reserved_names=(
+            "MAVEN_OPTS",
+            "GRADLE_USER_HOME",
+            "SBT_OPTS",
+            "MAVEN_MIRROR_URL",
+            "SANDBOX_JVM_MAX_RAM",
+        ),
         needs_procfs=java_home is None,
         reserved_prefixes=("JAVA_TOOL_OPTIONS", "_JAVA_OPTIONS", "CLASSPATH"),
     )

scripts/install-polyglot-launchers.py

@@ -23,6 +23,42 @@ def _library_path(directory: Path) -> str:
     return f"export LD_LIBRARY_PATH={shlex.quote(str(directory))}${{LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}}\n"
 
 
+# JDK tools that take JVM options through -J. `java` takes them directly.
+_J_OPTION_TOOLS = frozenset(
+    {
+        "javac",
+        "jar",
+        "jarsigner",
+        "javadoc",
+        "jdeps",
+        "jlink",
+        "jmod",
+        "jpackage",
+        "jshell",
+        "keytool",
+        "jdb",
+    }
+)
+
+
+def _jvm_memory(name: str) -> str:
+    """Size the JVM from the sandbox's memory limit, not the host's RAM.
+
+    Without a cgroup namespace the JVM sees the whole host and picks a default
+    heap of a quarter of it, far above the sandbox limit. The runtime exports
+    SANDBOX_JVM_MAX_RAM; it goes first so any flag the user passes later
+    (-Xmx, -XX:MaxRAM, -XX:MaxRAMPercentage) still wins. This is quiet, unlike
+    JAVA_TOOL_OPTIONS and JDK_JAVA_OPTIONS, which print a note to stderr.
+    """
+    if name == "java":
+        flag = '-XX:MaxRAM="$SANDBOX_JVM_MAX_RAM"'
+    elif name in _J_OPTION_TOOLS:
+        flag = '-J-XX:MaxRAM="$SANDBOX_JVM_MAX_RAM"'
+    else:
+        return ""
+    return f'if [ -n "${{SANDBOX_JVM_MAX_RAM:-}}" ]; then set -- {flag} "$@"; fi\n'
+
+
 def install_java(java_home: Path, destination: Path) -> None:
     """Expose a JDK-shaped JAVA_HOME, including wrappers for every JDK tool."""
     java_home = java_home.resolve(strict=True)
@@ -36,7 +72,9 @@ def install_java(java_home: Path, destination: Path) -> None:
         if binary.is_file() and os.access(binary, os.X_OK):
             _launcher(
                 destination / "bin" / binary.name,
-                _library_path(java_home / "lib") + f'exec {shlex.quote(str(binary))} "$@"\n',
+                _library_path(java_home / "lib")
+                + _jvm_memory(binary.name)
+                + f'exec {shlex.quote(str(binary))} "$@"\n',
             )
 
 

scripts/integration-test.sh

@@ -73,7 +73,8 @@ run_sandbox_tests() {
     "${TEST_IMAGE}" \
     uv run --no-sync pytest \
       tests/integration/test_bubblewrap_isolation.py \
-      tests/integration/test_template_isolation.py -v "$@"
+      tests/integration/test_template_isolation.py \
+      tests/integration/test_resource_limits.py -v "$@"
 }
 
 run_polyglot_tests() {

src/agent_sandbox/app.py

@@ -166,6 +166,7 @@ def relay_worker_response(response: httpx.Response) -> Response:
     "SANDBOX_EXEC_SCOPE_BUSY": 409,
     "SANDBOX_EXEC_SCOPE_LOCKED": 409,
     "SANDBOX_FILE_PATH_LOCKED": 409,
+    "SANDBOX_DISK_QUOTA_EXCEEDED": 507,
     # Suspend and resume. A suspended sandbox refuses work until it is resumed;
     # a suspend refuses while anything runs in it. Both are the caller's move.
     "SANDBOX_NOT_FOUND": 404,