[fliaping]

Per-sandbox CPU, memory and process limits, so one runaway or malicious command cannot take down the worker or its neighbours. Before this change only per-process rlimits existed and memory was unbounded.

Design

• cgroup v2 scheme (src/agent_sandbox/cgroups.py): the worker moves itself into a worker leaf, enables cpu memory pids for its subtree, and creates sandboxes/<id> with cpu.max, memory.max, memory.swap.max=0 and pids.max. Each execution runs in its own leaf (memory.oom.group=1) that it joins before prlimit/setpriv/bwrap, so no descendant can escape. When the execution ends, the leaf is killed (cgroup.kill) and removed. Release, suspend and shutdown remove the sandbox cgroup.
• Unprivileged delegation: Docker mounts the container cgroup read-only. The worker spawns unshare --user --map-root-user --mount --cgroup and mounts a fresh read-write cgroup2 rooted at its own cgroup, then uses it via /proc/<helper>/root. This needs only the existing seccomp=unconfined: no capabilities, no privileged mode, no host cgroup mount.
• Worker reserve: the worker leaf gets memory.min=SANDBOX_WORKER_MEMORY_RESERVE_BYTES, and the aggregate sandboxes/memory.max is set to the container limit minus that reserve.
• Tied to isolation level: basic and standard use cgroups when they can be managed, and otherwise fall back to an rlimit scheme: new RLIMIT_AS, existing RLIMIT_CPU/NPROC/NOFILE/FSIZE, the wall-clock timeout, and killing the sandbox UID's leftover processes. strict requires cgroups: auto drops to standard, and an explicit strict refuses to start. SANDBOX_CGROUP_MODE=auto|required|off.
• Results: an OOM returns status=OOM_KILLED and error_code=SANDBOX_MEMORY_LIMIT_EXCEEDED. Hitting the pids limit returns error_code=SANDBOX_PIDS_LIMIT_REACHED. /healthz resource_control reports scheme, the real sandbox_*_quota flags, and the effective limits.

Defaults

Setting	Default
SANDBOX_CGROUP_MODE	auto
SANDBOX_CPU_QUOTA	2 cores
SANDBOX_MEMORY_LIMIT_BYTES	2 GiB
SANDBOX_PIDS_LIMIT	512
SANDBOX_WORKER_MEMORY_RESERVE_BYTES	256 MiB
SANDBOX_ADDRESS_SPACE_LIMIT_BYTES (fallback only)	8 GiB

docs/ISOLATION.md has a level → limits table and the outer-runtime requirements.

Verification (Docker, outer container --memory 1536m --cpus 4, sandbox limits 1 core / 384 MiB / 128 pids)

Level / scheme	memory bomb	fork bomb + neighbour exec	4-core spin	leftovers
basic / cgroup (default Docker opts)	OOM_KILLED	bounded, neighbour ~0.1 s	4.0 s CPU in 4.3 s wall (capped to 1 core)	0
strict / cgroup (systempaths=unconfined)	OOM_KILLED	bounded	capped	0
standard / cgroup (required)	OOM_KILLED	bounded	capped	0
standard / rlimit (off)	MemoryError (RLIMIT_AS)	bounded	uncapped (15.5 s CPU)	0
basic / rlimit (off)	MemoryError	bounded	uncapped	0
strict + off	startup refused: resource limits: disabled by SANDBOX_CGROUP_MODE=off

The worker's /healthz stayed responsive in every case. ./scripts/integration-test.sh sandbox gives 29 passed, including the new tests/integration/test_resource_limits.py. Unit tests: 755 passed.

Found and fixed during verification: at basic/rlimit, fork-bomb survivors kept the sandbox UID's NPROC budget, so the next command failed with bwrap: Creating new namespace failed. Leftover UID processes are now killed after the sandbox's last execution.

Limitations

• No disk quota and no IO (io.max) limits yet.
• The rlimit scheme bounds memory per process, not per sandbox.
• Node runs under an 8 GiB RLIMIT_AS; JVM, Go and Rust under RLIMIT_AS were not tested here (the polyglot image wasn't built).
• Hosts with an AppArmor userns restriction, or cgroup v1, fall back to rlimits (reported in resource_control.cgroup.detail).
• Kubernetes is not verified.

Update: aggregate limits everywhere, plus disk and IO limits

• No cgroup available (cgroup v1, no user namespaces, or SANDBOX_CGROUP_MODE=off): a supervisor (src/agent_sandbox/supervisor.py) enforces the sandbox's memory and CPU totals across all of its processes. Every sandbox has its own UID, so it tracks that UID:
  • Memory: sums PSS (RSS where unreadable) and kills the whole UID when the sum is over the limit, returning SANDBOX_MEMORY_LIMIT_EXCEEDED.
  • CPU: a token bucket that SIGSTOPs and SIGCONTs the UID's processes.
  • IO: bytes from /proc/<pid>/io, duty-cycled the same way; this needs CAP_SYS_PTRACE.
  • RLIMIT_AS stays as a hard backstop, and RLIMIT_NPROC on the per-sandbox UID is already an aggregate kernel limit.
• Disk quota, every scheme (SANDBOX_DISK_LIMIT_BYTES, default 10 GiB): if usage goes over the quota and keeps growing, the running executions are killed with SANDBOX_DISK_QUOTA_EXCEEDED. File API writes are refused with HTTP 507. Deleting is always allowed.
• IO: under the cgroup scheme, io.max is set on the workspace block device, or on SANDBOX_IO_DEVICE (default 256 MiB/s read and write).
• /healthz: resource_control.enforcement says, for each dimension, whether the kernel enforces it, the supervisor enforces it (and how precisely), or nothing does.

Verified in Docker, 17 checks per run:

Configuration	Result
basic / strict / standard with cgroup	aggregate memory bomb OOM_KILLED, disk quota enforced, io.max 20.3 MiB/s at a 20 MiB/s limit, CPU 4.0 s in 4.2 s wall at 1 core
basic / standard with rlimit+supervisor, and a simulated cgroup v1 (tmpfs over /sys/fs/cgroup)	aggregate memory kill, disk quota enforced, CPU about 1.2 cores at a 1-core quota (1 s burst allowed); with SYS_PTRACE, IO 19.9 MiB/s at 20 MiB/s
Polyglot image (Go, Rust, JDK) at basic, standard and strict, both schemes	builds and runs; a JVM -Xmx4g heap bomb returns OOM_KILLED
kind (Kubernetes 1.31) pod, seccompProfile: Unconfined, runAsUser 0	cgroup scheme active, 16/16 checks

Integration suite: 37 passed, with both schemes parameterized. Unit tests: 762 passed.