Track managed sandbox lifecycle bugs: control pipe writes bypass timeouts #464

Current bug hunt: 2026-10-11

Revision: latest fetched origin/dev, 6020d5d49e2e4839feb46cbadbc8db7e8cf334e2.

This issue is reopened for a newly confirmed bug in the managed sandbox control client. The earlier concurrent-start bug was addressed by #476; the finding below is not a claim that its lifecycle-lock fix regressed.

The focused area was control-session framing and timeout/error handling, which the open bug-hunt issues #490 (guest artifact cache provenance) and #486 (live shares at /run targets) do not cover. Investigation stopped at the first confirmed bug, within the 45-minute budget. No repository files were edited and no commits were created.

• Bound managed control writes by the operation deadline and add a stalled-peer regression test.

1. Windows managed exec can hang beyond its control operation timeout

Severity: Medium. A stalled control endpoint can hang an orchestration call indefinitely rather than returning an error within the configured timeout. A guest workload timeout does not help while its exec request has not finished reaching the endpoint.

Source and caller trace

• The CLI documents --timeout as a control operation timeout, validates that it is finite and positive, and passes it to exec_workload as response_timeout: nvx.py timeout option, command_sandbox exec call.
• exec_workload establishes the authenticated session, then forwards that timeout to ControlSession.exec.
• ControlSession.connect selects _NamedPipeStream on Windows. The reproduction below completes its real capability-attach/READY exchange.
• ControlSession.exec calls _send_app before establishing the response deadline.
• _send_app ultimately calls the transport writer without a deadline.
• _NamedPipeStream.write_all uses synchronous os.write. When the connected server stops draining its pipe, a write larger than the available pipe quota blocks; there is no timer or cancellation path to enforce --timeout.

Trigger

On Windows, the control endpoint completes authentication and then stops reading. Send a valid exec request larger than its available inbound pipe quota. The reproduction uses a real byte-mode duplex named pipe with a 1 KiB quota and a valid 4,093-byte wire request: /bin/true plus one 4,000-byte argument. This is comfortably within both the 4,096-byte per-argument limit and the 64 KiB application-frame limit.

Expected versus actual

Expected: The control operation times out with an explicit timeout error, even if it is blocked sending the request. An incomplete frame must not leave a reusable session.

Actual: With response_timeout=0.05, the call remains blocked after approximately 0.35 seconds. It returns only after the reproduction closes the server handle, with a pipe-closure error rather than a timeout. If that peer stays open without reading, the synchronous write has no timeout bound.

Smallest reproduction

Windows, Python 3.14.7; no hypervisor, OpenVMM executable, guest images, external Python packages, or changes to the checkout are needed. Save this script outside the repository, then run:

python -B C:\Temp\repro_control_write_timeout.py C:\path\to\nvx

import sys
import threading
import time
import uuid
from pathlib import Path

import _winapi

sys.dont_write_bytecode = True
sys.path.insert(0, str(Path(sys.argv[1]).resolve() / "scripts"))
from nvx_tools import control_session


def read_exact(handle, length):
    data = bytearray()
    while len(data) < length:
        chunk, status = _winapi.ReadFile(handle, length - len(data))
        assert status == 0 and chunk, (status, chunk)
        data.extend(chunk)
    return bytes(data)


name = rf"\\.\pipe\nvx-write-timeout-{uuid.uuid4().hex}"
server = _winapi.CreateNamedPipe(
    name, _winapi.PIPE_ACCESS_DUPLEX, _winapi.PIPE_WAIT,
    1, 1024, 1024, 0, _winapi.NULL,
)
capability = b"\x11" * 32
instance = b"\x22" * 16
attached = threading.Event()
finished = threading.Event()
observed = {}
timeout = 0.05


def client():
    try:
        with control_session.ControlSession.connect(
            Path(name), capability, 2.0
        ) as session:
            observed["started"] = time.monotonic()
            attached.set()
            session.exec(
                ("/bin/true", "a" * 4000),
                timeout_ms=0,
                response_timeout=timeout,
            )
    except Exception as error:
        observed["error"] = f"{type(error).__name__}: {error}"
    finally:
        observed["finished"] = time.monotonic()
        finished.set()


worker = threading.Thread(target=client, daemon=True)
worker.start()
try:
    try:
        _winapi.ConnectNamedPipe(server)
    except OSError as error:
        if error.winerror != 535:  # ERROR_PIPE_CONNECTED
            raise
    header = control_session.OUTER_HEADER.unpack(
        read_exact(server, control_session.OUTER_HEADER.size)
    )
    assert header[:4] == (
        b"NVXS", 1, control_session.OUTER_HOST_ATTACH, 0
    ), header
    assert read_exact(server, header[-1]) == capability
    ready = control_session.OUTER_HEADER.pack(
        b"NVXS", 1, control_session.OUTER_READY, 0, instance, 1, 0, 0
    )
    written, status = _winapi.WriteFile(server, ready)
    assert status == 0 and written == len(ready), (written, status)
    assert attached.wait(2.0), observed
    blocked = not finished.wait(0.35)
    elapsed = time.monotonic() - observed["started"]
    print(f"authenticated attach completed: {attached.is_set()}")
    print(f"response_timeout: {timeout:.3f}s")
    print(f"still blocked after {elapsed:.3f}s: {blocked}")
    assert blocked, observed
finally:
    _winapi.CloseHandle(server)
    worker.join(2.0)
assert not worker.is_alive(), "client did not exit after server closed"
duration = observed["finished"] - observed["started"]
print(f"returned after server closed, at {duration:.3f}s")
print(f"result: {observed.get('error', 'unexpected success')}")
assert str(observed.get("error", "")).startswith(
    ("ControlEndpointClosed:", "BrokenPipeError:")
), observed
print("CONFIRMED: a blocked pipe write bypasses the operation timeout")

Observed output from executing the exact self-contained issue reproduction:

authenticated attach completed: True
response_timeout: 0.050s
still blocked after 0.351s: True
returned after server closed, at 0.352s
result: BrokenPipeError: [Errno 32] Broken pipe
CONFIRMED: a blocked pipe write bypasses the operation timeout

The self-contained reproduction exited with code 0. The server intentionally remains connected beyond the deadline and is closed only to release the blocked client during cleanup. Neither accepted closure exception is a timeout.

Existing tests and coverage gap

python -B -m unittest scripts\test_control_session.py -v ran 17 tests: 16 passed, 1 skipped (Linux-only Unix-domain-socket closure semantics).

python -B -m unittest scripts\test_nvx_tools.py -k sandbox_timeout -k sandbox_exec_forwards -v ran 3 tests, all passed, covering CLI timeout validation and forwarding of managed execution options.

• Invalid response-timeout tests ensure invalid values are rejected before sending; they do not cover a valid timeout while sending blocks.
• Ordinary exec test promptly drains the request.
• Named-pipe closure test closes the server before writing; it does not leave a connected, non-reading peer.
• CLI timeout-validation test covers finite/positive parsing, not transport backpressure.

Suggested fix

Establish the operation deadline before sending and propagate it into transport writes. On Windows, use deadline-aware overlapped pipe writes with cancellation/cleanup rather than an unbounded synchronous os.write. Close the session if a write times out after emitting only part of a frame. Apply the same bounded-write contract to attach, ping, and stop so that their operation timeouts do not have the same unbounded write path. Preserve existing capability authentication and closure/error reporting.

Add a regression test that authenticates over a real named pipe, stalls its reader, and sends a valid request larger than its quota. Assert that the operation terminates with a timeout within a small scheduling tolerance without closing the server to unblock it, and that a timed-out partial frame cannot be reused.

Checked, not filed

• Argument and environment bounds, omitted versus empty environments, supported exit categories, refusal diagnostics, and reset-versus-closure distinctions: traced and covered by the passing control-session tests; no additional bug confirmed.
• The connection and attach phases each calculate a fresh timeout budget: reviewed only, not independently reproduced or filed.
• Non-finite CLI timeout inputs: the current parser rejects them; the earlier low-impact unchecked input concern is not a current finding.

Validation limits and repository state

• This proves an unbounded host-side write using the real Windows named-pipe transport and unmodified NVX client, including its authenticated attach. A deliberately stalled protocol peer stands in for an unresponsive control endpoint. It does not demonstrate how frequently a real OpenVMM process stalls.
• No real OpenVMM or WHP guest was started. python -B scripts\nvx.py verify failed with initialized OpenVMM submodule not found: ...\openvmm\Cargo.toml; the submodule is uninitialized and was left untouched. No Linux/KVM/MSHV runtime or build gates were claimed.
• The worktree HEAD is c29f3d1b032938503369ba9c14a0063241317627, but the complete scripts tree is byte-identical to the fetched origin/dev revision used in all source permalinks: tree ID 4992e798a691eedaf2efea551cb3cfe804692d0d at both revisions. The intervening changes affect only performance baseline CSVs and distribution documentation.
• Bytecode generation was disabled; reproduction and issue-draft artifacts are outside the repository. Final git status --short was empty, matching the initial clean checkout; the submodule remained uninitialized at its original gitlink. No repository files were changed, no submodule was initialized, and no commits were made.

Earlier finding

• Concurrent sandbox start could orphan a running OpenVMM. Addressed by sandbox: serialize the lifecycle transitions of a state directory #476, which added per-state-directory lifecycle serialization and ownership-aware failed-start cleanup. Retained here as resolved history, not reported as a new or reproduced regression.