fix(guardian): reclaim stale Docker locks after container replace

docker-compose.yml

@@ -24,6 +24,9 @@ services:
       # only consumed by the CLIs running inside this container.
       - "47331:47331"
     volumes:
+      # Complete home. Guardian persists /data/state/machine-id so a
+      # replacement container can reclaim leftover locks from this volume.
+      # Do not mount this directory on two live containers.
       - openalice-data:/data
       # Optional: reuse host auth instead of authenticating inside the
       # container. Bind-mount your local credentials read-only.

docs/data-locations.md

@@ -3,7 +3,9 @@
 This guide owns OpenAlice data-location selection, desktop launcher
 preferences, and the isolation contract for concurrent local AliceProjects.
 Runtime lock recovery itself belongs to `packages/guardian-runtime/`; the
-persistent state layout belongs to [[docs/project-structure.md]].
+persistent state layout belongs to [[docs/project-structure.md]]. Docker
+container replacement and leftover `guardian.lock` / `runtime.lock` recovery
+belong to [[docs/docker-deployment.md]].
 
 ## One Complete Home
 

docs/docker-deployment.md

@@ -93,9 +93,53 @@ docker compose down
 docker compose up -d --build
 ```
 
+If a replacement container exits immediately with `already running` and a
+reason that names another machine, the volume still holds a leftover
+Guardian/runtime lock. See [Stale Locks After Container Replacement](#stale-locks-after-container-replacement).
+
 `docker compose down` preserves the named volume. `docker compose down -v` is
 a factory reset and permanently removes user data.
 
+## Stale Locks After Container Replacement
+
+Guardian records ownership in `/data/state/guardian.lock`,
+`/data/state/runtime.lock`, and the compatibility
+`/data/workspaces/state/runtime.lock`. A graceful `docker compose stop` or
+`docker stop` releases those leases. `docker kill`, an OOM, a host crash, or
+a compose recreate that interrupts shutdown can leave them behind.
+
+The lock stores a machine identity. Docker's `/etc/machine-id` is container
+local, so a replacement container looks like another machine even when it
+mounts the same volume. Desktop and CLI keep that cross-machine refusal:
+heartbeat expiry alone must not reclaim a copied or NFS-shared home, and
+`--takeover` must not signal a PID that belongs to another host.
+
+The server image is a single-writer contract:
+
+1. The first Docker boot writes `/data/state/machine-id` and uses it as
+   `OPENALICE_MACHINE_ID`, so later recreates keep the same identity and
+   reclaim a dead PID immediately.
+2. When a leftover lock still names a previous container identity and its
+   heartbeat is stale (default 90s), Docker Guardian quarantines the lock
+   without signaling that foreign PID.
+3. A leftover lock with a still-fresh heartbeat still refuses ordinary
+   start. Wait for the heartbeat to age, or start once with `--takeover` /
+   `OPENALICE_TAKEOVER=1` after confirming no other replica shares `/data`.
+
+Do not mount the same `/data` on two live containers. Two writers on one
+home remain unsupported; a volume-stable identity cannot see PIDs in
+another container's namespace.
+
+Existing deployments that already crash-loop on days-old locks recover on
+the first boot of this image: the heartbeat is stale, so the foreign lock
+is reclaimed automatically. The operator workaround of moving the three
+`owner.json` trees aside remains valid for older images.
+
+Custom compose files that bind-mount `OPENALICE_HOME` and
+`AQ_LAUNCHER_ROOT` separately must keep both `state/` trees on the same
+persistent volume. Quarantining only one home leaves the compatibility
+Workspace lock in place.
+
 ## Backup and Restore
 
 Stop the container before taking a filesystem-consistent volume snapshot:

docs/project-structure.md

@@ -357,6 +357,7 @@ sealing, and Broker Packs must remain coherent.
 │   │                          provenance, Agent conversation log, compatibility lock
 │   └── auto-quant-v2-mirror/  shared AutoQuant V2 source mirror
 ├── state/
+│   ├── machine-id             Docker/home-stable identity (optional)
 │   ├── guardian.lock          launcher ownership
 │   └── runtime.lock           shared writer ownership
 ├── runtime/

docs/remote-access.md

@@ -878,13 +878,13 @@ from roughly 24 seconds and many SSH sessions to roughly 2 seconds and one SSH
 session.
 
 The redeploy also confirmed the cross-machine safety boundary. The reattached
-volume still named the removed container as Guardian and Alice owner; ordinary
-start refused it, and explicit `--takeover` still refused to signal or reclaim
-an owner from another machine. After Railway independently reported the old
-deployment as removed, the operator quarantined all three foreign lock records
-before starting the new owner. This is an operational recovery observation,
-not automatic cross-machine failover: heartbeat expiry alone must never grant
-permission to reclaim a shared volume.
+volume still named the removed container as Guardian and Alice owner. Ordinary
+desktop/CLI start still refuses that foreign owner, and no path may signal a
+PID recorded on another machine. After the operator confirms the previous
+instance is gone, `--takeover` now quarantines those foreign lock records
+without signaling. Docker's single-writer image also reclaims a foreign lock
+whose heartbeat is already stale. Heartbeat expiry alone must never grant
+desktop or CLI permission to reclaim a copied or NFS-shared volume.
 
 ### Stage 3 — terminal transport optimization
 
@@ -924,7 +924,7 @@ permission to reclaim a shared volume.
 | graceful stop | Guardian stops children, releases owned lease/socket, exits in bound |
 | hung child | TERM precedes process-tree KILL; no orphan survives |
 | stale endpoint | recovered only with lease/ownership evidence |
-| cross-root or foreign owner | no normal-start kill; explicit takeover only |
+| cross-root or foreign owner | no normal-start kill; explicit takeover quarantines a foreign lock without signaling |
 | optional UTA absent | Server and browser Chat remain ready |
 | Electron running | `server start/stop` do not silently replace or terminate it |
 | packaged Electron smoke | local app, PTY, IPC, and shutdown remain healthy |

docs/remote-quickstart.md

@@ -208,9 +208,11 @@ openalice remote openalice-box \
 - For an ephemeral VM or container, place `--home` on persistent storage. The
   Runtime can be reinstalled; the home is the durable state that must survive.
 - A platform replacement can reattach a volume whose Guardian lock names the
-  removed machine. OpenAlice refuses cross-machine takeover automatically;
-  confirm the previous instance is gone before following the operator recovery
-  guidance in [[docs/remote-access.md]].
+  removed machine. Desktop and CLI still refuse automatic cross-machine
+  reclaim. Confirm the previous instance is gone, then use `--takeover` or
+  follow the operator recovery guidance in [[docs/remote-access.md]]. Docker
+  single-writer homes reclaim a stale foreign lock without signaling; see
+  [[docs/docker-deployment.md]].
 
 ## Docker Is a First-Class Alternative
 

packages/guardian-runtime/src/process-control.spec.ts

@@ -1,16 +1,27 @@
 import { spawn } from 'node:child_process'
 import { once } from 'node:events'
+import { mkdir, readFile, rm, writeFile } from 'node:fs/promises'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+
 import { afterEach, describe, expect, it } from 'vitest'
 
-import { normalizeProcessExitCode, terminateProcessTree } from './process-control.js'
+import {
+  ensureHomeMachineIdentity,
+  homeMachineIdPath,
+  normalizeProcessExitCode,
+  terminateProcessTree,
+} from './process-control.js'
 
 const cleanupPids = new Set<number>()
+let home: string
 
-afterEach(() => {
+afterEach(async () => {
   for (const pid of cleanupPids) {
     try { process.kill(pid, 'SIGKILL') } catch { /* already gone */ }
   }
   cleanupPids.clear()
+  if (home) await rm(home, { recursive: true, force: true, maxRetries: 5, retryDelay: 50 })
 })
 
 describe('normalizeProcessExitCode', () => {
@@ -60,6 +71,38 @@ describe('terminateProcessTree', () => {
   })
 })
 
+describe('ensureHomeMachineIdentity', () => {
+  it('creates, persists, and reuses a home-scoped machine identity', async () => {
+    home = join(tmpdir(), `guardian-machine-id-${process.pid}-${Math.random().toString(16).slice(2)}`)
+    await mkdir(home, { recursive: true })
+    const firstEnv: NodeJS.ProcessEnv = {}
+    const first = await ensureHomeMachineIdentity(home, firstEnv)
+    expect(firstEnv.OPENALICE_MACHINE_ID).toBe(first)
+    expect((await readFile(homeMachineIdPath(home), 'utf8')).trim()).toBe(first)
+
+    const secondEnv: NodeJS.ProcessEnv = {}
+    await expect(ensureHomeMachineIdentity(home, secondEnv)).resolves.toBe(first)
+    expect(secondEnv.OPENALICE_MACHINE_ID).toBe(first)
+  })
+
+  it('keeps an explicit OPENALICE_MACHINE_ID and writes it when the home file is missing', async () => {
+    home = join(tmpdir(), `guardian-machine-id-${process.pid}-${Math.random().toString(16).slice(2)}`)
+    await mkdir(home, { recursive: true })
+    const env: NodeJS.ProcessEnv = { OPENALICE_MACHINE_ID: 'compose-fixed' }
+    await expect(ensureHomeMachineIdentity(home, env)).resolves.toBe('compose-fixed')
+    expect((await readFile(homeMachineIdPath(home), 'utf8')).trim()).toBe('compose-fixed')
+  })
+
+  it('does not overwrite a persisted identity with a later env override absence', async () => {
+    home = join(tmpdir(), `guardian-machine-id-${process.pid}-${Math.random().toString(16).slice(2)}`)
+    await mkdir(join(home, 'state'), { recursive: true })
+    await writeFile(homeMachineIdPath(home), 'already-persisted\n', 'utf8')
+    const env: NodeJS.ProcessEnv = {}
+    await expect(ensureHomeMachineIdentity(home, env)).resolves.toBe('already-persisted')
+    expect(env.OPENALICE_MACHINE_ID).toBe('already-persisted')
+  })
+})
+
 function isAlive(pid: number): boolean {
   try {
     process.kill(pid, 0)

packages/guardian-runtime/src/process-control.ts

@@ -1,6 +1,8 @@
 import { execFile } from 'node:child_process'
-import { readFile } from 'node:fs/promises'
+import { randomUUID } from 'node:crypto'
+import { mkdir, readFile, writeFile } from 'node:fs/promises'
 import { hostname } from 'node:os'
+import { dirname, resolve } from 'node:path'
 import { promisify } from 'node:util'
 
 const execFileAsync = promisify(execFile)
@@ -148,6 +150,58 @@ async function readProcessStartedAt(pid: number): Promise<number | null> {
   }
 }
 
+export function homeMachineIdPath(userDataHome: string): string {
+  return resolve(userDataHome, 'state', 'machine-id')
+}
+
+/**
+ * Persist a volume-stable identity for single-writer Docker/server homes.
+ *
+ * Container `/etc/machine-id` changes on recreate, so a leftover lock looks
+ * foreign even though the previous process is gone. Writing the id into the
+ * home keeps replacement containers on the same identity. Desktop and CLI
+ * launchers must not call this: they keep host hardware identity so a copied
+ * home still looks foreign.
+ */
+export async function ensureHomeMachineIdentity(
+  userDataHome: string,
+  env: NodeJS.ProcessEnv = process.env,
+): Promise<string> {
+  const path = homeMachineIdPath(userDataHome)
+  await mkdir(dirname(path), { recursive: true })
+  const existingEnv = env['OPENALICE_MACHINE_ID']?.trim()
+  let persisted = ''
+  try {
+    persisted = (await readFile(path, 'utf8')).trim()
+  } catch (err) {
+    if (!isNodeErrno(err, 'ENOENT')) throw err
+  }
+  if (existingEnv) {
+    if (!persisted) await writeFile(path, `${existingEnv}\n`, 'utf8')
+    return existingEnv
+  }
+  if (persisted) {
+    env['OPENALICE_MACHINE_ID'] = persisted
+    return persisted
+  }
+  const created = randomUUID()
+  try {
+    await writeFile(path, `${created}\n`, { encoding: 'utf8', flag: 'wx' })
+    env['OPENALICE_MACHINE_ID'] = created
+    return created
+  } catch (err) {
+    if (!isNodeErrno(err, 'EEXIST')) throw err
+    const raced = (await readFile(path, 'utf8')).trim()
+    if (!raced) throw new Error(`OpenAlice home machine identity at ${path} is empty`)
+    env['OPENALICE_MACHINE_ID'] = raced
+    return raced
+  }
+}
+
+function isNodeErrno(err: unknown, code: string): boolean {
+  return err instanceof Error && 'code' in err && (err as NodeJS.ErrnoException).code === code
+}
+
 async function readMachineId(): Promise<string> {
   const override = process.env['OPENALICE_MACHINE_ID']?.trim()
   if (override) return `env:${override}`