diff --git a/docker-compose.yml b/docker-compose.yml index 38c58e8f4a..9a09c5a5eb 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -24,6 +24,9 @@ services: # only consumed by the CLIs running inside this container. - "47331:47331" volumes: + # Complete home. Guardian persists /data/state/machine-id so a + # replacement container can reclaim leftover locks from this volume. + # Do not mount this directory on two live containers. - openalice-data:/data # Optional: reuse host auth instead of authenticating inside the # container. Bind-mount your local credentials read-only. diff --git a/docs/data-locations.md b/docs/data-locations.md index 376b31aa47..56d7f1ad81 100644 --- a/docs/data-locations.md +++ b/docs/data-locations.md @@ -3,7 +3,9 @@ This guide owns OpenAlice data-location selection, desktop launcher preferences, and the isolation contract for concurrent local AliceProjects. Runtime lock recovery itself belongs to `packages/guardian-runtime/`; the -persistent state layout belongs to [[docs/project-structure.md]]. +persistent state layout belongs to [[docs/project-structure.md]]. Docker +container replacement and leftover `guardian.lock` / `runtime.lock` recovery +belong to [[docs/docker-deployment.md]]. ## One Complete Home diff --git a/docs/docker-deployment.md b/docs/docker-deployment.md index c63dd41b44..37e7a5c5a7 100644 --- a/docs/docker-deployment.md +++ b/docs/docker-deployment.md @@ -93,9 +93,53 @@ docker compose down docker compose up -d --build ``` +If a replacement container exits immediately with `already running` and a +reason that names another machine, the volume still holds a leftover +Guardian/runtime lock. See [Stale Locks After Container Replacement](#stale-locks-after-container-replacement). + `docker compose down` preserves the named volume. `docker compose down -v` is a factory reset and permanently removes user data. +## Stale Locks After Container Replacement + +Guardian records ownership in `/data/state/guardian.lock`, +`/data/state/runtime.lock`, and the compatibility +`/data/workspaces/state/runtime.lock`. A graceful `docker compose stop` or +`docker stop` releases those leases. `docker kill`, an OOM, a host crash, or +a compose recreate that interrupts shutdown can leave them behind. + +The lock stores a machine identity. Docker's `/etc/machine-id` is container +local, so a replacement container looks like another machine even when it +mounts the same volume. Desktop and CLI keep that cross-machine refusal: +heartbeat expiry alone must not reclaim a copied or NFS-shared home, and +`--takeover` must not signal a PID that belongs to another host. + +The server image is a single-writer contract: + +1. The first Docker boot writes `/data/state/machine-id` and uses it as + `OPENALICE_MACHINE_ID`, so later recreates keep the same identity and + reclaim a dead PID immediately. +2. When a leftover lock still names a previous container identity and its + heartbeat is stale (default 90s), Docker Guardian quarantines the lock + without signaling that foreign PID. +3. A leftover lock with a still-fresh heartbeat still refuses ordinary + start. Wait for the heartbeat to age, or start once with `--takeover` / + `OPENALICE_TAKEOVER=1` after confirming no other replica shares `/data`. + +Do not mount the same `/data` on two live containers. Two writers on one +home remain unsupported; a volume-stable identity cannot see PIDs in +another container's namespace. + +Existing deployments that already crash-loop on days-old locks recover on +the first boot of this image: the heartbeat is stale, so the foreign lock +is reclaimed automatically. The operator workaround of moving the three +`owner.json` trees aside remains valid for older images. + +Custom compose files that bind-mount `OPENALICE_HOME` and +`AQ_LAUNCHER_ROOT` separately must keep both `state/` trees on the same +persistent volume. Quarantining only one home leaves the compatibility +Workspace lock in place. + ## Backup and Restore Stop the container before taking a filesystem-consistent volume snapshot: diff --git a/docs/project-structure.md b/docs/project-structure.md index 224adce02a..67cbceb92e 100644 --- a/docs/project-structure.md +++ b/docs/project-structure.md @@ -357,6 +357,7 @@ sealing, and Broker Packs must remain coherent. │ │ provenance, Agent conversation log, compatibility lock │ └── auto-quant-v2-mirror/ shared AutoQuant V2 source mirror ├── state/ +│ ├── machine-id Docker/home-stable identity (optional) │ ├── guardian.lock launcher ownership │ └── runtime.lock shared writer ownership ├── runtime/ diff --git a/docs/remote-access.md b/docs/remote-access.md index 20b3cdc52b..05319d4d4e 100644 --- a/docs/remote-access.md +++ b/docs/remote-access.md @@ -878,13 +878,13 @@ from roughly 24 seconds and many SSH sessions to roughly 2 seconds and one SSH session. The redeploy also confirmed the cross-machine safety boundary. The reattached -volume still named the removed container as Guardian and Alice owner; ordinary -start refused it, and explicit `--takeover` still refused to signal or reclaim -an owner from another machine. After Railway independently reported the old -deployment as removed, the operator quarantined all three foreign lock records -before starting the new owner. This is an operational recovery observation, -not automatic cross-machine failover: heartbeat expiry alone must never grant -permission to reclaim a shared volume. +volume still named the removed container as Guardian and Alice owner. Ordinary +desktop/CLI start still refuses that foreign owner, and no path may signal a +PID recorded on another machine. After the operator confirms the previous +instance is gone, `--takeover` now quarantines those foreign lock records +without signaling. Docker's single-writer image also reclaims a foreign lock +whose heartbeat is already stale. Heartbeat expiry alone must never grant +desktop or CLI permission to reclaim a copied or NFS-shared volume. ### Stage 3 — terminal transport optimization @@ -924,7 +924,7 @@ permission to reclaim a shared volume. | graceful stop | Guardian stops children, releases owned lease/socket, exits in bound | | hung child | TERM precedes process-tree KILL; no orphan survives | | stale endpoint | recovered only with lease/ownership evidence | -| cross-root or foreign owner | no normal-start kill; explicit takeover only | +| cross-root or foreign owner | no normal-start kill; explicit takeover quarantines a foreign lock without signaling | | optional UTA absent | Server and browser Chat remain ready | | Electron running | `server start/stop` do not silently replace or terminate it | | packaged Electron smoke | local app, PTY, IPC, and shutdown remain healthy | diff --git a/docs/remote-quickstart.md b/docs/remote-quickstart.md index 941db4a674..7c9b40d7bd 100644 --- a/docs/remote-quickstart.md +++ b/docs/remote-quickstart.md @@ -208,9 +208,11 @@ openalice remote openalice-box \ - For an ephemeral VM or container, place `--home` on persistent storage. The Runtime can be reinstalled; the home is the durable state that must survive. - A platform replacement can reattach a volume whose Guardian lock names the - removed machine. OpenAlice refuses cross-machine takeover automatically; - confirm the previous instance is gone before following the operator recovery - guidance in [[docs/remote-access.md]]. + removed machine. Desktop and CLI still refuse automatic cross-machine + reclaim. Confirm the previous instance is gone, then use `--takeover` or + follow the operator recovery guidance in [[docs/remote-access.md]]. Docker + single-writer homes reclaim a stale foreign lock without signaling; see + [[docs/docker-deployment.md]]. ## Docker Is a First-Class Alternative diff --git a/packages/guardian-runtime/src/process-control.spec.ts b/packages/guardian-runtime/src/process-control.spec.ts index 714d9724b1..9c558f2b78 100644 --- a/packages/guardian-runtime/src/process-control.spec.ts +++ b/packages/guardian-runtime/src/process-control.spec.ts @@ -1,16 +1,27 @@ import { spawn } from 'node:child_process' import { once } from 'node:events' +import { mkdir, readFile, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' + import { afterEach, describe, expect, it } from 'vitest' -import { normalizeProcessExitCode, terminateProcessTree } from './process-control.js' +import { + ensureHomeMachineIdentity, + homeMachineIdPath, + normalizeProcessExitCode, + terminateProcessTree, +} from './process-control.js' const cleanupPids = new Set() +let home: string -afterEach(() => { +afterEach(async () => { for (const pid of cleanupPids) { try { process.kill(pid, 'SIGKILL') } catch { /* already gone */ } } cleanupPids.clear() + if (home) await rm(home, { recursive: true, force: true, maxRetries: 5, retryDelay: 50 }) }) describe('normalizeProcessExitCode', () => { @@ -60,6 +71,38 @@ describe('terminateProcessTree', () => { }) }) +describe('ensureHomeMachineIdentity', () => { + it('creates, persists, and reuses a home-scoped machine identity', async () => { + home = join(tmpdir(), `guardian-machine-id-${process.pid}-${Math.random().toString(16).slice(2)}`) + await mkdir(home, { recursive: true }) + const firstEnv: NodeJS.ProcessEnv = {} + const first = await ensureHomeMachineIdentity(home, firstEnv) + expect(firstEnv.OPENALICE_MACHINE_ID).toBe(first) + expect((await readFile(homeMachineIdPath(home), 'utf8')).trim()).toBe(first) + + const secondEnv: NodeJS.ProcessEnv = {} + await expect(ensureHomeMachineIdentity(home, secondEnv)).resolves.toBe(first) + expect(secondEnv.OPENALICE_MACHINE_ID).toBe(first) + }) + + it('keeps an explicit OPENALICE_MACHINE_ID and writes it when the home file is missing', async () => { + home = join(tmpdir(), `guardian-machine-id-${process.pid}-${Math.random().toString(16).slice(2)}`) + await mkdir(home, { recursive: true }) + const env: NodeJS.ProcessEnv = { OPENALICE_MACHINE_ID: 'compose-fixed' } + await expect(ensureHomeMachineIdentity(home, env)).resolves.toBe('compose-fixed') + expect((await readFile(homeMachineIdPath(home), 'utf8')).trim()).toBe('compose-fixed') + }) + + it('does not overwrite a persisted identity with a later env override absence', async () => { + home = join(tmpdir(), `guardian-machine-id-${process.pid}-${Math.random().toString(16).slice(2)}`) + await mkdir(join(home, 'state'), { recursive: true }) + await writeFile(homeMachineIdPath(home), 'already-persisted\n', 'utf8') + const env: NodeJS.ProcessEnv = {} + await expect(ensureHomeMachineIdentity(home, env)).resolves.toBe('already-persisted') + expect(env.OPENALICE_MACHINE_ID).toBe('already-persisted') + }) +}) + function isAlive(pid: number): boolean { try { process.kill(pid, 0) diff --git a/packages/guardian-runtime/src/process-control.ts b/packages/guardian-runtime/src/process-control.ts index 8128fd997f..5d4a7deb66 100644 --- a/packages/guardian-runtime/src/process-control.ts +++ b/packages/guardian-runtime/src/process-control.ts @@ -1,6 +1,8 @@ import { execFile } from 'node:child_process' -import { readFile } from 'node:fs/promises' +import { randomUUID } from 'node:crypto' +import { mkdir, readFile, writeFile } from 'node:fs/promises' import { hostname } from 'node:os' +import { dirname, resolve } from 'node:path' import { promisify } from 'node:util' const execFileAsync = promisify(execFile) @@ -148,6 +150,58 @@ async function readProcessStartedAt(pid: number): Promise { } } +export function homeMachineIdPath(userDataHome: string): string { + return resolve(userDataHome, 'state', 'machine-id') +} + +/** + * Persist a volume-stable identity for single-writer Docker/server homes. + * + * Container `/etc/machine-id` changes on recreate, so a leftover lock looks + * foreign even though the previous process is gone. Writing the id into the + * home keeps replacement containers on the same identity. Desktop and CLI + * launchers must not call this: they keep host hardware identity so a copied + * home still looks foreign. + */ +export async function ensureHomeMachineIdentity( + userDataHome: string, + env: NodeJS.ProcessEnv = process.env, +): Promise { + const path = homeMachineIdPath(userDataHome) + await mkdir(dirname(path), { recursive: true }) + const existingEnv = env['OPENALICE_MACHINE_ID']?.trim() + let persisted = '' + try { + persisted = (await readFile(path, 'utf8')).trim() + } catch (err) { + if (!isNodeErrno(err, 'ENOENT')) throw err + } + if (existingEnv) { + if (!persisted) await writeFile(path, `${existingEnv}\n`, 'utf8') + return existingEnv + } + if (persisted) { + env['OPENALICE_MACHINE_ID'] = persisted + return persisted + } + const created = randomUUID() + try { + await writeFile(path, `${created}\n`, { encoding: 'utf8', flag: 'wx' }) + env['OPENALICE_MACHINE_ID'] = created + return created + } catch (err) { + if (!isNodeErrno(err, 'EEXIST')) throw err + const raced = (await readFile(path, 'utf8')).trim() + if (!raced) throw new Error(`OpenAlice home machine identity at ${path} is empty`) + env['OPENALICE_MACHINE_ID'] = raced + return raced + } +} + +function isNodeErrno(err: unknown, code: string): boolean { + return err instanceof Error && 'code' in err && (err as NodeJS.ErrnoException).code === code +} + async function readMachineId(): Promise { const override = process.env['OPENALICE_MACHINE_ID']?.trim() if (override) return `env:${override}` diff --git a/packages/guardian-runtime/src/runtime-lock.spec.ts b/packages/guardian-runtime/src/runtime-lock.spec.ts index c2fa05dff7..6480727580 100644 --- a/packages/guardian-runtime/src/runtime-lock.spec.ts +++ b/packages/guardian-runtime/src/runtime-lock.spec.ts @@ -12,6 +12,7 @@ import { acquireRuntimeLock, inspectRuntimeLock, prepareOpenAliceRuntime, + reclaimStaleForeignRequested, runtimeLockDir, } from './runtime-lock.js' @@ -153,7 +154,7 @@ describe('runtime lock ownership', () => { await lock.release() }) - it('never signals or reclaims an owner recorded on another machine', async () => { + it('reclaims a foreign owner on explicit takeover without signaling it', async () => { controller.add(101, 10_000) controller.add(202, 20_000) const lockDir = join(home, 'runtime.lock') @@ -180,8 +181,156 @@ describe('runtime lock ownership', () => { takeover: true, heartbeatMs: 0, processController: controller, - })).rejects.toThrow(/another machine/) + })).resolves.toMatchObject({ owner: { pid: 202 } }) expect(controller.signals).toEqual([]) + await expect(inspectRuntimeLock(lockDir, { processController: controller })).resolves.toMatchObject({ + state: 'active', + owner: { pid: 202 }, + foreign: false, + }) + }) + + it('does not reclaim a fresh foreign owner even when reclaimStaleForeign is set', async () => { + controller.add(101, 10_000) + controller.add(202, 20_000) + const lockDir = join(home, 'runtime.lock') + await acquireRuntimeLock(lockDir, { + pid: 101, + processStartedAt: 10_000, + heartbeatMs: 0, + processController: controller, + }) + controller.currentMachineId = 'machine-b' + + await expect(inspectRuntimeLock(lockDir, { + processController: controller, + reclaimStaleForeign: true, + })).resolves.toMatchObject({ + state: 'active', + foreign: true, + heartbeatStale: false, + }) + await expect(acquireRuntimeLock(lockDir, { + pid: 202, + processStartedAt: 20_000, + reclaimStaleForeign: true, + heartbeatMs: 0, + processController: controller, + })).rejects.toBeInstanceOf(RuntimeAlreadyRunningError) + expect(controller.signals).toEqual([]) + }) + + it('reclaims a stale Docker-replacement lock without signaling the recorded pid', async () => { + controller.add(101, 10_000) + controller.add(202, 20_000) + const lockDir = join(home, 'runtime.lock') + await acquireRuntimeLock(lockDir, { + pid: 101, + processStartedAt: 10_000, + heartbeatMs: 0, + processController: controller, + }) + controller.alive.set(101, false) + controller.currentMachineId = 'machine-b' + + await expect(inspectRuntimeLock(lockDir, { + processController: controller, + staleHeartbeatMs: -1, + })).resolves.toMatchObject({ + state: 'active', + foreign: true, + heartbeatStale: true, + reason: expect.stringContaining('refusing automatic takeover'), + }) + await expect(acquireRuntimeLock(lockDir, { + pid: 202, + processStartedAt: 20_000, + heartbeatMs: 0, + processController: controller, + staleHeartbeatMs: -1, + })).rejects.toBeInstanceOf(RuntimeAlreadyRunningError) + + const fresh = await acquireRuntimeLock(lockDir, { + pid: 202, + processStartedAt: 20_000, + reclaimStaleForeign: true, + heartbeatMs: 0, + processController: controller, + staleHeartbeatMs: -1, + }) + expect(controller.signals).toEqual([]) + await expect(inspectRuntimeLock(lockDir, { processController: controller })).resolves.toMatchObject({ + state: 'active', + owner: { pid: 202 }, + foreign: false, + }) + await fresh.release() + }) + + it('Guardian preflight skips signaling a stale foreign runtime owner when reclaim is allowed', async () => { + controller.add(101, 10_000) + const lockDir = runtimeLockDir(home) + await acquireRuntimeLock(lockDir, { + pid: 101, + processStartedAt: 10_000, + heartbeatMs: 0, + processController: controller, + }) + controller.alive.set(101, false) + controller.currentMachineId = 'machine-b' + + await expect(prepareOpenAliceRuntime({ + userDataHome: home, + launcherRoot: join(home, 'workspaces'), + processController: controller, + staleHeartbeatMs: -1, + })).rejects.toBeInstanceOf(RuntimeAlreadyRunningError) + expect(controller.signals).toEqual([]) + + await expect(prepareOpenAliceRuntime({ + userDataHome: home, + launcherRoot: join(home, 'workspaces'), + processController: controller, + staleHeartbeatMs: -1, + reclaimStaleForeign: true, + })).resolves.toEqual(expect.arrayContaining([ + expect.objectContaining({ lockDir, state: 'stale', foreign: true }), + ])) + expect(controller.signals).toEqual([]) + }) + + it('reads reclaimStaleForeign from env and argv', async () => { + expect(reclaimStaleForeignRequested({}, [])).toBe(false) + expect(reclaimStaleForeignRequested({ OPENALICE_RECLAIM_STALE_FOREIGN: '1' }, [])).toBe(true) + expect(reclaimStaleForeignRequested({}, ['--reclaim-stale-foreign'])).toBe(true) + + controller.add(101, 10_000) + controller.add(202, 20_000) + const lockDir = join(home, 'runtime.lock') + await acquireRuntimeLock(lockDir, { + pid: 101, + processStartedAt: 10_000, + heartbeatMs: 0, + processController: controller, + }) + controller.alive.set(101, false) + controller.currentMachineId = 'machine-b' + const previous = process.env['OPENALICE_RECLAIM_STALE_FOREIGN'] + process.env['OPENALICE_RECLAIM_STALE_FOREIGN'] = '1' + try { + const fresh = await acquireRuntimeLock(lockDir, { + pid: 202, + processStartedAt: 20_000, + heartbeatMs: 0, + processController: controller, + staleHeartbeatMs: -1, + }) + expect(controller.signals).toEqual([]) + await fresh.release() + } finally { + if (previous === undefined) delete process.env['OPENALICE_RECLAIM_STALE_FOREIGN'] + else process.env['OPENALICE_RECLAIM_STALE_FOREIGN'] = previous + } }) it('performs a controlled takeover before acquiring the lock', async () => { @@ -358,6 +507,71 @@ describe('OpenAlice global + legacy lock composition', () => { await second.release() }) + it('reclaims leftover Docker guardian and runtime locks after container replacement', async () => { + controller.add(101, 10_000) + controller.add(202, 20_000) + const launcherRoot = join(home, 'workspaces') + const guardian = await acquireGuardianRuntime({ + userDataHome: home, + launcherRoot, + launcher: 'guardian-docker', + pid: 101, + processStartedAt: 10_000, + heartbeatMs: 0, + processController: controller, + }) + const alice = await acquireOpenAliceRuntimeLocks({ + userDataHome: home, + launcherRoot, + launcher: 'docker', + pid: 101, + processStartedAt: 10_000, + heartbeatMs: 0, + processController: controller, + }) + controller.alive.set(101, false) + controller.currentMachineId = 'replacement-container' + + await expect(acquireGuardianRuntime({ + userDataHome: home, + launcherRoot, + launcher: 'guardian-docker', + pid: 202, + processStartedAt: 20_000, + heartbeatMs: 0, + processController: controller, + staleHeartbeatMs: -1, + })).rejects.toBeInstanceOf(RuntimeAlreadyRunningError) + + const recovered = await acquireGuardianRuntime({ + userDataHome: home, + launcherRoot, + launcher: 'guardian-docker', + pid: 202, + processStartedAt: 20_000, + heartbeatMs: 0, + processController: controller, + staleHeartbeatMs: -1, + reclaimStaleForeign: true, + }) + const recoveredAlice = await acquireOpenAliceRuntimeLocks({ + userDataHome: home, + launcherRoot, + launcher: 'docker', + pid: 202, + processStartedAt: 20_000, + heartbeatMs: 0, + processController: controller, + staleHeartbeatMs: -1, + reclaimStaleForeign: true, + }) + expect(controller.signals).toEqual([]) + await guardian.release() + await alice.release() + await recovered.release() + await recoveredAlice.release() + }) + it('prevents two launcher roots from writing the same OPENALICE_HOME', async () => { controller.add(101, 10_000) controller.add(202, 20_000) diff --git a/packages/guardian-runtime/src/runtime-lock.ts b/packages/guardian-runtime/src/runtime-lock.ts index 5e73b3e9c2..67a824c94e 100644 --- a/packages/guardian-runtime/src/runtime-lock.ts +++ b/packages/guardian-runtime/src/runtime-lock.ts @@ -41,6 +41,7 @@ export interface RuntimeLockInspection { readonly heartbeatStale: boolean readonly directoryIdentity: string | null readonly reason: string + readonly foreign: boolean } export interface RuntimeProcessLock { @@ -56,6 +57,7 @@ export interface RuntimeLockOptions { readonly guardianPid?: number readonly guardianStartedAt?: number readonly takeover?: boolean + readonly reclaimStaleForeign?: boolean readonly heartbeatMs?: number readonly staleHeartbeatMs?: number readonly initializationGraceMs?: number @@ -80,6 +82,7 @@ export interface PrepareOpenAliceRuntimeOptions { readonly userDataHome: string readonly launcherRoot: string readonly takeover?: boolean + readonly reclaimStaleForeign?: boolean readonly processController?: ProcessController readonly staleHeartbeatMs?: number readonly initializationGraceMs?: number @@ -119,13 +122,32 @@ export function takeoverRequested(env: NodeJS.ProcessEnv = process.env, argv: re return /^(1|true|yes|on)$/i.test(env['OPENALICE_TAKEOVER']?.trim() ?? '') } +export function reclaimStaleForeignRequested(env: NodeJS.ProcessEnv = process.env, argv: readonly string[] = process.argv): boolean { + if (argv.includes('--reclaim-stale-foreign')) return true + return /^(1|true|yes|on)$/i.test(env['OPENALICE_RECLAIM_STALE_FOREIGN']?.trim() ?? '') +} + +export function canReclaimForeignOwner( + current: Pick, + opts: Pick = {}, +): boolean { + if (!current.foreign || !current.owner) return false + if (opts.takeover) return true + return Boolean(resolveReclaimStaleForeign(opts) && current.heartbeatStale) +} + +function resolveReclaimStaleForeign(opts: Pick = {}): boolean { + return opts.reclaimStaleForeign ?? reclaimStaleForeignRequested() +} + export async function inspectRuntimeLock( lockDir: string, - opts: Pick = {}, + opts: Pick = {}, ): Promise { const controller = opts.processController ?? defaultProcessController const staleMs = opts.staleHeartbeatMs ?? DEFAULT_STALE_HEARTBEAT_MS const initGraceMs = opts.initializationGraceMs ?? DEFAULT_INITIALIZATION_GRACE_MS + const reclaimStaleForeign = resolveReclaimStaleForeign(opts) let lockStat try { lockStat = await stat(lockDir) @@ -149,6 +171,18 @@ export async function inspectRuntimeLock( const heartbeatAgeMs = Number.isFinite(heartbeatAt) ? Math.max(0, Date.now() - heartbeatAt) : null const heartbeatStale = heartbeatAgeMs === null || heartbeatAgeMs > staleMs if (owner.machineId && owner.machineId !== await controller.machineId()) { + if (reclaimStaleForeign && heartbeatStale) { + return inspection( + lockDir, + 'stale', + owner, + heartbeatAgeMs, + heartbeatStale, + directoryIdentity, + 'owner belongs to another machine and its heartbeat is stale', + true, + ) + } return inspection( lockDir, 'active', @@ -159,6 +193,7 @@ export async function inspectRuntimeLock( heartbeatStale ? 'owner belongs to another machine and its heartbeat is stale; refusing automatic takeover' : 'owner belongs to another machine', + true, ) } if (!controller.isAlive(owner.pid)) { @@ -182,22 +217,23 @@ export async function acquireRuntimeLock( lockDir: string, opts: RuntimeLockOptions = {}, ): Promise { - const controller = opts.processController ?? defaultProcessController - const processStartedAt = opts.processStartedAt ?? currentProcessStartedAt() + const resolved = { ...opts, reclaimStaleForeign: resolveReclaimStaleForeign(opts) } + const controller = resolved.processController ?? defaultProcessController + const processStartedAt = resolved.processStartedAt ?? currentProcessStartedAt() const machineId = await controller.machineId() const now = new Date().toISOString() const owner: RuntimeLockOwner = { schemaVersion: 1, - pid: opts.pid ?? process.pid, + pid: resolved.pid ?? process.pid, hostname: hostname(), machineId, token: randomUUID(), - launcher: opts.launcher ?? process.env['OPENALICE_LAUNCHER'] ?? 'standalone', + launcher: resolved.launcher ?? process.env['OPENALICE_LAUNCHER'] ?? 'standalone', acquiredAt: now, heartbeatAt: now, processStartedAt: new Date(processStartedAt).toISOString(), - ...(opts.guardianPid ? { guardianPid: opts.guardianPid } : {}), - ...(opts.guardianStartedAt ? { guardianStartedAt: new Date(opts.guardianStartedAt).toISOString() } : {}), + ...(resolved.guardianPid ? { guardianPid: resolved.guardianPid } : {}), + ...(resolved.guardianStartedAt ? { guardianStartedAt: new Date(resolved.guardianStartedAt).toISOString() } : {}), } await mkdir(dirname(lockDir), { recursive: true }) @@ -205,18 +241,23 @@ export async function acquireRuntimeLock( try { await mkdir(lockDir) await writeOwnerAtomic(lockDir, owner, controller) - return makeLock(lockDir, owner, opts) + return makeLock(lockDir, owner, resolved) } catch (err) { if (!isErrno(err, 'EEXIST')) throw err } - const current = await inspectRuntimeLock(lockDir, opts) + const current = await inspectRuntimeLock(lockDir, resolved) if (current.state === 'missing' || current.state === 'initializing') { await controller.sleep(25) continue } + if (current.foreign && (current.state === 'stale' || canReclaimForeignOwner(current, resolved))) { + if (await claimAndRemove(current)) continue + await controller.sleep(25) + continue + } if (current.state === 'active') { - if (!opts.takeover || !current.owner) throw new RuntimeAlreadyRunningError(current) + if (!resolved.takeover || !current.owner) throw new RuntimeAlreadyRunningError(current) await recoverRuntimeOwner(current.owner, { processController: controller }) await controller.sleep(25) continue @@ -226,7 +267,7 @@ export async function acquireRuntimeLock( await controller.sleep(25) } - throw new RuntimeAlreadyRunningError(await inspectRuntimeLock(lockDir, opts)) + throw new RuntimeAlreadyRunningError(await inspectRuntimeLock(lockDir, resolved)) } export async function acquireOpenAliceRuntimeLocks(opts: OpenAliceRuntimeOptions): Promise { @@ -281,11 +322,13 @@ export async function acquireGuardianRuntime(opts: GuardianRuntimeOptions): Prom * The next Alice process performs the atomic stale-lock reclamation itself. */ export async function prepareOpenAliceRuntime(opts: PrepareOpenAliceRuntimeOptions): Promise { - const inspections = await inspectOpenAliceRuntime(opts) + const resolved = { ...opts, reclaimStaleForeign: resolveReclaimStaleForeign(opts) } + const inspections = await inspectOpenAliceRuntime(resolved) const active = dedupeOwners(inspections.filter((row) => row.state === 'active' && row.owner !== null)) - if (active.length > 0 && !opts.takeover) throw new RuntimeAlreadyRunningError(active[0]!) - for (const row of active) { - await recoverRuntimeOwner(row.owner!, { processController: opts.processController }) + const blocking = active.filter((row) => !canReclaimForeignOwner(row, resolved)) + if (blocking.length > 0 && !resolved.takeover) throw new RuntimeAlreadyRunningError(blocking[0]!) + for (const row of blocking) { + await recoverRuntimeOwner(row.owner!, { processController: resolved.processController }) } return inspections } @@ -461,8 +504,9 @@ function inspection( heartbeatStale: boolean, directoryIdentity: string | null, reason: string, + foreign = false, ): RuntimeLockInspection { - return { lockDir, state, owner, heartbeatAgeMs, heartbeatStale, directoryIdentity, reason } + return { lockDir, state, owner, heartbeatAgeMs, heartbeatStale, directoryIdentity, reason, foreign } } function dedupeOwners(rows: RuntimeLockInspection[]): RuntimeLockInspection[] { diff --git a/scripts/guardian/prod.mjs b/scripts/guardian/prod.mjs index a929e50986..7e44d06633 100644 --- a/scripts/guardian/prod.mjs +++ b/scripts/guardian/prod.mjs @@ -42,6 +42,9 @@ import { normalizeProcessExitCode, RestartBackoff, takeoverRequested, + reclaimStaleForeignRequested, + ensureHomeMachineIdentity, + RuntimeAlreadyRunningError, } from '@traderalice/guardian-runtime' import { planProdPorts, @@ -59,6 +62,8 @@ const NODE_BINARY = process.env.OPENALICE_NODE_BINARY?.trim() || process.execPat const BIND_HOST = process.env.OPENALICE_BIND_HOST?.trim() || '127.0.0.1' const GUARDIAN_STARTED_AT = currentProcessStartedAt() const TAKEOVER = takeoverRequested() +const RECLAIM_STALE_FOREIGN = LAUNCHER === 'docker' || reclaimStaleForeignRequested() +if (RECLAIM_STALE_FOREIGN) process.env.OPENALICE_RECLAIM_STALE_FOREIGN = '1' const SERVER_MODE = process.env.OPENALICE_SERVER_MODE?.trim() || 'foreground' const GUARDIAN_INSTANCE_ID = randomUUID() const RUNTIME_PROVIDER = resolveRuntimeProvider() @@ -589,11 +594,16 @@ async function startFlagWatcher() { } async function main() { + if (LAUNCHER === 'docker') { + const machineId = await ensureHomeMachineIdentity(DATA_HOME) + console.log(`[guardian/prod] home machine identity → ${machineId}`) + } guardianRuntimeLock = await acquireGuardianRuntime({ userDataHome: DATA_HOME, launcherRoot: LAUNCHER_ROOT, launcher: GUARDIAN_LAUNCHER, takeover: TAKEOVER, + reclaimStaleForeign: RECLAIM_STALE_FOREIGN, processStartedAt: GUARDIAN_STARTED_AT, onOwnershipLost: (err) => { console.error('[guardian/prod] runtime ownership lost:', err) @@ -652,6 +662,23 @@ async function main() { } main().catch((err) => { + if (err instanceof RuntimeAlreadyRunningError) { + const owner = err.inspection.owner + console.error(`[guardian/prod] ${err.message}`) + if (owner) { + console.error( + `[guardian/prod] owner → ${owner.launcher} pid=${owner.pid} host=${owner.hostname} machine=${owner.machineId ?? 'unknown'} heartbeat=${owner.heartbeatAt}`, + ) + } + if (err.inspection.foreign) { + console.error('[guardian/prod] this home still names a Guardian from another container or host.') + console.error('[guardian/prod] stop every replica that shares this volume, then start with --takeover or OPENALICE_TAKEOVER=1') + console.error('[guardian/prod] if that previous process is already gone, quarantine state/guardian.lock, state/runtime.lock, and workspaces/state/runtime.lock') + } else { + console.error('[guardian/prod] another OpenAlice runtime already owns this home; stop it or rerun with --takeover') + } + process.exit(2) + } console.error('[guardian/prod] fatal:', err) shutdown(1) }) diff --git a/src/main.ts b/src/main.ts index ef1a9a6018..011a59310c 100644 --- a/src/main.ts +++ b/src/main.ts @@ -1,5 +1,6 @@ import { acquireOpenAliceRuntimeLocks, + reclaimStaleForeignRequested, takeoverRequested, type OpenAliceRuntimeLock, } from '@traderalice/guardian-runtime' @@ -454,6 +455,7 @@ async function start(): Promise { launcherRoot: resolveLauncherRoot(), launcher: process.env['OPENALICE_LAUNCHER'] ?? 'standalone', takeover: takeoverRequested(), + reclaimStaleForeign: reclaimStaleForeignRequested(), ...(guardianPid ? { guardianPid } : {}), ...(guardianStartedAt ? { guardianStartedAt } : {}), onOwnershipLost: (err) => {