diff --git a/docs/adr/agent-lifecycle.md b/docs/adr/agent-lifecycle.md index 3c69301..642a934 100644 --- a/docs/adr/agent-lifecycle.md +++ b/docs/adr/agent-lifecycle.md @@ -24,7 +24,7 @@ is this agent in" that: any engineer reads at a glance; is identical regardless of the runtime underneath; and is what the control plane observes and the director acts on. -Humans direct; agents do the control. The control plane must classify every +Humans direct; agents do control. The control plane must classify every agent, at any moment, into **exactly one** state. ## 2. Decision Drivers @@ -41,13 +41,18 @@ agent, at any moment, into **exactly one** state. Every agent is in exactly one of **6 states**, discriminated by four observable axes — `desiredStatus` (running / stopped), `accepting_work` (bool), `health` -(in-sync & authorized / not), and `identity_verified` (a **latching** bit: set +(lease/probe/authz in-sync & authorized — **not** version-skew, which is healthy +and routes to Paused / not), and `identity_verified` (a **latching** bit: set true the first time the agent reaches Running, never cleared). The latch is what separates `Starting` (never verified) from `Unhealthy` (was verified, now faulted) — without it their `(desiredStatus, accepting_work, health)` tuples collide. It is CP-observable per runtime: ECS `lastStatus` ever reached RUNNING / k8s ever Ready / compose ever healthy. +`Stopping` and `Stopped` share `desiredStatus==stopped` and are **not** separated +by the four axes; the discriminator is a fifth signal — **process liveness / +terminality** (graceful window open vs. absorbing, per principle 3). + ```mermaid stateDiagram-v2 [*] --> Starting @@ -112,7 +117,7 @@ unobservable; death `cause` enum; turn-level busy/idle. the discriminators `(desiredStatus, accepting_work, health, identity_verified)`; the machine never changes per runtime. 6. **Two predicates, kept apart.** *Dispatch new work* = `state == Running` - (single field). *Doing in-flight work* = `Running ∪ Paused ∪ Stopping`(within + (single field). *Doing in-flight work* = `Running ∪ Paused ∪ Stopping` (within deadline) — a cordoned (Paused) agent still finishes its current turn / MCP call. Don't collapse them into one sentence. @@ -134,10 +139,10 @@ an ECS-only coincidence. | canonical | ECS | k8s / GKE | docker-compose | |---|---|---|---| -| Starting | PROVISIONING / PENDING / **ACTIVATING** (ENI + secret inject) | Pending / ContainerCreating / startupProbe pending | created / starting | +| Starting | **PROVISIONING** (ENI) / PENDING / ACTIVATING (image pull + secret inject) | Pending / ContainerCreating / startupProbe pending | created / starting | | Running | RUNNING + health OK + desiredStatus RUNNING | Running + readinessProbe True + lease valid | healthy *(healthcheck required)* | | Paused | RUNNING + health OK + CP/director cordon (`accepting_work=false`) | Ready but cordoned (CP/director) | running + CP/director cordon | -| Unhealthy | RUNNING + healthStatus UNHEALTHY / lease lost *(attribute, not a task state)* | readiness/liveness fail; **Unknown (node lost) → Unhealthy(fenced) + epoch fence**; CrashLoopBackOff | healthcheck fail; `docker pause` (SIGSTOP) → healthcheck stall → Unhealthy | +| Unhealthy | RUNNING + healthStatus UNHEALTHY / lease lost *(attribute, not a task state)* | readiness/liveness fail; **Unknown (node lost) → Unhealthy(fenced) + epoch fence**; CrashLoopBackOff | healthcheck fail; `docker pause` (cgroup freezer) → healthcheck stall → Unhealthy | | Stopping | desiredStatus STOPPED *(DEACTIVATING only if in a target group / service-discovery; else RUNNING→STOPPING)* | deletionTimestamp != null (Terminating: preStop + grace) | stop requested (stop_grace_period) | | Stopped | STOPPED + stopCode (enum) | deleted; *preempted* = the reclaim edge | exited | diff --git a/docs/adr/deployment-control-plane.md b/docs/adr/deployment-control-plane.md index 3414c6a..21284bd 100644 --- a/docs/adr/deployment-control-plane.md +++ b/docs/adr/deployment-control-plane.md @@ -51,7 +51,7 @@ Vendor-specific terms (ECS/task/ARN/S3/Fargate…) appear **only** inside a | **Spec** | Desired: `identity + version + scale + admission + runtime + configRef` | `.spec` | | **admission** | Spec field (`accepting_work`): may this Instance take new work | — (app/CP-level, orthogonal to `Ready`) | | **Instance.Status** | Observed per Instance: `phase + conditions + …` | Pod `.status` | -| **phase** | Field on an **Instance**'s Status; value is an `AgentState` — **Instance-level only** | Pod `.status.phase` | +| **phase** | Field on an **Instance**'s Status; value is an `AgentState` — **Instance-level only** | ≈ Pod `.status.phase` (ours is *derived* from phase + conditions + probes + deletionTimestamp; see §4 traps) | | **Deployment.Status** | Observed per Deployment: replica counters `desired/current/ready/available` + Conditions — **not** an `AgentState` | Deployment `.status` | | **AgentState** | The 6 lifecycle states (ADR-1) | — | | **Discriminators** | `desiredStatus · accepting_work · health · identity_verified` | ADR-1 | @@ -66,7 +66,9 @@ of Spec** (ADR-1). A driver observes native signals and projects them onto: - **Instance-level** — `Instance.status.phase` is one `AgentState` (ADR-1's `classify()`), plus `conditions[]` (ready, superseded, …). - **Deployment-level** — `Deployment.status` is **replica counters** - (`desired / current / ready / available`) + Deployment `conditions[]` (e.g. + (`current / ready / available` counters, plus `desired` — which in k8s is + spec-sourced (`.spec.replicas`), not a `.status` field; service-level in ECS) + + Deployment `conditions[]` (e.g. `Available`, `Progressing`). **It is NOT an `AgentState`**: a Deployment with N Instances (say one `Running`, one `Unhealthy`) has no single lifecycle value — it has *counts*. (Mirrors k8s exactly: a Pod has a `phase`; a Deployment has @@ -82,6 +84,8 @@ here (see §7). k8s has no single lifecycle enum; it is `phase + conditions + probes + deletionTimestamp`. Traps to document so k8s intuition doesn't misread us: +- **`Starting`** = k8s `phase=Pending` / `ContainerCreating`, or `Running` before + the `Ready` latch (pre-`identity_verified`) — the 6th state, mapped to pre-`Ready`. - **`Running` is stricter than k8s** — ours = k8s `phase=Running ∧ Ready=True`. - **`Paused` has no per-Pod k8s analog**, and is **not** k8s `Deployment.spec.paused` (that is rollout-pause). - **`Stopping`** = k8s "Terminating" (`deletionTimestamp≠null`), which is not a `.status.phase` value. @@ -121,7 +125,7 @@ One idempotent primitive: **`apply(Spec)`** — converge observed toward desired resumable. - **dry-run / diff**: every write supports a preview that returns *what would change* without mutating — a safety valve, and important for agents (look - before leap). See §6 for `dry_run` as a first-class tool parameter. + before you leap). See §6 for `dry_run` as a first-class tool parameter. ## 6. MCP adapter