Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions docs/adr/agent-lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ is this agent in" that: any engineer reads at a glance; is identical regardless
of the runtime underneath; and is what the control plane observes and the
director acts on.

Humans direct; agents do the control. The control plane must classify every
Humans direct; agents do control. The control plane must classify every
agent, at any moment, into **exactly one** state.

## 2. Decision Drivers
Expand All @@ -41,13 +41,18 @@ agent, at any moment, into **exactly one** state.

Every agent is in exactly one of **6 states**, discriminated by four observable
axes — `desiredStatus` (running / stopped), `accepting_work` (bool), `health`
(in-sync & authorized / not), and `identity_verified` (a **latching** bit: set
(lease/probe/authz in-sync & authorized — **not** version-skew, which is healthy
and routes to Paused / not), and `identity_verified` (a **latching** bit: set
true the first time the agent reaches Running, never cleared). The latch is what
separates `Starting` (never verified) from `Unhealthy` (was verified, now
faulted) — without it their `(desiredStatus, accepting_work, health)` tuples
collide. It is CP-observable per runtime: ECS `lastStatus` ever reached RUNNING /
k8s ever Ready / compose ever healthy.

`Stopping` and `Stopped` share `desiredStatus==stopped` and are **not** separated
by the four axes; the discriminator is a fifth signal — **process liveness /
terminality** (graceful window open vs. absorbing, per principle 3).

```mermaid
stateDiagram-v2
[*] --> Starting
Expand Down Expand Up @@ -112,7 +117,7 @@ unobservable; death `cause` enum; turn-level busy/idle.
the discriminators `(desiredStatus, accepting_work, health, identity_verified)`;
the machine never changes per runtime.
6. **Two predicates, kept apart.** *Dispatch new work* = `state == Running`
(single field). *Doing in-flight work* = `Running ∪ Paused ∪ Stopping`(within
(single field). *Doing in-flight work* = `Running ∪ Paused ∪ Stopping` (within
deadline) — a cordoned (Paused) agent still finishes its current turn / MCP
call. Don't collapse them into one sentence.

Expand All @@ -134,10 +139,10 @@ an ECS-only coincidence.

| canonical | ECS | k8s / GKE | docker-compose |
|---|---|---|---|
| Starting | PROVISIONING / PENDING / **ACTIVATING** (ENI + secret inject) | Pending / ContainerCreating / startupProbe pending | created / starting |
| Starting | **PROVISIONING** (ENI) / PENDING / ACTIVATING (image pull + secret inject) | Pending / ContainerCreating / startupProbe pending | created / starting |
| Running | RUNNING + health OK + desiredStatus RUNNING | Running + readinessProbe True + lease valid | healthy *(healthcheck required)* |
| Paused | RUNNING + health OK + CP/director cordon (`accepting_work=false`) | Ready but cordoned (CP/director) | running + CP/director cordon |
| Unhealthy | RUNNING + healthStatus UNHEALTHY / lease lost *(attribute, not a task state)* | readiness/liveness fail; **Unknown (node lost) → Unhealthy(fenced) + epoch fence**; CrashLoopBackOff | healthcheck fail; `docker pause` (SIGSTOP) → healthcheck stall → Unhealthy |
| Unhealthy | RUNNING + healthStatus UNHEALTHY / lease lost *(attribute, not a task state)* | readiness/liveness fail; **Unknown (node lost) → Unhealthy(fenced) + epoch fence**; CrashLoopBackOff | healthcheck fail; `docker pause` (cgroup freezer) → healthcheck stall → Unhealthy |
| Stopping | desiredStatus STOPPED *(DEACTIVATING only if in a target group / service-discovery; else RUNNING→STOPPING)* | deletionTimestamp != null (Terminating: preStop + grace) | stop requested (stop_grace_period) |
| Stopped | STOPPED + stopCode (enum) | deleted; *preempted* = the reclaim edge | exited |

Expand Down
10 changes: 7 additions & 3 deletions docs/adr/deployment-control-plane.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Vendor-specific terms (ECS/task/ARN/S3/Fargate…) appear **only** inside a
| **Spec** | Desired: `identity + version + scale + admission + runtime + configRef` | `.spec` |
| **admission** | Spec field (`accepting_work`): may this Instance take new work | — (app/CP-level, orthogonal to `Ready`) |
| **Instance.Status** | Observed per Instance: `phase + conditions + …` | Pod `.status` |
| **phase** | Field on an **Instance**'s Status; value is an `AgentState` — **Instance-level only** | Pod `.status.phase` |
| **phase** | Field on an **Instance**'s Status; value is an `AgentState` — **Instance-level only** | ≈ Pod `.status.phase` (ours is *derived* from phase + conditions + probes + deletionTimestamp; see §4 traps) |
| **Deployment.Status** | Observed per Deployment: replica counters `desired/current/ready/available` + Conditions — **not** an `AgentState` | Deployment `.status` |
| **AgentState** | The 6 lifecycle states (ADR-1) | — |
| **Discriminators** | `desiredStatus · accepting_work · health · identity_verified` | ADR-1 |
Expand All @@ -66,7 +66,9 @@ of Spec** (ADR-1). A driver observes native signals and projects them onto:
- **Instance-level** — `Instance.status.phase` is one `AgentState` (ADR-1's
`classify()`), plus `conditions[]` (ready, superseded, …).
- **Deployment-level** — `Deployment.status` is **replica counters**
(`desired / current / ready / available`) + Deployment `conditions[]` (e.g.
(`current / ready / available` counters, plus `desired` — which in k8s is
spec-sourced (`.spec.replicas`), not a `.status` field; service-level in ECS)
+ Deployment `conditions[]` (e.g.
`Available`, `Progressing`). **It is NOT an `AgentState`**: a Deployment with N
Instances (say one `Running`, one `Unhealthy`) has no single lifecycle value —
it has *counts*. (Mirrors k8s exactly: a Pod has a `phase`; a Deployment has
Expand All @@ -82,6 +84,8 @@ here (see §7).
k8s has no single lifecycle enum; it is `phase + conditions + probes +
deletionTimestamp`. Traps to document so k8s intuition doesn't misread us:

- **`Starting`** = k8s `phase=Pending` / `ContainerCreating`, or `Running` before
the `Ready` latch (pre-`identity_verified`) — the 6th state, mapped to pre-`Ready`.
- **`Running` is stricter than k8s** — ours = k8s `phase=Running ∧ Ready=True`.
- **`Paused` has no per-Pod k8s analog**, and is **not** k8s `Deployment.spec.paused` (that is rollout-pause).
- **`Stopping`** = k8s "Terminating" (`deletionTimestamp≠null`), which is not a `.status.phase` value.
Expand Down Expand Up @@ -121,7 +125,7 @@ One idempotent primitive: **`apply(Spec)`** — converge observed toward desired
resumable.
- **dry-run / diff**: every write supports a preview that returns *what would
change* without mutating — a safety valve, and important for agents (look
before leap). See §6 for `dry_run` as a first-class tool parameter.
before you leap). See §6 for `dry_run` as a first-class tool parameter.

## 6. MCP adapter

Expand Down
Loading