Skip to content

fix(read-model): UNKNOWN health with no defined check is Running, not Unhealthy - #32

Merged
brettchien merged 1 commit into
mainfrom
fix/unknown-health-not-unhealthy
Aug 13, 2026
Merged

brettchien merged 1 commit into
mainfrom
fix/unknown-health-not-unhealthy

Conversation

@brettchien

Copy link
Copy Markdown
Contributor

Drafted by Orca (ecs-claude) — flagging authorship (shared GitHub identity).

The bug (visible in a user screenshot): every instance of oab-prod-orca / oab-prod-mira showed Unhealthy (ready 0/1) while actually RUNNING + CONNECTED. Root cause is a read-model mapping, not real health.

Root cause. These services define no ECS container health check, so ECS reports healthStatus = UNKNOWN forever. The driver (agent-lifecycle::EcsDriver::project) faulted on any non-HEALTHY status:

let health = match task.health {
    EcsHealth::Healthy if task.lease_valid => Health::Ok,
    _ => Health::Faulted,   // ← UNKNOWN (no probe) treated the same as UNHEALTHY
};

So RUNNING + verified + Faulted → Unhealthy, for every service without a health check.

Fix — distinguish the two meanings of UNKNOWN (chosen approach A, health-check-aware):

  • No check defined → "no probe", benign → Running (a leased, running instance is Ok).
  • A defined check reporting UNKNOWN → lost signal / node lost → Unhealthy (the fencing the model intended is preserved).

Changes:

  • agent-lifecycle: EcsTask gains health_check_defined; UNKNOWN faults only when a check is defined (else Ok, still gated by lease). +2 driver tests.
  • oabctl: InstanceStatus carries health_check_defined, resolved via DescribeTaskDefinition (cached per task-def ARN; best-effort false on read error, so a missing ecs:DescribeTaskDefinition permission reads benign, not red — the oab-fleet-laptop policy already has this perm).
  • studio-cp: instance_phase plumbs the flag; regression tests for the exact orca/mira case.

Verification: agent-lifecycle tests run green locally (pure crate) — incl. the two new cases and the preserved node_lost fence (12/12). studio-cp / oabctl test execution + clippy left to CI (they statically link the aws-sdk).

Note: touches the ADR-1 6-state canon (health projection). @jellyfish — a canon sanity-check on the UNKNOWN split would be welcome.

🤖 Generated with Claude Code

… Unhealthy

Every OAB service with no ECS container health check showed all instances red
"Unhealthy" (ready 0/desired) even while RUNNING and CONNECTED — because ECS
reports healthStatus=UNKNOWN forever when no check is defined, and the driver
faulted on any non-HEALTHY status.

Distinguish the two meanings of UNKNOWN:
- no health check defined -> "no probe", benign -> Running (a leased, running
  instance is Ok).
- a *defined* check reporting UNKNOWN -> lost signal / node lost -> Unhealthy
  (the fencing the model intended is preserved).

- agent-lifecycle: EcsTask gains health_check_defined; UNKNOWN faults only when
  a check is defined (else Ok, still gated by lease). Two new driver tests.
- oabctl: InstanceStatus carries health_check_defined, resolved via
  DescribeTaskDefinition (cached per task-def ARN; best-effort false on error so
  a permission gap reads benign, not red).
- studio-cp: instance_phase plumbs the flag; regression tests for the exact
  orca/mira case (RUNNING + UNKNOWN + no check -> Running).

Verified: agent-lifecycle tests run green (incl. the two new cases + the
node-lost fence). studio-cp/oabctl test execution + clippy left to CI.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@brettchien
brettchien merged commit 7c506e8 into main Aug 13, 2026
2 checks passed
@brettchien
brettchien deleted the fix/unknown-health-not-unhealthy branch August 13, 2026 16:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant