Repository navigation
fix(read-model): UNKNOWN health with no defined check is Running, not Unhealthy - #32
Merged
Merged
Conversation
… Unhealthy Every OAB service with no ECS container health check showed all instances red "Unhealthy" (ready 0/desired) even while RUNNING and CONNECTED — because ECS reports healthStatus=UNKNOWN forever when no check is defined, and the driver faulted on any non-HEALTHY status. Distinguish the two meanings of UNKNOWN: - no health check defined -> "no probe", benign -> Running (a leased, running instance is Ok). - a *defined* check reporting UNKNOWN -> lost signal / node lost -> Unhealthy (the fencing the model intended is preserved). - agent-lifecycle: EcsTask gains health_check_defined; UNKNOWN faults only when a check is defined (else Ok, still gated by lease). Two new driver tests. - oabctl: InstanceStatus carries health_check_defined, resolved via DescribeTaskDefinition (cached per task-def ARN; best-effort false on error so a permission gap reads benign, not red). - studio-cp: instance_phase plumbs the flag; regression tests for the exact orca/mira case (RUNNING + UNKNOWN + no check -> Running). Verified: agent-lifecycle tests run green (incl. the two new cases + the node-lost fence). studio-cp/oabctl test execution + clippy left to CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Drafted by Orca (
ecs-claude) — flagging authorship (shared GitHub identity).The bug (visible in a user screenshot): every instance of
oab-prod-orca/oab-prod-mirashowedUnhealthy(ready 0/1) while actuallyRUNNING+CONNECTED. Root cause is a read-model mapping, not real health.Root cause. These services define no ECS container health check, so ECS reports
healthStatus = UNKNOWNforever. The driver (agent-lifecycle::EcsDriver::project) faulted on any non-HEALTHYstatus:So
RUNNING + verified + Faulted→Unhealthy, for every service without a health check.Fix — distinguish the two meanings of
UNKNOWN(chosen approach A, health-check-aware):Ok).UNKNOWN→ lost signal / node lost → Unhealthy (the fencing the model intended is preserved).Changes:
EcsTaskgainshealth_check_defined;UNKNOWNfaults only when a check is defined (elseOk, still gated by lease). +2 driver tests.InstanceStatuscarrieshealth_check_defined, resolved viaDescribeTaskDefinition(cached per task-def ARN; best-effortfalseon read error, so a missingecs:DescribeTaskDefinitionpermission reads benign, not red — theoab-fleet-laptoppolicy already has this perm).instance_phaseplumbs the flag; regression tests for the exact orca/mira case.Verification:
agent-lifecycletests run green locally (pure crate) — incl. the two new cases and the preservednode_lostfence (12/12).studio-cp/oabctltest execution + clippy left to CI (they statically link the aws-sdk).Note: touches the ADR-1 6-state canon (health projection). @jellyfish — a canon sanity-check on the
UNKNOWNsplit would be welcome.🤖 Generated with Claude Code