Skip to content

feat(cli): health-aware list/stop, startup failure diagnostics, port retry, doctor --probe#820

Open
AbirAbbas wants to merge 4 commits into
mainfrom
feat/cli-health-lifecycle
Open

feat(cli): health-aware list/stop, startup failure diagnostics, port retry, doctor --probe#820
AbirAbbas wants to merge 4 commits into
mainfrom
feat/cli-health-lifecycle

Conversation

@AbirAbbas

Copy link
Copy Markdown
Contributor

Problem

A live field test of the sub-harness flow hit four operational failures in one session, all of the same class: the platform knew something and didn't say it.

  1. af stop silently killed in-flight work. A graceful stop terminated a node that had a long-running execution in progress, with no warning — even though running executions are queryable from the control plane.
  2. af run startup failures were opaque and non-recoverable. When a node was assigned a port that looked free but lost the bind race (a just-stopped node's port lingering under mirrored networking), the SDK exited with the strict-port "assigned port unavailable" error and nothing retried. The CLI only printed "did not become ready within 30s"; the real traceback lived in a separate af logs.
  3. af list trusted the local registry blindly. Registry status is a claim, not a fact — a node the registry calls "running" can be dead on the control plane (or vice versa) and nothing reconciled the two views.
  4. af doctor reported providers as healthy from a PATH check alone. A provider CLI whose binary exists can still return instant empty completions (broken auth, model outage); doctor never actually probed provider health.

What changed

Four focused commits, one per failure class:

  • feat(cli): warn before af stop interrupts running executions — before signalling the process, af stop <node> queries the control plane for executions still running on the node. Found + TTY → list count/ids/age and prompt to confirm; found + non-TTY → warn and proceed (prior behavior); no executions or --force → stop immediately; control plane unreachable → note it and proceed (best-effort). Warnings go to stderr so --json stdout stays a clean envelope.
  • feat(cli): surface af run startup logs and retry once on strict-port bind conflict — on any startup failure the last ~15 lines of the node's log are printed inline plus a Full logs: af logs <node> pointer. When the failure is the SDK's strict-port bind conflict (detected from the node log), the run path retries exactly once on a fresh port (the failed port is reserved first so it is never reused), logging Port <p> unavailable, retrying on a fresh port. The port-alloc/start/wait section is refactored into attemptStart + startWithPortRetry so the retry decision and port-change logic are unit-testable without the real health poll.
  • feat(cli): reconcile af list registry status with control-plane healthaf list gains a HEALTH column that fetches GET /api/v1/nodes?show_all=true (so inactive nodes are included) and reconciles it with each node's registry status. Agreement shows plain health; disagreement is marked (mismatch) with an explanatory footer; a node absent from the control plane while the registry calls it running reads not on control plane (mismatch); an unreachable control plane yields unknown (control plane unreachable) and never errors. The same health / health_discrepancy fields are added to af list --json.
  • feat(cli): add af doctor --probe to smoke-test detected provider CLIs — new opt-in --probe flag runs a minimal one-shot prompt against each detected provider CLI (claude -p, codex exec, gemini -p, opencode run) with a per-provider 60s timeout, classifying the result as ok (non-empty), empty (exit 0 but no completion — the silently-broken case a PATH check misses), error (non-zero exit, stderr head captured), or timeout. Probes run only for providers doctor already detects; without --probe the command is unchanged. Help text and output note that a probe consumes a trivial amount of provider quota.

Validation Contract

Tests are derived from behavior, not implementation. Each contract item maps to named tests:

Contract item Tests
Stop with running executions: non-TTY warns with count; --force does not warn (and never queries); no executions is silent; unreachable proceeds; interactive decline aborts TestConfirmStop_NonInteractive_WarnsWithCount, TestConfirmStop_Force_NoWarnNoQuery, TestConfirmStop_NoRunningExecutions_Silent, TestConfirmStop_ControlPlaneUnreachable_Proceeds, TestConfirmStop_Interactive_DeclineAborts, TestConfirmStop_Interactive_AcceptProceeds, TestQueryRunningExecutions_ParsesEnvelope, TestQueryRunningExecutions_EmptyServerURL, TestReadAffirmative, TestFormatExecutionAge
Startup-failure path prints the tail of the node's log and the af logs pointer TestPrintStartupFailureDiagnostics, TestReadLogTailLines
Strict-port failure triggers exactly one retry on a different port; non-conflict failures do not retry; the failed port is excluded TestStartWithPortRetry_RetriesOnceOnPortConflict, TestStartWithPortRetry_NoRetryOnNonConflictFailure, TestStartWithPortRetry_SuccessRunsOnce, TestStartWithPortRetry_NoDistinctPortDoesNotRetry, TestFreshRetryPort_ExcludesFailedPort, TestLogIndicatesPortConflict
af list merges the registry with a mocked /api/v1/nodes: agreement renders plainly, discrepancy is flagged, unreachable yields unknown without erroring TestReconcileHealth, TestResolveNodeHealth_Merge, TestResolveNodeHealth_Unreachable, TestFetchControlPlaneNodes_Reachable, TestFetchControlPlaneNodes_Unreachable
Doctor probe classification of ok/empty/error/timeout from (exit code, stdout, timed-out) tuples; probes only detected providers; af doctor without --probe performs no probe TestClassifyProbe, TestProbeHarnessProvider_RealProcesses, TestRunHarnessProbes_SkipsUndetected, TestDoctorCommand_NoProbeByDefault

Test results

  • npm ci && npm run build (control-plane UI): pass
  • GOFLAGS=-buildvcs=false go build ./...: pass
  • go vet ./...: pass
  • go test ./internal/cli/... ./internal/core/services/...: pass (all new spec tests green)
  • gofmt -l on changed files: empty
  • golangci-lint run: changed files introduce zero new findings (errcheck counts unchanged vs. base for doctor.go and agent_service.go)

One unrelated test, internal/server/TestStartAndStopCoverAdditionalBranches, fails locally in this sandbox because the admin gRPC listener cannot bind here; it fails identically on the base commit and this branch touches no code in internal/server, so it is a pre-existing environment artifact, not a regression.

🤖 Generated with Claude Code

AbirAbbas and others added 4 commits July 22, 2026 15:02
`af stop <node>` gracefully killed nodes with long-running executions in
flight with no warning. Running executions are queryable from the control
plane, so query them before signalling the process:

- running executions found + TTY   → list count/ids/age and prompt to confirm
- running executions found + no TTY → warn (count/ids/age) and proceed
- --force                           → stop immediately, no query/warning/prompt
- control plane unreachable         → note it and proceed (best-effort, as before)

The check runs only once we have confirmed the process is genuinely ours and
alive, so a dead/stale node still reconciles cleanly without spurious queries.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bind conflict

Two operational failures in the `af run` path made a failed start opaque and
non-recoverable:

- On any startup failure the CLI only printed "did not become ready within 30s";
  the real traceback/exit reason lived in the node log, reachable only via a
  separate `af logs`. Now the last ~15 lines of the node's log are printed
  inline on failure, plus a "Full logs: af logs <node>" pointer.

- When a node was assigned a port that looked free but lost the bind race (a
  just-stopped node's port lingering under mirrored networking), the SDK exited
  with AGENTFIELD_STRICT_PORT "assigned port N is unavailable" and nothing
  retried. The run path now detects that strict-port exit from the node log and
  retries exactly once on a fresh port (the failed port is reserved first so it
  is never reused), logging "Port <p> unavailable, retrying on a fresh port".

The port-alloc/start/wait section is refactored into attemptStart +
startWithPortRetry so the retry decision and port-change logic are unit-testable
without the real health-poll.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`af list` showed only the local registry's view, which is a claim, not a fact:
a node the registry calls "running" can be dead on the control plane (or a
registry "stopped" node can still be live). Add a HEALTH column that fetches the
control plane's node view (GET /api/v1/nodes?show_all=true, so inactive nodes
are included) and reconciles it with each node's registry status:

- statuses agree           → plain health (e.g. "active")
- statuses disagree        → health + "(mismatch)" and a footer explaining it
- node absent from CP       → "not on control plane (mismatch)" when registry running
- control plane unreachable → "unknown (control plane unreachable)", never an error

The same health/health_discrepancy fields are added to `af list --json` for the
agent-driven flow. A missing control plane never fails the command.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`af doctor` reported a provider as available whenever its binary was on PATH,
but a present binary can still return instant empty completions (broken auth,
model outage) — the doctor called it healthy while every real call failed.

Add an opt-in `--probe` flag that runs a minimal one-shot prompt against each
DETECTED provider CLI (claude -p, codex exec, gemini -p, opencode run) with a
per-provider 60s timeout and classifies the result:

- ok      → non-empty completion
- empty   → exit 0 but no output (the silently-broken case a PATH check misses)
- error   → non-zero exit (stderr head captured)
- timeout → no response within the timeout

Probes run only for providers doctor already detects; without --probe the
command is unchanged. Output and help note that a probe consumes a trivial
amount of provider quota. The classifier is a pure function so ok/empty/error/
timeout are table-tested from (exit code, stdout, timed-out) tuples.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@AbirAbbas
AbirAbbas requested a review from a team as a code owner July 22, 2026 20:15
@github-actions

Copy link
Copy Markdown
Contributor

📊 Coverage gate

Thresholds from .coverage-gate.toml: per-surface ≥ 84%, aggregate ≥ 85%, max per-surface regression ≤ 1.0 pp, max aggregate regression ≤ 0.50 pp.

Surface Current Baseline Δ
control-plane 86.80% 87.40% ↓ -0.60 pp 🟡
sdk-go 92.50% 92.00% ↑ +0.50 pp 🟢
sdk-python 93.82% 93.73% ↑ +0.09 pp 🟢
sdk-typescript 91.05% 90.42% ↑ +0.63 pp 🟢
web-ui 84.76% 84.79% ↓ -0.03 pp 🟡
aggregate 85.51% 85.75% ↓ -0.24 pp 🟡

✅ Gate passed

No surface regressed past the allowed threshold and the aggregate stayed above the floor.

@github-actions

Copy link
Copy Markdown
Contributor

📐 Patch coverage gate

Threshold: 80% on lines this PR touches vs origin/main (from .coverage-gate.toml:thresholds.min_patch).

Surface Touched lines Patch coverage Status
control-plane 399 82.00%
sdk-go 0 ➖ no changes
sdk-python 0 ➖ no changes
sdk-typescript 0 ➖ no changes
web-ui 0 ➖ no changes

✅ Patch gate passed

Every surface whose lines were touched by this PR has patch coverage at or above the threshold.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant