feat(cli): health-aware list/stop, startup failure diagnostics, port retry, doctor --probe#820
Open
AbirAbbas wants to merge 4 commits into
Open
feat(cli): health-aware list/stop, startup failure diagnostics, port retry, doctor --probe#820AbirAbbas wants to merge 4 commits into
AbirAbbas wants to merge 4 commits into
Conversation
`af stop <node>` gracefully killed nodes with long-running executions in flight with no warning. Running executions are queryable from the control plane, so query them before signalling the process: - running executions found + TTY → list count/ids/age and prompt to confirm - running executions found + no TTY → warn (count/ids/age) and proceed - --force → stop immediately, no query/warning/prompt - control plane unreachable → note it and proceed (best-effort, as before) The check runs only once we have confirmed the process is genuinely ours and alive, so a dead/stale node still reconciles cleanly without spurious queries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bind conflict Two operational failures in the `af run` path made a failed start opaque and non-recoverable: - On any startup failure the CLI only printed "did not become ready within 30s"; the real traceback/exit reason lived in the node log, reachable only via a separate `af logs`. Now the last ~15 lines of the node's log are printed inline on failure, plus a "Full logs: af logs <node>" pointer. - When a node was assigned a port that looked free but lost the bind race (a just-stopped node's port lingering under mirrored networking), the SDK exited with AGENTFIELD_STRICT_PORT "assigned port N is unavailable" and nothing retried. The run path now detects that strict-port exit from the node log and retries exactly once on a fresh port (the failed port is reserved first so it is never reused), logging "Port <p> unavailable, retrying on a fresh port". The port-alloc/start/wait section is refactored into attemptStart + startWithPortRetry so the retry decision and port-change logic are unit-testable without the real health-poll. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`af list` showed only the local registry's view, which is a claim, not a fact: a node the registry calls "running" can be dead on the control plane (or a registry "stopped" node can still be live). Add a HEALTH column that fetches the control plane's node view (GET /api/v1/nodes?show_all=true, so inactive nodes are included) and reconciles it with each node's registry status: - statuses agree → plain health (e.g. "active") - statuses disagree → health + "(mismatch)" and a footer explaining it - node absent from CP → "not on control plane (mismatch)" when registry running - control plane unreachable → "unknown (control plane unreachable)", never an error The same health/health_discrepancy fields are added to `af list --json` for the agent-driven flow. A missing control plane never fails the command. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`af doctor` reported a provider as available whenever its binary was on PATH, but a present binary can still return instant empty completions (broken auth, model outage) — the doctor called it healthy while every real call failed. Add an opt-in `--probe` flag that runs a minimal one-shot prompt against each DETECTED provider CLI (claude -p, codex exec, gemini -p, opencode run) with a per-provider 60s timeout and classifies the result: - ok → non-empty completion - empty → exit 0 but no output (the silently-broken case a PATH check misses) - error → non-zero exit (stderr head captured) - timeout → no response within the timeout Probes run only for providers doctor already detects; without --probe the command is unchanged. Output and help note that a probe consumes a trivial amount of provider quota. The classifier is a pure function so ok/empty/error/ timeout are table-tested from (exit code, stdout, timed-out) tuples. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
📊 Coverage gateThresholds from
✅ Gate passedNo surface regressed past the allowed threshold and the aggregate stayed above the floor. |
Contributor
📐 Patch coverage gateThreshold: 80% on lines this PR touches vs
✅ Patch gate passedEvery surface whose lines were touched by this PR has patch coverage at or above the threshold. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A live field test of the sub-harness flow hit four operational failures in one session, all of the same class: the platform knew something and didn't say it.
af stopsilently killed in-flight work. A graceful stop terminated a node that had a long-running execution in progress, with no warning — even though running executions are queryable from the control plane.af runstartup failures were opaque and non-recoverable. When a node was assigned a port that looked free but lost the bind race (a just-stopped node's port lingering under mirrored networking), the SDK exited with the strict-port "assigned port unavailable" error and nothing retried. The CLI only printed "did not become ready within 30s"; the real traceback lived in a separateaf logs.af listtrusted the local registry blindly. Registry status is a claim, not a fact — a node the registry calls "running" can be dead on the control plane (or vice versa) and nothing reconciled the two views.af doctorreported providers as healthy from a PATH check alone. A provider CLI whose binary exists can still return instant empty completions (broken auth, model outage); doctor never actually probed provider health.What changed
Four focused commits, one per failure class:
feat(cli): warn before af stop interrupts running executions— before signalling the process,af stop <node>queries the control plane for executions still running on the node. Found + TTY → list count/ids/age and prompt to confirm; found + non-TTY → warn and proceed (prior behavior); no executions or--force→ stop immediately; control plane unreachable → note it and proceed (best-effort). Warnings go to stderr so--jsonstdout stays a clean envelope.feat(cli): surface af run startup logs and retry once on strict-port bind conflict— on any startup failure the last ~15 lines of the node's log are printed inline plus aFull logs: af logs <node>pointer. When the failure is the SDK's strict-port bind conflict (detected from the node log), the run path retries exactly once on a fresh port (the failed port is reserved first so it is never reused), loggingPort <p> unavailable, retrying on a fresh port. The port-alloc/start/wait section is refactored intoattemptStart+startWithPortRetryso the retry decision and port-change logic are unit-testable without the real health poll.feat(cli): reconcile af list registry status with control-plane health—af listgains a HEALTH column that fetchesGET /api/v1/nodes?show_all=true(so inactive nodes are included) and reconciles it with each node's registry status. Agreement shows plain health; disagreement is marked(mismatch)with an explanatory footer; a node absent from the control plane while the registry calls it running readsnot on control plane (mismatch); an unreachable control plane yieldsunknown (control plane unreachable)and never errors. The samehealth/health_discrepancyfields are added toaf list --json.feat(cli): add af doctor --probe to smoke-test detected provider CLIs— new opt-in--probeflag runs a minimal one-shot prompt against each detected provider CLI (claude -p,codex exec,gemini -p,opencode run) with a per-provider 60s timeout, classifying the result asok(non-empty),empty(exit 0 but no completion — the silently-broken case a PATH check misses),error(non-zero exit, stderr head captured), ortimeout. Probes run only for providers doctor already detects; without--probethe command is unchanged. Help text and output note that a probe consumes a trivial amount of provider quota.Validation Contract
Tests are derived from behavior, not implementation. Each contract item maps to named tests:
--forcedoes not warn (and never queries); no executions is silent; unreachable proceeds; interactive decline abortsTestConfirmStop_NonInteractive_WarnsWithCount,TestConfirmStop_Force_NoWarnNoQuery,TestConfirmStop_NoRunningExecutions_Silent,TestConfirmStop_ControlPlaneUnreachable_Proceeds,TestConfirmStop_Interactive_DeclineAborts,TestConfirmStop_Interactive_AcceptProceeds,TestQueryRunningExecutions_ParsesEnvelope,TestQueryRunningExecutions_EmptyServerURL,TestReadAffirmative,TestFormatExecutionAgeaf logspointerTestPrintStartupFailureDiagnostics,TestReadLogTailLinesTestStartWithPortRetry_RetriesOnceOnPortConflict,TestStartWithPortRetry_NoRetryOnNonConflictFailure,TestStartWithPortRetry_SuccessRunsOnce,TestStartWithPortRetry_NoDistinctPortDoesNotRetry,TestFreshRetryPort_ExcludesFailedPort,TestLogIndicatesPortConflictaf listmerges the registry with a mocked/api/v1/nodes: agreement renders plainly, discrepancy is flagged, unreachable yieldsunknownwithout erroringTestReconcileHealth,TestResolveNodeHealth_Merge,TestResolveNodeHealth_Unreachable,TestFetchControlPlaneNodes_Reachable,TestFetchControlPlaneNodes_Unreachableaf doctorwithout--probeperforms no probeTestClassifyProbe,TestProbeHarnessProvider_RealProcesses,TestRunHarnessProbes_SkipsUndetected,TestDoctorCommand_NoProbeByDefaultTest results
npm ci && npm run build(control-plane UI): passGOFLAGS=-buildvcs=false go build ./...: passgo vet ./...: passgo test ./internal/cli/... ./internal/core/services/...: pass (all new spec tests green)gofmt -lon changed files: emptygolangci-lint run: changed files introduce zero new findings (errcheck counts unchanged vs. base fordoctor.goandagent_service.go)One unrelated test,
internal/server/TestStartAndStopCoverAdditionalBranches, fails locally in this sandbox because the admin gRPC listener cannot bind here; it fails identically on the base commit and this branch touches no code ininternal/server, so it is a pre-existing environment artifact, not a regression.🤖 Generated with Claude Code