feat(quorum): deep-probe gate — tell a quota-dead slot from a healthy one (P1)

CHANGELOG.md

@@ -27,11 +27,14 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 ## [Unreleased]
 
 ### Added
+- feat(quorum): **deep-probe gate — preflight can now tell a quota-dead slot from a healthy one (P1 of #293, the last piece).** L1 (`--version`) proves a binary exists and L2 (`/models`) treats 401/403 as *reachable*, so neither notices a slot that is authenticated, reachable, and out of quota. Demonstrated live on 2026-08-08: preflight reported **7/7 available** while `claude-z-ai` (429, weekly limit) and `antigravity-1` (quota, 74h to reset) could not answer at all — a panel promising full quorum that then produces two UNAVAILs mid-round. `deepProbeSlot()` runs the slot's configured `deep_probe` and is deliberately **lopsided**: it downgrades *only* on a fast, explicit auth/quota signal in the slot's own output; a timeout, spawn error, or unrecognised response is `INCONCLUSIVE` and never downgrades. That asymmetry is the point — the opposite error, killing a slow-but-alive slot, is the one this codebase has already paid for three times (STALL-TIMEOUT-01..03). It runs only when the panel is already **degraded** *and* an explicit `--budget-ms` covers it (a naive auto-enable previously blew the `<8s` liveness budget with `spawnSync ETIMEDOUT`), or on demand via `--deep-probe`. Live-verified end to end: both dead slots are now downgraded with their real reasons and the gate reports `DEGRADED — 5 available`, where the cheap budget-less path still finishes in ~3s having skipped the probe entirely.
 - feat(quorum): **preflight now validates the roster and decides the quorum gate (P3 + P5 of #293, extracted fresh).** `validateProviders()` reports config faults that would otherwise surface mid-quorum as an opaque UNAVAIL — the two this repo has actually shipped are a slot with neither `cli` nor `mainTool` (spawn receives `null`, taking the whole quorum offline, #197) and an `args_template` with no `{prompt}` placeholder (the CLI runs with flags and no question). Also errors on duplicate names (silently last-wins everywhere downstream), missing/unknown `type`, and http slots without a base URL or any auth; warns on non-positive timeout fields and on third-party-routed slots with no `deep_probe`. Non-fatal by design: it reports on every `--all` and never blocks the preflight. `computeQuorumGate()` turns *"available < max_quorum_size → BLOCK unless `--force-quorum`"* from prose in `commands/nf/quorum.md` into a machine field — `{available_count, quorum_met, degraded, blocked, waiver_required, waiver_used, gate_reason}` — because a gate that depends on a model reading a paragraph correctly is not a gate. It keeps **degraded** and **blocked** distinct (below `max_quorum_size` but above `min_live_voters` still dispatches), and a `--force-quorum` waiver unblocks while being *recorded* rather than silently making the panel look healthy. 11 tests, each of the three core behaviours mutation-proven.
 - feat(release): **`Align dist-tags` — a manually dispatchable workflow that repairs `@next` drift without a one-time password.** `publish.yml` aligns `@next` automatically after every publish since #391, but that cannot help a release published *before* that fix, or an alignment that failed transiently. Doing it by hand needs `npm dist-tag add`, which on a 2FA-protected account fails with `EOTP` and demands an authenticator code — so it cannot be scripted, and in practice it simply does not get done (0.44.3 shipped with `next` still on 0.44.2). The `NPM_TOKEN` secret is 2FA-exempt, which is what automation tokens are for, so CI can do unattended what an interactive human cannot. Scope is deliberately narrow: it moves a dist-tag and cannot publish. It refuses a prerelease target (a prerelease holding `@next` would silently install for everyone under the alias policy), and — unlike `publish.yml`'s verify step, which must never fail a job whose package is already out — this one **fails** if the tag did not move, because a green manual run that changed nothing is the exact outcome worth preventing. Each `run:` block was executed against a stubbed `npm view` across all four branches (aligned / not-moved / unreadable / prerelease) rather than reviewed by eye.
 - feat(quorum): **timeout records now say which config produced the window (I5).** `resolveTtfbTimeout` / `resolveInterChunkCeiling` each compute a `source` (`per-slot` / `default` / `idle-budget` / `legacy-stall`) and `recordTelemetry` was dropping both it and the resolved values, so a TIMEOUT in `quorum-rounds-*.jsonl` could be seen but never attributed — the number alone never answers "did *my* config do this?", since today's per-slot value can be tomorrow's default. Records now carry `ttfb_timeout_ms`, `ttfb_source`, `inter_chunk_ceiling_ms` and `inter_chunk_source`, always present (null when unknown) so the schema stays stable for analysis. Passed as a trailing options object rather than a 15th positional parameter. 4 tests drive the **real child** and read the actual log, so the fields are proven to survive the whole dispatch path; mutation-proven. Accepted as improvement I5 by the 2026-08-07 quorum.
 
 ### Fixed
+- fix(quorum): **the shared error classifier missed `"quota reached"`** — it only knew `"quota exceeded"`, so antigravity's `"Individual quota reached. Resets in 74h38m39s"` was **not classified at all** and the slot was reported healthy with three days of cooldown left. Added `quota reached`, `limit exhausted` and `limit reached`, checked against real messages and against ordinary prose a model might emit (`"I reached the conclusion that the limit is fine"` must not match).
+- fix(provider-status): `main()` ran unconditionally on `require()`, so importing the module for its classifier would execute the entire probe sweep — the #198 side-effect-on-require class. Guarded with `require.main === module` and the classifier is now exported, so `quorum-preflight` shares one pattern list instead of forking the definition that decides whether a slot is declared dead.
 - fix(quorum): **a quota-dead slot is now held out until it actually resets, instead of being retried every 5 minutes for days (QUOTA-RESET-01).** Providers state their reset window, but in shapes `parseAvailabilityHint` did not match — `antigravity-1` says `"Resets in 95h26m37s."` and `claude-z-ai` says `"...will reset at 2026-08-09 10:01:44"`, and neither the `"in N hours"` nor the `"until <Month Day>"` pattern covers them. Both fell through to the 5-minute default, so a slot that was dead for **four days** burned a dispatch in every quorum round, dragging the live-voter count toward the `min_live_voters` floor. Two new shapes are parsed: a compact duration (`95h26m37s`, `32h54m`, `90s`) and an absolute `reset at YYYY-MM-DD HH:MM:SS`. Deliberately conservative in the dangerous direction — a plain `STALL`/`IDLE_TIMEOUT` still gets the 5-minute default, because over-matching would retire a healthy slot for days on one bad round. The compact parser is written as locate-token-then-decompose: a single pattern of all-optional unit groups matches the empty string, "succeeding" with every capture undefined and silently falling through to the very default it exists to replace (which is what the first version did). 6 tests, mutation-proven in both directions. This is P4 of the long-open #293, extracted fresh.
 - fix(install): **a stale backup no longer clobbers live slot config on every install.** When `bin/providers.json` ships empty (it always does), install reconstructs the user's `providers.json` from `~/.claude.json` and overlays what it can recover for each slot, gathering from four candidates documented as being "in order of freshness" — live `nf-bin/` first, then two `nf-local-patches/` backups, then the legacy live path. The gather loop called `map.set(name, entry)` unconditionally, so the **last** candidate won: the exact inversion of the stated order. On the machine that surfaced it, a `nf-local-patches/nf/bin/providers.json` dated **2026-05-13** overrode the live entry for 7 slots on every single install, silently reverting per-slot tuning (`ttfb_timeout_ms`, `inter_chunk_ceiling_ms`, `idle_timeout_ms`, `model`, `env`). It was found because freshly-set `ttfb_timeout_ms` values kept vanishing after `install.js --claude --global`, which also explains earlier "mystery" resets of `stall_timeout_ms`. First writer now wins, so backups do what they exist for — recovering slots the live file has **lost** — and never override it. 4 tests in `bin/install-providers-precedence.test.cjs` execute the extracted gather loop against stubbed candidates; mutation-proven, and verified against a real install that now preserves every tuned field.
 

bin/deep-probe-gate.test.cjs

@@ -0,0 +1,100 @@
+#!/usr/bin/env node
+'use strict';
+// bin/deep-probe-gate.test.cjs — P1 of #293, extracted fresh.
+//
+// L1 (`--version`) proves a binary exists; L2 (`/models`) treats 401/403 as reachable.
+// Neither can tell a quota-dead slot from a healthy one. Demonstrated live on
+// 2026-08-08: preflight reported 7/7 available while claude-z-ai (429, weekly limit)
+// and antigravity-1 (quota, 74h to reset) could not answer at all.
+//
+// The rule is deliberately LOPSIDED, because the opposite error is the one this codebase
+// has already paid for three times (STALL-TIMEOUT-01..03): killing a slow-but-alive slot.
+//   downgrade ONLY on a fast, explicit auth/quota signal from the slot's own output.
+//   timeout / spawn error / unrecognised output = INCONCLUSIVE, never a downgrade.
+
+const { test } = require('node:test');
+const assert = require('node:assert');
+const fs = require('node:fs');
+const os = require('node:os');
+const path = require('node:path');
+const { deepProbeSlot, DOWNGRADE_STATUSES } = require('./quorum-preflight.cjs');
+const { classifyOutput } = require('./provider-status.cjs');
+
+const TMP = [];
+process.on('exit', () => { for (const d of TMP) { try { fs.rmSync(d, { recursive: true, force: true }); } catch (_) {} } });
+
+function slotEmitting(body) {
+  const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'nf-dp-'));
+  TMP.push(dir);
+  const cli = path.join(dir, 'fake.cjs');
+  fs.writeFileSync(cli, body, 'utf8');
+  return { name: 't', type: 'subprocess', mainTool: 'node', cli: 'node',
+           args_template: [cli, '{prompt}'], deep_probe: { prompt: 'hi', timeout_ms: 30000 } };
+}
+
+test('DPG-1: an explicit quota signal downgrades', async () => {
+  const p = slotEmitting("process.stdout.write('API Error: Request rejected (429) Weekly Limit Exhausted\\n');process.exit(1);");
+  const r = await deepProbeSlot(p, 10000);
+  assert.strictEqual(r.downgrade, true);
+  assert.strictEqual(r.status, 'QUOTA_EXCEEDED');
+});
+
+test('DPG-2: a SLOW but alive slot is never downgraded — the whole point', async () => {
+  // The failure this must not reproduce. A 30s thinker probed with a 1s budget is
+  // inconclusive, not dead.
+  const p = slotEmitting("setTimeout(()=>process.stdout.write('PROBE_OK\\n'), 30000);");
+  const r = await deepProbeSlot(p, 1200);
+  assert.strictEqual(r.downgrade, false, 'a timeout must never downgrade a slot');
+  assert.strictEqual(r.status, 'INCONCLUSIVE');
+  assert.match(r.reason, /slow is not dead/);
+});
+
+test('DPG-3: a spawn failure is inconclusive, not a downgrade', async () => {
+  const p = { name: 't', type: 'subprocess', mainTool: 'definitely-not-a-real-binary-xyz',
+              cli: '/nonexistent/xyz', args_template: ['{prompt}'], deep_probe: { prompt: 'hi' } };
+  const r = await deepProbeSlot(p, 5000);
+  assert.strictEqual(r.downgrade, false, 'an unrunnable probe says nothing about slot health');
+});
+
+test('DPG-4: a healthy slot stays available', async () => {
+  const r = await deepProbeSlot(slotEmitting("process.stdout.write('PROBE_OK\\n');"), 10000);
+  assert.strictEqual(r.downgrade, false);
+  assert.strictEqual(r.status, 'OK');
+});
+
+test('DPG-5: a slot with no deep_probe is skipped, not guessed at', async () => {
+  const p = slotEmitting("process.stdout.write('PROBE_OK\\n');");
+  delete p.deep_probe;
+  const r = await deepProbeSlot(p, 5000);
+  assert.strictEqual(r.probed, false);
+});
+
+test('DPG-6: the downgrade reason is the MATCHING line, not trailing noise', async () => {
+  // First live run blamed "SessionEnd hook failed" for what was a 429, because it
+  // quoted the last line of output. A wrong cause is worse than no cause.
+  const p = slotEmitting(
+    "process.stdout.write('API Error: Request rejected (429) Limit Exhausted\\n');" +
+    "process.stdout.write('SessionEnd hook failed: Hook cancelled\\n');process.exit(1);");
+  const r = await deepProbeSlot(p, 10000);
+  assert.match(r.reason, /429/);
+  assert.doesNotMatch(r.reason, /SessionEnd/);
+});
+
+test('DPG-7: only auth/quota classes may downgrade', () => {
+  // A format error or an unknown response must not retire a slot.
+  assert.ok(DOWNGRADE_STATUSES.has('QUOTA_EXCEEDED'));
+  assert.ok(DOWNGRADE_STATUSES.has('AUTH_ERROR'));
+  assert.ok(!DOWNGRADE_STATUSES.has('FORMAT_ERROR'), 'a malformed request is not a dead slot');
+  assert.ok(!DOWNGRADE_STATUSES.has('TIMEOUT'));
+});
+
+test('DPG-8: the classifier recognises the real messages that motivated this', () => {
+  // 'quota exceeded' alone missed antigravity's "Individual quota reached" — the slot
+  // was reported healthy with 74h of cooldown remaining.
+  assert.strictEqual(classifyOutput('Error: Individual quota reached. Resets in 74h38m39s.'), 'QUOTA_EXCEEDED');
+  assert.strictEqual(classifyOutput('Weekly/Monthly Limit Exhausted'), 'QUOTA_EXCEEDED');
+  assert.strictEqual(classifyOutput('Failed to refresh token: 401 Unauthorized'), 'AUTH_ERROR');
+  // And must not fire on ordinary prose a model might emit.
+  assert.strictEqual(classifyOutput('I reached the conclusion that the limit is fine'), null);
+  assert.strictEqual(classifyOutput('PROBE_OK'), null);
+});

bin/provider-status.cjs

@@ -25,7 +25,10 @@ const { resolveArgsTemplate } = require('./provider-arg-templates.cjs');
 // ─── Error pattern classification ────────────────────────────────────────────
 
 const ERROR_PATTERNS = [
-  { status: 'QUOTA_EXCEEDED', patterns: ['quota exceeded', 'rate_limit', 'credit limit exceeded', 'too many requests', 'resource_exhausted', '429'] },
+  // 'quota reached' / 'limit exhausted' / 'limit reached' added 2026-08-08 from live
+  // output: antigravity says "Individual quota reached", which 'quota exceeded' misses
+  // entirely — the slot was reported healthy while it had 74h of cooldown left.
+  { status: 'QUOTA_EXCEEDED', patterns: ['quota exceeded', 'quota reached', 'limit exhausted', 'limit reached', 'rate_limit', 'credit limit exceeded', 'too many requests', 'resource_exhausted', '429'] },
   { status: 'AUTH_ERROR',     patterns: ['unauthorized', 'refresh_token_reused', 'token_expired', 'invalid api key', 'invalid_api_key', '401', '403', 'please try signing in again'] },
   { status: 'FORMAT_ERROR',   patterns: ['invalid_request_error', 'request validation error', '400'] },
   { status: 'NO_CREDITS',    patterns: ['credit limit', '402', 'payment required'] },
@@ -323,7 +326,16 @@ async function main() {
   }
 }
 
-main().catch(err => {
+// Guard main() so require()ing this module for its classifier does not run the whole
+// probe sweep — the #198 class the no-side-effects-on-require gate exists for.
+if (require.main === module) {
+  main().catch(err => {
   process.stderr.write(`[provider-status] ${err.message}\n`);
   process.exit(1);
 });
+}
+
+// Exported so bin/quorum-preflight.cjs can classify deep-probe output with the SAME
+// pattern list. A second copy would drift, and these patterns decide whether a slot is
+// declared dead — the one place a stale duplicate is least affordable.
+module.exports = { ERROR_PATTERNS, classifyOutput };