diff --git a/devlog/_plan/260926_kiro_lb_parity2/081_head_to_head_result.md b/devlog/_plan/260926_kiro_lb_parity2/081_head_to_head_result.md new file mode 100644 index 00000000000..89435bc1ef7 --- /dev/null +++ b/devlog/_plan/260926_kiro_lb_parity2/081_head_to_head_result.md @@ -0,0 +1,100 @@ +# 081 — Head-to-head with kiro-lb, from the landed tree + +Written by the method in `080_head_to_head_method.md`. opencodex at `origin/dev` `54ed02be17` +(after #5967, #5981, #5991, #5994, #5996, #6001, #6002); kiro-lb at `bee73b3` (no newer commits on +2026-09-27, so no new kiro-lb behaviour needed classifying). Every opencodex claim cites merged code and +names the regression test that proves it; every test file named exists at `54ed02be17`. kiro-lb was +studied for behaviour only; no AGPL code or text was copied. + +## Result in one paragraph + +Every one of the 20 adopted rows (18 adopted in 001, plus S1/F and P8/J, which landed on dev outside this unit) now reaches parity with kiro-lb or better, each with a named test; the two Keep rows (A7, M) hold, and the 13 Reject rows keep their reasons. +opencodex is ahead outright where it matters most for a pool of accounts: quota and exhaustion evidence +now survives restart and is fenced to the login that measured it (kiro-lb keys by account and never +fences a re-login), refresh stays leased where kiro-lb falls back to an unleased refresh, credentials are +never reloaded from an unrelated source, region values are validated, and polling is on demand; load +spreading is now at parity (deterministic, opt-in least-loaded against kiro-lb's weighted race or +deterministic most-credits mode). Kiro-lb still leads on eight axes listed +below. Each is a deliberate choice or needs live evidence we do not have, so **the goal criterion "ahead on +every compared axis" is not met as written**; "ahead or at parity on every adopted axis" is. + +## Row by row + +| ID | Axis | kiro-lb (`bee73b3`) | opencodex (`54ed02be17`) | Verdict | Proving test / reject reason | +|---|---|---|---|---|---| +| A1/A2/A9 | Native device onboarding | `kiro/device_login.py:49-56,119-141,218-276`; dashboard start at `kiro/dashboard.py:1188` | `src/oauth/kiro-device-login.ts:176,288,303`; `src/oauth/store.ts:943` (`appendKiroAccountFromDeviceLogin`); CLI `--method` at `src/cli/account-auth.ts:37,146` | parity (API/CLI) | `tests/providers/kiro/kiro-device-builder.test.ts` "register, authorize, pending, slow_down and approve without persisting a service profile"; `tests/providers/kiro/kiro-device-social.test.ts` "uses millisecond authorization timings and appends duplicate profile ARN with warning"; `tests/server/server-kiro-device-login.test.ts` "native only with a valid method; reauth and invalid methods are rejected"; `tests/cli/cli-account-kiro-device.test.ts` "--method starts native flow and prints only the public view" | +| A3 | Refresh window | `kiro/config.py:100-103` | `src/oauth/index.ts:83` (60 s) | parity | Reject holds: a policy difference with no evidence the shorter window fails. | +| A4 | Refresh contention | `kiro/auth.py:954-992` | `src/oauth/store.ts:420,431` (lock, 120 s stale) | ahead-ocx | Reject holds: an unleased refresh can lose a rotated token. | +| A5 | Terminal refresh rejection | `kiro/auth.py:645-676` | `src/oauth/kiro.ts:38` (code allowlist); `src/oauth/kiro-terminal-failover.ts:9` | parity | `tests/server/server-kiro-refusal-e2e.test.ts` "terminal refresh rejection marks reauth and admits the sibling before any A send", "post-401 terminal refresh rejects A generation and replays on B once", "an unlisted refresh error on 400 does not mark reauth or rotate" | +| A6 | External source reload | `kiro/auth.py:645-653` | `src/oauth/kiro.ts:615` (reload only when the ARN matches) | ahead-ocx | Reject holds: an unrestricted reload can import another account. | +| A7 | Builder ID profile | `kiro/device_login.py:286-293` | `src/oauth/kiro.ts:520` | parity | `tests/providers/kiro/kiro-builder-id-profile.test.ts` "the fallback never becomes the account's identity, region, or stored metadata" | +| A8 | Secrets in logs | `kiro/auth.py:822-834` | `src/oauth/kiro.ts:533-543` (allowlisted refresh-error codes) | ahead-ocx | Reject holds: kiro-lb logs the full failed OIDC response body; we log allowlisted codes only. | +| S1/F | Builder ID usage ARN | `kiro/usage.py:63-97` | `src/providers/kiro-usage.ts:226` (shared resolver) | parity | `tests/providers/kiro/kiro-usage-restart.test.ts` "probe and runtime use the same request-profile resolver" | +| S1b | No ARN, no probe | same | `src/providers/kiro-usage.ts:184` | parity | `tests/providers/kiro/kiro-usage-restart.test.ts` "non-OIDC missing ARN makes no usage request and keeps same-login last good bar" | +| B/P7 | Restart continuity | `kiro/store.py:314-350` | `src/providers/kiro-account-state-disk.ts:43`; `kiroAccountEvidence` at `src/providers/kiro-usage.ts:300` | ahead-ocx (login fence plus TTL/reset) | `tests/providers/kiro/kiro-account-state-disk.test.ts` "first routing request hydrates exhausted evidence without probing", "two different logins into one identity-less slot invalidate evidence across restart" | +| P6 | Polling | `main.py:356` | `src/providers/quota.ts:418-440` (pull on demand, joined per account) | ahead-ocx | Reject holds: kiro-lb polls in the background every 60 s or more even when polling is disabled; ours is pull-on-demand. | +| P9 | Served success clears a verdict | `kiro/account_manager.py:1411` | `src/providers/kiro-usage.ts:344`; `src/server/responses/adapter-delivery.ts:125` | parity | `tests/providers/kiro/kiro-refusal-failover.test.ts` "kiro later served completion clears older exhaustion but not a newer refusal or different identity" | +| E5 | Provider egress | `kiro/proxy_chain.py:176-198` | `src/adapters/kiro-retry.ts:169-196` | parity | `tests/providers/kiro/kiro-transport-parity.test.ts` "initial, reset, gateway alternate and 429 recovery use the supplied executor" | +| E2/H | 502/503/504 rotation | `kiro/http_client.py:603-653` | `src/adapters/kiro-retry.ts:290-296` | parity | `tests/providers/kiro/kiro-transport-parity.test.ts` "canonical HTTP %i rotates once and final failure is fixed text", "500, 521 and custom URLs do not rotate" | +| E6 | Timeout handling | `kiro/network_errors.py:65-145` | `src/adapters/kiro-retry.ts:200` | parity on header deadline → 504; ahead-lb on the connect/read split | `tests/providers/kiro/kiro-transport-parity.test.ts` "header deadline becomes 504 without rotating; caller abort preserves its reason" | +| E7 | Fixed public 5xx text | `kiro/exceptions.py:27-36` | `src/adapters/kiro-errors.ts:215` | parity | `tests/providers/kiro/kiro-transport-parity.test.ts` "upstream 5xx marker reaches neither client nor debug ring or stderr"; `tests/server/server-kiro-refusal-e2e.test.ts` "unrotated Kiro 5xx retains fixed public text after raw-error handoff" | +| E1 | Extra endpoint dialects | `kiro/endpoints.py:25-82` | runtime host plus `q.*` | ahead-lb | Reject holds: kiro-lb itself verifies only the runtime host (`kiro/endpoints.py:9-12`). | +| E4 | Endpoint probe | `kiro/endpoint_probe.py:2-8` | absent | ahead-lb | Reject holds: the probe spends real credits. | +| E8 | Region | `kiro/http_client.py:593-615` | `src/oauth/kiro-credentials.ts:14,115-123` (region pattern validation) | ahead-ocx | Reject holds: kiro-lb interpolates the region unvalidated; ours validates the region value and keeps it account-scoped. | +| E9 | Connection reuse | pooled clients | `src/adapters/kiro-retry.ts:188-193` (fresh connection on reset) | parity | Reject holds: equivalent contract, nothing to adopt. | +| P4/I | Refusal classes | `kiro/account_manager.py:1496,1539` | `src/adapters/kiro-refusal.ts:11`; `src/oauth/generic-account-failover.ts:426`; Kiro loop at `src/server/responses/adapter-dispatch.ts:907` | parity | `tests/providers/kiro/kiro-refusal.test.ts` "kiro refusal: 429 rate and 400 or 429 monthly reasons stay distinct"; `tests/providers/kiro/kiro-refusal-failover.test.ts` "kiro monthly refusal persists and skips only the refused account until reset or TTL" | +| P5/E3 | Suspension 403 | `kiro/account_manager.py:1514` | `src/oauth/generic-account-failover.ts:104` | parity | `tests/providers/kiro/kiro-refusal-failover.test.ts` "kiro suspended account rotates before output"; `tests/server/server-kiro-refusal-e2e.test.ts` "Kiro post-output refusal never resends the turn" | +| P1/P2 | Spreading healthy load | `kiro/account_manager.py:1162-1208` | `src/oauth/generic-account-failover.ts:601` (initial choice), 489 (rotator) | parity (ours: deterministic least-loaded, opt-in; kiro-lb: weighted race by default, deterministic `most_credits` mode at `kiro/account_manager.py:1209-1219`) | `tests/providers/kiro/kiro-leased-responses.test.ts` "least-loaded selects the less busy Kiro account on the first physical send", "least-loaded does not move a healthy request when proactive preference is off" | +| P3 | Per-account concurrency | `kiro/concurrency.py:2,95` | `src/oauth/kiro-account-load.ts:14,66` | parity | `tests/providers/kiro/kiro-account-load.test.ts` "a cap admits one, wakes one waiter, and never exceeds the limit"; `tests/providers/kiro/kiro-leased-responses.test.ts` "a full selected account waits then returns 503 account_capacity without a store write" | +| P10 | Affinity | `kiro/account_manager.py:1444` | none | parity | Reject holds: a global cursor is not conversation affinity. | +| P8/J | Per-request credits | `kiro/usage_tracking.py:55,80` | `src/adapters/kiro/stream.ts:609`; `src/types/request.ts:451` | parity | `tests/providers/kiro/kiro-metering-usage.test.ts` "repeated per-request readings replace the prior snapshot, including zero"; `tests/providers/kiro/kiro-metering-events.test.ts` "parses real precise sample metering event with unit and usage" | +| U1 | Multiplier estimates | `kiro/model_costs.py:8-22` | none | ahead-lb (advisory) | Reject holds: a second, weaker number beside measured credits. | +| C1/N | Per-account catalogue | `kiro/model_catalog.py:38-78` | `src/providers/kiro-model-catalog.ts:12,120` | parity | `tests/providers/kiro/kiro-model-catalog.test.ts` "management ListAvailableModels pairs each bearer with its own profile and region", "unrecognised or empty management replies preserve the last good list" | +| C2 | Membership as evidence | `kiro/model_resolver.py:309-355` | `src/oauth/generic-account-failover.ts:237` | parity | `tests/providers/kiro/kiro-model-preference.test.ts` "reactive Kiro rotation prefers model evidence after room filtering", "no catalogue evidence never moves a healthy active account" | +| C3 | Context limits | `kiro/model_costs.py:64-74` | `src/providers/kiro-model-catalog.ts:129`; `src/adapters/kiro/usage.ts:223`; `src/server/request-log.ts:1633` | parity when catalogue evidence exists; ahead-lb on the static fallback (272k GPT-5.6 window at `src/providers/kiro-models.ts:34` vs kiro-lb's 1M) | `tests/providers/kiro/kiro-model-catalog.test.ts` "a mixed known/unknown roster never reports more than the smallest known window" | +| W1/W2 | IDE fingerprint | `kiro/utils.py:20-31,74-85` | `src/adapters/kiro/wire.ts:8` (`KIRO_IDE_VERSION = "1.0.0"`) | ahead-lb | Reject holds: needs our own capture. | +| S2 | MCP web search | `kiro/mcp_tools.py:131-150` | none | n/a | Reject holds: a new feature, not auth or usage. | +| K | Quota metrics | `kiro/metrics.py:262` | `src/providers/kiro-quota-metrics.ts:16`; `src/server/request-metrics.ts:192-196` | parity | `tests/providers/kiro/kiro-quota-metrics.test.ts` "fresh cached credits yield four opaque gauges without probing", "future, stale, reset-passed, and identity-mismatched rows are omitted" | +| L | Auto-selection state | dashboard (`b176585`) | `src/oauth/generic-account-failover.ts:134`; `src/server/management/oauth-account-routes.ts:407`; `src/cli/account.ts:108` | parity (API/CLI) | `tests/providers/kiro/kiro-auto-selection.test.ts` "Kiro candidate and list projection agree on family-less exclusion states"; `tests/cli/cli-kiro-auto-selection.test.ts` "Kiro list prints a closed reason and carries it in JSON" | +| M | Opaque labels | `c0a7f98` | `src/codex/account-label.ts:36` | parity | Kept; `tests/oauth/oauth-account-attribution.test.ts` "an o-label is a valid persisted label and a p-label still is" | + +## Where kiro-lb still leads + +| Axis | Why we did not follow (yet) | +|---|---| +| W1/W2 IDE wire fingerprint | Changing the wire identity without our own capture risks every Kiro request. Needs a live capture. | +| E1 extra endpoint dialects | kiro-lb itself verifies only the runtime host for every credential type (`kiro/endpoints.py:9-12`); unverifiable offline. | +| E4 paid endpoint probe | Spends real credits; our 5xx rotation self-heals without spending. | +| S2 MCP web search side call | A new feature, not auth or usage. | +| U1 per-model credit estimates | Measured credits (`providerCredits`) are the source of truth; an estimate beside them is a weaker second number. | +| Dashboard device-login UI | Native device login ships in the CLI and management API (060, option a); the dashboard still uses kiro-cli. A device-code dialog is the follow-up. | +| Operations dashboard | Routable state and quota gauges are in the API, CLI and metrics export (070), not rendered in the GUI; the `health` field does not yet reflect Kiro suspension/exhaustion. | +| C3 static context fallback | Without catalogue evidence our static GPT-5.6 window is 272k (`src/providers/kiro-models.ts:34`) where kiro-lb records 1M; the observed catalogue raises it when an account reports it. | +| E6 connect vs read timeout split | A header deadline maps to 504 and never rotates on caller abort, but connect and read timeouts are not separated. | + +## Narrowed by recorded decisions (not failures) + +- **A1/A2/A9:** native login is add-only; neither method yields a verified identity, so native re-login is + refused and native accounts recover by remove and re-add. Flow binding is principal kind plus an + unguessable, unlisted flow id (no per-session identity exists in management auth). +- **P3:** a full account waits up to 250 ms and returns a retryable `503 account_capacity`; capacity never + moves a request or writes the stored selection. +- **L:** `autoSelectable`/`skipReason` are API and CLI fields; the GUI does not render them. + +## Evidence gaps (fixture-only, never exercised against a live Kiro account) + +- 030 refusal bodies: `MONTHLY_REQUEST_COUNT`, `TEMPORARILY_SUSPENDED` and the suspension message wordings. +- 050 `ListAvailableModels` reply fields (`models[].modelId`, `tokenLimits.maxInputTokens`). +- 060 device-flow replies: social field names and units, social terminal statuses, social `profileArn` + uniqueness, Builder ID replies and `x-amzn-errortype` spellings. +- Metering frame repetition within one send (the parser itself was built from kiro-cli captures). +- Egress through a real proxy, 502/503/504 rotation against the real `q.*` host, cross-account context + window accuracy. + +## Verification of this document + +Run on 2026-09-27 against branch `codex/kiro-lb2-081-head-to-head` at `origin/dev` `54ed02be17` with this file present: + +- `bun run privacy:scan` — "Privacy scan passed" (exit 0). +- `bun run structure:check` — "structure/ SSOT checks passed" (exit 0). +- Mechanical check (`.tmp/check081.mjs`, scratch): every `tests/...` path named above exists in `git ls-tree origin/dev`, and each of the 35 row IDs in 001 appears exactly once in the row table. diff --git a/tests/providers/kiro/kiro-auto-selection.test.ts b/tests/providers/kiro/kiro-auto-selection.test.ts index 6a055a9ae98..1dcd253a311 100644 --- a/tests/providers/kiro/kiro-auto-selection.test.ts +++ b/tests/providers/kiro/kiro-auto-selection.test.ts @@ -53,15 +53,18 @@ test("Kiro candidate and list projection agree on family-less exclusion states", exhausted: true, nextResetAt: now + 3600_000 }, kiroEvidenceIdentity(exhausted)); const expected = { reauth: "needs_reauth", suspended: "suspended", cooldown: "cooldown", exhausted: "quota_exhausted", unknown: undefined } as const; + // Evaluate after every write: evidence stamped later than the evaluation clock is rejected as + // future-dated, so reusing the pre-setup `now` made this test depend on millisecond timing. + const evalNow = Date.now(); const live = getAccountSet("kiro")!.accounts; - const eligible = eligibleFailoverAccounts("kiro", now); + const eligible = eligibleFailoverAccounts("kiro", evalNow); for (const account of live) { const name = account.credential.accountId as keyof typeof expected; - const projected = kiroAutoSelection(account, now); + const projected = kiroAutoSelection(account, evalNow); expect(projected.autoSelectable).toBe(eligible.includes(account.id)); expect(projected.skipReason).toBe(expected[name]); } expect(eligible).toEqual([byName("unknown").id]); - expect(kiroAutoSelection(byName("suspended"), now + 24 * 60 * 60_000 + 1)) + expect(kiroAutoSelection(byName("suspended"), evalNow + 24 * 60 * 60_000 + 1)) .toEqual({ autoSelectable: true }); });