Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,13 @@ jobs:
elif [ "$LATEST" = "$NEXT" ]; then
echo "invariant OK — next == latest == ${LATEST}"
else
echo "::warning::DIST-TAG DRIFT — latest=${LATEST} next=${NEXT}. Anyone installing @next gets the older build. Fix: npm dist-tag add @nforma.ai/nforma@${VERSION} next"
# @next is a DEPRECATED alias (CLAUDE.md, 2026-08-09) and cannot be kept in
# sync automatically — OIDC does not cover `npm dist-tag`. Drift is therefore
# EXPECTED, so this is a notice, not a defect report. Wording matters: a
# warning that fires every release and means nothing trains the reader to
# ignore warnings, which is how the false "unrecognized dispatch flag" alarm
# cost real debugging time (#385).
echo "::notice::DIST-TAG DRIFT (expected) — latest=${LATEST} next=${NEXT}. @next is a deprecated alias and may lag; @latest is authoritative. To align anyway: npm dist-tag add @nforma.ai/nforma@${VERSION} next"
fi

# ── Git tag + GitHub Release (after a successful publish) ──
Expand Down
7 changes: 4 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [Unreleased]

### Changed
- **`@next` is deprecated and scheduled for retirement — the `next == latest` invariant is retired with it.** `@next` was the prerelease channel; #366/#367 retired that flow, leaving a tag that by policy always equals `@latest`, so it carries **zero information** while costing a manual step after every release. Keeping it in sync *is* automatable — `.github/workflows/align-dist-tags.yml` does it — but only on a credential class npm is retiring: [npm's docs](https://docs.npmjs.com/trusted-publishers) scope OIDC to `npm publish` / `npm stage publish`, so `npm dist-tag` needs token auth, and per [GitHub's 2026-07-31 changelog](https://github.blog/changelog/2026-07-31-restricting-npm-bypass-2fa-granular-access-tokens/) 2FA-bypass tokens already cannot create tokens or change package access without an interactive challenge, and **from a targeted January 2027 they lose direct publishing capability entirely**. Automating this would mean investing in a mechanism npm is retiring on a published timeline. Deprecating `@next` instead removes the **last token dependency** from the release pipeline — publish is already OIDC, and the dist-tag step was the only thing still needing a credential. Consequences in force now: drift is **expected, not an incident**; `@latest` is the only supported channel; alignment is opportunistic (manually, or via the `Align dist-tags` workflow) and never gates a release. `publish.yml`'s drift report is downgraded from `::warning::` to `::notice::DIST-TAG DRIFT (expected)` accordingly — an every-release warning that means nothing is how readers learn to ignore warnings, which is exactly what the false "unrecognized dispatch flag" alarm cost in #385. **Not removing the tag yet:** ~1.2k downloads/month means someone may be pinned to `@next`, and `npm dist-tag rm @nforma.ai/nforma next` would make `npm i @nforma.ai/nforma@next` fail outright. It stays pointing at a real version; revisit after two or three releases.
- fix(quorum): **the stall timer no longer uses byte count as evidence of liveness (STALL-TIMEOUT-03).** `<500 bytes → apply a tighter kill window` was a proxy for "this slot is hung", and a live multi-model quorum falsified it the hard way — by killing three of its own five voters mid-answer. `claude-1` (idle 90s → 90s window), `copilot-1` (idle 30s → 30s), and `claude-z-ai` (90s) were all producing correct output when the timer fired; their partial answers were recoverable from `unavail_message`. `codex-1` and `antigravity-1` survived not because they were faster but because they **stream**, so every chunk rearmed the timer. The discriminator was never byte count. Worse, deriving the window from `idle_timeout_ms` (the #385 fix) made that field do double duty as "how long may this model think before output" — and for most slots the configured value (30s/90s) is far *smaller* than real think time, while the caller had budgeted 300s. There are now two windows, one per failure mode: **before the first byte**, `ttfb_timeout_ms` (default 30s) — zero output is the only honest evidence of a hang, so fast-fail belongs there; **after the first byte**, the inter-chunk ceiling — `inter_chunk_ceiling_ms` if the slot declares one, else the caller's idle budget. `stall_timeout_ms` is kept as a legacy alias for the ceiling, since the values already in that field were measured against post-preamble silence, so no config surface is added. Live-verified: `claude-1` answers in 134s where it previously died at 90s. Measuring also corrected an assumption — the `claude` CLI **buffers** (zero bytes until well past 30s) rather than emitting a preamble like the z-ai/minimax/kimi routes, which is why TTFB is per-slot configurable and why the claude-family slots carry `ttfb_timeout_ms: 150000`. Ratified by quorum (3/3 unanimous, Round 1: codex-1, antigravity-1, claude-minimax) over the alternative of keeping #386 as shipped; debate at `.planning/quorum/debates/2026-08-07-stall-timer-design-decision-a-vs-b.md`.

### Added
- feat(quorum): **deep-probe gate — preflight can now tell a quota-dead slot from a healthy one (P1 of #293, the last piece).** L1 (`--version`) proves a binary exists and L2 (`/models`) treats 401/403 as *reachable*, so neither notices a slot that is authenticated, reachable, and out of quota. Demonstrated live on 2026-08-08: preflight reported **7/7 available** while `claude-z-ai` (429, weekly limit) and `antigravity-1` (quota, 74h to reset) could not answer at all — a panel promising full quorum that then produces two UNAVAILs mid-round. `deepProbeSlot()` runs the slot's configured `deep_probe` and is deliberately **lopsided**: it downgrades *only* on a fast, explicit auth/quota signal in the slot's own output; a timeout, spawn error, or unrecognised response is `INCONCLUSIVE` and never downgrades. That asymmetry is the point — the opposite error, killing a slow-but-alive slot, is the one this codebase has already paid for three times (STALL-TIMEOUT-01..03). It runs only when the panel is already **degraded** *and* an explicit `--budget-ms` covers it (a naive auto-enable previously blew the `<8s` liveness budget with `spawnSync ETIMEDOUT`), or on demand via `--deep-probe`. Live-verified end to end: both dead slots are now downgraded with their real reasons and the gate reports `DEGRADED — 5 available`, where the cheap budget-less path still finishes in ~3s having skipped the probe entirely.
- feat(quorum): **preflight now validates the roster and decides the quorum gate (P3 + P5 of #293, extracted fresh).** `validateProviders()` reports config faults that would otherwise surface mid-quorum as an opaque UNAVAIL — the two this repo has actually shipped are a slot with neither `cli` nor `mainTool` (spawn receives `null`, taking the whole quorum offline, #197) and an `args_template` with no `{prompt}` placeholder (the CLI runs with flags and no question). Also errors on duplicate names (silently last-wins everywhere downstream), missing/unknown `type`, and http slots without a base URL or any auth; warns on non-positive timeout fields and on third-party-routed slots with no `deep_probe`. Non-fatal by design: it reports on every `--all` and never blocks the preflight. `computeQuorumGate()` turns *"available < max_quorum_size → BLOCK unless `--force-quorum`"* from prose in `commands/nf/quorum.md` into a machine field — `{available_count, quorum_met, degraded, blocked, waiver_required, waiver_used, gate_reason}` — because a gate that depends on a model reading a paragraph correctly is not a gate. It keeps **degraded** and **blocked** distinct (below `max_quorum_size` but above `min_live_voters` still dispatches), and a `--force-quorum` waiver unblocks while being *recorded* rather than silently making the panel look healthy. 11 tests, each of the three core behaviours mutation-proven.
Expand All @@ -38,9 +42,6 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
- fix(quorum): **a quota-dead slot is now held out until it actually resets, instead of being retried every 5 minutes for days (QUOTA-RESET-01).** Providers state their reset window, but in shapes `parseAvailabilityHint` did not match — `antigravity-1` says `"Resets in 95h26m37s."` and `claude-z-ai` says `"...will reset at 2026-08-09 10:01:44"`, and neither the `"in N hours"` nor the `"until <Month Day>"` pattern covers them. Both fell through to the 5-minute default, so a slot that was dead for **four days** burned a dispatch in every quorum round, dragging the live-voter count toward the `min_live_voters` floor. Two new shapes are parsed: a compact duration (`95h26m37s`, `32h54m`, `90s`) and an absolute `reset at YYYY-MM-DD HH:MM:SS`. Deliberately conservative in the dangerous direction — a plain `STALL`/`IDLE_TIMEOUT` still gets the 5-minute default, because over-matching would retire a healthy slot for days on one bad round. The compact parser is written as locate-token-then-decompose: a single pattern of all-optional unit groups matches the empty string, "succeeding" with every capture undefined and silently falling through to the very default it exists to replace (which is what the first version did). 6 tests, mutation-proven in both directions. This is P4 of the long-open #293, extracted fresh.
- fix(install): **a stale backup no longer clobbers live slot config on every install.** When `bin/providers.json` ships empty (it always does), install reconstructs the user's `providers.json` from `~/.claude.json` and overlays what it can recover for each slot, gathering from four candidates documented as being "in order of freshness" — live `nf-bin/` first, then two `nf-local-patches/` backups, then the legacy live path. The gather loop called `map.set(name, entry)` unconditionally, so the **last** candidate won: the exact inversion of the stated order. On the machine that surfaced it, a `nf-local-patches/nf/bin/providers.json` dated **2026-05-13** overrode the live entry for 7 slots on every single install, silently reverting per-slot tuning (`ttfb_timeout_ms`, `inter_chunk_ceiling_ms`, `idle_timeout_ms`, `model`, `env`). It was found because freshly-set `ttfb_timeout_ms` values kept vanishing after `install.js --claude --global`, which also explains earlier "mystery" resets of `stall_timeout_ms`. First writer now wins, so backups do what they exist for — recovering slots the live file has **lost** — and never override it. 4 tests in `bin/install-providers-precedence.test.cjs` execute the extracted gather loop against stubbed candidates; mutation-proven, and verified against a real install that now preserves every tuned field.

### Changed
- fix(quorum): **the stall timer no longer uses byte count as evidence of liveness (STALL-TIMEOUT-03).** `<500 bytes → apply a tighter kill window` was a proxy for "this slot is hung", and a live multi-model quorum falsified it the hard way — by killing three of its own five voters mid-answer. `claude-1` (idle 90s → 90s window), `copilot-1` (idle 30s → 30s), and `claude-z-ai` (90s) were all producing correct output when the timer fired; their partial answers were recoverable from `unavail_message`. `codex-1` and `antigravity-1` survived not because they were faster but because they **stream**, so every chunk rearmed the timer. The discriminator was never byte count. Worse, deriving the window from `idle_timeout_ms` (the #385 fix) made that field do double duty as "how long may this model think before output" — and for most slots the configured value (30s/90s) is far *smaller* than real think time, while the caller had budgeted 300s. There are now two windows, one per failure mode: **before the first byte**, `ttfb_timeout_ms` (default 30s) — zero output is the only honest evidence of a hang, so fast-fail belongs there; **after the first byte**, the inter-chunk ceiling — `inter_chunk_ceiling_ms` if the slot declares one, else the caller's idle budget. `stall_timeout_ms` is kept as a legacy alias for the ceiling, since the values already in that field were measured against post-preamble silence, so no config surface is added. Live-verified: `claude-1` answers in 134s where it previously died at 90s. Measuring also corrected an assumption — the `claude` CLI **buffers** (zero bytes until well past 30s) rather than emitting a preamble like the z-ai/minimax/kimi routes, which is why TTFB is per-slot configurable and why the claude-family slots carry `ttfb_timeout_ms: 150000`. Ratified by quorum (3/3 unanimous, Round 1: codex-1, antigravity-1, claude-minimax) over the alternative of keeping #386 as shipped; debate at `.planning/quorum/debates/2026-08-07-stall-timer-design-decision-a-vs-b.md`.

## [0.44.3] - 2026-08-07 — Quorum stall-window derivation, no shipped user config, and migration/portability guards

### Fixed
Expand Down
46 changes: 41 additions & 5 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,52 @@ nForma uses milestone-based semver (no separate prerelease channel — see dist-
Versions with a prerelease suffix (e.g. `0.40.2-rc.1`) **cannot ship** under the `@next == @latest` alias policy — a prerelease semver string published to `@latest` would silently install `0.40.2-rc.1` for every user doing `npm install @nforma.ai/nforma@latest`. `publish.yml` rejects any version with a `-` suffix.

Dist-tag mapping:
- `latest` — stable versions (0.40.1)
- `next` — alias for `latest` (always points to the same version; see invariant below)
- `latest` — stable versions (0.44.3). **The only channel.**
- `next` — **DEPRECATED alias, scheduled for retirement.** Best-effort mirror of `latest`; may lag.

**`@next` is deprecated (decided 2026-08-09).** It was the prerelease channel; #366/#367
retired that flow, leaving a tag that by policy always equals `@latest` — it therefore
carries **zero information** while costing a manual step after every release. Keeping it
in sync *can* be automated — `.github/workflows/align-dist-tags.yml` does exactly that —
but only on a credential class npm is retiring. OIDC does not cover `npm dist-tag` (see
"Aligning `@next`"),
so the only automated option is a long-lived publish-capable token in CI — precisely what
adopting trusted publishing was meant to eliminate, and it expires silently on a timer
(the current secret died exactly that way).

**npm is actively retiring that option.** Per
[GitHub's 2026-07-31 changelog](https://github.blog/changelog/2026-07-31-restricting-npm-bypass-2fa-granular-access-tokens/),
granular access tokens configured to bypass 2FA already cannot create/delete tokens or
change package access without an interactive 2FA challenge (this is why `npm token create`
now hangs on an OTP prompt in a non-interactive shell), and **from a targeted January 2027
they lose direct publishing capability altogether** — reduced to reading private packages
and staging a publish that a maintainer approves with 2FA. The recommended paths are
trusted publishing (OIDC) and staged publishing.

So the token route is not merely inconvenient: it is a mechanism npm is retiring, on a
published timeline. Deprecating `@next` removes the **last token dependency** from this
repo's release pipeline — publish is already OIDC, and the dist-tag step was the only
thing still needing a credential.

**Consequences, in force now:**
- Drift is **expected**, not a bug. Do not treat a stale `@next` as an incident.
- Align it opportunistically when convenient — see "Aligning `@next`" — or run the
`Align dist-tags` workflow. Neither is required for a release to be correct.
- `@latest` is the supported channel. All docs, install instructions and support should
say `@latest` or nothing at all.

**Retirement plan.** Do NOT run `npm dist-tag rm @nforma.ai/nforma next` yet. The package sees ~1.2k
downloads/month and someone may be pinned to `@next`; removing a tag makes
`npm i @nforma.ai/nforma@next` fail outright. Leave it pointing at a real version, revisit
after two or three releases, and remove it only if nothing appears to install from it.
Until then it is harmless — a slightly stale pointer to a real, published tarball.

**Invariant: `next` is an alias for `latest`** — both dist-tags point to the same tarball at all times; there is no separate prerelease channel. When `@latest` is updated, `@next` must be moved to the same version. This is automated in `publish.yml`, but the automation is currently blocked on an expired `NPM_TOKEN` secret, so alignment is a manual step for now — see "Aligning `@next`" below. Verify after every release:
```bash
npm view @nforma.ai/nforma dist-tags --json
# latest and next must show the same version
# `latest` is authoritative; `next` may lag and that is expected
```

When asked for a "new release", there is now only **one** channel — `@latest` (`@next` mirrors it). Confirm the target version string, then check `npm view @nforma.ai/nforma dist-tags --json` to determine the next version number.
When asked for a "new release", there is only **one** channel — `@latest` (`@next` is a deprecated alias that may lag; it does not gate a release). Confirm the target version string, then check `npm view @nforma.ai/nforma dist-tags --json` to determine the next version number.

### Release process

Expand Down
8 changes: 6 additions & 2 deletions bin/release-guards.test.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,11 @@ describe('publish.yml — @next alignment actually authenticates (DIST-TAG-01)',

it('verifies the invariant after aligning, instead of only printing dist-tags', () => {
assert.match(YML, /- name: Verify @next == @latest/, 'publish.yml must verify the alias invariant');
assert.match(YML, /DIST-TAG DRIFT/, 'a drifted tag must be called out explicitly, not left to be eyeballed');
// Still reported — but as a ::notice::, because @next is a deprecated alias whose
// drift is expected. An every-release ::warning:: that means nothing is how readers
// learn to ignore warnings (#385's false alarm cost real debugging time).
assert.match(YML, /DIST-TAG DRIFT \(expected\)/, 'drift must still be reported, just not as a defect');
assert.match(YML, /::notice::DIST-TAG DRIFT/, 'drift of a deprecated alias is a notice, not a warning');
});

it('keeps the publish step on OIDC — no NODE_AUTH_TOKEN in any non-comment line', () => {
Expand Down Expand Up @@ -265,7 +269,7 @@ describe('publish.yml — the @next verification actually discriminates (execute

it('reports DRIFT when the tags differ, naming the fix command', () => {
const r = runWith('{"latest":"0.44.3","next":"0.44.2"}');
assert.match(r.out, /DIST-TAG DRIFT/);
assert.match(r.out, /DIST-TAG DRIFT \(expected\)/);
assert.match(r.out, /npm dist-tag add @nforma\.ai\/nforma@0\.44\.3 next/);
assert.doesNotMatch(r.out, /invariant OK/);
});
Expand Down
Loading