Skip to content

Update the dependencies the cooldown admits, and pin the site past five advisories published on 29 September - #1085

Merged
jmrplens merged 4 commits into
mainfrom
deps-2026-09-30
Sep 30, 2026
Merged

jmrplens merged 4 commits into
mainfrom
deps-2026-09-30

Conversation

@jmrplens

@jmrplens jmrplens commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

I went through every dependency this repository pins, against the latest release of each upstream, and updated everything the cooldown in dependabot.yml admits: seven days for a patch, fourteen for a minor, counted from the release. I applied the same rule to the pins Dependabot does not manage. There were no open Dependabot pull requests, so every change here is by hand.

Security fixes in the site's dependency tree

  • brace-expansion (first commit): GHSA-6j4f-fj2g-mc7p and GHSA-qhr7-859c-m2p7, both high, reach the 1.1.18 and 5.0.9 the overrides pinned, through pa11y-ci, htmlhint and eslint, and they failed the site dependency audit on every pull request. Both pins move to the last release of their line, 1.1.21 and 5.0.12, published on 2026-09-14. Those two releases are also the first to fix GHSA-q2hr-2g5m-vwhr (moderate), published on 29 September with the other two, which is why the pins are 1.1.21 and 5.0.12 rather than 1.1.20 and 5.0.11; the comment beside them in site/pnpm-workspace.yaml now records all three.
  • ip-address 10.5.1 to 10.7.2: GHSA-j6r3-76f7-8jcv (isInSubnet compares addresses of different families as one address space) and GHSA-h3mg-xc3c-68pw (an Address6 parse diagnostic with no length bound), both moderate and published on 29 September, reach the 10.5.1 the override pinned, through pa11y-ci, puppeteer, @puppeteer/browsers, proxy-agent and socks-proxy-agent to socks. Both are fixed in 10.7.1. 10.7.2 is the last 10.x release and stays inside the ^10.1.1 socks asks for.

With both, pnpm audit --audit-level=high reports nothing but the two recorded extract-zip exceptions.

What moved

Go modules, all indirect:

  • github.com/ebitengine/purego v0.11.0 to v0.11.1
  • github.com/power-devops/perfstat to its 2026-09-16 commit
  • golang.org/x/telemetry to its 2026-09-22 commit, the newest one at least seven days old
  • google.golang.org/genproto/googleapis/api and /rpc to their 2026-09-21 commit, the newest one at least seven days old

The site:

  • @astrojs/starlight 0.42.2 to 0.42.3
  • astro 7.3.3 to 7.3.4, which itself moves obug to 3.0.0, tinyclip to 1.0.3 and its own @astrojs/markdown-satteri to 0.4.2 (Starlight keeps 0.4.1)
  • prettier 3.9.8 to 3.9.9
  • html-validate 11.15.0 to 11.16.0
  • The esbuild override 0.28.1 to 0.28.2, the last 0.28.x release, with the reason it is still needed now written beside it: GHSA-g7r4-m6w7-qqqr reaches 0.27.3 to 0.28.0, and astro's ^0.28.0 admits 0.28.0. The high advisory published beside it, GHSA-gv7w-rqvm-qjhr, was withdrawn on 2026-06-17.

I resolved the site lockfile with a seven-day minimum release age, so no transitive package younger than the cooldown came in with these.

Overrides no longer needed

I checked each override in site/pnpm-workspace.yaml against the range its consumers ask for. Two are no longer needed:

  • lodash: its one consumer, pa11y-ci, asks for ~4.18.1, which is past both advisories, so the tree holds the same 4.18.1 without the override.
  • yaml and yaml-language-server@1.20.0>yaml: the scoped entry named a release the tree no longer holds (it has 1.23.0), and every consumer now asks for a release past GHSA-48c2-rrv3-qjmp. Without the global pin, yaml-language-server gets the 2.8.3 it pins exactly, the first patched release, and the rest stay on 2.9.0.

lodash-es stays, because chevrotain 11.1.2 still asks for exactly 4.17.23, and I wrote that beside it. The others were already at the last patched release of their line: fast-uri 3.1.8, js-yaml 4.3.2 and undici 6.28.1 and 8.10.2, whose next releases (6.29.0, 8.11.x) are minors inside the cooldown.

Already current

Go 1.27.1 is still the newest Go release, go-sdk v1.8.0 and every other direct Go requirement are at their newest version, and Node 24.21.0 is the newest Node 24. Every GitHub Actions pin but one is on the newest release of its action. GoReleaser v2.18.2, syft v1.52.0, oras 1.3.4, cosign v3.1.3, the .NET SDK 10.0.401 (the image digest make validate-nuget pins is unchanged), mcp-publisher v1.8.1, the mcpb CLI 2.1.2, the LobeHub CLI 0.0.41, gremlins v0.6.0, gobco v1.3.4, twine 7.0.0 and release-cli v0.24.0 are all the newest releases. The builder, runtime and Dockerfile frontend digests in the Dockerfile are the ones their tags pointed to on 2026-09-30.

Held, and why

Inside the cooldown, with the date each becomes eligible:

  • github.com/vektah/gqlparser/v2 v2.5.58, a patch released 2026-09-23 at 18:03 UTC, eligible 2026-09-30 at 18:03 UTC. Its one functional change compares composite argument values when judging overlapping fields. I tried it anyway: with it, the full unit suite and make check-graphql-documents pass, so it can go in as soon as it clears.
  • gitlab.com/gitlab-org/api/client-go/v3 v3.15.0, a minor released 2026-09-28, eligible 2026-10-12.
  • google.golang.org/grpc v1.84.0 (minor, eligible 2026-10-01) and github.com/grpc-ecosystem/grpc-gateway/v2 v2.31.0 (minor, eligible 2026-10-09), both indirect.
  • The genproto commit of 2026-09-28 and the x/telemetry commits of 2026-09-23 and 2026-09-24.
  • github/codeql-action v4.38.2, released 2026-09-24, eligible 2026-10-01.
  • golangci-lint v2.14.0, a minor released 2026-09-24, eligible 2026-10-08.
  • The OpenTelemetry collector image 0.162.0, released 2026-09-29 and not yet on Docker Hub on the morning of 2026-09-30, eligible 2026-10-13, since 0.161 to 0.162 is a minor.
  • npm 11.20.0 in the release job, a minor released 2026-09-22, eligible 2026-10-06.
  • pnpm: every minor after 12.4 is younger than fourteen days. 12.5.1 is eligible 2026-10-02 and 12.8.1 on 2026-10-12.
  • The site: astro 7.3.5 and Starlight 0.42.4 (2026-10-01), eslint-plugin-astro 3.2.0 and 3.2.1 (2026-09-30 and 2026-10-01), eslint 10.11.0 (2026-10-02), sharp 0.35.5 and html-validate 11.16.1 (2026-10-04), typescript-eslint 8.71.0 (2026-10-12).

Majors, each a change of its own, as in #1058: npm 12 in the release job, whose trusted publishing exchange only a real publish exercises; Node 24 in the release job; Bitbucket 10 in the e2e fixture; the HAProxy 3.4 LTS branch in the balancer test; and TypeScript 7, which astro check refuses for the reason pnpm-workspace.yaml records. The server's major tracks client-go's, so a client-go v4 would be held regardless.

Checks

  • go build ./..., go vet ./... and go vet -tags e2e ./test/e2e/... pass.
  • The full unit suite, go test ./cmd/... ./internal/... -count=1 -shuffle=on, passes: 276 packages.
  • make golangci-lint at v2.13.2 passes.
  • The e2e suites compile: ./test/e2e/gitlab/... and the harness under e2e, and the httpe2e, stdioe2e and collectore2e modules.
  • make test-e2e-http passes, 566 tests with 7 skipped: the nginx and HAProxy layers need Docker, which the machine that ran it lacks, and CI runs them. make test-e2e-stdio passes, 90 tests.
  • govulncheck reports no vulnerability our code reaches. The one module-level entry is GO-2026-5932 in golang.org/x/crypto, which has no fix and is documented in scripts/govulncheck.sh.
  • make check-supply-chain, make check-server-json and go run ./cmd/gen_testing_docs/ --check pass.
  • make gen-stats changes nothing, and make check-stats and make check-em-dash pass.
  • The site: pnpm install --frozen-lockfile, pnpm audit --audit-level=high (only the two extract-zip exceptions), pnpm run build (105 pages, internal links valid, Mermaid rendered through Playwright's Chromium) and pnpm run lint all pass.
  • The release rehearsal on this branch, run 36688038543 (gh workflow run release.yml --ref deps-2026-09-30 -f tag=v3.1.0), passed. It covers what CI does not: the E2E gate against an ephemeral GitLab, the image build, GoReleaser with syft's SBOMs, npm's dry publish, the NuGet validation and the race gate with the collector module.

GHSA-6j4f-fj2g-mc7p and GHSA-qhr7-859c-m2p7 (both high) reach the
1.1.18 and 5.0.9 the overrides pinned, through pa11y-ci, htmlhint and
eslint, and fail the site dependency audit on every pull request. A
third, GHSA-q2hr-2g5m-vwhr (moderate), published with them, is first
fixed in 1.1.21 and 5.0.12. Both pins move to those releases, the last
of their line, published on 2026-09-14.
GHSA-j6r3-76f7-8jcv (isInSubnet compares addresses of different
families as one address space) and GHSA-h3mg-xc3c-68pw (an Address6
parse diagnostic with no length bound), both moderate, reach the 10.5.1
the override pinned, through pa11y-ci, puppeteer, @puppeteer/browsers,
proxy-agent and socks-proxy-agent to socks. Both are fixed in 10.7.1.
The pin moves to 10.7.2, the last 10.x release (published 2026-09-15),
which stays inside the ^10.1.1 socks asks for. The audit now reports
nothing but the two recorded extract-zip exceptions.
Direct dependencies, each past the cooldown dependabot.yml sets (seven
days for a patch, fourteen for a minor):

- @astrojs/starlight 0.42.2 to 0.42.3 (2026-09-22)
- astro 7.3.3 to 7.3.4 (2026-09-22), which itself moves obug to 3.0.0,
  tinyclip to 1.0.3 and its own @astrojs/markdown-satteri to 0.4.2
  (Starlight keeps 0.4.1)
- prettier 3.9.8 to 3.9.9 (2026-09-23)
- html-validate 11.15.0 to 11.16.0 (2026-09-15)

Overrides:

- esbuild 0.28.1 to 0.28.2, the last 0.28.x release, with the reason it
  is still pinned now written beside it: GHSA-g7r4-m6w7-qqqr reaches
  0.27.3 to 0.28.0, and astro's ^0.28.0 admits 0.28.0. The high
  advisory published beside it, GHSA-gv7w-rqvm-qjhr, was withdrawn on
  2026-06-17.
- lodash removed. Its one consumer, pa11y-ci, asks for ~4.18.1, which
  is past both advisories, so the tree holds the same 4.18.1 without it.
- yaml and yaml-language-server@1.20.0>yaml removed. The scoped entry
  named a release the tree no longer holds (it has 1.23.0), and every
  consumer now asks for a release past GHSA-48c2-rrv3-qjmp: without the
  global pin, yaml-language-server gets the 2.8.3 it pins exactly, the
  first patched release, and the rest stay on 2.9.0.

The lockfile was resolved with a seven-day minimum release age, so no
transitive package younger than the cooldown came in with the bumps.
Node 24.21.0 and pnpm 12.4.2 stay: the first is the newest Node 24, and
every pnpm minor after 12.4 is younger than fourteen days.

Held for the cooldown: astro 7.3.5 and Starlight 0.42.4 (2026-10-01),
eslint-plugin-astro 3.2.0 and 3.2.1 (2026-09-30 and 2026-10-01),
eslint 10.11.0 (2026-10-02), sharp 0.35.5 and html-validate 11.16.1
(2026-10-04), typescript-eslint 8.71.0 (2026-10-12). TypeScript 7
stays out for the reason pnpm-workspace.yaml records.
- github.com/ebitengine/purego v0.11.0 to v0.11.1 (2026-09-18)
- github.com/power-devops/perfstat to its 2026-09-16 commit
  (22a1a467d9f0)
- golang.org/x/telemetry to its 2026-09-22 commit (c6096d2f4a49), the
  newest one at least seven days old
- google.golang.org/genproto/googleapis/api and /rpc to their
  2026-09-21 commit (b14227669459), the newest one at least seven days
  old; each declares go 1.26.0, below the module's 1.27.1

Held for the cooldown dependabot.yml sets (seven days for a patch,
fourteen for a minor):

- gitlab.com/gitlab-org/api/client-go/v3 v3.15.0, a minor released
  2026-09-28, eligible 2026-10-12. The server's major tracks
  client-go's, so a v4 would be held regardless.
- github.com/vektah/gqlparser/v2 v2.5.58, a patch released 2026-09-23
  at 18:03 UTC, eligible 2026-09-30 at 18:03 UTC. Its one functional
  change compares composite argument values when judging overlapping
  fields.
- google.golang.org/grpc v1.84.0 (minor, 2026-09-17, eligible
  2026-10-01) and github.com/grpc-ecosystem/grpc-gateway/v2 v2.31.0
  (minor, 2026-09-25, eligible 2026-10-09).
- The genproto commit of 2026-09-28 and the x/telemetry commits of
  2026-09-23 and 2026-09-24.

Go 1.27.1 is still the newest Go release, and go-sdk v1.8.0 and every
other direct requirement are at their newest version.
@jmrplens jmrplens added this to the 3.1.0 milestone Sep 30, 2026
@jmrplens jmrplens added security Security-related issue dependencies Pull requests that update a dependency file site The documentation site's build, components and scripts, not its content v3.1.0 Targeted at the 3.1.0 release labels Sep 30, 2026

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @jmrplens, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 2 days and 1 hour by commenting @sourcery-ai review. Upgrade to get a review now.

@coderabbitai

coderabbitai Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: jmrplens/gitlab-mcp-server/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 564317d5-0d10-44ed-a2a7-76a0d12d9e00

📥 Commits

Reviewing files that changed from the base of the PR and between 77e72c4 and 6a64915.

⛔ Files ignored due to path filters (2)
  • go.sum is excluded by !**/*.sum
  • site/pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (3)
  • go.mod
  • site/package.json
  • site/pnpm-workspace.yaml

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Summary

Summary by CodeRabbit

  • Chores
    • Updated supporting components used by the application and documentation site to newer versions. These updates include changes to build tooling, site packages, and other software components. No user-facing features or behavior changes are indicated in this release.

Walkthrough

The pull request updates indirect Go dependency versions and site package version ranges. It also changes several site workspace overrides, including updated pins and removed overrides.

Changes

Dependency updates

Layer / File(s) Summary
Go dependency updates
go.mod
Several indirect dependencies, including purego, perfstat, x/telemetry, and Google genproto modules, were updated.
Site dependency declarations and overrides
site/package.json, site/pnpm-workspace.yaml
Package version ranges and workspace pins were updated. The direct lodash override and both yaml overrides were removed. The lodash-es override remains.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Merge Risk: ⚪ Minimal · up to 6a649

No actionable merge-blocking risk was identified in the dependency updates.

Architecture Summary

Architecture risk: 🔵 Low · up to 6a649

The change affects 2 systems.

Changed systems: site, go.mod

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — site (service) was modified; 2 changed files map to changed impact.
  • observed — go.mod (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in go.mod: The indirect github.com/ebitengine/purego dependency was updated from v0.11.0 to v0.11.1.
  • observed — Modified behavior in go.mod: The indirect github.com/power-devops/perfstat dependency was updated to a newer pseudo-version.
  • observed — Modified behavior in go.mod: The indirect golang.org/x/telemetry and Google genproto API/RPC dependencies were updated to newer pseudo-versions; x/term, x/text, and x/vuln remain unchanged.
  • observed — Modified behavior in site/package.json: Updates the declared version ranges for @astrojs/starlight and astro.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the dependency updates and the five site advisories addressed. It is somewhat awkward but remains specific and relevant to the main changes.
Description check ✅ Passed The description is comprehensive and directly explains the dependency updates, security fixes, retained and removed overrides, cooldown decisions, and validation results. It does not follow every temp…
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sonarqubecloud

Copy link
Copy Markdown

@jmrplens
jmrplens merged commit b44ab96 into main Sep 30, 2026
6 checks passed
@jmrplens
jmrplens deleted the deps-2026-09-30 branch September 30, 2026 09:02
jmrplens added a commit that referenced this pull request Sep 30, 2026
…token (#1087)

## What was wrong

In oauth mode two things on the admission path grew with whatever
callers sent. The verified-identity cache was a map that lost an entry
only to its TTL, a lazy read or the sweep, so it held one entry for
every credential verified inside one TTL, and a valid credential is
cheap to mint on GitLab. The verifier sent `GET /api/v4/user` and up to
two introspection requests for every token the cache did not hold, with
no ceiling anywhere: the pool's probe ceiling counts only the pool's own
`GET /user`, which runs after these have been sent, and the per-address
budgets bound one source at a time. An oauth deployment reachable from
many addresses relayed up to three GitLab requests per invented token.
The register recorded both as F-29 (`ADM-005`) and F-30 (`ADM-002`,
`POL-006`).

## What changed

**The identity cache holds at most 10000 identities**
(`tenancy.OAuthCacheCapacity`, the largest pool `--max-http-clients`
allows, held equal to `PoolSizeMax` by the values test). When full it
drops an expired identity if there is one, and otherwise the one used
least recently, to hold the one GitLab has just verified: refusing the
newcomer would refuse a credential GitLab accepted. It walks for an
expired identity only once the earliest expiry it holds has passed, so a
flood arriving at a cache of live identities costs one comparison each.
Every read counts as a use, so what pushes a live identity out is 10000
other distinct credentials used since its last request, read or verified
alike; on a deployment serving close to that many, every new
verification does. The identity dropped costs its credential a
re-verification and, while its address is blocked, the exemption a
cached identity gives it. The expiry is judged under the same lock that
removes the entry. It is a take across keys, so `ADM-005` records the
decision (`EvictAcrossKeys`, decided in issue 950) and `INV-005` names
it beside the pool entry.

**The verifier's round trips run under slots of its own**, keyed on the
process and not configurable: sixteen verifications at once
(`tenancy.OAuthVerifications`), separate from the pool's sixteen probes,
so neither kind of work occupies the other's slots. It bounds
concurrency, not rate: at 50 ms a round trip, sixteen slots send about
320 requests a second, which is above GitLab.com's allowance for
unauthenticated traffic from one address, so what it holds is the load
relayed in proportion to the instance's speed rather than to how many
tokens arrive at once. The cache is read before a slot is asked for, so
a cached credential never waits, and read again when the wait ends and
once a slot is held, so a request that waited behind another carrying
the same new token finds the identity that one verified rather than
repeating its round trips or being refused. The order cache, then `GET
/user`, then introspection is kept, with the slot held through
introspection. A free slot is taken before anything else is looked at,
so a request whose context has already ended is not refused with a slot
free.

**A new token waits up to five seconds**
(`tenancy.OAuthVerificationWait`) and is then refused with
`oauth.ErrVerificationBusy`, which the bearer guard answers from the
same return and in the same words as a verification with no verdict:
503, `-50300`, `Retry-After: 30`, "GitLab could not verify this token
right now. Retry shortly. The token itself has not been rejected.", not
cached and charged to no budget. The new register row is `ADM-014`, a
process ceiling that waits then refuses; its refusal is `ADM-002`'s, as
`POL-006`'s is `ADM-001`'s, so the failure table keeps its 21 returns.

**What a refused caller can learn.** One that presents a new token to an
instance it knows to be healthy and is refused after the slot wait can
infer that other callers are verifying. That is the one bit `INV-019`
accepts for a bound keyed on the process, and `ADM-014` is listed there
and in ADR-0023 NEG-007 beside `HLD-002`, `HLD-004` and `RTC-007`; the
wording adds nothing to it. The operator tells the causes apart in the
log: `token verification refused: every verification slot stayed busy`
(WARN), `token verification abandoned: the request ended before it was
verified` (INFO), `token verification failed` (ERROR); the first two are
throttled and name neither the caller nor the token.

**What it costs.** A legitimate credential presented for the first time
while a flood holds every slot waits in the flood's queue and is served
only if a slot frees within its five seconds, so under a sustained flood
about the slots' share of new credentials is served and the rest are
refused like the flood. The ceiling protects the instance's load at the
cost of admitting new credentials during a flood. Credentials already
cached never wait for a slot, but they share the listener with the
waiting flood. This layer's pages (the HTTP server mode guide, the
security concept page, the troubleshooting pages, the site in English
and Spanish, and the specification) state the cost as a refusal for as
long as the flood lasts; a later layer of this stack, the OAuth driver
for `make bench-fairness`, measures it (42 and 40 of 120 new credentials
served at 100 ms under 400 invented tokens a second) and corrects them,
with the descriptor and memory conditions on cached credentials.

F-29 and F-30 stay in `AllFindings` as issue 950's and no row carries
them; `TestDecisions_AFindingNoRowCarries_IsAnsweredByItsIssue` names
them in the answered set.

Two refactors ride along, both behaviour-neutral. The identity request
moved out of the verifier closure into `askIdentity`, which brought the
closure back under the cognitive complexity bound once it took a slot.
And `describeScopeShortfall`'s tagless switch became two conditions:
gremlins reads a case expression as outside every covered block, so its
mutants were reported as not covered however the tests drove them.

One test fix rides along too, in a test that was already on main:
`TestLimit_ProcessBoundsListingAcrossCredentials` decided its arm with
the rate limit off by the time measured after the drive returned, which
also counts the time the workers took to stop, and on windows-latest,
which outlists the process's refill by a narrow margin, that failed a
run 77 tools short of a crossing the drive had already made. The verdict
is now kept from the one instant both figures were read at. It sits in
the bottom layer so that every layer above it, each of which runs that
suite on Windows, carries it.

## Figures

Measured through the verifier against a stand-in GitLab on truenas, 16
workers, distinct valid tokens, retained heap after GC:

| Tokens  | Entries before | Heap before | Entries after | Heap after |
| ------: | -------------: | ----------: | ------------: | ---------: |
| 10      | 10             | 0.30 MB     | 10            | 0.33 MB    |
| 100     | 100            | 0.45 MB     | 100           | 0.40 MB    |
| 1000    | 1000           | 0.96 MB     | 1000          | 1.10 MB    |
| 10000   | 10000          | 6.58 MB     | 10000         | 7.11 MB    |
| 100000  | 100000         | 60.5 MB     | 10000         | 7.04 MB    |

About 605 bytes an entry before and 700 after (the recency list adds its
element).

A flood of invented but valid tokens all at once, 50 ms a request at the
stand-in:

| Flood | Peak in flight before | Requests before | Peak in flight after
| Requests after | Verified after | Refused busy after |
| ----: | --------------------: | --------------: | -------------------:
| -------------: | -------------: | -----------------: |
| 200 | 200 | 600 | 16 | 600 | 200 | 0 |
| 2000 | 2000 | 6000 | 16 | 1584 | 528 | 1472 |

`test/e2e/http` pins the ceiling on the wire: twenty tokens, each
forwarded by a trusted proxy for an address of its own, against an
instance that holds each identity request; sixteen reach it and no more,
four are refused 503 `-50300` with `Retry-After: 30`, no challenge and
`ADM-002`'s words, a credential already cached is served while every
slot is held, and with the per-address and distinct-credential budgets
both blocking on their first charge a new token from every one of those
twenty addresses is served afterwards.

## Validation checklist (tenant-policy-spec.md)

- **VAL-001, question**: both rows answer Admit.
- **VAL-002, key**: `ADM-005` stays on `KeyVerified` (a verified
credential, cheap to mint: a personal access token is a form with no
administrator); `ADM-014` is on `KeyProcess`, which no caller can mint.
No new key.
- **VAL-003, class**: `ADM-005` class C, `ADM-014` class P; neither is
class D, and `ADM-014`'s stated reason ("counting work rather than
callers") is in the process's unit.
- **VAL-004, ceiling or share**: `ADM-014` is a ceiling; `ADM-005` a
lifetime with a capacity. No share on any key.
- **VAL-005, process partner**: `ADM-014` is itself keyed on the
process, protects the process's upstream load, and is a constant because
an operator who could raise it could undo it (INV-004). The cache
capacity is a constant on the one process-wide cache.
- **VAL-006, at capacity**: `ADM-005` evicts across keys, expired first,
then least recently used (decided in issue 950: refusing would refuse a
credential GitLab accepted); `ADM-014` waits five seconds, then refuses.
- **VAL-007, refusal**: `ADM-014` on the HTTP gate only (OAuth mode is
HTTP-only; `StdioKey` none): 503, JSON-RPC `-50300`, `Retry-After` fixed
at 30 s, no challenge, retry-later class, `ADM-002`'s text. `ADM-005`
refuses nothing.
- **VAL-008, charges**: nothing. The failure table's
`verification-failed` return is uncharged and G7 holds it; the unit test
drives all three budgets (per-address, transport source, distinct
credential) at a limit of one and the httpe2e test the two a flag
reaches.
- **VAL-009, values**: `OAuthCacheCapacity` 10000, `OAuthVerifications`
16, `OAuthVerificationWait` 5 s; constants, zero not applicable, no flag
or variable, so no malformed-value policy; all three frozen in
`TestValues_HoldTheirPins`.
- **VAL-010, telemetry**: no metric was added; the two slot log lines
carry the cause only, no token and no address.
- **VAL-011, stdio**: none; OAuth verification runs only in HTTP mode.
- **VAL-012, measurement**: `ADM-014` does not claim to make a
population better off than without it; it bounds what the process relays
to GitLab. Its cost to the one population it reaches is measured in a
later layer of this stack, which adds the OAuth driver to `make
bench-fairness` (`BOUND=oauth-verification`): against a stand-in
answering in 100 ms and a flood of 400 invented tokens a second, once
the flood has settled into its queue, 42 and 40 of 120 new credentials
were served in two runs, after 5.5 s rather than 0.5 s, and the rest
refused; no cached request was refused in any arm, and their medians lay
between 9 and 16 ms with the ceiling and without it; and the instance
received about 160 requests a second, sixteen slots over the round trip,
with at most 20 at once, against about 420 a second with up to 56 at
once without the ceiling. A cached credential is outside the ceiling by
construction (the cache is read before a slot is asked for), which the
unit and httpe2e tests pin and the measurement confirms, as long as the
process has a descriptor and memory for its connection: the waiting
flood holds a descriptor and about 50 KiB for each waiting request, and
the measurement ran under a descriptor limit of 1048576 with no memory
limit reached.

## Checks

- Tests: `internal/oauth`, `internal/tenancy`, `cmd/audit_tenancy` and
`cmd/server` pass with 100.0% statement coverage each; the whole
`test/e2e/http` suite passes, and the flood test passed three runs in a
row; `internal/oauth` passes `-race` and `-shuffle=on -count=3`, and the
guard's tests pass `-race`.
- Conditions (gobco v1.3.4): `internal/oauth` 214/214,
`internal/tenancy` 408/408, `cmd/audit_tenancy` 1002/1002. `cmd/server`
could not be instrumented in place when this was measured (#1017); the
conditions this change adds there (`logUnverified`'s two,
`describeScopeShortfall`'s two) are each driven both ways by a named
test case.
- Mutants (gremlins, `--workers 2 --timeout-coefficient 20`):
`internal/oauth` killed 109, lived 0, not covered 0, timed out 0;
`internal/tenancy` killed 226, lived 0, timed out 0, not covered 66, all
arithmetic on constant initializers in `values.go`, `codes.go` and
`decision.go`, which carry no statement counter (the known tool-artifact
class). `cmd/server`: `bearer_guard.go` mutated against the whole
package's suite with every other file excluded (the package's other
files are unchanged here apart from one comment in `main.go`): killed
65, lived 0, not covered 0, timed out 0.
- `make check-tenancy`: 82 rows, 21 failures and 280 declared sites over
202 packages (21 refusal returns, 76 refusals, 12 reasons and 31
settings read); 0 findings, 56 declarations exempted.
- golangci-lint (e2e, httpe2e tags) on the five packages,
`check-test-subtests`, `check-test-goroutines`, `check-test-file-names`,
`check-doc-tool-names`, `format_md_tables --check` and markdownlint on
the changed pages: clean.
- The figures above were measured on this layer before it was rebased
onto main at 5da9699. At its place in this stack on 50ef2cf, it
built and vetted, the e2e gitlab, http and stdio packages compiled, the
unit suites of `cmd/server`, `internal/tenancy`, `internal/oauth` and
`cmd/audit_tenancy` passed, and `make check-tenancy` read the same 82
rows and 280 sites. The stack now sits on b44ab96, which adds #1083,
#1084 and #1085; they change the upstream register, indirect module
versions and the site's package pins, none of them a file this stack
writes, and every commit of the stack applied unchanged.

## Decisions I took here

- The cache capacity is a constant 10000 rather than a setting: it
equals `PoolSizeMax`, the largest `--max-http-clients`, so no
configurable pool serves more credentials than the cache remembers, and
at about 700 bytes an entry that is about 7 MB.
- The verification ceiling mirrors `POL-006`'s numbers (sixteen at once,
a five-second wait, then refuse) on slots of its own, constant and not
configurable (INV-004).
- The ceiling is a new row, `ADM-014`, rather than a change to
`ADM-002`, whose shape is a rule; `ADM-002` and `POL-006` drop F-30.
- `ADM-005` records its eviction as a take across keys
(`EvictAcrossKeys`), and the specification's `INV-005` and ADR-0023 name
it beside the pool entry. No new capacity kind was added.
- The refusal reuses `ADM-002`'s words and return, following `RTC-007`
reusing `RTC-003`'s, and `ADM-014` joins `INV-019`'s one-bit exceptions
and ADR-0023 NEG-007. Both are decided: the listing because no wording
closes the bit (NEG-007), and the words because what they tell the
caller to do, retry shortly with its token intact, is right for a
request GitLab never saw.
- A request whose own context ends while it waits for a slot gets its
own error rather than `ErrVerificationBusy`, and is answered by the same
uncharged 503 path rather than the generic verification failure, which
logs at ERROR.
- The slot is held for the whole verification, introspection included,
so all three requests a verification may send are counted.
- VAL-012 is answered by a measurement rather than a statement: a later
layer of this stack builds the OAuth driver `make bench-fairness`
lacked, and the specification cites what it measured.

## Not done

- `POL-006`, the pool's probe ceiling, discloses the same one bit and is
not listed in `INV-019` or NEG-007. It was so before this change, and
listing it, with the reason its refusal gives nothing more away than
`ADM-014`'s, is a follow-up of its own rather than part of this stack.
- No per-key single flight for concurrent verifications of the same new
token, and no handoff of the guard's verified identity to the SDK's own
verify, which would close the narrow window in which that second verify
can miss the stored identity. Both are described in the code comments.
- The flood e2e test does not show eviction on the wire, which would
take more than 10000 verifications through the binary; it is pinned at
unit level, including 10000 flood entries into a cache of 100 while a
credential in steady use keeps its entry.
- No whole-package gremlins run on `cmd/server`: only `bearer_guard.go`
was mutated, against the whole suite.
- The first commit's message still carries the wording the review
corrected; the second commit says what changed, and a squash takes this
description.

Closes #950.
jmrplens added a commit that referenced this pull request Sep 30, 2026
…or limit (#1088)

A `tools/call` holds its `POST` for as long as the call runs: up to an
hour for a pipeline wait, and any call for as long as GitLab keeps it
waiting. The rate limit bounds how often a credential calls, not how
many of its calls are still running, and the pool bounds entries, not
what an entry holds, so nothing bounded how many calls one credential,
or all of them, could keep open (finding F-31). Under a hard descriptor
limit of 1024 the process held about five hundred and then stopped
accepting connections at all, `/health` among them.

This is the second of issue 951's three layers. It adds register row
`HLD-011`: a ceiling on the calls the process holds open, keyed on the
process and set by no flag. The stateful sessions, the other half of
F-31, are the third layer.

**What it is sized from.** The descriptor limit the process runs under,
read once at startup: an eighth of it spare, one descriptor for each of
the 512 listen streams (`HLD-002`), and two for each held call (the
caller's connection and the one to GitLab). The Go runtime raises the
soft limit to the hard one before `main`, so the figure follows the hard
limit: 192 under 1024, 1536 under 4096, 229120 under the 524288 a
systemd service gets by default, 458496 under 1048576. Windows has no
such limit to read and gets the figure for 1024. No flag moves it;
raising the descriptor limit raises it together with what it protects.
The startup line announces it as `held_requests_per_process`.

**What it counts.** Every call that reaches GitLab (the methods
`MeterFor` charges to the tool-call and completion buckets:
`tools/call`, `resources/read`, `resources/subscribe`, `prompts/get`,
`completion/complete`), a `subscriptions/listen` aside, counted where
the SDK dispatches it. So each call of a JSON-RPC batch counts on its
own, and a response a client sends to a request of the server's own (its
answer to an elicitation), a notification, or a listen on any revision
never does. On protocol 2026-07-28 or later, where the SDK holds
`Mcp-Method` to the body and refuses batches, the gate takes the slot
itself after admission and hands it to the call through the carrier, so
the refusal there happens before the SDK reads a byte.

**How it refuses.** Always in the words `This server is busy. Retry
later.`, naming no bound; the log line (`request refused: too many
requests held across the process`) names `scope=process` and the figure.
On 2026-07-28 or later: a gate 503, `-50300`, `Retry-After: 30`, and the
connection closed, so refused callers do not accumulate descriptors of
the limit the ceiling protects. On older revisions: the way the rate
limit refuses the same method (a `tools/call` as a result with
`isError`; `resources/read`, `resources/subscribe` and `prompts/get`
with `-42900`; a completion as an empty completion). Nothing is charged,
and the credential's rate is not spent: the middleware runs before the
rate limit.

## Measured

Truenas, `cmd/bench_resources -held` against a stand-in GitLab that
holds every project read, headers first. Per held call, linear to 4000
calls: 2.00 descriptors, 6.00 goroutines, about 51 KiB of live heap and
about 190 KiB resident.

| Run | Offered | Held | Served | Refused | Failed | Descriptors |
`/health` |
| ------------------------------------------ | ------: | ---: | -----: |
------: | -----: | ----------: | ------------------ |
| before, 1 credential, hard 1024 | 1000 | 503 | 558 | 0 | 442 | 1024 |
unanswered for 5 s |
| before, 1 credential, hard 1024 | 4000 | 503 | 3466 | 0 | 534 | 1024 |
unanswered for 5 s |
| this change, 1 credential, hard 1024 | 4000 | 192 | 192 | 3808 | 0 |
394 | 200 in 0 ms |
| this change, 100 credentials, hard 1024 | 4000 | 192 | 192 | 3808 | 0
| 394 | 200 in 0 ms |
| this change, 1 credential, hard 1048576 | 4000 | 4000 | 4000 | 0 | 0 |
8010 | 200 in 0 ms |
| this change, 100 credentials, hard 1048576 | 4000 | 4000 | 4000 | 0 |
0 | 8010 | 200 in 0 ms |

The first cut of this layer reached 1141 to 1280 descriptors with a
hundred credentials, because the refused connections stayed open; the
refusal now closes its connection. `make bench-held` runs the ladder
(`HELD_CREDENTIALS=100` for the second row). `test/e2e/http` holds the
same behaviour against the real binary started under `ulimit -n 1024`:
192 calls held, a modern call refused in the gate with the connection
closed, an older revision's call refused as a tool error, a batch of two
refused call by call, a notification accepted, a listen acknowledged,
600 more calls offered at once all refused while `/health` answers, and
every held call served once GitLab answers.

## What the review of the first cut found, and what changed

- **A POST is not a call.** A JSON-RPC batch below 2025-06-18 carried as
many calls as the body held on one slot; a POST carrying only a response
or a notification was counted and refused at a full ceiling, which kept
an elicitation's answer from the call waiting on it; and an older
revision's listen was counted twice. The count moved to where the SDK
dispatches calls, as above.
- **192 was sized for a soft limit no Go process runs under.** On a
default systemd host it cut what the process could hold to about a
twentieth and let one credential's pipeline waits have everyone else
refused for up to an hour. The ceiling is now derived from the hard
limit, and the pages that called 1024 the common case say what the limit
actually is.
- **The many-credential measurement under 1024 was missing** and the
first cut would have failed it; it is the fourth row above.
- **The refusal named the bound**; it now says only that the server is
busy (item 5 under Decided on issue 951).
- **PAT-003 was cited for the opposite order**; the departure is
recorded instead (item 4 under Decided on issue 951).
- **The OAuth wiring was untested**;
`TestRegisterOAuthMCPHandlers_RefusesPastTheProcessHeldCeiling` fails if
`registerOAuthMCPHandlers` stops passing the count.
- **POL-003's answer was unpinned**;
`TestHeldCall_TakesOneSlotAndLeavesItsEntryEvictable` pins that a held
call leaves its entry evictable and that the call outlives the eviction.
- **The e2e `/health` assertion could not fail**; the binary now runs
under `ulimit -n 1024`, where 792 held calls would not fit.
- **Docs**: the sizing pages no longer call the failed-authentication
budget unconfigurable, security.md no longer says every call past the
limit failed (558 of 1000 were served), and the listen exemption is
stated without a condition because there is none left. The enterprise
guide now counts the two unconfigurable limits the way the sizing pages
do (the watcher ceilings, then the held calls), with the configurable
budget after them.
- **The house gate was measured over the files this layer writes only.**
`cmd/bench_resources` is now measured whole and `cmd/server`'s `main.go`
is measured too; the results, and what closed each survivor, are under
Gates.

## Decided on issue 951

The six questions this layer put are answered, each as proposed here, so
nothing in this layer's row, pins, gate, e2e expectation or pages
changes. The layer of this stack that records issue 951's decisions
writes each answer and its reason where the tree still presented the
question: the register row's comment, the specification, ADR-0023, the
HTTP server, enterprise, remote deployment, security and resource
consumption pages, and the site in English and Spanish.

1. **The value stays derived** from the hard descriptor limit. No
operator setting moves it, and raising the limit raises it together with
what it protects. **Windows keeps 192** (`FallbackDescriptorLimit` 1024)
rather than no ceiling: 192 protects no descriptor there, and it is kept
so the platform is not left unbounded by default.
2. **No per-credential ceiling stands beside it**, as `HLD-001` stands
beside `HLD-002`. The credential is a key a caller can mint (`INV-003`),
so a per-credential number multiplies with every token a caller mints
and bounds nothing the process ceiling does not. One credential can
therefore fill the ceiling where the limit is small, in about fifteen
seconds at the default rate (a burst of 40, then ten a second), and the
operator pages say so.
3. **The server keeps no memory cap.** At about 190 KiB a held call, a
memory limit set outside the server is what bounds held calls where the
descriptor limit is large. I named the container's limit or
`GOMEMLIMIT`; the pages set the container's as the bound and describe
`GOMEMLIMIT` as a soft target the runtime collects toward and live held
calls pass, so it is a lever and not that bound.
4. **The slot is taken after admission**, the departure from issue 951's
request to take the process slot before anything per key, accepted for
its reason: a slot taken first would let a caller with no credential
hold one for as long as its verification takes. A refused newcomer has
spent its admission by then: in legacy mode its pool entry (with
`POL-006`'s probe and, at the pool's bound, another credential's quiet
entry evicted, `POL-002`), in oauth mode one `ADM-014` slot, and none of
its rate.
5. **INV-019.** With no per-caller ceiling beside it, any refusal tells
its caller the process is full whatever the words, which is the one bit
the INV-019 decision accepted for a bound keyed on the process, and
`HLD-011` falls under that exception rather than needing its own.
6. **Two refusal channels**, accepted as implemented, because only a
modern POST's headers prove what it carries: a gate 503 on 2026-07-28 or
later, in-band before it. An older revision's in-band refusal does not
close its connection, since the middleware has no connection to close;
the ceiling bounds calls, not connections, and the guide says so.

The two confirmations issue 951 asked for: a held call does not make its
pool entry busy (`POL-003` unchanged, and now pinned by a test); the
process listing bucket follows `RTC-003` (the first layer, 951a).

## Decisions I took here

- The ceiling is derived from the hard descriptor limit read at startup,
not fixed: an eighth spare, one descriptor for each of the 512 listen
streams, two for each held call, with a floor of one call. On Windows,
which has no such limit to read, it is sized as if the limit were 1024
(decided on issue 951, item 1).
- It is keyed on the process alone, with no per-credential partner, and
no flag or variable moves it (INV-004, INV-018; decided on issue 951,
item 2).
- The slot is taken after admission and the session ownership check,
which departs from issue 951's request to take the process slot first;
the departure is recorded on the row (decided on issue 951, item 4).
- The refusal reuses `UpstreamRetryAfter` (30 s, fixed) and
`CodeUnavailable` (`-50300`) rather than adding values, and says only
that the server is busy; the log line alone names the scope and the
figure (decided on issue 951, item 5).
- Calls are counted where the SDK dispatches them, so a batch counts
call by call and an elicitation's answer, a notification and a listen
never count; only on 2026-07-28 or later does the gate take the slot
itself, because only there do the headers prove what a POST carries
(decided on issue 951, item 6).
- A held slot does not make a pool entry busy and does not delay its
eviction (`POL-003`).
- I changed `cmd/bench_resources` to remove equivalent mutants rather
than record them (`admit` now runs over one count, the sampler's count
of answered processes is a bool, the goroutine-dump scanner has no
initial buffer, the first chart tick candidate lost its `+1`), moved the
latency stop's edge into `crossesLatencyCeiling` so a test can reach it,
and gave the warm-up its own ceiling (`warmUpTimeout`); each change
preserves behaviour for every input that reaches it, and the package
stays at 100% of statements and 1322 of 1322 conditions.
- `auth_gate.go:793` (`effectiveWindow <= 0`) stays classified rather
than deleted: retiring that three-way window fallback is issue 958's, as
`AUB-002` and #1015 already record.

## Not done

- Idle keep-alive client connections are not bounded by this ceiling,
which bounds calls, not connections; an older revision's in-band refusal
does not close its connection.
- Memory is not bounded by it, and issue 951 decided the server adds no
cap for it: about 190 KiB a held call, so under a large limit the
container's memory limit bounds held calls first.
- Listen streams and held calls filling the process together were not
measured; the sizing arithmetic says they fit.
- `cmd/server` was mutated over the four files this layer writes, not
whole; `main.go`'s survivors are the ones #1015 recorded on identical
code.
- The stateful sessions, the other half of F-31, are the next layer.

## Validation checklist (tenant-policy-spec.md)

- **VAL-001, question**: Allow.
- **VAL-002, key**: `KeyProcess`, which no caller can mint. No new key.
- **VAL-003, class**: P. Its reason ("only a ceiling keyed on the
process bounds the process") and the reason's unit are both the process.
- **VAL-004, ceiling or share**: a ceiling. It promises no caller a
share.
- **VAL-005, process partner**: it is itself keyed on the process,
protects the process's descriptors, and is set by no flag, because an
operator who could raise it could undo the bound (INV-004, INV-018). No
per-caller row stands beside it, which issue 951 decided (item 2 above).
- **VAL-006, at capacity**: refuses the newcomer; never evicts or ends a
held call.
- **VAL-007, refusal**: gate on protocol 2026-07-28 or later (503,
`-50300`, fixed 30 s `Retry-After`, no challenge, connection closed);
in-band on older revisions (`tools/call` as a tool error;
`resources/read`, `resources/subscribe` and `prompts/get` as RPC
`-42900`; `completion/complete` as an empty completion). Retry-later
class throughout, text `This server is busy. Retry later.`. On stdio:
none.
- **VAL-008, charges**: nothing. The refusal comes after admission and
before the rate limit, so it spends no budget and no rate token.
- **VAL-009, value**: `Source: Derived`, from `RLIMIT_NOFILE` read at
startup and three pinned constants (`HeldRequestDescriptors` 2,
`DescriptorSpareDivisor` 8, `FallbackDescriptorLimit` 1024) beside
`ListenStreamsPerProcess`; a floor of one call; zero is not applicable;
no flag or variable, so no malformed-value policy.
- **VAL-010, telemetry**: the reason only. The refusal's log line
carries the reason, the scope and the figure and nothing about the
caller, and it is throttled like every refusal a caller can cause at
will. An in-band refusal is an ordinary result or error on the call's
span, under whatever identity policy the deployment set for every call.
No metric counts the refusals; one was not added in this layer.
- **VAL-011, stdio**: not applied (`StdioKey` none). The middleware
counts only calls that arrived on an HTTP POST, so a stdio server
carries it and counts nothing.
- **VAL-012, measurement**: the ceiling does not claim to protect one
population from another. It protects the process's descriptors, and it
is measured by the held mode above, which is what that claim needs;
`make bench-fairness` answers a claim this ceiling does not make.

## Gates

Measured on this layer before it was rebased onto main at 5da9699: the
line numbers below are that tree's, and the one commit cited below
(fef2e45) is named as it sits in this stack; main's server card change
(#1061) moved `main.go`'s lines and none of the code these figures
describe. The stack was then rebased onto 50ef2cf, whose only change
to a file this layer writes is the serialtypecheck blank import in
`cmd/audit_tenancy/main_test.go`, and at its place in this stack on that
base the layer built and vetted, the e2e gitlab, http and stdio packages
compiled, the unit suites of `cmd/server`, `internal/tenancy`,
`internal/oauth` and `cmd/audit_tenancy` passed, and `make
check-tenancy` read the same 83 rows and 299 sites. The stack now sits
on b44ab96, which adds #1083, #1084 and #1085; they change the
upstream register, indirect module versions and the site's package pins,
none of them a file this stack writes, and every commit of the stack
applied unchanged.

- `go test -shuffle=on`: 100.0% of statements in `cmd/server`,
`internal/tenancy`, `cmd/bench_resources` and `cmd/audit_tenancy`;
`-race` on the held tests; the whole `test/e2e/http` module (578 tests,
the Docker-only proxy and balancer cases skipped); the e2e gitlab and
stdio packages compile.
- golangci-lint with the `e2e` and `httpe2e` tags: 0 issues.
markdownlint and `format_md_tables --check`: clean.
- `make check-tenancy`: 83 rows, 21 failures and 299 declared sites over
202 packages (21 refusal returns, 80 refusals, 13 reasons and 31
settings read); 0 findings, 56 declarations exempted.
`check-dead-consts`, `check-test-file-names`, `check-test-subtests`,
`check-test-goroutines`, `check-doc-tool-names` and `check-action-ids`
pass.
- gobco: `internal/tenancy` 408/408, `cmd/bench_resources` 1322/1322,
`cmd/audit_tenancy` 1002/1002. `cmd/server` is measurable through a
staged copy without its non-Linux twins (the issue 1017 approach):
2040/2082 (2039 on one of three runs, a timing-dependent condition
outside these files), with none of the open conditions in `held.go`,
`descriptor_limit.go` or the lines this layer changed in `auth_gate.go`
and `main.go`; all of them predate this change, `auth_gate.go:793` among
them (the unobservable guard below).
- gremlins (`--workers 2 --timeout-coefficient 20`, `INVERT_LOGICAL`
on):
- `internal/tenancy`, whole package: 226 killed, 0 lived, 0 timed out,
66 not covered, every one of them arithmetic in a constant declaration
(`codes.go` 26, `values.go` 33, `decision.go` 7), which has no statement
for coverage to count; each value is pinned by
`TestValues_HoldTheirPins`, and the count is the base's.
- `cmd/server`, scoped to the four files this layer writes (the house
rule allows it for this package), in two runs:
- `held.go`, `descriptor_limit.go` and `auth_gate.go`: 99 killed, 1
lived, 8 not covered, 0 timed out (`held.go` and `descriptor_limit.go`
alone: 23 killed and nothing else). The first pass over the held files
timed out three mutants, because two tests hung instead of failing when
the gate refused a POST they expected to pass; both now fail at once
(fef2e45). The transport budget's two window-edge boundaries (`charge`
re-opening a pair at exactly one window, and the sweep dropping it then)
are now killed: two tests land a charge exactly one window later on a
fake clock (`testing/synctest`), which the cmd/server sweep (#1015) had
recorded as a clock no test could schedule. The one lived is
`window()`'s `effectiveWindow <= 0` (`auth_gate.go:793`):
`newTransportBudget` never stores a window of zero or less, so the guard
is unobservable, and #1015 recorded it and left it for issue 958, which
retires that three-way fallback; deleting it here would be half of 958.
The 8 not covered are constant initializers: the two JSON-RPC code
literals, whose sign flips
`TestRefusalCodes_MirrorTheirStatusAndSitWhereTheSpecificationAllows`
kills when they are applied by hand, and `+` to `-` between string
constants, which does not compile.
- `main.go`: 338 killed, 4 lived, 25 not covered, 0 timed out. This
layer's change to it carries no mutable token (a middleware
registration, two struct fields, a log attribute and a help line). The 4
lived are the four boundaries #1015 recorded, on identical code (at
lines 1899, 2605, 3244 and 3744 in #1015's tree, and 1911, 2629, 3273
and 3776 in the one measured here); the 25 not covered are 15 constant
and variable initializers, 9 `switch` case expressions and the
equivalent `802:19` #1015 recorded, which is where #1015 left the file.
- `cmd/bench_resources`, whole package, with the script's default
`--exclude-files=/` so the out-of-process stand-in under
`testdata/standin` is not counted. Before this pass: 1173 killed, 24
lived, 67 not covered, 2 timed out, all in files this layer does not
otherwise touch. After it: **1174 killed, 17 lived, 67 not covered, 0
timed out.**
- The two timeouts were `coldList`'s retry rule mutated to ask again
after an answer it should take: every test that admits against a
stand-in refusing `tools/list` then retried until `callTimeout`, five
minutes a warm-up, and the suite timed out instead of failing the test
that pins the rule. The warm-up now has its own ceiling
(`warmUpTimeout`, `callTimeout` in a run and fifteen seconds under the
tests); both mutants are killed.
- Killed by a new test: `waitUntil` at an instant that is exactly now on
a cancelled run (a fake clock); the latency stop's edge, moved into
`crossesLatencyCeiling` so a tail exactly at the ceiling can be asked
for; `admit`'s `want-have` (twice), now the one count the loop runs
over; and two `switch` case edges gremlins reports as not covered that
no test pinned, `msLabel` at 1 and 10 ms and `cpuFigures` on equal
samples.
- Removed rather than recorded: the `+1` on `linearXTicks`' first
candidate (the spacing test already turns the origin away), the
sampler's count of answered processes (read only for zero, now a bool)
and the initial buffer of the goroutine-dump scanner (only the line
ceiling matters).
- The 17 lived, each one of the shapes
`docs/development/testing/README.md` records. Two sides that agree at
the boundary: `chart.go:276` (`Log10(0)` is `-Inf`, which clamps to the
floor where `v = lo` lands), `chart.go:330` and `proc_sample.go:271`
(the assignment writes the value already held), `chart.go:570` (a
candidate at `maxX` is turned away by the spacing test), `chart.go:718`
and `:721` (a shift by zero), `fairness.go:242` (at share zero the
division is `+Inf` for every bound, all of which meter a positive rate,
and `min` caps it at the same default), `fairness.go:617` (zero times
anything), `fairness_result.go:550` (a floor of zero is met by any busy
figure, never negative), `figures.go:402` and `render.go:230` (a string
of exactly the slice's length is returned whole either way),
`proc_sample.go:32` (a probe whose deadline has passed falls back to
100, what `getconf` answers on Linux). Unreachable by construction:
`chart.go:299` (ticks are whole steps and the bound is half a step past
the last), `fairness_result.go:715` (reached only once the claim exceeds
a spread of zero or more), `fairness_result.go:734` twice (a zero delta
returns before either comparison, the tie-break shape), `theme.go:96`
(the index is taken over text that begins with the selector, so it is
never 0).
- The 67 not covered: 29 constant and variable initializers and 36
`switch` case expressions, which carry no statement counter, and 2 at
`fairness_run.go:277`, a call argument after a function literal that the
profile attributes to no block. Applied by hand, the
`fairness_run.go:277` mutant is killed by
`TestDrive_SpreadsEachPopulationAcrossOnePeriod` and the two protocol
codes' sign flips by
`TestClassifyOutcome_KeepsRefusalsApartFromSuccessAndFailure`; every
`switch` case boundary has a test at its edge now (the two that had none
are above); `+` to `-` between string constants does not compile. gobco
holds every case condition both ways (1322/1322).

Part of #951.
jmrplens added a commit that referenced this pull request Sep 30, 2026
… stream when it opens (#1089)

On `--stateless=false` the SDK keeps every session a client opens until
the client deletes it, the pool evicts its credential or it has sat idle
for `--session-timeout` (30 minutes by default), and `initialize` is
metered to no bucket (`RTC-004`), so nothing bounded how many sessions a
caller, or all of them, could open. Each session may also hold its
standalone stream, a `GET` the SDK serves for the session's life. Under
a hard descriptor limit of 1024 the process kept every session it was
offered until those streams had taken every descriptor, at about a
thousand sessions, and from there answered nothing, `/health` included.
The register recorded this as finding F-31 on `HLD-010` and `IDN-010`.

This is the third of issue 951's three layers. It makes `HLD-010` a
ceiling on the stateful sessions the process keeps, keyed on the process
and set by no flag, and has each session reserve a held-call slot
(`HLD-011`) for its standalone stream when it opens.

**What it is sized from.** Half of `HLD-011`'s figure
(`SessionHeldDivisor`, 2): 96 under a hard descriptor limit of 1024, 768
under 4096, 114560 under the 524288 a systemd service gets by default,
229248 under 1048576, and 96 on Windows, which is sized as 1024. The
`POST` that opens a session takes the session's slot and one held-call
slot for its stream together (`mcpServerGate.takeSessionSlot`), and the
session keeps both until it ends, whether its stream is open or not. The
sessions therefore hold at most half the held-call slots: the descriptor
budget `HLD-011` was derived from holds as it was sized, a call on an
open session is still served when every session slot is taken, and a
session the process keeps is never refused its stream. The standalone
`GET` takes no slot of its own. The startup line announces the figure as
`stateful_sessions_per_process` on a deployment that keeps sessions; on
the default stateless transport nothing is counted and nothing is
announced.

**What it counts.** Every `POST` carrying no `Mcp-Session-Id` on a
revision before 2026-07-28, which is what the SDK opens a session for.
The gate takes the slots once the credential is admitted and before the
SDK creates anything, the same order `HLD-011` records. The first
request the SDK dispatches on the new session keeps them, found through
the per-POST carrier, and a goroutine waiting on `ServerSession.Wait`
gives them back when the session ends, however it ends (a `DELETE`,
`--session-timeout`, the credential's eviction). A `POST` whose session
did not survive it (not an `initialize`, or one the SDK refused) gives
them back as the gate returns. One flag guards both ends, so the slots
are released once. A `POST` on 2026-07-28 or later takes no slot: the
stateful transport answers it, `server/discover` included, with `400`,
`-32022` and the revisions it serves, and keeps no session for it, so
the client falls back.

**How it refuses.** In the gate, before a session exists, with
`HLD-011`'s answer word for word: `503`, JSON-RPC `-50300`,
`Retry-After: 30`, the connection closed, and `This server is busy.
Retry later.`. It costs the credential no rate-limit token and is
charged to nothing. The log line `request refused: too many stateful
sessions across the process` names `scope=process` and
`limit_stateful_sessions`; a `POST` that finds a session slot and no
held slot for its stream is refused in the same words and logs the
held-call ceiling's line.

**Why the stream is reserved at open.** The first version of this layer
counted the stream as a held call when its `GET` arrived and refused it
at a full held ceiling. The go-sdk client never asks again for a
standalone stream it was refused: it logs a warning and gives the stream
up, and the server then drops every message meant for that session
outside a response, resource updates and every `list_changed` among
them, while the session carries on and its subscribe still reports
success.
`TestStatefulSessions_AGoSDKClientKeepsItsStreamWithEveryHeldSlotTaken`
drives the SDK's own client with every held slot taken and holds that
its stream is served, and that the next client's connect fails in the
gate.

**What it bounds, and what it costs.** Descriptors, and memory only
where the hard limit is small: an idle session costs four goroutines and
88 to 110 KiB of resident set, which the descriptor limit does not
raise, so the 114560 sessions a hard limit of 524288 allows come to
about ten to twelve GiB idle and the container's memory limit bounds
them first. And filling the ceiling costs a caller nothing, since
`initialize` spends no rate and an idle session holds no connection: one
credential can open every session, and while it holds them every other
tenant's `initialize` is refused for up to `--session-timeout`, half an
hour by default and a day at most. Before this ceiling an idle session
refused nobody. With `--session-timeout=0` (`END-005`, which keeps its
meaning) a session nobody deletes keeps its slots until the pool evicts
its credential, after `--pool-idle-timeout` without a request or to make
room at `--max-http-clients`, and startup now warns about that. Issue
951 decided that none of the three levers stands beside this row: no
per-credential ceiling, no metering of `initialize` and no fixed cap
beside the derived figure, for the reasons under Decided on issue 951
below.

`heldRequestsFailure` becomes `processBusyFailure`, the refusal both
rows share, and `heldRequests` becomes `processSlots`, the count both
ceilings use, whose `acquire` takes an optional hook between its read
and its swap so a test drives the retry on purpose. The startup line's
attributes move into `httpStartupAttrs`, which keeps `serveHTTPOn` under
the maintainability threshold the new attribute pushed it past.

## Measured

Truenas, the new sessions mode of `cmd/bench_resources` (`make
bench-sessions`, first commit of this layer), against the stand-in
GitLab, before this layer and at its tip. Per session over the idle
process, read at 500 to 4000 sessions: before, an idle session held 3
goroutines, 10 to 17 KiB of live heap, no descriptor and 56 to 119 KiB
of resident set, and one holding its stream 6 goroutines, 24 to 25 KiB,
one descriptor and 95 to 124 KiB. After, from one credential and from a
hundred, an idle session holds 4 goroutines (the fourth gives its slots
back), 10 to 20 KiB and 88 to 110 KiB, and one holding its stream 7
goroutines, 25 to 28 KiB, one descriptor and 66 to 108 KiB.

| Run | Offered | Open | Live | Refused | Failed | Descriptors |
`/health` |
| ------------------------------------------ | ------: | ---: | ---: |
------: | -----: | ----------: | ------------------ |
| before, 1 credential, streams, hard 1024 | 2000 | 1013 | 0 | 0 | 0 |
1024 | unanswered for 5 s |
| before, 1 credential, streams, hard 1024 | 4000 | 1012 | 1012 | 0 |
1973 | 1024 | unanswered for 5 s |
| before, 1 credential, streams, inherited | 4000 | 4000 | 4000 | 0 | 0
| 4013 | 200 in 0 ms |
| before, 100 credentials, idle, inherited | 4000 | 4000 | 4000 | 0 | 0
| 811 | 200 in 0 ms |
| after, 1 credential, streams, hard 1024 | 4000 | 96 | 96 | 3904 | 0 |
112 | 200 in 1 ms |
| after, 100 credentials, streams, hard 1024 | 4000 | 96 | 96 | 3904 | 0
| 183 | 200 in 1 ms |
| after, 1 credential, idle, hard 1024 | 4000 | 96 | 96 | 3904 | 0 | 19
| 200 in 0 ms |
| after, 1 credential, streams, inherited | 4000 | 4000 | 4000 | 0 | 0 |
4011 | 200 in 0 ms |
| after, 100 credentials, idle, inherited | 4000 | 4000 | 4000 | 0 | 0 |
811 | 200 in 0 ms |

"Open" is the bench's count of sessions open at the settled sample;
"Live" is how many of them answered a ping, each waiting no longer than
the `/health` probe, stopping at the first that got no answer. In the
first row the process accepted no connection at the sample, so none
answered though 1013 were open. "Inherited" is the host's container hard
limit, 1048576. The SDK's session map and `sessionOwners` hold one entry
per open session and drop it when the session ends
(`TestSessionOwners_ADisconnectedSession_IsForgotten`), so both stop at
the ceiling; that is inferred from the test and the ceiling, not sampled
by the bench.

`test/e2e/http` holds the same behaviour against the real binary under
`ulimit -n 1024` with `--stateless=false`: 96 announced and 192 held
calls, 96 sessions opened each with its stream, the 97th `initialize`
refused in the gate with the connection closed and no session id, a call
on an open session still served, 600 more `initialize`s at once all
refused while `/health` is probed and answers throughout, and a
session's slots back once it is deleted.

## Decided on issue 951

The seven questions this layer put are answered, each as proposed here,
so nothing in this layer's row, pins, gate, e2e expectation or pages
changes. The layer of this stack that records issue 951's decisions
writes each answer and its reason where the tree still presented the
question, and records the metering answer on `RTC-004` itself.

1. **Half the held-call ceiling** (`SessionHeldDivisor` 2, 96 under a
hard limit of 1024), with the stream's slot reserved when the session
opens, rather than a descriptor reservation of its own that would lower
`HLD-011`'s figure on a stateful deployment. The cost stays as stated: a
session holds a held slot while its stream is closed, so calls have half
the held slots on a deployment whose sessions are all open.
2. **No per-credential ceiling stands beside it, and `initialize` stays
unmetered**, for 951b's reason: the credential is a key a caller can
mint (`INV-003`), so a per-credential number, or a price in the opener's
own rate, multiplies with every token a caller mints and bounds nothing
the process ceiling does not. One credential can therefore open every
session at no rate cost and hold them for up to `--session-timeout`, and
the operator pages say so, with what to do about it.
3. **No fixed cap stands beside the derived figure.** At 524288
descriptors the figure admits about ten to twelve GiB of idle sessions,
and the container's memory limit bounds that memory; a fixed cap no flag
moves would be sized for one host. The pages say to set a memory limit
on the container of a deployment that keeps sessions.
4. **The slots are taken after admission**, the departure `HLD-011`
records from issue 951's request (PAT-003), accepted with its cost here:
admission at the pool's bound can evict another credential's quiet
entry, ending its sessions, before this row refuses the newcomer.
5. **INV-019.** The refusal says what `HLD-011`'s does, and with no
per-caller ceiling beside it any refusal says the process is full, the
one bit the exception accepts; `HLD-010` stands under it beside
`HLD-011`.
6. **One goroutine more per session**, bounded by the ceiling, rather
than riding on `sessionOwners`' waiter and tying the two rows' sites
together, accepted as implemented.
7. **The issue is complete.** With 951a (`RTC-007`), 951b (`HLD-011`)
and this layer the register carries F-03 and F-31 on no row, and the
answers to 2 and 3 here and to 951b's per-credential and memory
questions add no row, so this layer closes #951.

## Validation checklist (tenant-policy-spec.md)

- **VAL-001, question**: Allow.
- **VAL-002, key**: `KeyProcess`, which no caller can mint. No new key.
- **VAL-003, class**: P. Its reason ("only a ceiling keyed on the
process bounds the sessions it keeps") and the reason's unit are both
the process.
- **VAL-004, ceiling or share**: a ceiling. It promises no caller a
share.
- **VAL-005, process partner**: it is itself keyed on the process and
protects the process's descriptors, and its memory only where the hard
limit is small, the container's memory limit bounding the rest (decided
on issue 951, item 3); no flag sets it, because an operator who could
raise it could undo the bound (INV-004, INV-018). No per-caller row
stands beside it, which issue 951 decided (item 2).
- **VAL-006, at capacity**: refuses the newcomer; never evicts or ends a
session. `IDN-010` now states the same.
- **VAL-007, refusal**: gate, legacy era only (a 2026-07-28 `POST` takes
no slot): 503, `-50300`, fixed 30 s `Retry-After`, no challenge,
connection closed; retry-later class, text `This server is busy. Retry
later.`. On stdio: none.
- **VAL-008, charges**: nothing. The refusal comes after admission and
before anything metered.
- **VAL-009, value**: `Source: Derived`, half of `HLD-011`'s derived
figure through the pinned `SessionHeldDivisor` (2), with a floor of one
session; zero is not applicable; no flag or variable, so no
malformed-value policy. `--session-timeout=0` keeps its meaning under
`END-005`, and startup warns what it costs here.
- **VAL-010, telemetry**: the reason only. The log line carries the
reason, the scope and the figure, nothing about the caller, and is
throttled like every refusal a caller can cause at will. No metric
counts the refusals.
- **VAL-011, stdio**: not applied (`StdioKey` none). The gate counts
only on a stateful HTTP deployment, and the middleware finds no slot on
any other request.
- **VAL-012, measurement**: the ceiling does not claim to protect one
population from another; it protects the process's descriptors, and it
is measured by the sessions mode above, which is what that claim needs;
`make bench-fairness` answers a claim this ceiling does not make.

## Gates

Measured on this layer before it was rebased onto main at 5da9699: the
line numbers below are that tree's, and the one commit cited below
(a0a03fe) is named as it sits in this stack. At its place in this
stack on 50ef2cf, the layer built and vetted, the e2e gitlab, http and
stdio packages compiled, the unit suites of `cmd/server`,
`internal/tenancy`, `internal/oauth` and `cmd/audit_tenancy` passed, and
`make check-tenancy` read the same 83 rows and 308 sites. The stack now
sits on b44ab96, which adds #1083, #1084 and #1085; they change the
upstream register, indirect module versions and the site's package pins,
none of them a file this stack writes, and every commit of the stack
applied unchanged.

- `go test -shuffle=on`: 100.0% of statements in `cmd/server`,
`internal/tenancy`, `cmd/audit_tenancy` and `cmd/bench_resources`;
`-race` on the session, gate and held tests of `cmd/server` and on
`cmd/bench_resources`; the `test/e2e/http` limit, held, session and
health cases against the binary built at the tip, without the
Docker-only proxy and balancer cases, which need a daemon this lane did
not start; the e2e gitlab packages compile.
- `go vet` and golangci-lint with the `e2e` and `httpe2e` tags: 0
issues. markdownlint, `format_md_tables --check` and `check-em-dash`:
clean.
- `make check-tenancy`: 83 rows, 21 failures and 308 declared sites over
202 packages (21 refusal returns, 81 refusals, 14 reasons and 31
settings read); 0 findings, 56 declarations exempted.
`check-dead-consts`, `check-test-file-names`, `check-test-subtests`,
`check-test-goroutines`, `check-sdk-context` and `check-doc-tool-names`
pass. The godoc audit finds nothing this layer wrote; the five findings
in the packages it touches predate it (three test comments in
`cmd/server/main_test.go`, and `cmd/bench_resources/fairness.go`'s
package comment, twice).
- gobco: `internal/tenancy` 408/408, `cmd/bench_resources` 1420/1420.
`cmd/server` through a staged copy without its non-Linux twins (the
issue 1017 approach): 2082/2124, every condition of `sessions.go` and
`held.go` both ways, `acquire`'s retry now taken on purpose through its
hook. The 42 open are the ones 951b's run left, none on a line this
layer wrote: `auth_gate.go` 3 (lines 728, 818, 916), `bearer_guard.go`
5, `etag.go` 2, `first_run.go` 1, `health.go` 1, `jsonrpc_id.go` 3,
`listen.go` 2, `main.go` 11, `pprof.go` 1, `probe.go` 3, `readiness.go`
1, `sdk_log.go` 2, `shape.go` 2, `socket_mode_unix.go` 2,
`subscriptions.go` 2, `telemetry.go` 1.
- gremlins (`--workers 2 --timeout-coefficient 20`, `INVERT_LOGICAL`
on):
- `internal/tenancy`, whole package: 226 killed, 0 lived, 0 timed out,
66 not covered, every one arithmetic in a constant declaration, the same
66 as the base.
- `cmd/server`, scoped to the four files this layer writes
(`sessions.go`, `auth_gate.go`, `held.go`, `main.go`), at the review fix
(a0a03fe, whose code the later commits leave as it is): 459 killed, 5
lived, 33 not covered, 0 timed out. Every mutant of `sessions.go` and
`held.go` is killed. The 5 lived and 33 not covered are exactly 951b's
on the same code, moved by the lines this layer inserted:
`auth_gate.go`'s unobservable `effectiveWindow <= 0` (issue 958) and its
8 constant initializers, and `main.go`'s four recorded boundaries and 25
initializers, `switch` cases and the equivalent `807:19`.
- `cmd/bench_resources`, the four files this layer writes (`held.go`,
`main.go`, `sessions.go`, `target.go`), on a tree whose package differs
from the tip only in how `sessions.go` builds its one constant handshake
message: 253 killed, 0 lived, 0 not covered, 0 timed out. `sessions.go`
again at the tip: 55 killed, 0 lived, 0 timed out, and 2 not covered,
the two concatenations in that constant's initializer, which the
compiler folds and no test can reach, the class of `internal/tenancy`'s
66.

## Decisions I took here

- `HLD-010` is half of `HLD-011` (`SessionHeldDivisor`, 2, floor one),
derived, keyed on the process and not configurable. Sessions get no
descriptor reservation of their own, so 951b's derivation of `HLD-011`
is unchanged (decided on issue 951, item 1).
- A session reserves a held-call slot for its standalone stream when it
opens, rather than the `GET` taking one when it arrives, which changes
what `HLD-011` counts on a `--stateless=false` deployment and leaves
calls at least half the held slots there when every session is open
(decided on issue 951, item 1).
- `HLD-011`'s gate refusal is no longer limited to the 2026-07-28 era,
since a legacy `POST` that would open a session is refused there when no
held slot is left for its stream.
- The session slot is taken after admission and before the SDK creates
the session, the departure 951b records (decided on issue 951, item 4).
- The refusal is `HLD-011`'s gate failure word for word, the log line
alone naming the scope and the figure, and `HLD-010` is listed under
`INV-019`'s one-bit exception (decided on issue 951, item 5).
- No per-credential session ceiling and no metering of `initialize` were
added, and no fixed cap stands beside the derived figure (decided on
issue 951, items 2 and 3).
- Each session's slots are given back by a goroutine of its own waiting
on `ServerSession.Wait`, bounded by the ceiling, rather than by
`sessionOwners`' waiter (decided on issue 951, item 6).
- F-31 is treated as answered: it leaves `HLD-010` and `IDN-010`, joins
the answered set in the specification and the tests, and `IDN-010`
refuses the newcomer at capacity, decided in issue 951.
- The startup attributes of `serveHTTPOn` moved into `httpStartupAttrs`
so the new attribute does not push it past the maintainability
threshold; behaviour is unchanged.
- This layer closes #951: with 951a and 951b it bounds everything the
issue named, and every question its layers put is answered without a new
row (item 7).

## Not done

- The SDK's session map and `sessionOwners` were not sampled by the
bench; that both stop at the ceiling is inferred from
`TestSessionOwners_ADisconnectedSession_IsForgotten` and the ceiling.
- `make bench-fairness` was not run: the ceiling protects the process's
descriptors rather than one population from another, so the comparison
of two populations it runs answers nothing this row claims.
- The Docker-only proxy and balancer cases of `test/e2e/http` did not
run, and the Docker GitLab suites were compiled but not run.
- `cmd/server` was mutated over the four files this layer writes, not
whole.
- The godoc findings in the packages this layer touches predate it and
are left as they are.

Closes #951.
jmrplens added a commit that referenced this pull request Sep 30, 2026
…the ones it cannot ignore safely (#1090)

Every flag `main` registers into `httpConfig` is parsed on every run,
and a stdio run never reads that struct: `main` hands `runWithContext` a
nil one unless the transport is HTTP, and `runStdio` builds its
configuration from `config.Load` and the environment alone. So a stdio
user who passed `--rate-limit-rps=5` to switch the limiter on got no
limiter and no message, one who passed `--read-only` got a server that
writes, and one whose client configuration carried `--gitlab-url
https://gitlab.corp.example` with the corporate token in its environment
sent that token to `https://gitlab.com`, the instance stdio falls back
to when `GITLAB_URL` is unset, at startup and before any tool was
called. `docs/development/documentation-audit-2026-09.md` (D8) found the
same for `--action-timeout` and left the warning as a code decision.

**What a stdio run does now.** Once the transport is explained, it names
every flag it was given that lands in `httpConfig`, each with the
variable stdio reads instead where there is one (`--rate-limit-rps (on
stdio set GITLAB_MCP_RATE_LIMIT_RPS)`, `--max-request-body-bytes (on
stdio set GITLAB_MCP_STDIO_MAX_LINE_BYTES)`, the line ceiling that
bounds the same inbound message, `--http-addr (stdio has no such
setting)`), and goes on ignoring them:

- at `WARN` when the operator chose stdio, by default, with
`--transport=stdio` or with `--http=false`;
- under `--transport=auto`, a flag stdio has no setting for at all (the
listener, the pool, the authentication gate) at `INFO` on a line of its
own, since that command line was written for either transport and the
container image's own command carries `--http-addr`: a warning on every
`docker run -i` session of the published image would teach the reader of
that log to skip warnings. A flag stdio has a variable for stays at
`WARN` there, since a setting written once for either transport was
plausibly meant for both.

**The flags that refuse instead.** The condition is what ignoring the
value would cost, the rule `config.RetiredEnvUses` applies to the
retired variable names. Ignoring most of these flags costs a setting.
Two kinds cost more, and a stdio run given one exits 1 before serving,
under `--transport=auto` as well, with one `ERROR` line each naming the
flag and what to set instead:

- `--read-only` or `--safe-mode` set to `true`, or an `--exclude-tools`
naming anything: the three settings register row `AUT-004` decides
together, each withholding part of what the server serves, so ignoring
one serves it (`set_instead=GITLAB_MCP_READ_ONLY=true`,
`GITLAB_MCP_EXCLUDE_TOOLS=project.delete`).
- a `--gitlab-url` none of whose instances is the one the stdio run
connects to: ignoring it sends `GITLAB_TOKEN` to an instance the command
line did not name, which is the reason HTTP mode refuses to choose an
instance on a caller's behalf, applied here to the operator
(`set_instead=GITLAB_URL`). The comparison is in the canonical form the
HTTP allow-list uses, so a `--gitlab-url` naming that same instance
however it is spelled is only named, and the instance stdio connects to
is resolved the way `config.Load` resolves it.

A value that asks for nothing (`--read-only=false`, an `--exclude-tools`
of blanks, an empty `--gitlab-url`) costs nothing when ignored and is
only named. A refused flag is refused even when its variable already
asks for the same thing, as a retired name is beside its replacement,
and the refusal line is worded to hold then too: the flag has no effect
here, so set the variable and remove the flag.

**How the set is kept.** `httpOnlyFlags` in
`cmd/server/http_only_flags.go` names each flag with its stdio variable
and, for the four above, the refusal.
`TestHTTPOnlyFlags_AreTheFlagsBoundToHTTPConfig` reads the package's
source for every `flag.XxxVar(&hcfg.field, ...)` and holds the table to
it in both directions. `TestHTTPOnlyFlags_NameVariablesStdioReads` sets
each named variable to two values and requires what stdio reads at
startup, `config.Load` and the transport's own limits, to differ.
`TestHTTPOnlyFlags_NameEachFlagsOwnVariable` holds each variable to the
flag it is named for, through the HTTP environment overlay: set to two
values with no flag passed, the one field it moves must be the field the
flag is registered into, which a table sending `--tier` to
`GITLAB_MCP_TOOL_SURFACE` fails (checked by making exactly that edit).
`--max-request-body-bytes`, whose stdio variable HTTP mode has no
variable for, is declared with its reason and held to moving nothing.
The retired-name report, the transport line and this check run from one
function, `reportStartupConfiguration`, since `main` was at the limit
the maintainability linter accepts.

`test/e2e/stdio/http_only_flags_test.go` holds the behaviors against the
binary: `--rate-limit-rps=0.001 --rate-limit-burst=1` named at `WARN`
and still ignored (three tool calls served, no limiter line); the
image's `--transport auto --http-addr 0.0.0.0:8080` named at `INFO` with
no warning, and a `--tool-surface` added to it named at `WARN`;
`--read-only`, `--read-only` beside `GITLAB_MCP_READ_ONLY=true`, and
`--exclude-tools` each exiting 1 before serving with nothing on stdout;
a `--gitlab-url` naming another instance exiting 1 with the instance
`GITLAB_URL` names counting that no request reached it; and a
`--gitlab-url` naming the same instance with a trailing slash served and
named.

The curated `-h` says which flags a stdio server reads and which it
refuses, and now lists `-allow-private-instances` and
`-description-substitutions`, which it never did;
`TestPrintHelp_DocumentsEveryFlag` reads the env-backed flags from their
table so that cannot recur. The two `cmd/server` test comments that gave
the stdio rate-limit reason as latency, which issue #1045 lists as its
own work because #959's layer could not edit `cmd/server`, now give the
one the rest of the tree states: the limiter refuses and never delays,
so on stdio it would only refuse its one user's own calls (the same
comment in `internal/config/config.go` was #1042's, which main now
carries). Twelve documentation lines that said stdio ignores
`--rate-limit-rps` "without a warning" are corrected, and the CLI and
configuration references, the environment reference, the troubleshooting
guide and the site's configuration and troubleshooting pages (English
and Spanish), the exit-code table, `CLAUDE.md` and
`.github/copilot-instructions.md` state the rule.
`internal/tenancy/decisions_allow.go` (RTC-001) still says stdio accepts
the flag and ignores it, which stays true; changing register text would
make this a register change for no gain.

**Not a limit.** No register row, value, refusal or site changes: the
refusal is a startup refusal of the operator's configuration, like the
retired-name one, and not a refusal of a caller. `make check-tenancy`
passes unchanged. The validation checklist therefore has nothing to
answer: VAL-001 to VAL-009 describe a row there is none of, VAL-010 has
no telemetry to carry, VAL-011 is this change (stdio is where it acts),
and VAL-012 has no population to protect and no measurement to take.

## Verified

Measured on this layer before it was rebased onto main at 5da9699,
where main's server card change (#1061) sits in the same file; the two
merged without a conflict. At its place in this stack on 50ef2cf, the
layer built and vetted, the e2e gitlab, http and stdio packages
compiled, the unit suites of `cmd/server`, `internal/tenancy`,
`internal/oauth` and `cmd/audit_tenancy` passed, and `make
check-tenancy` read the same 83 rows and 308 sites as the layer below.
The stack now sits on b44ab96, which adds #1083, #1084 and #1085; they
change the upstream register, indirect module versions and the site's
package pins, none of them a file this stack writes, and every commit of
the stack applied unchanged.

- `cmd/server`, shuffled: coverage 100.0% of statements; race on the new
and neighboring tests; `go vet`; `golangci-lint --build-tags e2e` and
`--build-tags stdioe2e` clean.
- gobco on `cmd/server` (a staged copy without the non-Linux twins,
since gobco cannot instrument the package in place): 2118 of 2160
conditions both ways. The 42 left are the base's 42 at the same code
(2082 of 2124 before this change); every condition this change adds,
`http_only_flags.go` whole, is evaluated both ways.
- gremlins (`--workers 2 --timeout-coefficient 20`) over `cmd/server`,
scoped to the two files this change writes, at the layer's last commit
before the rebase: 367 killed, 4 lived, 25 not covered, 0 timed out.
`http_only_flags.go` has no survivor. All 29 are in `main.go`, none on a
line this change writes, and they are the base's set on the same code
shifted by the help text added here.
- `main.go` therefore does not meet the gate on its own. Its 11 one-way
conditions, 4 lived and 25 not-covered mutants are carried to #1017,
whose `cmd/server` half is the package's closing pass, with the list;
most of the not-covered ones are arithmetic in constant initializers,
the class `http_only_flags.go` removed by spelling its variables out.
- `test/e2e/stdio`, the whole module against the binary built from this
tree: pass. `test/e2e/gitlab/common` and `test/e2e/http` compile.
- `make check-tenancy`, `check-dead-consts`, `check-test-file-names`,
`check-test-subtests`, `check-test-goroutines`, `check-doc-tool-names`,
`check-em-dash` against both `main` and the layer below: pass.
`format_md_tables --check` and `markdownlint-cli2` on the changed pages:
clean.

## The top of the stack

This layer was the top of the tenancy stack (#950, the two parts of #951
and this one) when it was written, so it also carries the refresh of the
committed artifacts the layers below leave stale, in its own commit; the
layers since stacked above it refresh them again at the new top. The
README statistics and the testing reference move, since the stack adds
tests; the golden snapshots, the llms files, the token footprint, the
request inventory, the LobeHub manifest and the site stats were
regenerated and came out unchanged, since no layer moves the served
surface or a request a handler sends. `CLAUDE.md` and
`docs/development/cmd-utilities.md` take the text the lower layers
proposed: the identity cache and verification ceiling (`ADM-005`,
`ADM-014`) and the held-call and stateful session ceilings (`HLD-011`,
`HLD-010`) in the environment table, the HTTP flag table and the OAuth
routing section, and the `bench_resources` held and sessions modes with
their make targets.

At the top, with the freshness gates checked rather than deferred: every
check of CI's generated-artifacts job passes (`check-openplugin`,
`check-stats`, `check-doc-links`, `check-plan-untracked` and
`check-em-dash` run locally, the rest on the remote runner), as do
`check-e2e-static`, `check-tenancy`, `check-request-inventory` and the
e2e harness's own tests; the whole unit suite passes (275 packages);
golangci-lint with the `e2e`, `httpe2e` and `stdioe2e` tags over
`cmd/audit_tenancy`, `cmd/bench_resources`, `cmd/server`,
`internal/oauth`, `internal/tenancy`, `test/e2e/http` and
`test/e2e/stdio` reports nothing; the unit suites of those packages pass
shuffled twice; `test/e2e/http` passes (605 tests, the 7 Docker-only
proxy and balancer cases skipped) and `test/e2e/stdio` passes (107
tests), both against the binary built at the top; markdownlint is clean
on every page the stack changes.

## Decisions I took here

- `--read-only` and `--safe-mode` set to true, an `--exclude-tools`
naming anything, and a `--gitlab-url` naming no instance stdio connects
to refuse a stdio start (exit 1) instead of warning. The issue allows a
warning or a refusal; the rule `config.RetiredEnvUses` already applies
to the retired names is what ignoring the value would cost, and nobody
reads a stdio server's stderr.
- A start that still sets the retired bare `EXCLUDE_TOOLS` refuses as
well, by the same rule: ignoring it serves what the operator removed,
exactly as ignoring `--exclude-tools` does. The rule lives in
`config.protectionNames` in `internal/config`, so a layer of its own
higher in this stack makes that change, beside the retired
`GITLAB_READ_ONLY` and `GITLAB_SAFE_MODE` it already refused.
- Under `--transport=auto` a flag stdio has no setting for is named at
INFO, not WARN, because the published image's own command passes
`--http-addr` and a warning would fire on every `docker run -i` session.
A flag with a stdio variable stays at WARN there, and the refusals apply
under auto too.
- The refusal is a startup refusal of the operator's configuration, not
a caller refusal, so no register row was added and `check-tenancy` is
unchanged.
- "HTTP-only" means exactly the flags registered into `httpConfig`, held
in both directions by a test that reads the source. `--tool-surface`,
`--tier` and `--tls-cert` are also read by `--tool-search` and
`--probe`, which exit before a transport is chosen.
- One WARN line lists every ignored flag with its stdio variable, and
each refused flag gets an ERROR line of its own. A refused flag is
refused even when its variable already asks for the same thing.
- `--gitlab-url` is compared in the canonical form the HTTP allow-list
uses; an empty value, or one naming the same instance however it is
spelled, is only named, and a spelling either side cannot normalize
matches nothing and refuses. The instance stdio connects to is resolved
in `cmd/server` the way `config.Load` resolves it (trim `GITLAB_URL`,
fall back to `config.DefaultGitLabURL`), held together by
`TestStdioInstance_IsTheOneConfigLoadResolves`, rather than by exporting
config's resolver.
- The stdio variable names in `httpOnlyFlags` are written as literals,
not as `config.EnvPrefix` plus a name, so gremlins has no unreachable
initializer mutants there; a test holds each spelling to a variable
`config.Load` reads.
- `reportRetiredEnvNames`, the transport line and this check run from
one function, `reportStartupConfiguration`, in that order, to keep
`main` under the maintainability limit.
- `CLAUDE.md` was edited in this layer (the `GITLAB_MCP_RATE_LIMIT_RPS`
row and the paragraph before the HTTP flag table), because the issue is
about that text. The orphaned test comment for a
`TestRateLimit_DefaultsDifferByTransport` that does not exist was
removed with its stale latency reason.
- The resolution of D8 in
`docs/development/documentation-audit-2026-09.md` is marked "Resolved by
issue 1045" rather than by a release number, which is not known yet.

## Not done

- A stdio run does not start honouring the HTTP flags (writing
`GITLAB_MCP_READ_ONLY` from `--read-only`, for instance); that would
change stdio's configuration model beyond what the issue asks, so it
names or refuses them instead.
- `internal/tenancy/decisions_allow.go` (`RTC-001`) still says stdio
accepts `--rate-limit-rps` and ignores it, which stays true; changing
register text would make this a register change for no gain.
- `main.go` does not meet the house gate on its own: its 11 one-way
conditions, 4 lived and 25 not-covered mutants predate this change and
are carried to #1017's `cmd/server` half with the list.

Closes #1045.
jmrplens added a commit that referenced this pull request Sep 30, 2026
…-fairness (#1091)

Follows #950.

#950 put sixteen verification slots and a five-second wait in front of
GitLab (register row `ADM-014`) and stated what that costs a legitimate
credential presented for the first time during a flood, because `make
bench-fairness` had no OAuth driver and could not measure it. The
decision on #950 was to build the driver rather than accept the
statement. This layer is that driver and what it measured.

## What the driver does

`make bench-fairness BOUND=oauth-verification` (or `go run
./cmd/bench_resources/ -fairness oauth-verification`) runs the two arms
every bound gets, with the ceiling in force and without it, and reports
the quiet population's served and refused counts apart, with latency
percentiles over served requests only.

**The arm without the ceiling is a build of its own.** The ceiling is a
constant no operator can move, and it must not grow a switch: a flag
that turns off a security bound is a flag somebody sets in production.
So both arms start with the same flags (`--auth-mode=oauth`, a
`--public-url`, `--trusted-proxies=127.0.0.0/8` with
`--trusted-proxy-header=X-Forwarded-For`, the rate limit off) and the
arm without the ceiling runs a second build of the same checkout in
which `const verificationSlots = ` in `internal/oauth/verifier.go` reads
`1 << 20`, made through the go command's `-overlay`. The tree on disk is
never touched, the two builds differ in that line alone, the record
names the line (`variant_off`), and a checkout that no longer declares
it, or declares it twice, stops the run rather than comparing the binary
with itself. The bound refuses `-binary`, since one binary cannot be
both arms.

**The stand-in GitLab** answers `GET /api/v4/user` for every token
except one starting `bench-invented-`, which it answers 401, answers
`/api/v4/personal_access_tokens/self` and `/oauth/token/info` 404 the
way an instance that will not describe a token does, holds each of those
requests for `-fairness-upstream-delay` (100 ms unless given), and
counts how many it answered and how many it held at once, for every
token and for invented ones apart. Those counts include the pool's own
two requests to the same endpoints when it admits a new credential,
which run under the pool's probe slots rather than the verification
slots.

**The two populations.** The noisy one lists tools with a new invented
token on every request, each forwarded for an address of its own. It
sends from as many transport sources in 127.2.0.0/16 as keep each one
under half of the transport-source failure budget (`AUB-002`, 500
distinct failing credentials a source a minute), because from one source
that budget blocks the flood, and the quiet population behind it, long
before the slots fill. Linux answers every address in 127.0.0.0/8 on its
loopback; elsewhere every flood request fails and says why. The quiet
population calls tools with its own cached credential three times in
four and presents a new valid token the fourth, using it once it is
served, so its two rows are the onboarding the ceiling can cost and the
cached credentials it must not.

**The stand-in server** the package's whole-run tests start
(`testdata/standin`) gains an OAuth gate that mirrors the bearer guard:
a token it has verified before is admitted at once, a new one after the
instance answers `GET /api/v4/user` for it, under `STANDIN_VERIFY_SLOTS`
slots with a `STANDIN_VERIFY_WAIT` wait (100 ms unless set), refused in
the guard's codes and words, 503 for a slot that never freed and 401 for
a token the instance refuses.

**The phase follows the lead-in on one clock.** The lead-in runs into a
tally of its own, and the phase's schedule begins where the lead-in's
clock puts it, whether or not the lead-in's requests have landed; the
resident set peak, the stand-in's counters and the processor samples are
reset at that instant. Run one after the other, as the first version of
this layer did, the flood stopped at the end of the lead-in, every
waiter drained, and each phase opened on an empty queue whose first
seconds served every new credential.

**What the record and the verdict say.** A 401 for an invented token is
counted as refused otherwise, a fifth outcome beside served, refused,
failed and timed out, and only on the flood's row: the same 401 to a
cached or a new valid credential is a failure. The per-arm summary says
what the instance answered. The bound carries its own plan defaults
(eight noisy credentials at fifty tokens a second, a thirty-second phase
after a ten-second lead-in, a fifteen-second deadline, 100 ms at the
instance), and the plan is refused when the flood could not hold every
slot at the round trip given, when the lead-in is shorter than the time
the flood takes to settle into its queue from empty (the wait times the
flood over its excess above what the slots finish: 8.3 seconds at the
defaults, 25 at 50 ms), when the deadline is shorter than the wait plus
a new credential's round trips and a second, or when the quiet
population's own new credentials would keep half the slots busy. The
ceiling protects the instance rather than the quiet tenant, so the
verdict says what it cost the tenant instead of calling the tenant the
one it was meant to protect, and the saturation gate, which asks whether
a per-credential bound reached the quiet tenant through the processor,
does not apply to a bound whose contended resource is its slots.

## Measured

On truenas through `rgo`: Intel i5-14400, sixteen threads, the container
held to five cores, Go 1.27.1, a descriptor limit of 1048576. Two
repetitions per configuration, arms alternated, thirty-second phases,
each opening on the lead-in's clock after the flood has settled into its
queue: a ten-second lead-in, and thirty seconds at 50 ms, where the
queue takes twenty-five to settle. Eight quiet credentials present four
new tokens a second between them (120 in a phase) beside twelve requests
a second on cached credentials (360). "Instance" is every request the
stand-in answered at the three endpoints from the instant the phase
opened until the last request of either window landed: the lead-in's
waiters that a slot served inside the window are in it, since that is
the window's load, and so are the pool's own two requests for each new
credential it admits. Each pair is the two repetitions.

| Round trip | Flood | New served with the ceiling | New median with /
without | Cached median with / without | Instance requests with /
without | At once with / without | Resident set peak with / without |
| ---------: | -------------: | --------------------------: |
------------------------: | ---------------------------: |
-------------------------------: | ---------------------: |
-------------------------------: |
| 100 ms | 400 a second | 42, 40 of 120 | 5.52, 5.51 s / 0.52 s | 14.0,
13.6 / 14.6, 12.5 ms | 5,642, 5,644 / 12,600, 12,600 | 20, 19 / 56, 56 |
277, 273 / 175, 178 MiB |
| 50 ms | 400 a second | 80, 82 of 120 | 5.27, 5.26 s / 0.27 s | 14.3,
15.6 / 15.5, 12.0 ms | 11,054, 11,149 / 12,600, 12,600 | 18, 18 / 35, 41
| 280, 280 / 172, 171 MiB |
| 200 ms | 400 a second | 17, 17 of 120 | 6.01, 6.01 s / 1.01 s | 9.9,
9.6 / 11.6, 10.3 ms | 2,845, 2,841 / 12,600, 12,600 | 19, 19 / 92, 90 |
265, 269 / 175, 176 MiB |
| 100 ms | 1,000 a second | 17, 17 of 120 | 5.51, 5.51 s / 0.51 s | 9.1,
9.1 / 9.5, 9.5 ms | 5,606, 5,599 / 30,600, 30,600 | 17, 18 / 110, 110 |
438, 436 / 188, 188 MiB |

Every new credential the ceiling did not serve was refused 503 with the
rest of the flood; none failed and none timed out, and no cached request
was refused in any arm. Without the ceiling all 120 new credentials were
served in every run. The verdict of every run is `worse`, in the first
configuration's words: "the bound refused 78 of the quiet population's
120 tools/call (new credential) requests. This bound protects the GitLab
instance the verifications reach rather than the quiet tenant, and that
is what it cost the tenant".

What the numbers say:

- **The cost falls where #950 said, and it tracks the slots' share of
what arrives, a little below it.** A new credential waits in the flood's
queue and is served only when a slot frees before its five seconds do,
which is why the ones served took the whole wait and their round trips,
5.5 s rather than 0.5 s, and why the share served moves with the round
trip: 35% and 33% at 100 ms, where the slots finish two in five of the
flood, 67% and 68% at 50 ms (four in five), 14% at 200 ms (one in five),
and 14% at 100 ms under the larger flood (about one in six). The flood's
own refusals match the slots' share closely (61%, 24%, 80% and 84%
refused), and the newcomers' shortfall below it comes from the driver's
schedule rather than from the ceiling: in a queue whose waiters all give
up at the same five seconds, a freed slot serves the oldest waiter still
inside its wait, so a request's chance grows with the gap before it, and
the flood arrives on an even grid (one request every 2.5 ms) while the
new credentials arrive in clumps on fixed ticks. A queue model of the
gate reproduces the shortfall with that schedule (36% against 39% at 100
ms, 67% to 71% against 78% at 50 ms) and removes it when both streams
arrive at random (39.5% against 38.9%, 78.6% against 77.8%, 19.8%
against 19.4% at 200 ms).
- **Cached credentials never wait for a slot**, as the code says: no
cached request was refused in any arm of any configuration, and their
medians lay between 9 and 16 ms with the ceiling and without it.
- **What it buys is the instance's load, and that is independent of the
flood.** At 100 ms the instance received about 160 requests a second
under 400 invented tokens a second and under 1,000, sixteen slots over
the round trip, never more than 20 at once, against about 420 and 1,020
a second and up to 110 at once without the ceiling. Over the window each
arm counted that is 5,640 and 5,600 requests in the 35 seconds from the
phase's first request to the answer of its last waiter, against 12,600
and 30,600 in 30, and the rates match the slot arithmetic in every
configuration: 160.8 a second at 100 ms, 315 and 318 at 50 ms, 80.8 at
200 ms and 160.0 under the larger flood. Of the 12,000 invented tokens
of the first configuration's first run, 4,630 reached the instance and
7,370 were refused 503 without reaching it.
- **The queue moves into the process, and nothing bounds it.** Each
waiting request holds its connection for up to the five seconds, so the
server holds the arrival rate times the wait, about 2,000 requests at
400 a second, and its resident set peaked at 273 and 277 MiB against 175
and 178, and at 436 and 438 against 188 at 1,000 a second. Nothing but
the arrival rate bounds how many wait, and their connections are
descriptors outside the held calls `HLD-011` counts, since the wait
comes before the gate admits a request. These runs had a descriptor
limit of 1048576 and did not reach it; a process under a low hard limit
would, cached credentials' connections included, and I did not measure
what it does then. The waiters hold memory as well: the resident set
rose by about 100 MiB for some 2,000 waiting and 250 MiB for some 5,000,
about 50 KiB a waiting request, so a process whose memory limit is below
the arrival rate times five seconds times that is ended by the flood,
cached credentials with it (1 GiB is some 21,000 waiters, a flood of
about 4,200 invented tokens a second). The pages now say both wherever
they say a cached credential never waits for a slot, the troubleshooting
rows and the site's HTTP server step among them.

**What changed from the first measurement.** The first version of this
layer ran the lead-in and the phase one after the other, so every phase
opened on an empty queue. Its figures were biased toward the ceiling: 50
of 120 at 100 ms against 42 and 40 now, 104 and 100 at 50 ms against 80
and 82 at a median of 4.2 s against 5.3, 26 and 29 at 200 ms against 17,
and 24 and 27 under the larger flood against 17. The records showed it:
in all eight runs with the ceiling the flood's 401 count equalled the
stand-in's invented requests exactly, which holds only if no lead-in
waiter reached the instance during the phase. The instance's load, which
the slots saturate from the first second either way, barely moved.

To reproduce: `make bench-fairness BOUND=oauth-verification` for the
first row, and `go run ./cmd/bench_resources/ -fairness
oauth-verification` with `-fairness-upstream-delay 50ms
-fairness-lead-in 30s`, `-fairness-upstream-delay 200ms` or
`-fairness-noisy 20` for the others. Each run took three to four and a
half minutes on that host. The documents are `bench/fairness.json` and
are not committed.

## Documentation

- `docs/development/tenant-policy-spec.md`: the findings paragraph on
issue 950 cites the measurement and the command instead of saying the
driver speaks no OAuth, and states the unbounded queue.
- `docs/guides/http-server-mode.md`, `docs/concepts/security.md` and the
site's security page in English and Spanish: the price of the ceiling
now carries the measured figures, and the guide says why the descriptor
limit matters to it. Where they say a cached credential never waits for
a slot, they also say it shares the listener with the waiting flood, so
under a hard descriptor limit below the arrival rate times five seconds
its connection is delayed or refused with the flood, and that the
measurement ran under a limit of 1048576; and that the waiting flood
holds about 50 KiB a request, so under a memory limit below the arrival
rate times five seconds times that the flood ends the process and every
cached credential with it. The instance's load is quoted as a rate over
the window each arm counted, about 160 requests a second against about
420 and 1,020, with the counts and their windows beside it, since 5,640
requests over 35 seconds and 12,600 over 30 are not comparable totals.
The spec and the guide say the new credentials' share sits a few points
under the slots' share because of the driver's evenly spaced schedule.
- `docs/guides/troubleshooting.md` and the site's troubleshooting page
in English and Spanish: the row for a 503 on a new token said
credentials the server holds are still served and a new one gets through
once the flood stops; it now says a held credential never waits for a
slot while the process has descriptors and memory left for the waiting
requests, and that a new one is served only if a slot frees within its
five seconds (about a third of them under the measured flood), so it may
take several retries. The site's HTTP server page in both languages,
whose OAuth step said a cached token never waits, now says it shares the
listener with the requests waiting for a slot and links to the price of
the ceiling on the security page.
- `internal/oauth/verifier.go`: the comment on the verification slots
said a new credential is refused like the flood for as long as the flood
lasts; it now says what the measurement shows, that a new credential is
served when a slot frees within its wait, about the slots' share under a
sustained flood, and names the command.
- `docs/development/cmd-utilities.md`: the new bound,
`-fairness-upstream-delay`, the bound defaults, the fifth outcome and
the row it applies to, the lead-in on the phase's clock and the
settled-queue floor, and how the arm without the ceiling is built;
`Makefile` help lists the `BOUND` spellings and what
`oauth-verification` needs.
- ADR-0023 NEG-004 said the benchmark drives two bounds and that no
switch could exist for a bound no operator can turn off; it now names
the four it drives and says how the one with no switch is compared.
`CLAUDE.md` says the same in one clause.

## Checks

All through `rgo` on truenas unless said otherwise, on this layer before
the stack was rebased onto main at 50ef2cf, which changes no file this
layer writes. The stack now sits on b44ab96, which adds #1083, #1084
and #1085; they change the upstream register, indirect module versions
and the site's package pins, none of them a file this stack writes, and
every commit of the stack applied unchanged.

- `go test ./cmd/bench_resources/`: passes with 100.0% statement
coverage; `-race -shuffle=on` passes. `go vet`, `go build ./...` and
golangci-lint with the `e2e` tag on the package: clean.
- Conditions (gobco, `make coverage-conditions
PKG=./cmd/bench_resources`): 1558/1558. The first reading was 1565/1566:
the stand-in's gauge raised its peak with a compare-and-swap loop whose
retry only contention takes, which no test can arrange, so the gauge now
counts under a mutex.
- Mutants (gremlins, `--workers 2 --timeout-coefficient 20`, staged by
`make coverage-mutants`), over the ten files of the package this layer
writes (`fairness`, `fairness_result`, `fairness_run`, `main`,
`presenter`, `rpc`, `run`, `stubs`, `target`, `variant`); the stand-in
server under `testdata/standin`, a separate program the script's default
exclusion of files below the package leaves out, as it did for #951's
layers, is exercised through the `oauth-verification` bound by
`TestRunFairness_OAuthVerification_MeasuresTheCeilingAgainstABuildWithout`,
which drives its gate with and without slots. First run, at the driver's
commit: 627 killed, 16 lived, 27 not covered, 0 timed out. After the
fixes: **632 killed, 5 lived, 27 not covered, 0 timed out.**
- Killed by a new test: the flood-source limit on its edge, the source
count for a flood inventing every other token, a bucket bound's
description naming no instance and no sources, the deadline floor a
millisecond under it (the old boundary cases sat on it and well below
it, so its round-trip count was never pinned), the onboarding refusal's
share, a row's dispatched sum, and a declaration on a file's first line.
- Removed rather than recorded: a bucket bound's `above <= 0` in
`refusalsExpected`, now `max(0, above)` as the slot branch already was,
`pause`'s guard on a zero delay, which a timer of zero makes redundant,
and the gauge's `now <= peak` short cut, which went with its
compare-and-swap loop.
- The 5 lived are the shapes #951's layer recorded on the same,
unchanged code, at lines this layer moved: `fairness.go:360` (at share
zero the division is `+Inf` and `min` caps it at the same default),
`fairness_result.go:610` (a floor of zero), `fairness_result.go:777`
(reached only once the claim exceeds a spread of zero or more) and
`fairness_result.go:796` twice (the tie-break shape).
- The 27 not covered are constant and variable initializers and `switch`
case expressions, which carry no statement counter. 18 are on lines this
layer does not change and are the ones #951's layer read. The 9 on lines
it writes were applied by hand: the four bound defaults
(`fairness.go:606`, `:607`) die in
`TestFairnessPlanFor_TakesTheBoundsDefaultsWhereNoFlagWasTyped`, which
states each literal; `case err == nil` in `classifyExpecting` dies in
`TestClassifyExpecting_FilesTheRunsOwnRefusalsApart` and
`TestClassifyOutcome_KeepsRefusalsApartFromSuccessAndFailure`;
`maxFloodSources`' two die in
`TestSourceAddress_AndForwardedAddress_AreDistinctPerIndex`; the
stand-in's `case body == ""` dies in
`TestStubGitLab_Verification_AnswersAfterItsRoundTripAndCountsWhatIsInFlight`;
and the `+` between the proxy header flag and its value does not compile
as a `-`.
- After the review fixes (the invented-credential classification, the
lead-in and phase on one clock, the settled-queue floor), at the fixes'
commit: `go test ./cmd/bench_resources/` at 100.0% of statements, `-race
-shuffle=on` passing, golangci-lint clean on `cmd/bench_resources` and
`internal/oauth`; conditions 1560/1560, the two new ones among them;
gremlins over the two files the fixes touch (`fairness`,
`fairness_run`), **256 killed, 1 lived, 19 not covered, 0 timed out**.
Every mutant of the new code died. The one lived is `fairness.go:363`,
the share-zero shape recorded above at its old line, and the 19 not
covered are the constant initializers, composite-literal fields,
`switch` case expressions and the `go` statement's argument the first
run classified, at the lines the fixes moved them to.
- After the second review, at the layer's head: the whole-run test of
the verification bound wrote its two wrapped stand-ins inside the build
seams and ran `captureStdout` inside `finishWithin`, so `t.TempDir`,
`t.Fatalf` and `t.Cleanup` ran on the goroutine `finishWithin` starts
(`make check-test-goroutines` does not follow calls through helpers or
function-valued seams); both scripts are now written on the test
goroutine and `captureStdout` wraps `finishWithin`, leaving only
`runFairness` on the spawned goroutine. The `Unwrap` this layer gave
`httpStatusError` let `coldList`, which asks a warm-up listing again
while a rate bound refuses it, match the authentication gate's
per-address lockout, which carries the same `-42900` at HTTP 429, so a
credential admitted from a locked-out address would have been asked
again every 500 ms for the five-minute warm-up; `coldList` now asks
again only for the refusal answered at 200, and its table carries the
lockout case. `go test ./cmd/bench_resources/` passes at 100.0% of
statements, `-race -shuffle=on` passes, golangci-lint is clean;
conditions 1562/1562, the two parts of the new status test among them;
gremlins over `series.go`, the file the fix touches, **61 killed, 0
lived, 7 not covered, 0 timed out**. The first run of it reported one
timed out, the new condition's second `||` turned into `&&`, which asks
again a listing that was served; the table ran those cases on
`t.Context()`, so the mutant hung the test rather than failing it, and
every case the context does not end now runs under a ten-second
deadline, against which that mutant, applied by hand, fails in ten
seconds. The 7 not covered are two variable initializers and a constant
initializer (`series.go:57`, `:265`, `:371`) and four `switch` case
expressions (`:534`, `:536` twice, `:538`), which carry no statement
counter; none is on a line this layer writes.
- `make check-test-goroutines`, `make check-test-subtests`, `make
check-test-file-names`, `make check-doc-tool-names` and `make
check-tenancy` (83 rows, 0 findings): pass.
- Locally: `gofumpt`, `go run ./cmd/format_md_tables/ --check` and
markdownlint on every changed page: clean, and no em or en dash in
anything this layer adds.
- The four measurement runs above, plus a ten-second smoke run of the
default before them.

## Decisions I took here

- The arm without the ceiling is an overlay build rather than a flag, an
environment variable or a build tag. A flag or a variable is a switch
somebody sets in production; a build tag would put a second value of the
ceiling in the tree for the register gate and every reader to account
for. The overlay leaves the tree as it is and makes the two binaries
differ in one line.
- The replacement is `1 << 20` slots rather than no gate at all, so the
arm without the ceiling runs the same code path, a slot taken and given
back per verification, with a count no flood here can fill.
- The flood comes from many transport sources, each under half of
`AUB-002`'s budget, because that budget is not what this measures and
from one source it answers first. The per-address budgets are kept out
of the way the same way, by a new forwarded address per request.
- The bound's verdict keeps the benchmark's rule that a quiet request
refused is `worse`, and says in the same sentence that the ceiling
protects the instance and that this is its price. Calling it anything
else would hide the one thing the measurement is for.
- The stand-in answers introspection 404 for valid tokens, so a new
credential costs its slot three requests and the server assumes `api`,
which is the path the verifier takes against an instance that will not
describe a token.
- The lead-in and the phase run as two schedules on one clock, the
lead-in in a goroutine of its own, rather than as one schedule whose
observations are split by their intended instant. Each window keeps
exactly its own tick count, which the whole-run tests hold, and a
lead-in whose length is not a whole number of periods cannot move ticks
from one window to the other.
- The plan's floor for a slot bound's lead-in is the time the queue
takes to settle, the wait times the flood over its excess above what the
slots finish, rather than the time it takes to reach the wait, the wait
times capacity over the excess. The second is when a request arriving
would wait the whole wait; the first is a wait later, when that request
is refused and the queue holds as many waiters as it ever will, so the
resident set the phase reports is the sustained one too. It is never
shorter than the wait, and at the defaults it is 8.3 seconds, inside the
ten-second lead-in.
- The run's own 401 is asked of the invented-credential row alone, in
the tally, rather than through a credential field on the refusal shape:
the tally already knows the row, and a shape that named a credential
would need every other bound's shapes to say none.

## Not done

- The queue in the process is a finding and not fixed here: nothing
bounds how many requests wait for a verification slot. A bound on the
waiters (or an immediate refusal once the queue is deeper than the slots
can drain within the wait) is a change of policy and, by `INV-021`, a
change of its own. It is drafted as an issue of its own and not yet
filed.
- The fairness mode has no descriptor limit flag like `-held-nofile`, so
what a process under a low hard limit does while a flood waits is
derived above and not measured.
- The stand-in's round trip is fixed; a real instance's varies, and
GitLab.com would also rate-limit the flood itself, which the stand-in
does not.
- The flood needs Linux, for the loopback addresses it sends from.
jmrplens added a commit that referenced this pull request Sep 30, 2026
…red read-only switches already do (#1092)

Follows #1045.

#1045 made a stdio run refuse to start when it is given
`--exclude-tools`, because ignoring the flag serves what the operator
removed. It left one inconsistency open on purpose: the retired bare
`EXCLUDE_TOOLS`, which 3.1.0 stopped reading, was still only named at
`WARN`, so a deployment carrying `EXCLUDE_TOOLS=project.delete` started
and served `project.delete`. `config.RetiredEnvUses` refuses the retired
`GITLAB_READ_ONLY` and `GITLAB_SAFE_MODE` for exactly that reason. The
answer to the open question was to refuse the retired exclusion too, and
this layer does that.

## What changed

`config.protectionNames` (`internal/config/env_name.go`) gains
`EXCLUDE_TOOLS`, so `RetiredEnvUses` files it with the two retired
switches it refuses rather than with the names it warns about. The three
are the settings register row `AUT-004` decides together, each
withholding part of what the server serves, and the rule is the one the
package already states: what ignoring the name would cost, not where it
came from. Nothing else in the startup path changes:
`reportRetiredEnvNames` in `cmd/server` already ends the run with exit
code 1 before any transport starts when the refuse list is not empty, so
the retired exclusion now stops a stdio and an HTTP start alike, with
the refusal line the other two already print:

```text
EXCLUDE_TOOLS is no longer read (removed in 3.1.0): rename it to GITLAB_MCP_EXCLUDE_TOOLS, and this deployment will not be started under a capability it did not ask for
```

**The bare name refuses even though it may not be ours.**
`EXCLUDE_TOOLS` is generic enough that another tool in the same shell
could own it, which is the reason the review of #1045 gave for waiting.
The server cannot tell whose it is, and of the two guesses, the one that
it belongs to somebody else is the guess that serves what an operator
removed. The refusal costs whoever set it one variable renamed, or
removed from the server's environment, and the documentation now says
how.

**The refusal reads the name, never the value**, as it does for the
other two: a retired `GITLAB_READ_ONLY=false` has always refused, and so
does an `EXCLUDE_TOOLS` set to empty. That differs on purpose from the
`--exclude-tools` flag, which is judged by what it asks for, because a
retired name is not read at all and so nothing about its value can make
ignoring it safe. The troubleshooting pages say to remove a foreign
`EXCLUDE_TOOLS` with `env -u` rather than blank it.

Comments that named two refusing names now name three:
`reportRetiredEnvNames`, `httpOnlyFlags`' doc comment, the bench
driver's environment scrub and the HTTP e2e harness, whose list of
stripped generic names had said "two of these retired names now stop the
server starting" when neither of the two was on that list;
`EXCLUDE_TOOLS` is.

No register row, value, refusal or site changes: this is a startup
refusal of the operator's configuration, like the two it joins, and
`make check-tenancy` reads the same 83 rows and 308 sites as the layer
below.

## Tests

- `internal/config`:
`TestRetiredEnvUses_SplitsByWhatIgnoringOneWouldCost` holds a retired
exclusion to the refuse list, once naming an action and once set to
empty, and its prefixed spelling to neither list. `cmd/server`'s
`TestReportRetiredEnvNames_RefusesWhenAProtectionWasRetired` covers all
three names.
- `test/e2e/stdio/retired_env_test.go` is new and holds the split
against the binary, which nothing did: each of the three names, set once
in the environment and once in the file `GITLAB_MCP_ENV_FILE` names,
exits non-zero at `ERROR` with the refusal line, nothing on stdout and
no "starting MCP server" line; a retired `TOOL_SURFACE=individual` is
named at `WARN` and a `gitlab_execute_action` call is served, which also
shows the value was ignored, since the individual surface would not
register that tool. Nothing in the stdio module pinned the old warning,
so no existing case changed; `awaitRefusal` now says "a setting" rather
than "a flag", since it judges a variable too.
- With the `EXCLUDE_TOOLS` entry removed from `protectionNames`, both
exclusion cases of the new stdio test and the two refusing cases of the
table test fail, and every other case passes.

## Checks

Run on this branch at its last commit before the stack was rebased onto
main at 50ef2cf, through the remote runner unless marked local. That
base adds a `serialtypecheck` entry to a dependency list in
`cmd/server/main_test.go`, which this layer also writes, and moves the
README statistics and the testing reference, which this layer's refresh
commit regenerated there. The stack now sits on b44ab96, which adds
#1083, #1084 and #1085, none of which touches a file this stack writes,
and every commit of the stack applied unchanged.

- `internal/config`: `go test -shuffle=on`, 100.0% of statements;
conditions (`make coverage-conditions`) 424/424; mutants (`make
coverage-mutants PKG=./internal/config`, the whole package) 205 killed,
0 lived, 0 timed out, 9 not covered. The nine are arithmetic in
`config.go`'s package-level constant initializers (lines 21, 22, 40, 53
and 56: `2 * 1024 * 1024 * 1024`, `24 * time.Hour` and the like), a file
this branch does not touch, and are the tool artifact the testing guide
classifies: an initializer carries no statement counter. `env_name.go`
has no survivor.
- `cmd/server` and `cmd/bench_resources`: `go test -shuffle=on`, 100.0%
of statements each. The change there is comments and one test table, so
gremlins was not rerun on either; #1045's figures for `cmd/server`
stand.
- `test/e2e/stdio`, the whole module against a binary built from this
tree (`make test-e2e-stdio`): 118 tests pass, the 107 of #1045 and the
11 added here.
- golangci-lint with the `e2e`, `httpe2e` and `stdioe2e` tags over
`internal/config`, `cmd/server`, `cmd/bench_resources`, `test/e2e/stdio`
and `test/e2e/http`: clean. `gofumpt` and `goimports` (local): clean.
- `make check-tenancy` (83 rows, 308 sites, 0 findings),
`check-doc-tool-names`, `check-test-file-names`, `check-test-subtests`,
`check-test-goroutines`, `check-dead-consts`: pass.
- Local: markdownlint on every changed Markdown page, `go run
./cmd/format_md_tables/ --check`, and `scripts/check-em-dash.sh` against
the layer below and against `main`: clean. No em or en dash in anything
this layer adds.

## Documentation

- `docs/reference/configuration.md`, `docs/reference/env.md` and
`CLAUDE.md`'s Environment variables paragraph name three refusing names
where they named two, and say why the bare one refuses and that the
refusal reads the name and not the value.
- `docs/reference/cli.md` still said the unprefixed spelling "still
works and is removed in 3.1.0" and that the `GITLAB_`-prefixed switches
keep their bare names; both were wrong since 3.1.0 and 2.8.0, and the
paragraph now says what 3.1.0 does.
- The site's configuration page, English and Spanish, had the same stale
"still works" paragraph and listed the renamed `GITLAB_MCP_` switches
and `GITLAB_MCP_YOLO_MODE` among the names that "stay bare"; the section
now matches the reference, with the refusal rule where the stale
sentence was.
- `docs/guides/troubleshooting.md` and the site's troubleshooting page,
English and Spanish, gain a row for the refusal line: which names cause
it, and the fix, including `env -u EXCLUDE_TOOLS` for one that belongs
to another tool.

## After review

- `RetiredEnvUses`' comment stated a wider rule than the code applies:
that refuse names any setting whose absence would leave the deployment
able to do more than it was configured for. The code refuses the three
names `protectionNames` holds, the settings `AUT-004` decides together,
and warns about every other retired name. The comment now says so, and
names the retired bounds among the warnings. The set is not widened.
- The collector harness (`test/e2e/collector/harness_test.go`) started
the server with the whole inherited environment, so a foreign
`EXCLUDE_TOOLS` exported in the developer's shell, which this layer
turns into a refusal, would have failed every collector test at startup,
as an exported `GITLAB_READ_ONLY` already did. It now filters the
environment the way the HTTP module's harness does (`GITLAB_` and
`OTEL_` names and the generic retired names) and passes
`GITLAB_MCP_LOG_LEVEL=info` and `GITLAB_MCP_TOOL_SURFACE=dynamic`
instead of the retired `LOG_LEVEL` and `TOOL_SURFACE`, which configured
nothing. The module was compiled, vetted and linted with its tag; it
skips on the remote runner, which has no Docker, so the filter was not
driven against a collector.
- A second review found the HTTP module in the same state, while its
harness comment implied it was protected. Only `startServer` and
`runServerExpectingExit` started from `configFreeEnviron`;
`shutdownStartServer` and `startServerWithClient` passed the whole
inherited environment, so with `EXCLUDE_TOOLS=project.delete` exported
every `TestListener_*` test, the shutdown test on open listen streams
and the probe, health and balancer cases built on those starters failed
after about 45 seconds with "the server never became healthy". Both now
start from `configFreeEnviron`, and every starter in the module passes
`GITLAB_MCP_LOG_LEVEL=info` and `GITLAB_MCP_TOOL_SURFACE=dynamic`
instead of the retired spellings, which configured nothing and only
added a warning to each start; the values are the defaults. The harness
comment says every starter does this. With
`EXCLUDE_TOOLS=project.delete` exported, the whole `test/e2e/http`
module passes through `rgo` (286 s), and the listener and shutdown tests
pass without it too; `go vet` and golangci-lint with the `httpe2e` tag
are clean.

## Question left open

By the wider rule the old comment stated, four retired resource bounds
would refuse too, and they warn: a retired `UPLOAD_MAX_FILE_SIZE=10MB`
becomes the 2 GB default, `RATE_LIMIT_RPS` and `RATE_LIMIT_BURST` fall
back to 10 and 40 in HTTP mode and to no limit on stdio, and
`MAX_HTTP_CLIENTS` falls back to 100. Each was read bare by 2.7.5 and
still honoured by 3.0.0. I have not decided whether they should refuse
like the three protection names, which would widen `protectionNames`
past `AUT-004`'s three, or stay warnings, which keeps the rule as the
comment now states it. This layer leaves them as warnings.

## The top of the stack

This layer sits on the OAuth fairness layer, which added eight test
files after the refresh #1045 carried, so it was the new top when
written and carries the refresh in a commit of its own (the layer that
records issue 951's decisions now sits above it and refreshes them once
more): the README statistics and the testing reference move. The testing
reference was regenerated with `-skip-coverage`, which carries the
recorded coverage values forward and refreshes the counts and package
rows the CI gate compares. The site stats, the llms files, the LobeHub
manifest and the request inventory were checked and are unchanged, since
no layer above #1045 moves the served surface or a request a handler
sends.

## Decisions I took here

- The retired `EXCLUDE_TOOLS` refuses, as the question left by #1045's
review was answered: the rule `RetiredEnvUses` applies is what ignoring
a name would cost, and ignoring this one serves what the operator
removed.
- The refusal stays on presence, exactly as it is for `GITLAB_READ_ONLY`
and `GITLAB_SAFE_MODE`, rather than on a value that names something as
the `--exclude-tools` flag is judged. A second rule for one retired name
would have to be explained on every page that states the first, and the
value of a variable nothing reads cannot make ignoring it safe.
- The refusal text is unchanged. It already reads correctly for an
exclusion (serving an action the operator removed is starting "under a
capability it did not ask for"), and rewording it would change the two
existing refusals' lines for no gain.
- A stdio e2e file was added although nothing there pinned the old
warning, because the retired-name split had no hold against the binary
at all, and the dotenv path is where a long-lived deployment keeps a
retired name.
- The site's stale "still works" paragraph and its table of bare names
were corrected here rather than left, since they are the passage the
refusal rule belongs in and contradicted it.

## Not done

- The coverage values in the testing reference are carried forward, not
recomputed; a full `make gen-testing-docs` would measure them again.
jmrplens added a commit that referenced this pull request Sep 30, 2026
…nd say one credential can fill them (#1093)

On 2026-09-29 I decided the questions the layers of this stack had put
to me, and this layer writes each answer and its reason where the stack
still presented it as open: the register, the specification, ADR-0023,
the server's comments, `CLAUDE.md` and the operator pages. It changes no
value, key, refusal or site of any row, and no behaviour; `make
check-tenancy` reads the same 83 rows and 308 sites as the layer below.

## What was decided, and where it now stands

For the held-call ceiling (`HLD-011`, the 951b layer), every answer is
the one the layer proposed:

- The value stays derived from the hard descriptor limit, and no
operator setting moves it: raising the limit raises it together with
what it protects.
- Windows keeps 192 (`FallbackDescriptorLimit` 1024) rather than no
ceiling, so that platform is not left unbounded by default.
`descriptor_limit_other.go` and the row's comment say so.
- No per-credential ceiling stands beside it. The credential is a key a
caller can mint (`INV-003`), so a per-credential number multiplies with
every token a caller mints and bounds nothing the process ceiling does
not. One credential can therefore fill it where the limit is small, and
the operator pages now say that in as many words.
- The server keeps no memory cap. At about 190 KiB a held call, a memory
limit set outside the server is what bounds held calls where the
descriptor limit is large. I named the container's limit or
`GOMEMLIMIT`. The pages tell operators to set the memory limit the
process runs under, a memory limit on the container or `MemoryMax` on
the systemd unit, since a unit without `MemoryMax` is bounded only by
the host, and describe `GOMEMLIMIT` for what it does: the runtime
collects harder as the heap nears it, but it refuses nothing, so live
held calls pass it. It is recorded as a lever and not as the bound.
- The slot taken after admission, the one bit `INV-019` accepts, and the
two refusal channels are accepted as implemented.

For the stateful session ceiling (`HLD-010`, the 951c layer), again as
proposed:

- Half the held-call ceiling (`SessionHeldDivisor` 2, 96 under a hard
limit of 1024), with the stream's slot reserved when the session opens.
- No per-credential session ceiling and no metering of `initialize`, for
the held-call ceiling's reason: a number or a price in the opener's own
rate on a key a caller can mint multiplies with every token it mints.
`RTC-004`, the row that leaves `initialize` charged to no bucket, now
records issue 951 in `Decided`.
- No fixed cap beside the derived figure: the memory limit the process
runs under (a container's, or a systemd unit's `MemoryMax`) bounds the
ten to twelve GiB of idle sessions a large descriptor limit admits, and
a cap no flag moves would be sized for one host.
- The order of admission and slot, the one bit, and the goroutine a
session keeps to give its slots back are accepted as implemented.
- With these answers issue 951 is complete: the 951c layer's body now
closes it, and the 951b layer's stays a part of it.

For #950 and #1045 the tree already stated the decisions, and only the
pull request bodies changed: `ADM-014` stays listed under `INV-019` and
NEG-007 and keeps `ADM-002`'s refusal words; VAL-012 is answered by the
measurement the OAuth driver layer took, not by a statement; the retired
bare `EXCLUDE_TOOLS` refuses a start, which the layer below implements.
The pool's probe ceiling (`POL-006`) discloses the same one bit as
`ADM-014` and is not listed under `INV-019`; that is a follow-up of its
own rather than part of this stack, drafted as an issue and not yet
filed. The queue of requests waiting for a verification slot, which the
OAuth driver measured, is drafted as an issue of its own in the same
way.

## What changed in the tree

- `internal/tenancy`: the comments of `HLD-011`, `HLD-010` and `RTC-004`
state the decisions and their reasons, and `RTC-004` records issue 951
in `Decided`. Two pins hold what was decided:
`TestDecisions_HeldRequests_AreBoundedOnTheProcessByIssue951` now
requires `HLD-011`'s values to be the three descriptor figures and
nothing else, so a memory cap cannot join them unnoticed, and the new
`TestDecisions_Initialize_StaysUnmeteredByIssue951` requires
`initialize` to stay unmetered with the decision recorded. The session
test already held `HLD-010` to `SessionHeldDivisor` alone and to no
partner.
- `cmd/server`: comments only, in `sessions.go` and
`descriptor_limit_other.go`.
- `docs/development/tenant-policy-spec.md`, ADR-0023 and `CLAUDE.md`
(the held-call paragraph of the HTTP routing section) state the
decisions where they had put the questions.
- The operator pages say that one credential can fill either ceiling and
that the memory limit the process runs under bounds memory, a
container's or a systemd unit's `MemoryMax`: the HTTP server mode,
enterprise and remote deployment guides, the security concept page, the
resource consumption reference, and on the site the HTTP server,
security, sizing and remote deployment pages in English and Spanish.
- The site's troubleshooting tables, English and Spanish, are padded to
their widest row again; rows added lower in the stack had left them
unpadded, and `format_md_tables --check` reported both. Whitespace only,
in a commit of its own.
- This layer was the top of the stack when it was written and adds a
test, so it carries the refresh of the README statistics and the testing
reference, in a commit of its own; the testing reference was regenerated
with `-skip-coverage`, and the site stats were checked and are
unchanged.

## After review

- The `HLD-010` and `HLD-011` comments, the `sessionHeldDivisor` comment
in `cmd/server/sessions.go` and the `HLD-011` test's message said the
container's memory limit bounds what held calls and idle sessions keep,
right after the example of the 524288 descriptors a systemd service
gets. A systemd unit is not a container, and its `MemoryMax` defaults to
infinity, so they now say the memory limit the process runs under (a
container's, or a systemd unit's `MemoryMax`) bounds it where one is
set, and that a unit without one is bounded only by the host, which is
the case the decision accepts. Comments and a test message only; `make
check-tenancy` reads the same 83 rows and 308 sites, `internal/tenancy`
stays at 100.0% and golangci-lint is clean on both packages. A second
review found the specification, `CLAUDE.md` and every operator page
still saying the container's memory limit, often right after the systemd
example, and the remote deployment guide, whose recommended shape is a
unit with `MemoryMax=512M`, telling its reader to set a memory limit on
the container. They were changed the same way in a commit of their own:
the specification's three places, `CLAUDE.md`'s held-call paragraph, the
HTTP server mode, enterprise and remote deployment guides, the security
concept page and the resource consumption reference, and on the site the
HTTP server, security, sizing and remote deployment pages in English and
Spanish, the remote deployment pages now telling a systemd operator to
keep `MemoryMax` on the unit or to set a memory limit on the container.
markdownlint, `format_md_tables --check` and the em dash check are clean
on them.
- The lower layers gained review fixes (the OAuth driver's settled-queue
measurement and its classifier, and the 1045 layer's comment and
collector harness), so this layer, still the top, carries a second
refresh of the README statistics and the testing reference, in a commit
of its own, regenerated with `-skip-coverage`; `gen_stats -check`,
`gen_testing_docs --check -skip-coverage` and the site stats check pass.
The second review's fixes below it (a test moved onto the test goroutine
and a case added in the OAuth driver layer, and the HTTP module's
starters in the 1045 layer) moved the README statistics again, so a
third refresh followed at the top; the testing reference, the site stats
and the llms files were checked and were unchanged. When the stack was
rebased onto 50ef2cf (#1081 and #1082), the README statistics and the
testing reference were regenerated at each refresh commit to resolve the
conflicts there, which left that third refresh empty, and it was
dropped. The stack now sits on b44ab96, which adds #1083, #1084 and
#1085; they change the upstream register, indirect module versions and
the site's package pins, none of them a file this stack writes, and
every commit of the stack applied unchanged. At the new top, a5401f1,
it builds and vets, every e2e gitlab package compiles, the unit suites
of the six packages the stack touches pass, `make check-tenancy` reads
83 rows and 308 sites, and the freshness gates CI runs pass again.
- On that base `CLAUDE.md` still named 3.0.0 in the project structure
beside a `VERSION` of 3.1.0, and its scale table's counts predated the
packages main has gained since, `internal/testutil/serialtypecheck`
among them. A `docs:` commit brings them to the tree: 286 packages from
`go list ./...` and 301 directories in the README's count, the same
fifteen apart, and 471 test files under `internal/tools` rather than
373. The Copilot instructions repeat two of the figures `CLAUDE.md`
states and one of their own, and all three had fallen further behind
(177 `internal/tools` packages, client-go v2.62.0 and 31 utility
commands); they now say 179, v3.14.0 and 42.

## Checks

- `internal/tenancy`: tests pass shuffled, at 100.0% of statements, and
under `-race`; conditions 408 of 408; mutants
(`scripts/coverage-mutants.sh`, whole package) 226 killed, 0 lived, 0
timed out, 66 not covered, the same 66 as the base: constant arithmetic
in `codes.go` (26), `values.go` (33) and `decision.go` (7), which
carries no statement counter.
- `cmd/audit_tenancy`: tests pass at 100.0%. `make check-tenancy`: 83
rows, 21 failures and 308 declared sites over 202 packages, 0 findings,
56 declarations exempted.
- `cmd/server`: comments only; the package builds and vets for Linux and
builds for Windows, where `descriptor_limit_other.go` compiles.
- golangci-lint (`e2e` tag) over `internal/tenancy` and `cmd/server`:
clean. `gofumpt` and `goimports`: clean.
- `check-test-subtests`, `check-test-file-names`,
`check-test-goroutines`, `check-doc-tool-names`, `check-doc-links`,
`check-site-stats`, `gen_stats -check` and `gen_testing_docs --check
-skip-coverage`: pass.
- markdownlint on every changed Markdown and MDX page, `format_md_tables
--check`, and `check-em-dash` against the layer below and against the
stack's base: clean.
- At this layer's head on 50ef2cf, while it was the top: it builds and
vets (plain and under the `e2e`, `httpe2e`, `stdioe2e` and
`collectore2e` tags), every e2e package compiles, and the
`serialtypecheck` gate #1082 added passes, under `-race` too; the whole
unit suite passes shuffled with the freshness gates checked (276
packages); every make target of CI's generated-artifacts job passes,
with `check-request-inventory` on a recording of the whole suite (1625
rows, 1085 of 1085 actions owned by a package that recorded a request,
unchanged); `make golangci-lint` and `make govulncheck` report nothing;
`test/e2e/http` passes (605 tests, the 7 Docker-only proxy and balancer
cases skipped) and `test/e2e/stdio` passes (118 tests).

## Not done

- `make tenancy-code-identity` was not run. The layer changes comments
in `cmd/server`, one `Decided` entry and the register's tests, and moves
no value; `check-tenancy` holds every site where it was.
- The complete Docker runs were taken at the top of the stack rather
than at this layer's head. The CE run of this layer's head before the
rebase onto b44ab96 (47ad92602, GitLab 19.4.1) failed one end-to-end
scenario, the Pages unpublish, which raced a second pipeline of its own;
the layer above fixes it, and both complete runs at that top (e600a73d0,
also before the rebase) then passed with no skip (19.4.1-ee, 1332 tests;
19.4.1, 1067 tests).

Part of #951.
jmrplens added a commit that referenced this pull request Sep 30, 2026
…lands after the unpublish (#1094)

The complete CE run of this stack's top (47ad92602, the
issue-951-decisions head before the stack was rebased onto b44ab96;
GitLab 19.4.1) failed one end-to-end scenario, and the failure is
neither in the server nor in anything this stack changes:
`TestPages_Unpublish_RemovesTheDeployment` raced its own second
pipeline. This layer, the new top of the stack, fixes the scenario.

## Two pipelines deployed every site

The scenario commits a Pages configuration to the default branch and
then creates a pipeline, so every project ran two: the push started one
and the fixture the other, which is why the pipelines the three surfaces
waited for were 33, 35 and 37. Both run the `pages` job, and both deploy
the site.

The wait for a deployment was therefore met by whichever deployment came
first. On the meta surface, the unpublish and the read after it
completed within 200 ms of the read that met the wait, and that last
read still listed a deployment created in the same second: the other
pipeline's, deployed after `Pages::DeleteService` had deactivated every
active one. The dynamic and individual surfaces passed on the same run,
and the same scenario passed on the stack's earlier Docker runs.

## The fix

`deployPages` now commits the configuration with `[skip ci]`, as the
model evaluation's `commitCIConfiguration` already commits the
configurations of the worlds it builds, so the pipeline the fixture
creates is the only one and its deployment is the one the wait sees.
GitLab ignores the marker for a pipeline created through the API
(`lib/api/ci/pipelines.rb` calls `Ci::CreatePipelineService` with
`ignore_skip_ci: true`), so that pipeline still runs. The push still
records a pipeline of its own, in the skipped state and with no job
(`Gitlab::Ci::Pipeline::Chain::Skip`), so each project still takes two
pipeline IDs and the one the fixture waits for is every other one (32,
34 and 36 on the passing CE run, 33, 35 and 37 on the EE run). What the
scenario holds is unchanged: a deployment exists before the unpublish,
and after it the settings list none or the read is refused as not found.

Other scenarios also commit a CI configuration and then create a
pipeline through the API, so they run a second pipeline too: the job
lifecycle, the resource group and the SAST report pipeline, and every
scenario built on `fixture.CIFile`. None of them reads anything the
push's pipeline can change after its wait: the job reads are pinned to
the created pipeline or resolved to the newest successful one, the
resource group scenario asserts only the shape of what is queued, the
SAST findings are read for the created pipeline's iid, and the pipeline
lifecycle already accepts a latest pipeline no older than the one it
created. None of them failed on these runs, and this layer leaves them
as they are. The release job scenario is not one of them: its one job
runs only on a tag, so the branch push starts nothing, and the tag
pipeline it waits for runs on the commit that carries the configuration,
where the marker would skip the only pipeline it has.

## Checks

- **Docker, complete runs at this layer before the rebase onto b44ab96
(e600a73d0 on the previous base 50ef2cf, the same patches as e4d21e7
on b44ab96), with the skip gate on:** the licensed run on GitLab
19.4.1-ee passed, 1332 tests with no failure and no skip; the unlicensed
run on 19.4.1 passed, 1067 tests with no failure and no skip. The skip
gate read 0 skipped, 0 declared, 0 undeclared and 0 stale on both. The
Pages scenario passed on all three surfaces of both runs.
- **The run that found it:** the CE run of 47ad92602, the layer below
before the rebase, on the same 19.4.1: 1067 tests, 2 failures (the
scenario and its meta subtest), 0 skipped. Nothing else failed.
- **Coverage record:** re-recorded from the two passing runs in a commit
of its own. The levels are unchanged (CE L1 834 of 872, L2 730, L3 714;
EE L1 1028 of 1092, L2 924, L3 908), the test calls grow to 3524 and
4671, and every package entry names e600a73d0, the pre-rebase tree the
runs tested. `make check-e2e-coverage-record` and `make
check-e2e-coverage-page` pass.
- **At e600a73d0, before the rebase, with the freshness gates checked**
(the record commit above it changes only the two coverage files, whose
gates, `check-e2e-static` and the freshness checks were run again after
it): it builds and vets (plain and under the `e2e`, `httpe2e`,
`stdioe2e` and `collectore2e` tags), every e2e package compiles, and the
`serialtypecheck` gate passes, under `-race` too; the whole unit suite
passes shuffled (276 packages); every make target of CI's
generated-artifacts job passes, with `check-request-inventory` on a
recording of the whole suite (1625 rows, unchanged); `make
golangci-lint` and `make govulncheck` report nothing; `test/e2e/http`
passes (605 tests, the 7 Docker-only proxy and balancer cases skipped)
and `test/e2e/stdio` passes (118 tests).
- **After the rebase onto b44ab96** (#1083, #1084 and #1085, none of
which touches a file this stack writes): at a5401f1 it builds and vets
(plain and under the `e2e`, `httpe2e`, `stdioe2e` and `collectore2e`
tags), every e2e gitlab package compiles, the unit suites of the six
packages the stack touches pass shuffled, `make check-tenancy` reads 83
rows and 308 sites, and the freshness gates CI runs pass: the README
statistics, the llms files, the LobeHub manifest, the model evaluation
corpus and results, the e2e coverage record and page, the token
footprint, the site stats, the benchmark charts, the action catalog
manifest, the testing reference, the request inventory and the golden
snapshots. `check-e2e-static`, `brand-check`, `check-install-buttons`,
`check-doc-tool-names`, `check-doc-links` and `check-plan-untracked`
pass there too.
- **A docs commit at the top (a5401f1):** `CLAUDE.md` opened its HTTP
routing section with "Nine invariants" over eight bullets (main says six
over seven, and the held-call bullet this stack adds moved the figure by
three), and said 19 skills in its project structure and its skills
heading, as the Copilot instructions did, where `.github/skills` holds
18 and the skills table lists 18. They now say eight and 18; no link
targets the heading's anchor. markdownlint on both files,
`format_md_tables --check`, `check-em-dash` against `main` and the layer
below, `gen_stats -check`, `check-llms`, `check-doc-tool-names`,
`check-doc-links` and `audit-catalog-first`, which reads both files,
pass.
- **Local:** `gen_stats -check`, `check-openplugin`, `check-doc-links`,
`check-plan-untracked`, `check-mcpb`, markdownlint on the changed pages,
`format_md_tables --check` and `check-em-dash` against `main` pass, and
the site-lint job's gates pass in `site/`.
- **Generated artifacts:** the README's end-to-end line count moves with
the comment, in the refresh commit at this top; every other generator
came out unchanged.

## Review fixes from #1093

The two review threads on #1093 are answered here, at the top, so that
no layer below is rebased for a wording change (9185eac):

- **What a refusal of `HLD-010` or `HLD-011` establishes.** `INV-019`
read a refusal of either as showing that other callers hold slots. It
establishes only that the process is full: one credential can fill
`HLD-010` by itself, and `HLD-011` where the descriptor limit is small
or the rate limit is off, and the refusal names no figure, so a caller
can conclude that others hold slots only from its own count, plainly
when it holds none. The specification now says so, and its lead sentence
draws the inference that others are holding only where a caller is still
under a bound of its own or knows the upstream to be healthy, which
leaves these two rows out. ADR-0023 NEG-007 now calls their refusal at
most the same bit, and the comment on the session refusal in
`cmd/server/sessions.go` no longer says that words naming the ceiling
would tell a caller others hold sessions; the register rows and the
held-call refusal already stated only that the process is full.
- **The memory limit a systemd unit inherits.** Every page, register
comment and code comment that said a unit without `MemoryMax` is bounded
only by the host was wrong for a unit whose slices set one, since cgroup
v2 applies the `memory.max` of every ancestor. They now say it is
bounded by the slices above it where one of them sets one, and otherwise
only by the host, in English and Spanish, and the operator guides name
`systemctl show -p EffectiveMemoryMax <unit>`, which reports the
tightest of those limits on systemd 256 and later and prints nothing on
earlier versions (checked on 252).

Checked at 9185eac: `go test ./internal/tenancy`, the `cmd/server`
build and `make check-tenancy` (83 rows, 308 sites, 0 findings) pass
through the remote runner; markdownlint on the changed pages,
`format_md_tables --check`, `gen_llms -check`, the site stats check and
`gen_stats -check`, after regenerating the README statistics for the
four comment lines added, pass locally, and no em or en dash is added.

## Stack

Part of the tenancy stack for 3.1.0; no issue of its own. The other
scenarios that start a second pipeline, and the fixtures the model
evaluation shares with them, are recorded in #1096.

This branch was previously deployed

1 inactive deployment
release — 6a649151 Deployed Sep 30, 2026 by jmrplens
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file security Security-related issue site The documentation site's build, components and scripts, not its content v3.1.0 Targeted at the 3.1.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant