Skip to content

Bound the calls the process holds open at once, sized by its descriptor limit - #1088

Merged
jmrplens merged 13 commits into
issue-950-oauth-boundsfrom
issue-951b-held-requests
Sep 30, 2026
Merged

jmrplens merged 13 commits into
issue-950-oauth-boundsfrom
issue-951b-held-requests

Conversation

@jmrplens

@jmrplens jmrplens commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

A tools/call holds its POST for as long as the call runs: up to an hour for a pipeline wait, and any call for as long as GitLab keeps it waiting. The rate limit bounds how often a credential calls, not how many of its calls are still running, and the pool bounds entries, not what an entry holds, so nothing bounded how many calls one credential, or all of them, could keep open (finding F-31). Under a hard descriptor limit of 1024 the process held about five hundred and then stopped accepting connections at all, /health among them.

This is the second of issue 951's three layers. It adds register row HLD-011: a ceiling on the calls the process holds open, keyed on the process and set by no flag. The stateful sessions, the other half of F-31, are the third layer.

What it is sized from. The descriptor limit the process runs under, read once at startup: an eighth of it spare, one descriptor for each of the 512 listen streams (HLD-002), and two for each held call (the caller's connection and the one to GitLab). The Go runtime raises the soft limit to the hard one before main, so the figure follows the hard limit: 192 under 1024, 1536 under 4096, 229120 under the 524288 a systemd service gets by default, 458496 under 1048576. Windows has no such limit to read and gets the figure for 1024. No flag moves it; raising the descriptor limit raises it together with what it protects. The startup line announces it as held_requests_per_process.

What it counts. Every call that reaches GitLab (the methods MeterFor charges to the tool-call and completion buckets: tools/call, resources/read, resources/subscribe, prompts/get, completion/complete), a subscriptions/listen aside, counted where the SDK dispatches it. So each call of a JSON-RPC batch counts on its own, and a response a client sends to a request of the server's own (its answer to an elicitation), a notification, or a listen on any revision never does. On protocol 2026-07-28 or later, where the SDK holds Mcp-Method to the body and refuses batches, the gate takes the slot itself after admission and hands it to the call through the carrier, so the refusal there happens before the SDK reads a byte.

How it refuses. Always in the words This server is busy. Retry later., naming no bound; the log line (request refused: too many requests held across the process) names scope=process and the figure. On 2026-07-28 or later: a gate 503, -50300, Retry-After: 30, and the connection closed, so refused callers do not accumulate descriptors of the limit the ceiling protects. On older revisions: the way the rate limit refuses the same method (a tools/call as a result with isError; resources/read, resources/subscribe and prompts/get with -42900; a completion as an empty completion). Nothing is charged, and the credential's rate is not spent: the middleware runs before the rate limit.

Measured

Truenas, cmd/bench_resources -held against a stand-in GitLab that holds every project read, headers first. Per held call, linear to 4000 calls: 2.00 descriptors, 6.00 goroutines, about 51 KiB of live heap and about 190 KiB resident.

Run Offered Held Served Refused Failed Descriptors /health
before, 1 credential, hard 1024 1000 503 558 0 442 1024 unanswered for 5 s
before, 1 credential, hard 1024 4000 503 3466 0 534 1024 unanswered for 5 s
this change, 1 credential, hard 1024 4000 192 192 3808 0 394 200 in 0 ms
this change, 100 credentials, hard 1024 4000 192 192 3808 0 394 200 in 0 ms
this change, 1 credential, hard 1048576 4000 4000 4000 0 0 8010 200 in 0 ms
this change, 100 credentials, hard 1048576 4000 4000 4000 0 0 8010 200 in 0 ms

The first cut of this layer reached 1141 to 1280 descriptors with a hundred credentials, because the refused connections stayed open; the refusal now closes its connection. make bench-held runs the ladder (HELD_CREDENTIALS=100 for the second row). test/e2e/http holds the same behaviour against the real binary started under ulimit -n 1024: 192 calls held, a modern call refused in the gate with the connection closed, an older revision's call refused as a tool error, a batch of two refused call by call, a notification accepted, a listen acknowledged, 600 more calls offered at once all refused while /health answers, and every held call served once GitLab answers.

What the review of the first cut found, and what changed

  • A POST is not a call. A JSON-RPC batch below 2025-06-18 carried as many calls as the body held on one slot; a POST carrying only a response or a notification was counted and refused at a full ceiling, which kept an elicitation's answer from the call waiting on it; and an older revision's listen was counted twice. The count moved to where the SDK dispatches calls, as above.
  • 192 was sized for a soft limit no Go process runs under. On a default systemd host it cut what the process could hold to about a twentieth and let one credential's pipeline waits have everyone else refused for up to an hour. The ceiling is now derived from the hard limit, and the pages that called 1024 the common case say what the limit actually is.
  • The many-credential measurement under 1024 was missing and the first cut would have failed it; it is the fourth row above.
  • The refusal named the bound; it now says only that the server is busy (item 5 under Decided on issue 951).
  • PAT-003 was cited for the opposite order; the departure is recorded instead (item 4 under Decided on issue 951).
  • The OAuth wiring was untested; TestRegisterOAuthMCPHandlers_RefusesPastTheProcessHeldCeiling fails if registerOAuthMCPHandlers stops passing the count.
  • POL-003's answer was unpinned; TestHeldCall_TakesOneSlotAndLeavesItsEntryEvictable pins that a held call leaves its entry evictable and that the call outlives the eviction.
  • The e2e /health assertion could not fail; the binary now runs under ulimit -n 1024, where 792 held calls would not fit.
  • Docs: the sizing pages no longer call the failed-authentication budget unconfigurable, security.md no longer says every call past the limit failed (558 of 1000 were served), and the listen exemption is stated without a condition because there is none left. The enterprise guide now counts the two unconfigurable limits the way the sizing pages do (the watcher ceilings, then the held calls), with the configurable budget after them.
  • The house gate was measured over the files this layer writes only. cmd/bench_resources is now measured whole and cmd/server's main.go is measured too; the results, and what closed each survivor, are under Gates.

Decided on issue 951

The six questions this layer put are answered, each as proposed here, so nothing in this layer's row, pins, gate, e2e expectation or pages changes. The layer of this stack that records issue 951's decisions writes each answer and its reason where the tree still presented the question: the register row's comment, the specification, ADR-0023, the HTTP server, enterprise, remote deployment, security and resource consumption pages, and the site in English and Spanish.

  1. The value stays derived from the hard descriptor limit. No operator setting moves it, and raising the limit raises it together with what it protects. Windows keeps 192 (FallbackDescriptorLimit 1024) rather than no ceiling: 192 protects no descriptor there, and it is kept so the platform is not left unbounded by default.
  2. No per-credential ceiling stands beside it, as HLD-001 stands beside HLD-002. The credential is a key a caller can mint (INV-003), so a per-credential number multiplies with every token a caller mints and bounds nothing the process ceiling does not. One credential can therefore fill the ceiling where the limit is small, in about fifteen seconds at the default rate (a burst of 40, then ten a second), and the operator pages say so.
  3. The server keeps no memory cap. At about 190 KiB a held call, a memory limit set outside the server is what bounds held calls where the descriptor limit is large. I named the container's limit or GOMEMLIMIT; the pages set the container's as the bound and describe GOMEMLIMIT as a soft target the runtime collects toward and live held calls pass, so it is a lever and not that bound.
  4. The slot is taken after admission, the departure from issue 951's request to take the process slot before anything per key, accepted for its reason: a slot taken first would let a caller with no credential hold one for as long as its verification takes. A refused newcomer has spent its admission by then: in legacy mode its pool entry (with POL-006's probe and, at the pool's bound, another credential's quiet entry evicted, POL-002), in oauth mode one ADM-014 slot, and none of its rate.
  5. INV-019. With no per-caller ceiling beside it, any refusal tells its caller the process is full whatever the words, which is the one bit the INV-019 decision accepted for a bound keyed on the process, and HLD-011 falls under that exception rather than needing its own.
  6. Two refusal channels, accepted as implemented, because only a modern POST's headers prove what it carries: a gate 503 on 2026-07-28 or later, in-band before it. An older revision's in-band refusal does not close its connection, since the middleware has no connection to close; the ceiling bounds calls, not connections, and the guide says so.

The two confirmations issue 951 asked for: a held call does not make its pool entry busy (POL-003 unchanged, and now pinned by a test); the process listing bucket follows RTC-003 (the first layer, 951a).

Decisions I took here

  • The ceiling is derived from the hard descriptor limit read at startup, not fixed: an eighth spare, one descriptor for each of the 512 listen streams, two for each held call, with a floor of one call. On Windows, which has no such limit to read, it is sized as if the limit were 1024 (decided on issue 951, item 1).
  • It is keyed on the process alone, with no per-credential partner, and no flag or variable moves it (INV-004, INV-018; decided on issue 951, item 2).
  • The slot is taken after admission and the session ownership check, which departs from issue 951's request to take the process slot first; the departure is recorded on the row (decided on issue 951, item 4).
  • The refusal reuses UpstreamRetryAfter (30 s, fixed) and CodeUnavailable (-50300) rather than adding values, and says only that the server is busy; the log line alone names the scope and the figure (decided on issue 951, item 5).
  • Calls are counted where the SDK dispatches them, so a batch counts call by call and an elicitation's answer, a notification and a listen never count; only on 2026-07-28 or later does the gate take the slot itself, because only there do the headers prove what a POST carries (decided on issue 951, item 6).
  • A held slot does not make a pool entry busy and does not delay its eviction (POL-003).
  • I changed cmd/bench_resources to remove equivalent mutants rather than record them (admit now runs over one count, the sampler's count of answered processes is a bool, the goroutine-dump scanner has no initial buffer, the first chart tick candidate lost its +1), moved the latency stop's edge into crossesLatencyCeiling so a test can reach it, and gave the warm-up its own ceiling (warmUpTimeout); each change preserves behaviour for every input that reaches it, and the package stays at 100% of statements and 1322 of 1322 conditions.
  • auth_gate.go:793 (effectiveWindow <= 0) stays classified rather than deleted: retiring that three-way window fallback is issue 958's, as AUB-002 and Close the mutation and condition sweep of cmd/server #1015 already record.

Not done

  • Idle keep-alive client connections are not bounded by this ceiling, which bounds calls, not connections; an older revision's in-band refusal does not close its connection.
  • Memory is not bounded by it, and issue 951 decided the server adds no cap for it: about 190 KiB a held call, so under a large limit the container's memory limit bounds held calls first.
  • Listen streams and held calls filling the process together were not measured; the sizing arithmetic says they fit.
  • cmd/server was mutated over the four files this layer writes, not whole; main.go's survivors are the ones Close the mutation and condition sweep of cmd/server #1015 recorded on identical code.
  • The stateful sessions, the other half of F-31, are the next layer.

Validation checklist (tenant-policy-spec.md)

  • VAL-001, question: Allow.
  • VAL-002, key: KeyProcess, which no caller can mint. No new key.
  • VAL-003, class: P. Its reason ("only a ceiling keyed on the process bounds the process") and the reason's unit are both the process.
  • VAL-004, ceiling or share: a ceiling. It promises no caller a share.
  • VAL-005, process partner: it is itself keyed on the process, protects the process's descriptors, and is set by no flag, because an operator who could raise it could undo the bound (INV-004, INV-018). No per-caller row stands beside it, which issue 951 decided (item 2 above).
  • VAL-006, at capacity: refuses the newcomer; never evicts or ends a held call.
  • VAL-007, refusal: gate on protocol 2026-07-28 or later (503, -50300, fixed 30 s Retry-After, no challenge, connection closed); in-band on older revisions (tools/call as a tool error; resources/read, resources/subscribe and prompts/get as RPC -42900; completion/complete as an empty completion). Retry-later class throughout, text This server is busy. Retry later.. On stdio: none.
  • VAL-008, charges: nothing. The refusal comes after admission and before the rate limit, so it spends no budget and no rate token.
  • VAL-009, value: Source: Derived, from RLIMIT_NOFILE read at startup and three pinned constants (HeldRequestDescriptors 2, DescriptorSpareDivisor 8, FallbackDescriptorLimit 1024) beside ListenStreamsPerProcess; a floor of one call; zero is not applicable; no flag or variable, so no malformed-value policy.
  • VAL-010, telemetry: the reason only. The refusal's log line carries the reason, the scope and the figure and nothing about the caller, and it is throttled like every refusal a caller can cause at will. An in-band refusal is an ordinary result or error on the call's span, under whatever identity policy the deployment set for every call. No metric counts the refusals; one was not added in this layer.
  • VAL-011, stdio: not applied (StdioKey none). The middleware counts only calls that arrived on an HTTP POST, so a stdio server carries it and counts nothing.
  • VAL-012, measurement: the ceiling does not claim to protect one population from another. It protects the process's descriptors, and it is measured by the held mode above, which is what that claim needs; make bench-fairness answers a claim this ceiling does not make.

Gates

Measured on this layer before it was rebased onto main at 5da9699: the line numbers below are that tree's, and the one commit cited below (fef2e45) is named as it sits in this stack; main's server card change (#1061) moved main.go's lines and none of the code these figures describe. The stack was then rebased onto 50ef2cf, whose only change to a file this layer writes is the serialtypecheck blank import in cmd/audit_tenancy/main_test.go, and at its place in this stack on that base the layer built and vetted, the e2e gitlab, http and stdio packages compiled, the unit suites of cmd/server, internal/tenancy, internal/oauth and cmd/audit_tenancy passed, and make check-tenancy read the same 83 rows and 299 sites. The stack now sits on b44ab96, which adds #1083, #1084 and #1085; they change the upstream register, indirect module versions and the site's package pins, none of them a file this stack writes, and every commit of the stack applied unchanged.

  • go test -shuffle=on: 100.0% of statements in cmd/server, internal/tenancy, cmd/bench_resources and cmd/audit_tenancy; -race on the held tests; the whole test/e2e/http module (578 tests, the Docker-only proxy and balancer cases skipped); the e2e gitlab and stdio packages compile.
  • golangci-lint with the e2e and httpe2e tags: 0 issues. markdownlint and format_md_tables --check: clean.
  • make check-tenancy: 83 rows, 21 failures and 299 declared sites over 202 packages (21 refusal returns, 80 refusals, 13 reasons and 31 settings read); 0 findings, 56 declarations exempted. check-dead-consts, check-test-file-names, check-test-subtests, check-test-goroutines, check-doc-tool-names and check-action-ids pass.
  • gobco: internal/tenancy 408/408, cmd/bench_resources 1322/1322, cmd/audit_tenancy 1002/1002. cmd/server is measurable through a staged copy without its non-Linux twins (the issue 1017 approach): 2040/2082 (2039 on one of three runs, a timing-dependent condition outside these files), with none of the open conditions in held.go, descriptor_limit.go or the lines this layer changed in auth_gate.go and main.go; all of them predate this change, auth_gate.go:793 among them (the unobservable guard below).
  • gremlins (--workers 2 --timeout-coefficient 20, INVERT_LOGICAL on):
    • internal/tenancy, whole package: 226 killed, 0 lived, 0 timed out, 66 not covered, every one of them arithmetic in a constant declaration (codes.go 26, values.go 33, decision.go 7), which has no statement for coverage to count; each value is pinned by TestValues_HoldTheirPins, and the count is the base's.
    • cmd/server, scoped to the four files this layer writes (the house rule allows it for this package), in two runs:
      • held.go, descriptor_limit.go and auth_gate.go: 99 killed, 1 lived, 8 not covered, 0 timed out (held.go and descriptor_limit.go alone: 23 killed and nothing else). The first pass over the held files timed out three mutants, because two tests hung instead of failing when the gate refused a POST they expected to pass; both now fail at once (fef2e45). The transport budget's two window-edge boundaries (charge re-opening a pair at exactly one window, and the sweep dropping it then) are now killed: two tests land a charge exactly one window later on a fake clock (testing/synctest), which the cmd/server sweep (Close the mutation and condition sweep of cmd/server #1015) had recorded as a clock no test could schedule. The one lived is window()'s effectiveWindow <= 0 (auth_gate.go:793): newTransportBudget never stores a window of zero or less, so the guard is unobservable, and Close the mutation and condition sweep of cmd/server #1015 recorded it and left it for issue 958, which retires that three-way fallback; deleting it here would be half of 958. The 8 not covered are constant initializers: the two JSON-RPC code literals, whose sign flips TestRefusalCodes_MirrorTheirStatusAndSitWhereTheSpecificationAllows kills when they are applied by hand, and + to - between string constants, which does not compile.
      • main.go: 338 killed, 4 lived, 25 not covered, 0 timed out. This layer's change to it carries no mutable token (a middleware registration, two struct fields, a log attribute and a help line). The 4 lived are the four boundaries Close the mutation and condition sweep of cmd/server #1015 recorded, on identical code (at lines 1899, 2605, 3244 and 3744 in Close the mutation and condition sweep of cmd/server #1015's tree, and 1911, 2629, 3273 and 3776 in the one measured here); the 25 not covered are 15 constant and variable initializers, 9 switch case expressions and the equivalent 802:19 Close the mutation and condition sweep of cmd/server #1015 recorded, which is where Close the mutation and condition sweep of cmd/server #1015 left the file.
    • cmd/bench_resources, whole package, with the script's default --exclude-files=/ so the out-of-process stand-in under testdata/standin is not counted. Before this pass: 1173 killed, 24 lived, 67 not covered, 2 timed out, all in files this layer does not otherwise touch. After it: 1174 killed, 17 lived, 67 not covered, 0 timed out.
      • The two timeouts were coldList's retry rule mutated to ask again after an answer it should take: every test that admits against a stand-in refusing tools/list then retried until callTimeout, five minutes a warm-up, and the suite timed out instead of failing the test that pins the rule. The warm-up now has its own ceiling (warmUpTimeout, callTimeout in a run and fifteen seconds under the tests); both mutants are killed.
      • Killed by a new test: waitUntil at an instant that is exactly now on a cancelled run (a fake clock); the latency stop's edge, moved into crossesLatencyCeiling so a tail exactly at the ceiling can be asked for; admit's want-have (twice), now the one count the loop runs over; and two switch case edges gremlins reports as not covered that no test pinned, msLabel at 1 and 10 ms and cpuFigures on equal samples.
      • Removed rather than recorded: the +1 on linearXTicks' first candidate (the spacing test already turns the origin away), the sampler's count of answered processes (read only for zero, now a bool) and the initial buffer of the goroutine-dump scanner (only the line ceiling matters).
      • The 17 lived, each one of the shapes docs/development/testing/README.md records. Two sides that agree at the boundary: chart.go:276 (Log10(0) is -Inf, which clamps to the floor where v = lo lands), chart.go:330 and proc_sample.go:271 (the assignment writes the value already held), chart.go:570 (a candidate at maxX is turned away by the spacing test), chart.go:718 and :721 (a shift by zero), fairness.go:242 (at share zero the division is +Inf for every bound, all of which meter a positive rate, and min caps it at the same default), fairness.go:617 (zero times anything), fairness_result.go:550 (a floor of zero is met by any busy figure, never negative), figures.go:402 and render.go:230 (a string of exactly the slice's length is returned whole either way), proc_sample.go:32 (a probe whose deadline has passed falls back to 100, what getconf answers on Linux). Unreachable by construction: chart.go:299 (ticks are whole steps and the bound is half a step past the last), fairness_result.go:715 (reached only once the claim exceeds a spread of zero or more), fairness_result.go:734 twice (a zero delta returns before either comparison, the tie-break shape), theme.go:96 (the index is taken over text that begins with the selector, so it is never 0).
      • The 67 not covered: 29 constant and variable initializers and 36 switch case expressions, which carry no statement counter, and 2 at fairness_run.go:277, a call argument after a function literal that the profile attributes to no block. Applied by hand, the fairness_run.go:277 mutant is killed by TestDrive_SpreadsEachPopulationAcrossOnePeriod and the two protocol codes' sign flips by TestClassifyOutcome_KeepsRefusalsApartFromSuccessAndFailure; every switch case boundary has a test at its edge now (the two that had none are above); + to - between string constants does not compile. gobco holds every case condition both ways (1322/1322).

Part of #951.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @jmrplens, your pull request is larger than the review limit of 150,000 diff characters

@coderabbitai

coderabbitai Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: jmrplens/gitlab-mcp-server/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: ddfd141a-8e9e-4cf5-bfcf-09a6c228bd4b

📥 Commits

Reviewing files that changed from the base of the PR and between 7e64430 and 7cbe64d.

📒 Files selected for processing (64)
  • Makefile
  • cmd/audit_tenancy/main_test.go
  • cmd/bench_resources/chart.go
  • cmd/bench_resources/fairness_run_test.go
  • cmd/bench_resources/figures_test.go
  • cmd/bench_resources/held.go
  • cmd/bench_resources/held_test.go
  • cmd/bench_resources/main.go
  • cmd/bench_resources/main_test.go
  • cmd/bench_resources/matrix.go
  • cmd/bench_resources/matrix_test.go
  • cmd/bench_resources/proc_sample.go
  • cmd/bench_resources/rpc.go
  • cmd/bench_resources/rpc_test.go
  • cmd/bench_resources/run.go
  • cmd/bench_resources/run_test.go
  • cmd/bench_resources/series.go
  • cmd/bench_resources/series_test.go
  • cmd/bench_resources/stubs.go
  • cmd/bench_resources/stubs_test.go
  • cmd/bench_resources/target.go
  • cmd/bench_resources/target_test.go
  • cmd/bench_resources/testdata/standin/main.go
  • cmd/server/auth_gate.go
  • cmd/server/auth_gate_test.go
  • cmd/server/descriptor_limit.go
  • cmd/server/descriptor_limit_other.go
  • cmd/server/descriptor_limit_test.go
  • cmd/server/held.go
  • cmd/server/held_test.go
  • cmd/server/main.go
  • docs/concepts/security.md
  • docs/development/adr/adr-0023-tenant-policy-is-declared-once.md
  • docs/development/tenant-policy-spec.md
  • docs/guides/enterprise-deployment.md
  • docs/guides/http-server-mode.md
  • docs/guides/remote-deployment.md
  • docs/guides/troubleshooting.md
  • docs/reference/cli.md
  • docs/reference/configuration.md
  • docs/reference/env.md
  • docs/reference/resource-benchmark.md
  • docs/reference/resource-consumption.md
  • internal/tenancy/decision.go
  • internal/tenancy/decisions.go
  • internal/tenancy/decisions_allow.go
  • internal/tenancy/decisions_test.go
  • internal/tenancy/values.go
  • internal/tenancy/values_test.go
  • site/src/content/docs/configuration.mdx
  • site/src/content/docs/es/configuration.mdx
  • site/src/content/docs/es/operations/http-server.mdx
  • site/src/content/docs/es/operations/remote-deployment.mdx
  • site/src/content/docs/es/operations/security.mdx
  • site/src/content/docs/es/operations/troubleshooting.mdx
  • site/src/content/docs/es/performance/sizing.mdx
  • site/src/content/docs/operations/http-server.mdx
  • site/src/content/docs/operations/remote-deployment.mdx
  • site/src/content/docs/operations/security.mdx
  • site/src/content/docs/operations/troubleshooting.mdx
  • site/src/content/docs/performance/sizing.mdx
  • test/e2e/http/harness_test.go
  • test/e2e/http/held_test.go
  • test/e2e/http/limits_test.go

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added transport stdio and HTTP transports, the server process, and the transport e2e modules tooling The audit and generator commands under cmd/, and the Makefile targets that run them labels Sep 30, 2026
@jmrplens
jmrplens added this pull request to stack #1095 September 30, 2026 10:18
@jmrplens jmrplens added the v3.1.0 Targeted at the 3.1.0 release label Sep 30, 2026
@jmrplens jmrplens added this to the 3.1.0 milestone Sep 30, 2026
@jmrplens jmrplens changed the title issue 951b held requests Bound the calls the process holds open at once, sized by its descriptor limit Sep 30, 2026
jmrplens added a commit that referenced this pull request Sep 30, 2026
The fix for #945 (#948) removed the workflow's own `actions/cache` step
for `~/.cache/golangci-lint` and left a second cache of that directory
in place: `golangci-lint-action` keeps one itself. `install-only: true`
returns after the install and nowhere else, so the action still restores
that cache before installing (`run()` calls `restoreCache` first) and
saves it again in its post step, under
`golangci-lint.cache-<os>-<interval>-<go.mod hash>` keys. The comment in
`ci.yml` said the cache was not restored and the static analysis page
that it was not kept between runs, and neither was true.

On 2026-09-30 the lint of #1088 to #1094, opened minutes before,
reported fifty unused `//nolint:staticcheck` directives on each layer,
in files the stack does not touch. Every one of those jobs restored, by
exact key, the entry main's push run of b44ab96 had saved at 09:07 UTC:
the key names only the OS, a seven-day interval and the hash of
`go.mod`, which the stack shares with main, so the analysis of main's
tree matched a different tree exactly. With that entry deleted, #1087's
second run and a rerun of #1088's commit fell back through the restore
key to main's entry of 2026-09-25 and linted clean, and a cold `make
golangci-lint` of the stack's tip reports nothing. That is the failure
#945 describes, a restored analysis disagreeing with the tree, coming
from the cache #948 did not see, and this time from main's entry rather
than the pull request's own.

`skip-cache: true` turns off both the restore and the save. The comment
above the step now says why it is needed alongside `install-only` and
what the key leaves out, and `docs/development/static-analysis.md` says
the same. The job's header comment no longer says the action caches the
linter binary, which it downloads on every run. The Go build and module
caches setup-go keeps are unaffected.

What it costs, measured on the whole job as #945 asks: the eight pushes
to main between 29 September 11:27 and 30 September 09:02 UTC each
restored a cache and took 214 to 319 s, the `make golangci-lint` step
190 to 298 s of it; this pull request's run, the only one with the
change so far, restored nothing and took 313 s, the step 287 s. All
eight restored the entry main saved on 25 September, which the action
does not save again while its key still matches, so the analysis they
reused was of a tree days older than theirs: it was not buying time the
spread between runs can show.

Closes #945
@jmrplens
jmrplens force-pushed the issue-951b-held-requests branch from 28dad9d to 8e73aaa Compare September 30, 2026 12:05
Every mode of the benchmark drives requests the server answers as fast as
it can, so none of them could say what a request the server holds open
costs, or what a process holding too many of them does. Issue 951 needs
both to size a ceiling on held requests, so this adds a mode that asks.

-held <counts> offers a ladder of gitlab_execute_action calls running
project.get, spread round robin across -held-credentials, against a
stand-in GitLab that now answers a project read and, while a hold is on,
keeps it waiting after its headers. Headers first, because the server
gives GitLab a minute to answer with headers and would abandon and resend
a read held before them, and a step would measure the retry. A step is
sampled once every call is held or has come back, or once that count has
been still for three seconds (a process out of descriptors accepts no
more connections, so some calls are neither), or after two minutes; the
wait reads a clock the tests replace, so each end is held to its instant.
The
sample is the process's open descriptors from /proc/<pid>/fd, goroutines
and live heap from the profiling listener, the resident set, and a
/health probe on a connection of its own, since a probe riding an old
keep-alive connection would answer for a process that no longer accepts
one. Every call is then filed as served, refused (a 503) or failed, and
one held call is priced as the step's growth over the idle process.

-held-nofile starts the server through /bin/sh and ulimit -n, which sets
the soft and the hard limit together: Go raises its soft limit to the
hard one at startup, so lowering the soft one alone would be undone
before the server served anything. The shell execs the server in its own
place, so the pid sampled is the server's.

The mode writes bench/held.json, uncommitted, draws nothing, and refuses
a -held-json that names the published record, like the fairness mode.

Calling gitlab_execute_action exposed a gap in the harness's client: from
protocol 2026-07-28 the transport refuses a call whose parameter header
is missing, and the tool's schema mirrors its action argument into
Mcp-Param-Action. The client now sends it, only where the value exists.
The stand-in server learns the same call: it reads the project through
the --gitlab-url it is given, checks the header, and answers past
STANDIN_HELD_LIMIT with a 503, so the whole ladder runs in the unit
tests. parseSteps becomes parseCounts, shared by -clients and -held.

Part of #951.
A tools/call holds its POST for as long as the call runs: an hour for a
pipeline wait, and any call for as long as GitLab keeps it waiting. The
rate limit bounds how often a credential calls, not how many of its calls
are still running, and the pool bounds entries, not what an entry holds,
so nothing bounded how many requests one credential, or all of them,
could keep open. Issue 951 recorded that as finding F-31.

Measured with the new held mode of cmd/bench_resources on truenas (the
real server, a stand-in GitLab holding every project read):

- One held call costs 2.00 descriptors (the caller's connection and the
  one to GitLab), 6 goroutines, about 51 KiB of live heap and about
  190 KiB of resident set, linearly: 4000 held calls took 8010
  descriptors, 24013 goroutines and 873 MiB resident.
- Started under a descriptor limit of 1024 and offered 1000 calls, the
  process held 503 and then stopped accepting connections at all,
  /health among them, until they ended; 442 calls failed with "too many
  open files" at their dial to GitLab. At 4000 offered, 534 failed.
- With this change, under the same limit, 4000 calls offered: 192 held
  and served, 3808 refused with 503, none failed, 402 descriptors, and
  /health answered in a millisecond. The same at every step from 250.

The ceiling is 192 across the process (tenancy.HeldRequestsPerProcess),
sized against a descriptor limit of 1024, the default soft limit on most
Linux hosts, beside the other kind of request the process holds open:
512 listen streams at one descriptor, 192 held requests at two, and an
eighth of the limit spare for the idle process, /health and connections
being refused. It is keyed on the process because only a ceiling keyed
on the process bounds the process: a credential is one API call to mint,
so a per-caller number multiplies by however many a caller holds. And it
is not configurable, for the reason the 512 listen streams and watchers
are not: an operator who could raise it could undo the bound.

Decisions taken with it:

- The slot is taken last in mcpServerGate.middleware, after admission and
  the session ownership check (PAT-003), so a request refused for its
  credential, its instance or its session never spends one, and it is
  given back when the handler returns.
- Every POST counts except a subscriptions/listen whose Mcp-Method header
  says so on protocol 2026-07-28 or later, where the SDK holds the header
  to the body; the listen ceilings count those. An older client's listen
  is counted here, since its header is not checked against its body.
  GET and DELETE are not counted: the default stateless transport
  answers them 405, and a stateful session's standalone stream is one per
  session, which stays with HLD-010 and F-31 for the third part.
- The refusal is a gate 503 with -50300 and the fixed 30 s Retry-After
  the upstream refusals already advertise (UpstreamRetryAfter), in words
  that name no credential: "This server is holding as many requests as
  it serves at once. Retry later." It discloses one bit about the
  process, which INV-019 accepts for a process bound. The log line names
  the scope and the figure.
- A held call makes no pool entry busy and delays no eviction: the slot
  counts held work, never a caller.

The register gains row HLD-011 (Allow, Ceiling, process-keyed, valued,
not configurable) with its value pin and requirement ID; HLD-010 now
describes stateful sessions alone. make check-tenancy: 83 rows, 21
failures and 287 declared sites (77 refusals, 13 reasons), 0 findings.
The HTTP e2e module holds the whole of it against the real binary: 192
calls held, the next refused in the register's words with its
Retry-After and log line, /health and a listen still answered, and every
held call served once GitLab answers. The specification, ADR-0023, the
HTTP guide, the references and the site pages in English and Spanish say
what the ceiling is and what it is not.

Part of #951.
… descriptor limit

The first cut of HLD-011 counted POSTs in the gate against a fixed 192,
and the review of the layer found both halves wrong.

A POST is not a call (truth finding 1, house finding 3, truth findings 3
and 4). Below protocol 2025-06-18 go-sdk accepts a JSON-RPC batch, so one
slot carried as many calls as the body held, up to the 4 MiB limit with
the rate limit off. A POST on an older revision can carry only a
response or a notification, which holds nothing, and refusing it at a
full ceiling kept a client's answer to an elicitation from the call
waiting on it, so the slots never freed. And a listen on an older
revision was counted by the listen ceilings and again here. The calls
are now counted in an MCP middleware, where the SDK has read the body:
every call that reaches GitLab (MeterFor's tool-call and completion
buckets) other than a subscriptions/listen, so each call of a batch
takes a slot, a response or notification takes none, and a listen takes
none on any revision. It runs before the rate limit and refuses the way
the rate limit refuses the same method. On protocol 2026-07-28 or later,
where the SDK holds Mcp-Method to the body and refuses a batch, the gate
still takes the slot itself after admission and hands it to the call
through the carrier, so that refusal is a 503 the SDK never reads, and
it now closes its connection (truth finding 5): a refused caller that
kept its connection held a descriptor of the limit the ceiling protects,
and a hundred credentials reached 1141 to 1280 descriptors that way.

192 was sized for a soft limit of 1024 that no Go process runs under
(truth finding 2, house finding 4): the runtime raises the soft
RLIMIT_NOFILE to the hard limit before main. On a default systemd
service or container that cut what the process can hold to about a
twentieth and let one credential's pipeline waits have every other
tenant refused for up to an hour. The ceiling is now derived at startup
from the limit the process actually has: an eighth spare, one descriptor
for each of the 512 listen streams, two per held call. That is 192 under
a hard limit of 1024 and 229120 under 524288; where the platform has no
limit to read (Windows) it is the figure for 1024. The row is Source
Derived, and HeldRequestDescriptors, DescriptorSpareDivisor and
FallbackDescriptorLimit are pinned in place of HeldRequestsPerProcess.

Every refusal now says only "This server is busy. Retry later.", naming
no bound (truth finding 8); the log line names the scope and the figure,
and the spec puts the INV-019 question to the maintainer instead of
stating the exception on the maintainer's behalf. The slot is taken after admission,
which departs from issue 951's request to take the process slot before
anything per key; the row, the spec and the gate now say so and what a
refused newcomer has spent by then, instead of citing PAT-003 for the
opposite order (truth finding 7, house finding 2).

Tests: the OAuth mode's wiring has a test of its own, which fails if
registerOAuthMCPHandlers stops passing the count (truth finding 6, house
finding 1). A held call leaves its entry evictable under POL-003, and a
test pins that and that the call outlives the eviction (truth finding
11). The e2e test starts the binary under ulimit -n 1024, so its
/health assertion can now fail (truth finding 12). The comment that
said a stateful GET was bounded is gone with holdsRequest (house finding
11). Lint: the gosec G115 conversion carries its reason, the header
copies use maps.Copy, the nil-carrier case is reached through a lookup,
and the e2e test is split into helpers to stay under gocognit's bound.

make check-tenancy: 83 rows, 21 failures and 299 declared sites over 202
packages (21 refusal returns, 80 refusals, 13 reasons and 31 settings
read); 0 findings.

Part of #951.
…nt mutants

The review of the held mode found the package short of the mutation gate
on lines the mode reaches (house finding 8), and the mode documented
only as proposed text for the integrator (house finding 13).

The survivors were equivalent mutants the code invited, so the code
stopped inviting them rather than the tests pretending to kill them. The
two scanners in rpc.go started from a 64 KiB buffer whose size changes
nothing but the first allocation, so they now start from none and grow
to the longest line. firstLine cut at a newline by hand and capped at
200 by hand, whose index >= 0 and len > 200 boundaries no body tells
apart; it now reads strings.Cut and min. withoutConfig sized a slice
with arithmetic no test can observe and appended AUTOPILOT last; it now
starts from AUTOPILOT. healthWait is time.Minute rather than
60*time.Second, and the held mode's four pauses come from heldWaits,
which a test holds to their figures before any test replaces them, so no
package-level initializer is left uncovered.

The stand-in speaks the ceiling's new words, "This server is busy. Retry
later.", so the ladder's refusals are recognised as the real server's.
docs/reference/resource-benchmark.md gains the held mode's section and
flag table, and make bench-held runs the ladder the ceiling was
measured with, under a hard descriptor limit of 1024, from
HELD_CREDENTIALS credentials.

Part of #951.
The pages this layer wrote described the first cut of HLD-011, and the
review of the layer found several of their claims false.

They sized 192 for a descriptor limit of 1024 called the default soft
limit on most Linux hosts and the smallest a container is commonly
given (truth finding 2, house finding 4). A Go process never runs under
that soft limit, since the runtime raises it to the hard one, so the
pages now say the ceiling follows the hard limit read at startup, give
the figures for 1024, 4096, 524288 and 1048576 and for Windows, and say
what the ceiling costs where the limit is small (one credential can
fill it in about fifteen seconds at the default rate) and where it is
large (memory runs out first, at about 190 KiB a held call).

They said without condition that a listen takes no slot while the code
counted an older revision's (truth finding 4, house finding 12); now
none is counted on any revision, and the pages say so and that each
call of a batch counts on its own while a response to an elicitation
never does. They said /health keeps answering under a limit of 1024 on
the strength of one credential while a hundred reached 1141 to 1280
descriptors (truth finding 5, house finding 5); the claim now rests on
a run from one credential and from a hundred under that limit, both at
394 descriptors with the refused connections closed, and the guide says
the ceiling bounds calls and not idle connections. The security page
said every call past the limit failed at its dial to GitLab, while 558
of the 1000 were served and 442 failed (house finding 6). And the
sizing pages, in rewriting the sentence on limits that are not
configurable, kept the failed-authentication budget among them, which
--auth-failure-limit has configured since issue 790 (truth finding 10,
house finding 9); the enterprise deployment guide said the same and is
corrected with it.

The HTTP server guide, security, resource consumption, troubleshooting,
the enterprise and remote deployment guides, the cli, env and
configuration references and the site pages in English and Spanish are
brought to the derived ceiling, the two refusal channels and the busy
words.

Part of #951.
Widening a table re-adds every row of it, so the rows that carried an
em dash before this layer widened three tables (the rejection codes in
the HTTP guide and the troubleshooting tables of the guide and the site)
now count as lines the branch adds, and make check-em-dash fails on
them. Each dash becomes a colon, a pair of parentheses, or "none" in the
headers column, and format_md_tables re-pads the tables.

Part of #951.
… POST

Mutating the held-call code showed two tests that hang instead of
failing, and gremlins files a hang as TIMED OUT, which is no kill: the
gate's condition with either && turned to ||, and the ceiling's sizing
with its division turned to a multiplication (a ceiling of one), all
three timed out.

The gate test that sends POSTs it must not count waited on a channel
the handler writes to; when a mutant refused the POST in the gate, no
handler ran and the receive waited for ever. ServeHTTP has returned by
then, so the receive is now immediate and a missing record is the
failure. The whole-chain tests registered the stand-in GitLab's release
before the MCP server's close, so the close ran first and waited for a
held call that only the release could end; the release is now
registered after the close and runs before it.

Part of #951.
The paragraph opened with "two further limits are not configurable", named
the two watcher ceilings, then called the configurable failed-authentication
budget "a third" and added the held-call ceiling as "another". By its own
count three limits were not configurable, and the sizing page counts the
watcher ceilings as one limit and the held calls as the second.

The two unconfigurable limits are now the watcher ceilings and the held
calls, each in its own paragraph, followed by the configurable budget and
its trap behind a proxy. The overlong line the earlier rewrite left inside
the 80-column paragraph is rewrapped with it.
The transport budget remembers a (source, key) pair for one window and the
limiter behind it counts a failure for one window, both inclusive of the
last instant. Two boundary mutants of that edge lived in auth_gate.go,
recorded by the cmd/server sweep as clock boundaries no test could
schedule: charge re-opening a pair at exactly one window (`>` to `>=`),
and the sweep dropping it at that instant. Either would charge the source
a second time for a client the limiter still holds against it.

testing/synctest can schedule that instant, since neither the budget nor
the limiter touches a socket or starts a goroutine: a charge, a sleep of
exactly one window, and a second charge (or a sweep and then a charge)
must leave a two-client budget unblocked. Both mutants now die.

The third boundary, window()'s `effectiveWindow <= 0`, stays recorded:
newTransportBudget never stores a window of zero or less, so the guard is
unobservable, and retiring that three-way fallback is issue 958's.
…package

The branch had been measured over the files it writes only; the whole
package still carried 24 lived, 67 not covered and 2 timed out mutants,
all older than the held mode. This closes or classifies every one.

Timed out: coldList's retry rule (series.go) had two mutants that ask
again after an answer they should take. The tests that admit against a
stand-in refusing tools/list then retried until callTimeout, five minutes
per warm-up, so the suite timed out instead of failing on the test that
pins the rule. The warm-up now has a ceiling of its own, warmUpTimeout,
callTimeout in a run and fifteen seconds under the tests, where the
stand-in answers at once; both mutants now fail the suite in under two
minutes.

Killed with a test:
- waitUntil at an instant that is exactly now, on a cancelled run, asked
  on a fake clock (testing/synctest): waiting there raced a fired timer
  against the cancellation.
- the latency stop's edge, moved into crossesLatencyCeiling so a tail
  exactly at the ceiling can be asked for; a measured tail cannot be.
- admit runs its loop over the outcome slots, so the count of clients
  started and the count read back are one number; the two mutants of
  want-have were equivalent while they were two.
- msLabel's band edges at one and ten milliseconds and cpuFigures on equal
  samples, two switch-case boundaries gremlins reported as not covered and
  no test pinned.

Removed rather than recorded: the +1 on linearXTicks' first candidate
(the spacing test already turns away the origin), the sampler's count of
answered processes (only ever read for zero, now a bool), and the initial
buffer of the goroutine-dump scanner (only the line ceiling matters).

The rest are the recorded shapes of docs/development/testing/README.md,
listed in the pull request.
Two tests failed on Windows. The validation cases for a held run that
measures set a descriptor limit, which validate refuses on Windows
because the shell that would set it is not there, so the host decided
the verdict; the cases now pin the platform through the runtimeGOOS
seam, and the refusal keeps the test of its own. The /health probe
test demanded a positive elapsed time from a loopback answer, which
can measure 0 where the monotonic clock advances in timer ticks; the
handler now waits past a tick before it answers.
The held run reaches the server it started, over plain HTTP on the
loopback interface, and wrote each of those addresses as a string
beginning "http://", which SonarCloud reads as clear-text traffic
(go:S5332) on every new line that does it. loopbackURL builds them
from a url.URL and says in one comment why plain HTTP is the right
scheme there, so the next mode of the benchmark reuses the reason
instead of repeating the literal.
The held-call flood opens six hundred loopback connections at once to
outrun a descriptor limit, and on windows-latest, which has none, only
444 of 600 were answered with 503 and the test could not say what the
rest got. Most likely the burst overflowed the listen queue, which
refuses a connection with WSAECONNREFUSED before the server accepts it.
flood now keeps every connection in flight at once where the limit is
what is measured and goes in waves of 64 on Windows, and a failure names
each ending that was not a 503.
@jmrplens
jmrplens force-pushed the issue-951b-held-requests branch from 8e73aaa to 7cbe64d Compare September 30, 2026 13:04
@sonarqubecloud

Copy link
Copy Markdown

@jmrplens
jmrplens merged commit 73a0bd7 into main Sep 30, 2026
6 checks passed
@jmrplens
jmrplens deleted the issue-951b-held-requests branch September 30, 2026 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tooling The audit and generator commands under cmd/, and the Makefile targets that run them transport stdio and HTTP transports, the server process, and the transport e2e modules v3.1.0 Targeted at the 3.1.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant