Skip to content

Bound the stateful sessions the process keeps, and reserve each one's stream when it opens - #1089

Merged
jmrplens merged 9 commits into
issue-951b-held-requestsfrom
issue-951c-stateful-sessions
Sep 30, 2026
Merged

jmrplens merged 9 commits into
issue-951b-held-requestsfrom
issue-951c-stateful-sessions

Conversation

@jmrplens

@jmrplens jmrplens commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

On --stateless=false the SDK keeps every session a client opens until the client deletes it, the pool evicts its credential or it has sat idle for --session-timeout (30 minutes by default), and initialize is metered to no bucket (RTC-004), so nothing bounded how many sessions a caller, or all of them, could open. Each session may also hold its standalone stream, a GET the SDK serves for the session's life. Under a hard descriptor limit of 1024 the process kept every session it was offered until those streams had taken every descriptor, at about a thousand sessions, and from there answered nothing, /health included. The register recorded this as finding F-31 on HLD-010 and IDN-010.

This is the third of issue 951's three layers. It makes HLD-010 a ceiling on the stateful sessions the process keeps, keyed on the process and set by no flag, and has each session reserve a held-call slot (HLD-011) for its standalone stream when it opens.

What it is sized from. Half of HLD-011's figure (SessionHeldDivisor, 2): 96 under a hard descriptor limit of 1024, 768 under 4096, 114560 under the 524288 a systemd service gets by default, 229248 under 1048576, and 96 on Windows, which is sized as 1024. The POST that opens a session takes the session's slot and one held-call slot for its stream together (mcpServerGate.takeSessionSlot), and the session keeps both until it ends, whether its stream is open or not. The sessions therefore hold at most half the held-call slots: the descriptor budget HLD-011 was derived from holds as it was sized, a call on an open session is still served when every session slot is taken, and a session the process keeps is never refused its stream. The standalone GET takes no slot of its own. The startup line announces the figure as stateful_sessions_per_process on a deployment that keeps sessions; on the default stateless transport nothing is counted and nothing is announced.

What it counts. Every POST carrying no Mcp-Session-Id on a revision before 2026-07-28, which is what the SDK opens a session for. The gate takes the slots once the credential is admitted and before the SDK creates anything, the same order HLD-011 records. The first request the SDK dispatches on the new session keeps them, found through the per-POST carrier, and a goroutine waiting on ServerSession.Wait gives them back when the session ends, however it ends (a DELETE, --session-timeout, the credential's eviction). A POST whose session did not survive it (not an initialize, or one the SDK refused) gives them back as the gate returns. One flag guards both ends, so the slots are released once. A POST on 2026-07-28 or later takes no slot: the stateful transport answers it, server/discover included, with 400, -32022 and the revisions it serves, and keeps no session for it, so the client falls back.

How it refuses. In the gate, before a session exists, with HLD-011's answer word for word: 503, JSON-RPC -50300, Retry-After: 30, the connection closed, and This server is busy. Retry later.. It costs the credential no rate-limit token and is charged to nothing. The log line request refused: too many stateful sessions across the process names scope=process and limit_stateful_sessions; a POST that finds a session slot and no held slot for its stream is refused in the same words and logs the held-call ceiling's line.

Why the stream is reserved at open. The first version of this layer counted the stream as a held call when its GET arrived and refused it at a full held ceiling. The go-sdk client never asks again for a standalone stream it was refused: it logs a warning and gives the stream up, and the server then drops every message meant for that session outside a response, resource updates and every list_changed among them, while the session carries on and its subscribe still reports success. TestStatefulSessions_AGoSDKClientKeepsItsStreamWithEveryHeldSlotTaken drives the SDK's own client with every held slot taken and holds that its stream is served, and that the next client's connect fails in the gate.

What it bounds, and what it costs. Descriptors, and memory only where the hard limit is small: an idle session costs four goroutines and 88 to 110 KiB of resident set, which the descriptor limit does not raise, so the 114560 sessions a hard limit of 524288 allows come to about ten to twelve GiB idle and the container's memory limit bounds them first. And filling the ceiling costs a caller nothing, since initialize spends no rate and an idle session holds no connection: one credential can open every session, and while it holds them every other tenant's initialize is refused for up to --session-timeout, half an hour by default and a day at most. Before this ceiling an idle session refused nobody. With --session-timeout=0 (END-005, which keeps its meaning) a session nobody deletes keeps its slots until the pool evicts its credential, after --pool-idle-timeout without a request or to make room at --max-http-clients, and startup now warns about that. Issue 951 decided that none of the three levers stands beside this row: no per-credential ceiling, no metering of initialize and no fixed cap beside the derived figure, for the reasons under Decided on issue 951 below.

heldRequestsFailure becomes processBusyFailure, the refusal both rows share, and heldRequests becomes processSlots, the count both ceilings use, whose acquire takes an optional hook between its read and its swap so a test drives the retry on purpose. The startup line's attributes move into httpStartupAttrs, which keeps serveHTTPOn under the maintainability threshold the new attribute pushed it past.

Measured

Truenas, the new sessions mode of cmd/bench_resources (make bench-sessions, first commit of this layer), against the stand-in GitLab, before this layer and at its tip. Per session over the idle process, read at 500 to 4000 sessions: before, an idle session held 3 goroutines, 10 to 17 KiB of live heap, no descriptor and 56 to 119 KiB of resident set, and one holding its stream 6 goroutines, 24 to 25 KiB, one descriptor and 95 to 124 KiB. After, from one credential and from a hundred, an idle session holds 4 goroutines (the fourth gives its slots back), 10 to 20 KiB and 88 to 110 KiB, and one holding its stream 7 goroutines, 25 to 28 KiB, one descriptor and 66 to 108 KiB.

Run Offered Open Live Refused Failed Descriptors /health
before, 1 credential, streams, hard 1024 2000 1013 0 0 0 1024 unanswered for 5 s
before, 1 credential, streams, hard 1024 4000 1012 1012 0 1973 1024 unanswered for 5 s
before, 1 credential, streams, inherited 4000 4000 4000 0 0 4013 200 in 0 ms
before, 100 credentials, idle, inherited 4000 4000 4000 0 0 811 200 in 0 ms
after, 1 credential, streams, hard 1024 4000 96 96 3904 0 112 200 in 1 ms
after, 100 credentials, streams, hard 1024 4000 96 96 3904 0 183 200 in 1 ms
after, 1 credential, idle, hard 1024 4000 96 96 3904 0 19 200 in 0 ms
after, 1 credential, streams, inherited 4000 4000 4000 0 0 4011 200 in 0 ms
after, 100 credentials, idle, inherited 4000 4000 4000 0 0 811 200 in 0 ms

"Open" is the bench's count of sessions open at the settled sample; "Live" is how many of them answered a ping, each waiting no longer than the /health probe, stopping at the first that got no answer. In the first row the process accepted no connection at the sample, so none answered though 1013 were open. "Inherited" is the host's container hard limit, 1048576. The SDK's session map and sessionOwners hold one entry per open session and drop it when the session ends (TestSessionOwners_ADisconnectedSession_IsForgotten), so both stop at the ceiling; that is inferred from the test and the ceiling, not sampled by the bench.

test/e2e/http holds the same behaviour against the real binary under ulimit -n 1024 with --stateless=false: 96 announced and 192 held calls, 96 sessions opened each with its stream, the 97th initialize refused in the gate with the connection closed and no session id, a call on an open session still served, 600 more initializes at once all refused while /health is probed and answers throughout, and a session's slots back once it is deleted.

Decided on issue 951

The seven questions this layer put are answered, each as proposed here, so nothing in this layer's row, pins, gate, e2e expectation or pages changes. The layer of this stack that records issue 951's decisions writes each answer and its reason where the tree still presented the question, and records the metering answer on RTC-004 itself.

  1. Half the held-call ceiling (SessionHeldDivisor 2, 96 under a hard limit of 1024), with the stream's slot reserved when the session opens, rather than a descriptor reservation of its own that would lower HLD-011's figure on a stateful deployment. The cost stays as stated: a session holds a held slot while its stream is closed, so calls have half the held slots on a deployment whose sessions are all open.
  2. No per-credential ceiling stands beside it, and initialize stays unmetered, for 951b's reason: the credential is a key a caller can mint (INV-003), so a per-credential number, or a price in the opener's own rate, multiplies with every token a caller mints and bounds nothing the process ceiling does not. One credential can therefore open every session at no rate cost and hold them for up to --session-timeout, and the operator pages say so, with what to do about it.
  3. No fixed cap stands beside the derived figure. At 524288 descriptors the figure admits about ten to twelve GiB of idle sessions, and the container's memory limit bounds that memory; a fixed cap no flag moves would be sized for one host. The pages say to set a memory limit on the container of a deployment that keeps sessions.
  4. The slots are taken after admission, the departure HLD-011 records from issue 951's request (PAT-003), accepted with its cost here: admission at the pool's bound can evict another credential's quiet entry, ending its sessions, before this row refuses the newcomer.
  5. INV-019. The refusal says what HLD-011's does, and with no per-caller ceiling beside it any refusal says the process is full, the one bit the exception accepts; HLD-010 stands under it beside HLD-011.
  6. One goroutine more per session, bounded by the ceiling, rather than riding on sessionOwners' waiter and tying the two rows' sites together, accepted as implemented.
  7. The issue is complete. With 951a (RTC-007), 951b (HLD-011) and this layer the register carries F-03 and F-31 on no row, and the answers to 2 and 3 here and to 951b's per-credential and memory questions add no row, so this layer closes The process has no ceiling of its own on held requests, stateful sessions or catalog listings #951.

Validation checklist (tenant-policy-spec.md)

  • VAL-001, question: Allow.
  • VAL-002, key: KeyProcess, which no caller can mint. No new key.
  • VAL-003, class: P. Its reason ("only a ceiling keyed on the process bounds the sessions it keeps") and the reason's unit are both the process.
  • VAL-004, ceiling or share: a ceiling. It promises no caller a share.
  • VAL-005, process partner: it is itself keyed on the process and protects the process's descriptors, and its memory only where the hard limit is small, the container's memory limit bounding the rest (decided on issue 951, item 3); no flag sets it, because an operator who could raise it could undo the bound (INV-004, INV-018). No per-caller row stands beside it, which issue 951 decided (item 2).
  • VAL-006, at capacity: refuses the newcomer; never evicts or ends a session. IDN-010 now states the same.
  • VAL-007, refusal: gate, legacy era only (a 2026-07-28 POST takes no slot): 503, -50300, fixed 30 s Retry-After, no challenge, connection closed; retry-later class, text This server is busy. Retry later.. On stdio: none.
  • VAL-008, charges: nothing. The refusal comes after admission and before anything metered.
  • VAL-009, value: Source: Derived, half of HLD-011's derived figure through the pinned SessionHeldDivisor (2), with a floor of one session; zero is not applicable; no flag or variable, so no malformed-value policy. --session-timeout=0 keeps its meaning under END-005, and startup warns what it costs here.
  • VAL-010, telemetry: the reason only. The log line carries the reason, the scope and the figure, nothing about the caller, and is throttled like every refusal a caller can cause at will. No metric counts the refusals.
  • VAL-011, stdio: not applied (StdioKey none). The gate counts only on a stateful HTTP deployment, and the middleware finds no slot on any other request.
  • VAL-012, measurement: the ceiling does not claim to protect one population from another; it protects the process's descriptors, and it is measured by the sessions mode above, which is what that claim needs; make bench-fairness answers a claim this ceiling does not make.

Gates

Measured on this layer before it was rebased onto main at 5da9699: the line numbers below are that tree's, and the one commit cited below (a0a03fe) is named as it sits in this stack. At its place in this stack on 50ef2cf, the layer built and vetted, the e2e gitlab, http and stdio packages compiled, the unit suites of cmd/server, internal/tenancy, internal/oauth and cmd/audit_tenancy passed, and make check-tenancy read the same 83 rows and 308 sites. The stack now sits on b44ab96, which adds #1083, #1084 and #1085; they change the upstream register, indirect module versions and the site's package pins, none of them a file this stack writes, and every commit of the stack applied unchanged.

  • go test -shuffle=on: 100.0% of statements in cmd/server, internal/tenancy, cmd/audit_tenancy and cmd/bench_resources; -race on the session, gate and held tests of cmd/server and on cmd/bench_resources; the test/e2e/http limit, held, session and health cases against the binary built at the tip, without the Docker-only proxy and balancer cases, which need a daemon this lane did not start; the e2e gitlab packages compile.
  • go vet and golangci-lint with the e2e and httpe2e tags: 0 issues. markdownlint, format_md_tables --check and check-em-dash: clean.
  • make check-tenancy: 83 rows, 21 failures and 308 declared sites over 202 packages (21 refusal returns, 81 refusals, 14 reasons and 31 settings read); 0 findings, 56 declarations exempted. check-dead-consts, check-test-file-names, check-test-subtests, check-test-goroutines, check-sdk-context and check-doc-tool-names pass. The godoc audit finds nothing this layer wrote; the five findings in the packages it touches predate it (three test comments in cmd/server/main_test.go, and cmd/bench_resources/fairness.go's package comment, twice).
  • gobco: internal/tenancy 408/408, cmd/bench_resources 1420/1420. cmd/server through a staged copy without its non-Linux twins (the issue 1017 approach): 2082/2124, every condition of sessions.go and held.go both ways, acquire's retry now taken on purpose through its hook. The 42 open are the ones 951b's run left, none on a line this layer wrote: auth_gate.go 3 (lines 728, 818, 916), bearer_guard.go 5, etag.go 2, first_run.go 1, health.go 1, jsonrpc_id.go 3, listen.go 2, main.go 11, pprof.go 1, probe.go 3, readiness.go 1, sdk_log.go 2, shape.go 2, socket_mode_unix.go 2, subscriptions.go 2, telemetry.go 1.
  • gremlins (--workers 2 --timeout-coefficient 20, INVERT_LOGICAL on):
    • internal/tenancy, whole package: 226 killed, 0 lived, 0 timed out, 66 not covered, every one arithmetic in a constant declaration, the same 66 as the base.
    • cmd/server, scoped to the four files this layer writes (sessions.go, auth_gate.go, held.go, main.go), at the review fix (a0a03fe, whose code the later commits leave as it is): 459 killed, 5 lived, 33 not covered, 0 timed out. Every mutant of sessions.go and held.go is killed. The 5 lived and 33 not covered are exactly 951b's on the same code, moved by the lines this layer inserted: auth_gate.go's unobservable effectiveWindow <= 0 (issue 958) and its 8 constant initializers, and main.go's four recorded boundaries and 25 initializers, switch cases and the equivalent 807:19.
    • cmd/bench_resources, the four files this layer writes (held.go, main.go, sessions.go, target.go), on a tree whose package differs from the tip only in how sessions.go builds its one constant handshake message: 253 killed, 0 lived, 0 not covered, 0 timed out. sessions.go again at the tip: 55 killed, 0 lived, 0 timed out, and 2 not covered, the two concatenations in that constant's initializer, which the compiler folds and no test can reach, the class of internal/tenancy's 66.

Decisions I took here

  • HLD-010 is half of HLD-011 (SessionHeldDivisor, 2, floor one), derived, keyed on the process and not configurable. Sessions get no descriptor reservation of their own, so 951b's derivation of HLD-011 is unchanged (decided on issue 951, item 1).
  • A session reserves a held-call slot for its standalone stream when it opens, rather than the GET taking one when it arrives, which changes what HLD-011 counts on a --stateless=false deployment and leaves calls at least half the held slots there when every session is open (decided on issue 951, item 1).
  • HLD-011's gate refusal is no longer limited to the 2026-07-28 era, since a legacy POST that would open a session is refused there when no held slot is left for its stream.
  • The session slot is taken after admission and before the SDK creates the session, the departure 951b records (decided on issue 951, item 4).
  • The refusal is HLD-011's gate failure word for word, the log line alone naming the scope and the figure, and HLD-010 is listed under INV-019's one-bit exception (decided on issue 951, item 5).
  • No per-credential session ceiling and no metering of initialize were added, and no fixed cap stands beside the derived figure (decided on issue 951, items 2 and 3).
  • Each session's slots are given back by a goroutine of its own waiting on ServerSession.Wait, bounded by the ceiling, rather than by sessionOwners' waiter (decided on issue 951, item 6).
  • F-31 is treated as answered: it leaves HLD-010 and IDN-010, joins the answered set in the specification and the tests, and IDN-010 refuses the newcomer at capacity, decided in issue 951.
  • The startup attributes of serveHTTPOn moved into httpStartupAttrs so the new attribute does not push it past the maintainability threshold; behaviour is unchanged.
  • This layer closes The process has no ceiling of its own on held requests, stateful sessions or catalog listings #951: with 951a and 951b it bounds everything the issue named, and every question its layers put is answered without a new row (item 7).

Not done

  • The SDK's session map and sessionOwners were not sampled by the bench; that both stop at the ceiling is inferred from TestSessionOwners_ADisconnectedSession_IsForgotten and the ceiling.
  • make bench-fairness was not run: the ceiling protects the process's descriptors rather than one population from another, so the comparison of two populations it runs answers nothing this row claims.
  • The Docker-only proxy and balancer cases of test/e2e/http did not run, and the Docker GitLab suites were compiled but not run.
  • cmd/server was mutated over the four files this layer writes, not whole.
  • The godoc findings in the packages this layer touches predate it and are left as they are.

Closes #951.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @jmrplens, your pull request is larger than the review limit of 150,000 diff characters

@coderabbitai

coderabbitai Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: jmrplens/gitlab-mcp-server/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 04349c66-55cf-4a68-8139-931ff3a12498

📥 Commits

Reviewing files that changed from the base of the PR and between 7cbe64d and fb0a1d1.

📒 Files selected for processing (50)
  • Makefile
  • cmd/audit_tenancy/main_test.go
  • cmd/bench_resources/held.go
  • cmd/bench_resources/main.go
  • cmd/bench_resources/main_test.go
  • cmd/bench_resources/sessions.go
  • cmd/bench_resources/sessions_test.go
  • cmd/bench_resources/target.go
  • cmd/bench_resources/target_test.go
  • cmd/bench_resources/testdata/standin/main.go
  • cmd/server/auth_gate.go
  • cmd/server/held.go
  • cmd/server/held_test.go
  • cmd/server/main.go
  • cmd/server/main_test.go
  • cmd/server/sessions.go
  • cmd/server/sessions_test.go
  • docs/concepts/security.md
  • docs/development/adr/adr-0023-tenant-policy-is-declared-once.md
  • docs/development/tenant-policy-spec.md
  • docs/guides/enterprise-deployment.md
  • docs/guides/http-server-mode.md
  • docs/guides/remote-deployment.md
  • docs/guides/troubleshooting.md
  • docs/reference/cli.md
  • docs/reference/configuration.md
  • docs/reference/env.md
  • docs/reference/resource-benchmark.md
  • docs/reference/resource-consumption.md
  • internal/tenancy/decision.go
  • internal/tenancy/decisions_allow.go
  • internal/tenancy/decisions_end.go
  • internal/tenancy/decisions_identify.go
  • internal/tenancy/decisions_test.go
  • internal/tenancy/values.go
  • internal/tenancy/values_test.go
  • site/src/content/docs/configuration.mdx
  • site/src/content/docs/es/configuration.mdx
  • site/src/content/docs/es/operations/http-server.mdx
  • site/src/content/docs/es/operations/remote-deployment.mdx
  • site/src/content/docs/es/operations/security.mdx
  • site/src/content/docs/es/operations/troubleshooting.mdx
  • site/src/content/docs/es/performance/sizing.mdx
  • site/src/content/docs/operations/http-server.mdx
  • site/src/content/docs/operations/remote-deployment.mdx
  • site/src/content/docs/operations/security.mdx
  • site/src/content/docs/operations/troubleshooting.mdx
  • site/src/content/docs/performance/sizing.mdx
  • test/e2e/http/held_test.go
  • test/e2e/http/sessions_test.go

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added transport stdio and HTTP transports, the server process, and the transport e2e modules tooling The audit and generator commands under cmd/, and the Makefile targets that run them labels Sep 30, 2026
@jmrplens
jmrplens added this pull request to stack #1095 September 30, 2026 10:18
@jmrplens jmrplens added the v3.1.0 Targeted at the 3.1.0 release label Sep 30, 2026
@jmrplens jmrplens added this to the 3.1.0 milestone Sep 30, 2026
@jmrplens jmrplens changed the title issue 951c stateful sessions Bound the stateful sessions the process keeps, and reserve each one's stream when it opens Sep 30, 2026
@jmrplens
jmrplens force-pushed the issue-951c-stateful-sessions branch from 2d12f9a to 695822e Compare September 30, 2026 12:05
The held mode measures the requests the server holds open; nothing
measured what the --stateless=false transport keeps between them, the
session. The SDK keeps every session a client opens, with the goroutines
serving it, until the client deletes it, the pool evicts its credential
or it has sat idle for --session-timeout, and each session may open one
standalone stream, a GET held open for the session's life. Issue 951
needs both figures to size a ceiling on sessions.

-sessions <counts> starts the server with --stateless=false and opens a
ladder of sessions, each an initialize on protocol 2025-11-25 followed by
notifications/initialized, spread round robin across
-sessions-credentials; with -sessions-stream each also opens its
standalone stream and keeps it open. Before the first step every
credential opens a session, lists its tools and deletes it, so the pool
entry and its catalog are built before the idle process is sampled and
are not priced as part of the sessions. A step is settled and sampled
the way a held step is, then every session the server gave an id is
pinged, each ping waiting no longer than the /health probe, and the
count of answers is the number of sessions the server still holds, seen
from outside. The count stops at the first ping nobody answered, since a
process out of descriptors accepts no more connections and asking a
thousand sessions in turn would hold the step for as long as they all
took to time out. Every open is filed as opened, refused (a 503) or
failed, one session is priced over the idle process, and every stream is
closed and every session deleted before the next step.

-sessions-nofile starts the server under a descriptor limit as
-held-nofile does, and the mode writes bench/sessions.json, uncommitted,
draws nothing, refuses a -sessions-json that names the published record
and cannot be combined with -held or -fairness. make bench-sessions runs
the ladder under a limit of 1024 with every stream open.

The held and the sessions modes price what they hold with one function,
costPer, so a figure means the same thing in both documents. The
stand-in learns the stateful transport the tests drive: --stateless=false
gives out session ids, answers a request on an id it does not hold with
404, holds a standalone stream open until the session ends, deletes a
session on DELETE, and refuses an initialize past STANDIN_SESSION_LIMIT
with the 503 the real ceiling answers with.

A refused standalone stream's body is quoted, and read for no longer
than the /health probe waits: a refusal is a short body, but nothing
promises that one ends, and a response left open would otherwise hold
the step. A test holds a 503 whose body stays open to that bound.

Part of #951.
On --stateless=false the SDK keeps every session a client opens, with
the three goroutines serving it and its owner record, until the client
deletes it, the pool evicts its credential or it sits idle for
--session-timeout, half an hour by default. initialize is metered to no
bucket (RTC-004), so nothing bounded how many a caller opened, and each
session may also hold its standalone stream, a GET the SDK serves for as
long as the session lives. The register recorded that as finding F-31
on HLD-010 and IDN-010.

Measured on 951b's tip with the sessions ladder of cmd/bench_resources:
an idle session costs 3 goroutines, 10 to 17 KiB of heap and no
descriptor, and one holding its stream 6 goroutines, 24 to 25 KiB and
one descriptor. Under a hard descriptor limit of 1024 the base server
kept every session it was offered until the streams had taken all 1024
descriptors, at 1012 to 1016 sessions; from there /health went
unanswered and, in two runs, between 985 and 2984 opens of a step failed
at the connection.

The gate now takes a session slot for every POST that would open a
session, a POST carrying no Mcp-Session-Id on a stateful deployment,
once the credential is admitted and before the SDK creates anything,
and refuses it with the held-call ceiling's 503 when none is free. The
first request the SDK dispatches on the new session keeps the slot, and
a goroutine waiting on the session gives it back when the session ends,
however it ends; a POST whose session did not survive it (no initialize,
a refused one) gives it back as the gate returns. The two ends take one
flag, so a slot is released once.

The ceiling is half the held-call ceiling, 96 under a limit of 1024,
rather than a number of its own, and the gate now counts a session's
standalone GET as a held request: every stream the process keeps then
sits inside the descriptor budget HLD-011 was sized from, and the
streams of every session can take at most half of its slots, so a call
on a session that is open is still served with every session slot
taken. No flag moves it, for HLD-011's reason: it follows the descriptor
limit, the one lever that also raises what it protects.

The refusal is word for word the held ceiling's, since the next action
is the same and the one bit INV-019 accepts is that the process is full;
the log line names the scope and the figure. heldRequestsFailure becomes
processBusyFailure, the refusal both rows share, and the startup line
announces stateful_sessions_per_process beside held_requests_per_process
from a helper of its own, which keeps serveHTTPOn under the
maintainability threshold.

HLD-010 becomes a valued ceiling on the process with a declared site for
every place it is enforced, SessionHeldDivisor is pinned at 2, IDN-010
refuses the newcomer at capacity instead of carrying F-31, and F-31 joins
the findings the register answers. test/e2e/http pins the whole of it on
the wire under a limit of 1024.

Part of #951.
HLD-010 bounds the sessions a --stateless=false deployment keeps across
every credential, and nothing in the pages said a stateful deployment
had any such bound: the guides described --session-timeout as the only
thing that ends a session, and the flag rows for --max-http-clients
said the pool bounds entries and not sessions, which was true and left
the sessions bounded by nothing.

The HTTP guide and the site's HTTP page gain a section of their own
beside the held calls: what a session costs, measured through the
sessions mode (three goroutines and 10 to 17 KiB idle, a descriptor
and three goroutines more with its standalone stream), what the base
process did under a hard limit of 1024 (kept every session until the
streams had taken all 1024 descriptors, then answered nothing), the
ceiling as half the held-call one (96 under 1024, 114560 under
524288), what counts, the refusal, and what it costs a deployment. The
held-call sections now say that a standalone stream is counted there.

The rejection table, the troubleshooting tables (guide and site, EN
and ES), the startup line example, the flag rows for
--max-http-clients, --session-timeout and --stateless, the security
pages, the sizing and resource-consumption pages, the enterprise and
remote deployment guides say the same in their own places. The
specification's INV-019 lists HLD-010 among the process bounds whose
refusal discloses one bit, the answered-findings paragraph records
F-31 as answered by HLD-011 and HLD-010 together, and ADR-0023 NEG-004
and NEG-007 name the sessions mode and HLD-010 beside HLD-011.

Part of #951.
The first version of the session ceiling counted a session's standalone
stream as a held call when its GET arrived, and refused that GET with
the held ceiling's 503 when every held slot was taken. The go-sdk
client does not ask again for a standalone stream it is refused: it
logs a warning on the JSON body and gives the stream up, and the server
then drops every message meant for that session outside a response, a
resource update and every list_changed among them, while the session
carries on and its subscribe still reports success. Calls could take
every held slot, so a kept session was never guaranteed its stream.

The POST that opens a session now takes the stream's held slot together
with the session's, and the session keeps both until it ends, whether
the stream is open or not; the GET takes none. With the sessions at
half of the held slots, calls keep at least the other half, and a
session the process keeps is never refused its stream. A POST that
finds no held slot left is refused in the gate instead, with the held
ceiling's 503 and log line, which a client sees as a failed connect.
TestStatefulSessions_AGoSDKClientKeepsItsStreamWithEveryHeldSlotTaken
drives the SDK's own client through both halves.

A POST on protocol 2026-07-28 or later no longer takes a session slot.
The SDK answers it, server/discover included, with the revisions the
stateful transport serves and closes its session with the POST, so
counting it spent a slot no session could keep and, at a full ceiling,
replaced the answer that tells a client to fall back with one that
tells it to retry. HLD-010's refusal is then of the legacy era alone,
as the register already said.

--session-timeout=0 leaves the SDK no idle timer, so a session nobody
deletes holds its slot until the pool evicts its credential, which is
never while that credential stays active or with
--pool-idle-timeout=0. Zero keeps its meaning (END-005), and startup
now says what it costs on a stateful deployment. The startup line
announces stateful_sessions_per_process only where sessions are kept,
and the curated help names the session ceiling beside the held-call
figure and on -session-timeout.

The count's type is processSlots now, since it counts the sessions as
well as the held calls, and acquire runs an optional hook between its
read and its swap so a test takes the retry on purpose instead of by
the scheduler's luck.

The register says what the ceiling costs and bounds: one credential's
initializes, which spend no rate, can take every slot for up to
--session-timeout, which is put to the maintainer with the
per-credential question; the derived figure bounds descriptors, and
memory only where the limit is small (56 to 110 KiB of resident set per
idle session, six to twelve GiB at the 114560 sessions a hard limit of
524288 gives); admission at the pool's bound can evict another
credential's quiet entry and end its sessions before this row refuses
the newcomer; and IDN-010's bound lives only in its comment.

test/e2e/http probes /health while the six hundred extra initializes
are in flight rather than after they return.

Part of #951.
The pages described the first version of the session ceiling: a
standalone stream counted as a held call when its GET arrived, and a
full held ceiling refused it. The server now reserves the stream's
held slot on the POST that opens the session and keeps both slots
until the session ends, so the held-call sections say that instead, and
the session sections say why: a client refused its stream does not ask
for it again.

The session sections (HTTP guide, site EN and ES) also carry what the
review found the first version left out. An idle session costs four
goroutines once the ceiling exists, the fourth giving its slots back,
and 56 to 110 KiB of resident set. A POST on protocol 2026-07-28 or
later takes no slot, since the stateful transport answers it with the
revisions it serves. The ceiling bounds descriptors, and memory only
where the hard limit is small: the 114560 sessions a limit of 524288
allows come to six to twelve GiB idle, so the container's memory limit
bounds them first, and whether a fixed cap should stand beside the
derived figure is an open question. Filling the ceiling costs a caller
nothing, since initialize spends no rate-limit token and an idle
session holds no connection, and a session nobody deletes keeps its
slot for up to --session-timeout, or with 0 until the pool evicts its
credential, which startup now warns about.

The rejection table, the startup line example (which carries the
session figure only where sessions are kept), the troubleshooting rows,
the --session-timeout flag rows in the CLI, configuration and
environment references, the sizing, security and resource-consumption
pages, the enterprise deployment guide and the specification's HLD-010
and HLD-011 paragraphs say the same in their own places.

Part of #951.
The resident set this layer stated for an idle session, 56 to 110 KiB,
joined the lowest sample of the build before the ceiling to the highest
of the build after it, and left out the highest before it (119 KiB at
500 sessions under a hard limit of 1024). Each figure now describes one
build, read at 500 to 4000 sessions, where a per-session figure is not
dominated by what a few sessions share.

Before the ceiling an idle session held 56 to 119 KiB of resident set,
which is what the pages that describe that build now say. With it, and
its fourth goroutine, an idle session holds 10 to 20 KiB of live heap
and 88 to 110 KiB of resident set, from one credential and from a
hundred, so the 114560 sessions a hard limit of 524288 allows come to
about ten to twelve GiB idle rather than six to twelve. The server's
comment and the register's HLD-010 comment say the same.

Part of #951.
With --session-timeout=0 a session nobody deletes keeps its slot until
the pool evicts its credential's entry. The comments and pages said
that happens never while the credential stays active or with
--pool-idle-timeout=0, and the second half is not what the pool does:
with the idle sweep off, the size bound still evicts the least recently
used entry to make room at --max-http-clients.

They now name both ways an entry leaves: after --pool-idle-timeout
without a request (never with 0), or to make room at
--max-http-clients. The warning's own words were already right, since
they say only that the slot is kept until the entry is evicted.

Part of #951.
The sessions run built its two addresses on the server it started as
"http://" strings, the shape SonarCloud reads as clear-text traffic
(go:S5332). It now takes them from loopbackURL, which the held run
added for the server it starts and which carries the reason plain HTTP
is right there.
The session flood had its own copy of the held-call flood and the same
failure on windows-latest, 475 of 600 initializes answered with 503. It
now goes through flood, in waves on Windows and all at once elsewhere,
and a failure names each ending that was not a 503.
@jmrplens
jmrplens force-pushed the issue-951c-stateful-sessions branch from 695822e to fb0a1d1 Compare September 30, 2026 13:04
@sonarqubecloud

Copy link
Copy Markdown

@jmrplens
jmrplens merged commit b174740 into main Sep 30, 2026
6 checks passed
@jmrplens
jmrplens deleted the issue-951c-stateful-sessions branch September 30, 2026 15:13
jmrplens added a commit that referenced this pull request Oct 6, 2026
…es only Windows and macOS build (#1206)

#1064 made the condition gate able to read `cmd/server` at all, by
measuring it in a staged copy of the module without the files Linux does
not build, and #1017 stayed open for what that left. Two things: the
conditions and mutants of `cmd/server` that no test decided, and the
files a Linux run leaves out, which nothing measured anywhere. This
closes both.

## cmd/server on Linux

At e19b579 gobco measured 2229 of 2270 conditions of `cmd/server` both
ways. The 41 left are the list #1089 carried, less one closed since.
Every one is answered here: the package now measures 2236 of 2236, the
total lower because some conditions were removed rather than tested. The
whole package was then put through gremlins, which had only ever run on
the files a change touched. That run found 10 lived mutants and 1 timed
out, all answered below. Its not-covered mutants sit where gremlins has
no statement counter to read, and every one of them that compiles was
applied by hand and fails the suite (see Measured).

Most of the answer is tests, each for a case nothing exercised: a
shorter block checked after a longer one (`longestAuthBlock`), the
guard's two refusals with no rejected-token cache, a challenge with no
scope or no metadata URL, a capture of a handler that wrote nothing, a
pseudo-version whose patch overflows an int, request ids a decoder never
hands over (an unterminated string, `+1`, `4-2`), both credentials on a
terminal, a public URL that does not parse, an instance list entry with
no host, the singular `gitlabURL` field, a prebuilt catalog on the
dynamic and meta surfaces, a second `WriteHeader` and a second stop of
the SSE heartbeat, a card preflight that names no header, a lone `-`
among a peer's arguments, a probe target that does not parse, a peer
that exited between being listed and having its command line read,
`WithGroup("")`, `forget` among two shapes, a shape registry with no
registration hook, a staging descriptor that names something other than
a directory, closing a socket whose path is already gone, a listen
carrying prompts or resource list-changed beside its resources, and the
telemetry tool-name flag not registered at all. The Cygwin half of the
terminal check is false on every platform but Windows, so it is reached
through a seam, `isCygwinTerminal`, as `osExecutable` already is.

Some conditions could never take their other value, and those are code
changes rather than tests:

- `transportBudget.window` tested `effectiveWindow <= 0`, which
`newTransportBudget` has already resolved to the default, so a budget
that exists answers its own window. That is the lived mutant #1017
named, and the window now falls back in two places rather than the three
`AUB-002` described, so its comment in `internal/tenancy` says two (#958
owns the rest of that).
- `requireExplicitInstance` tested for a nil request, which an HTTP
handler is never given; `securityHeadersMiddleware` tested for a nil
body, which net/http never hands a server; `newReadinessGate` tested for
a nil context, which no caller passes; `listenUnix` tested
`filepath.Dir` for an empty string, which it never returns; and both
`Serve` loops tested their error for nil, which `Serve` never returns.
- `isUnixSocketAddr` asked about the native separator and then about
`/`, so on every platform but Windows the second question was reached
only when the first had said no to the same byte. One
`strings.ContainsAny` asks both.
- `sdkLogHandler.WithGroup("")` now returns the receiver, as slog's
contract for the method says. Passing the empty name on was not
harmless: the JSON handler opens a group with no name for it, so the
attributes after it were nested under `""` while this handler, seeing no
group, still renamed them. The Logger never forwards an empty name, so
nothing served changes.

The rest are mutants that no input can tell from the original, which is
the code saying it carries something it does not need: a capacity hint
in `publicPaths`, a `servers` counter in `runProbe` that only ever meant
"a failure was recorded", an `n > 0` guard before a scan that answers
false for no bytes, a `len(commit) > 7` before a cut that `min` makes,
an `IndexByte` boundary that `strings.Cut` replaces, a deadline
comparison `context.WithTimeout` already makes, a `sort.Slice` `<` that
`cmp.Compare` replaces, the two branches of `metricHostsFor` that
answered the environment's instance the same way, and the check that
skips the catalog text rewriter when there is nothing to substitute,
which now lives in `catalogTextRewriter` with a test of its own. The SSE
heartbeat's idleness boundary took a parameter instead: `writeKeepAlive`
is handed the tick it answers, so a test holds the exact interval, which
a call to `time.Since` cannot. The readiness gate's exemption for
unmarked connections timed out rather than dying, because its mutant
deadlocks registration and the first test to register waited for it
without a bound; that helper now waits at most
`testHTTPLivenessTimeout`, and the mutant is killed in 37 seconds.

Eleven tagless switches had a case expression gremlins reports as not
covered, since a case expression carries no statement counter of its
own. They are rewritten as the chains of returns or ifs the repository
already uses for that reason (`isRequestID`, `validateTLSFiles`,
`metricHostsFor`, `prepareStdioCatalog` twice, `shutdownHTTPServer`,
`parseProbeTarget`, `sanitizedInput.Read` and `readLine`,
`logKeyringChoice`, `inferTransport`). `sanitizedInput.Read` gave its
line handling to an `accept` method on the way, which golangci-lint's
nesting rule asked for.

## The Windows and macOS halves, in CI

The cross-platform job's Windows and macOS legs now run
`scripts/coverage-conditions.sh` on `cmd/server` and `internal/toolutil`
through bash, not make, with `GOBCO_GATE=beyond:linux/amd64`. **The step
gates.** It fails the leg on a condition not evaluated both ways in a
file that leg builds and linux/amd64 does not, and prints, counts and
passes one in a file Linux builds too. Holding the shared files there as
well would fail a leg over a test the platform skips, for a condition
the Linux run decides both ways: measured on Windows, that is 20
conditions of `cmd/server` and 12 of `internal/toolutil`, every one of
them both ways on Linux. That Linux run is the local one,
`GOBCO_GATE=all make coverage-conditions`, which every touched package
passes before it is committed. No CI step runs gobco on Linux, before
this change or after it, so the shared files are held by that run and
not by CI; the step's comment and the testing README say so.

The script gained what the step needs. `GOBCO_GATE` makes the report a
gate: unset it passes whatever it found, as before; `all` fails on any
condition gobco names as never true, never false or never evaluated,
which is the gate a touched package is held to locally;
`beyond:GOOS/GOARCH` holds only the files the named platform does not
build, asked of the go command under that platform and the same tags. A
value it cannot read is refused before gobco runs. `GOBCO_TEST_FLAGS`
hands gobco's `go test` flags of its own, here `-timeout=30m`, since
gobco runs it with go test's ten-minute default and `cmd/server` is
about three times slower on Windows.

Windows builds four sources of `cmd/server` Linux does not
(`conn_refused_windows.go`, `socket_mode_windows.go`, `probe_other.go`,
`descriptor_limit_other.go`) and two of `internal/toolutil`
(`download_output_other.go`, `file_utils_other.go`); macOS builds only
`probe_other.go` beyond Linux. Two of them carry conditions. The
connection-refused check's two operands were already decided by its own
test. The Windows listen's failure was decided by nothing, so
`socket_mode_windows_test.go` now binds under a directory that does not
exist, and binds where it can. Running golangci-lint over the package
with `GOOS=windows`, which nothing in CI does, reported two findings in
those files, both corrected here: Winsock's refusal constant is renamed
`errWSAEConnRefused`, as errname asks of a sentinel error, and a
misspelling in `socket_mode_windows.go`'s file comment is fixed.

## Measured

On the Windows Server 2025 test VM (Go 1.27.1, PortableGit 2.55.0's bash
5.3, 4 cores), the step as CI runs it: `cmd/server` 2180/2206, every
condition of the four Windows-only files both ways, 20 shared conditions
left to Linux; `internal/toolutil` 3836/3848, both Windows-only files
both ways, 12 left to Linux; exit 0, 15 minutes for the two. The leg
took 20 minutes before, so it stays well inside its 60. That run was on
the tree before the last mutant fixes, which changed only files Linux
builds too, before the rename above, which changes no condition, and
before this branch moved onto the two layers below it in this stack,
whose only change to these two packages is `internal/toolutil/errors.go`
(the bound on a reflected message and Workhorse's Orbit query refusal),
a file Linux builds too. That file adds 20 conditions to
`internal/toolutil` which the Windows figure predates, all of them in
the shared files the step does not gate; what it gates on Windows, the
six files Linux does not build, is the same at the head.

On linux/amd64 at the head: `cmd/server` 2236/2236 and
`internal/toolutil` 3870/3870 with `GOBCO_GATE=all`, and
`internal/tenancy`, whose comments changed, 444/444. The register row
and the testing README carry the 3870.

gremlins over the whole of `cmd/server`, `--workers 2`, in three groups
of files run side by side on the same tree (main.go with the flag files;
the gate, guard, probe, subscription, stdio and session-owner files; the
rest): Killed 1249, Lived 10, Not covered 47, Timed out 1, of 1307
mutants. Every file a fix then changed was run again on the final tree:
`main.go` Killed 354, Lived 0, Not covered 16; `probe.go` 73, 0, 2;
`health.go` and `readiness.go` 50, 0, 0; and `stdio.go`, whose group had
started before its line handling moved into `accept`, 58, 0, 0. None
timed out. A gremlins dry run on the head lists the same 47 not-covered
mutants (1242 runnable), every one of them in a place that has no
statement counter of its own: a constant or package-level variable
initializer, or the two Windows files, which gremlins mutates although
Linux never compiles them. Not covered is not the same as asserted, so I
applied the initializer mutants by hand and ran the whole `cmd/server`
suite on each. The 23 that turn a `+` between strings into a `-` do not
compile. The other 22, in 19 distinct forms (a timeout's `*` turned into
`/`, a refusal code losing its sign, `maxUnixPathLen`'s `- 1` becoming
`+ 1`), each fail the suite: the HTTP server timeouts through
`TestHTTPServerTimeouts_AreDeadlinesRatherThanTheAbsenceOfOne`, all
three forms of `idleTimeoutDisabled` through
`TestIdleTimeoutDisabled_OutlastsEveryOtherDeadline`, the two refusal
codes through
`TestRefusalCodes_MirrorTheirStatusAndSitWhereTheSpecificationAllows`,
the socket path limit through
`TestMaxUnixPathLen_IsTheLongestPathAnAddressCanCarry`,
`stdioRereadTimeout` through
`TestRefreshStdioAuthority_FollowsTheInstanceUntilTheContextEnds`,
`httpShutdownTimeout` through four tests,
`TestHTTPShutdownTimeout_SitsInsideTheBracketItsCommentArgues` among
them, the probe's timeout and budget and the telemetry, startup-drain,
stale-socket, pprof and shutdown-grace timeouts each through tests of
their own, and `sseKeepAliveInterval` at zero by panicking
`time.NewTicker`.

The script's tests (26, among them nine new for the gate, the test flags
and the CI step) pass under bash 5 and under bash 3.2.57, the version
macOS ships as `/bin/bash`, in a `bash:3.2` container.

## Gates

This is the third layer of a stack: it sits on the Orbit query layer for
#1031, which sits on the layer raising the bound on GitLab's reflected
message, which sits on main at fbf81ff. Since the base this branch was
first measured on, 862e393, nothing below it has changed `cmd/server`,
and what main had gained before that, since the base the mutation runs
measured, changes only string literals there (help text, a flag's usage,
the first-run guidance), which no mutator touches, so the `cmd/server`
mutation figures carry. Below it, #1199 changed IDN-013's text and added
one of its declared sites in `internal/tenancy`, and the two layers
changed `internal/toolutil/errors.go`; those two packages' figures were
measured again. Everything below ran on the head of this branch; the one
conflict of the rebase was `docs/development/testing/testing.md`, which
was regenerated rather than merged.

- `go build ./...`, and `go test -count=1 -cover`: `cmd/server`,
`internal/tenancy` and `internal/toolutil` 100.0% of statements.
- `GOBCO_GATE=all make coverage-conditions`: `cmd/server` 2236/2236,
`internal/toolutil` 3870/3870, `internal/tenancy` 444/444. The
`internal/toolutil` figure was taken again after the layer below began
refusing a null message in Workhorse's query refusal, which added its
four conditions.
- `make coverage-mutants PKG=./internal/tenancy`: Killed 239, Lived 0,
Not covered 67 (every one an `iota` or constant declaration), Timed out
0. `cmd/server` as above.
- `golangci-lint run --build-tags e2e` on `./cmd/server/...` and
`./internal/tenancy/...`: clean, and clean on `cmd/server` under
`GOOS=windows` and `GOOS=darwin`.
- `go vet` of `cmd/server` and `internal/toolutil` for windows and
darwin on amd64 and arm64: clean.
- `make check-test-subtests`, `check-test-goroutines`,
`check-test-file-names`, `check-action-ids`, `check-md-escaping`,
`check-doc-tool-names`, `check-surface-quality`, `check-tenancy`,
`check-sdk-context`, `check-dead-consts`, `check-e2e-static`,
`check-orbit-record`, `check-gateway-chars`, `audit-catalog-first`,
`check-action-grants-derivation`, `audit-1to1`, `audit-1to1-sdk`,
`audit-1to1-paths` and `audit-1to1-grants`: all pass.
- `check-testing-docs`, `check-llms`, `check-tool-reference`,
`check-lhm-manifest`, `check-footprint`, `check-site-stats` and
`check-action-grants`: all pass.
- `go run ./cmd/format_md_tables/ --check`, `node
scripts/check-doc-links.mjs`, markdownlint-cli2 on the changed Markdown,
`make check-em-dash`: pass.
- `python3 -m unittest scripts/coverage_conditions_sh_test.py`: 26 tests
pass under bash 5.2. The run under bash 3.2.57 in a `bash:3.2` container
was before the rebase, which changed neither the script nor its tests.
- actionlint v1.7.7 on `ci.yml`: no findings. That run had no shellcheck
on its path, so shellcheck 0.9.0 was run on its own over the step's
`run` block and over `scripts/coverage-conditions.sh`: no findings.
Before the rebase, actionlint v1.7.7 with shellcheck 0.10.0 had found
none either.

## Decisions I took here

- The cross-platform step gates rather than reports, and gates only the
files Linux does not build, for the reason above. The shared files stay
with the local `GOBCO_GATE=all` run on Linux that a touched package
passes before it is committed; this change adds no Linux gobco step to
CI.
- The step runs on Windows and macOS only. Linux builds none of the
files it would hold.
- A Go runtime crash of #467 fails this step the way it fails the unit
suite's first run, and the answer is the same: rerun the failed job. The
step does not reproduce the crash-aware runner's retry.
- Conditions no input can reach were removed rather than tested through
their impossible state, and equivalent mutants were answered by
rewriting the code they sat in rather than by declaring them.
- The two stale test comments the godoc audit reported in `main_test.go`
were corrected on the way, since the package was being held to its gates
anyway.

## Not done

- The macOS leg was not run: no macOS machine was available. What it
depends on was checked instead: the script's tests under bash 3.2.57,
and its tools being the ones a macOS runner has (BSD awk, tar, mktemp
and tr behave the way the tests exercise them). macOS builds only
`probe_other.go` beyond Linux, which carries no condition, so the leg
holds nothing today and is there for the next file only it builds.
- gremlins does not honor build constraints either, so the two Windows
files' mutants are reported not covered on Linux and measured nowhere. A
gremlins run on Windows would answer them, and that is not this change.
- The e2e harness and retiring the staged copy wait on gobco honoring
build tags, register row 73, and stay with #1099.

Closes #1017.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tooling The audit and generator commands under cmd/, and the Makefile targets that run them transport stdio and HTTP transports, the server process, and the transport e2e modules v3.1.0 Targeted at the 3.1.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The process has no ceiling of its own on held requests, stateful sessions or catalog listings

1 participant