Repository navigation
Bound the stateful sessions the process keeps, and reserve each one's stream when it opens - #1089
Conversation
There was a problem hiding this comment.
Sorry @jmrplens, your pull request is larger than the review limit of 150,000 diff characters
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 59 minutes. View limit detailsLimit details: You’ve used the included review currently available. Review configuration: ⚙️ Run configurationConfiguration used: Repository: jmrplens/gitlab-mcp-server/.coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (50)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
2d12f9a to
695822e
Compare
The held mode measures the requests the server holds open; nothing measured what the --stateless=false transport keeps between them, the session. The SDK keeps every session a client opens, with the goroutines serving it, until the client deletes it, the pool evicts its credential or it has sat idle for --session-timeout, and each session may open one standalone stream, a GET held open for the session's life. Issue 951 needs both figures to size a ceiling on sessions. -sessions <counts> starts the server with --stateless=false and opens a ladder of sessions, each an initialize on protocol 2025-11-25 followed by notifications/initialized, spread round robin across -sessions-credentials; with -sessions-stream each also opens its standalone stream and keeps it open. Before the first step every credential opens a session, lists its tools and deletes it, so the pool entry and its catalog are built before the idle process is sampled and are not priced as part of the sessions. A step is settled and sampled the way a held step is, then every session the server gave an id is pinged, each ping waiting no longer than the /health probe, and the count of answers is the number of sessions the server still holds, seen from outside. The count stops at the first ping nobody answered, since a process out of descriptors accepts no more connections and asking a thousand sessions in turn would hold the step for as long as they all took to time out. Every open is filed as opened, refused (a 503) or failed, one session is priced over the idle process, and every stream is closed and every session deleted before the next step. -sessions-nofile starts the server under a descriptor limit as -held-nofile does, and the mode writes bench/sessions.json, uncommitted, draws nothing, refuses a -sessions-json that names the published record and cannot be combined with -held or -fairness. make bench-sessions runs the ladder under a limit of 1024 with every stream open. The held and the sessions modes price what they hold with one function, costPer, so a figure means the same thing in both documents. The stand-in learns the stateful transport the tests drive: --stateless=false gives out session ids, answers a request on an id it does not hold with 404, holds a standalone stream open until the session ends, deletes a session on DELETE, and refuses an initialize past STANDIN_SESSION_LIMIT with the 503 the real ceiling answers with. A refused standalone stream's body is quoted, and read for no longer than the /health probe waits: a refusal is a short body, but nothing promises that one ends, and a response left open would otherwise hold the step. A test holds a 503 whose body stays open to that bound. Part of #951.
On --stateless=false the SDK keeps every session a client opens, with the three goroutines serving it and its owner record, until the client deletes it, the pool evicts its credential or it sits idle for --session-timeout, half an hour by default. initialize is metered to no bucket (RTC-004), so nothing bounded how many a caller opened, and each session may also hold its standalone stream, a GET the SDK serves for as long as the session lives. The register recorded that as finding F-31 on HLD-010 and IDN-010. Measured on 951b's tip with the sessions ladder of cmd/bench_resources: an idle session costs 3 goroutines, 10 to 17 KiB of heap and no descriptor, and one holding its stream 6 goroutines, 24 to 25 KiB and one descriptor. Under a hard descriptor limit of 1024 the base server kept every session it was offered until the streams had taken all 1024 descriptors, at 1012 to 1016 sessions; from there /health went unanswered and, in two runs, between 985 and 2984 opens of a step failed at the connection. The gate now takes a session slot for every POST that would open a session, a POST carrying no Mcp-Session-Id on a stateful deployment, once the credential is admitted and before the SDK creates anything, and refuses it with the held-call ceiling's 503 when none is free. The first request the SDK dispatches on the new session keeps the slot, and a goroutine waiting on the session gives it back when the session ends, however it ends; a POST whose session did not survive it (no initialize, a refused one) gives it back as the gate returns. The two ends take one flag, so a slot is released once. The ceiling is half the held-call ceiling, 96 under a limit of 1024, rather than a number of its own, and the gate now counts a session's standalone GET as a held request: every stream the process keeps then sits inside the descriptor budget HLD-011 was sized from, and the streams of every session can take at most half of its slots, so a call on a session that is open is still served with every session slot taken. No flag moves it, for HLD-011's reason: it follows the descriptor limit, the one lever that also raises what it protects. The refusal is word for word the held ceiling's, since the next action is the same and the one bit INV-019 accepts is that the process is full; the log line names the scope and the figure. heldRequestsFailure becomes processBusyFailure, the refusal both rows share, and the startup line announces stateful_sessions_per_process beside held_requests_per_process from a helper of its own, which keeps serveHTTPOn under the maintainability threshold. HLD-010 becomes a valued ceiling on the process with a declared site for every place it is enforced, SessionHeldDivisor is pinned at 2, IDN-010 refuses the newcomer at capacity instead of carrying F-31, and F-31 joins the findings the register answers. test/e2e/http pins the whole of it on the wire under a limit of 1024. Part of #951.
HLD-010 bounds the sessions a --stateless=false deployment keeps across every credential, and nothing in the pages said a stateful deployment had any such bound: the guides described --session-timeout as the only thing that ends a session, and the flag rows for --max-http-clients said the pool bounds entries and not sessions, which was true and left the sessions bounded by nothing. The HTTP guide and the site's HTTP page gain a section of their own beside the held calls: what a session costs, measured through the sessions mode (three goroutines and 10 to 17 KiB idle, a descriptor and three goroutines more with its standalone stream), what the base process did under a hard limit of 1024 (kept every session until the streams had taken all 1024 descriptors, then answered nothing), the ceiling as half the held-call one (96 under 1024, 114560 under 524288), what counts, the refusal, and what it costs a deployment. The held-call sections now say that a standalone stream is counted there. The rejection table, the troubleshooting tables (guide and site, EN and ES), the startup line example, the flag rows for --max-http-clients, --session-timeout and --stateless, the security pages, the sizing and resource-consumption pages, the enterprise and remote deployment guides say the same in their own places. The specification's INV-019 lists HLD-010 among the process bounds whose refusal discloses one bit, the answered-findings paragraph records F-31 as answered by HLD-011 and HLD-010 together, and ADR-0023 NEG-004 and NEG-007 name the sessions mode and HLD-010 beside HLD-011. Part of #951.
The first version of the session ceiling counted a session's standalone stream as a held call when its GET arrived, and refused that GET with the held ceiling's 503 when every held slot was taken. The go-sdk client does not ask again for a standalone stream it is refused: it logs a warning on the JSON body and gives the stream up, and the server then drops every message meant for that session outside a response, a resource update and every list_changed among them, while the session carries on and its subscribe still reports success. Calls could take every held slot, so a kept session was never guaranteed its stream. The POST that opens a session now takes the stream's held slot together with the session's, and the session keeps both until it ends, whether the stream is open or not; the GET takes none. With the sessions at half of the held slots, calls keep at least the other half, and a session the process keeps is never refused its stream. A POST that finds no held slot left is refused in the gate instead, with the held ceiling's 503 and log line, which a client sees as a failed connect. TestStatefulSessions_AGoSDKClientKeepsItsStreamWithEveryHeldSlotTaken drives the SDK's own client through both halves. A POST on protocol 2026-07-28 or later no longer takes a session slot. The SDK answers it, server/discover included, with the revisions the stateful transport serves and closes its session with the POST, so counting it spent a slot no session could keep and, at a full ceiling, replaced the answer that tells a client to fall back with one that tells it to retry. HLD-010's refusal is then of the legacy era alone, as the register already said. --session-timeout=0 leaves the SDK no idle timer, so a session nobody deletes holds its slot until the pool evicts its credential, which is never while that credential stays active or with --pool-idle-timeout=0. Zero keeps its meaning (END-005), and startup now says what it costs on a stateful deployment. The startup line announces stateful_sessions_per_process only where sessions are kept, and the curated help names the session ceiling beside the held-call figure and on -session-timeout. The count's type is processSlots now, since it counts the sessions as well as the held calls, and acquire runs an optional hook between its read and its swap so a test takes the retry on purpose instead of by the scheduler's luck. The register says what the ceiling costs and bounds: one credential's initializes, which spend no rate, can take every slot for up to --session-timeout, which is put to the maintainer with the per-credential question; the derived figure bounds descriptors, and memory only where the limit is small (56 to 110 KiB of resident set per idle session, six to twelve GiB at the 114560 sessions a hard limit of 524288 gives); admission at the pool's bound can evict another credential's quiet entry and end its sessions before this row refuses the newcomer; and IDN-010's bound lives only in its comment. test/e2e/http probes /health while the six hundred extra initializes are in flight rather than after they return. Part of #951.
The pages described the first version of the session ceiling: a standalone stream counted as a held call when its GET arrived, and a full held ceiling refused it. The server now reserves the stream's held slot on the POST that opens the session and keeps both slots until the session ends, so the held-call sections say that instead, and the session sections say why: a client refused its stream does not ask for it again. The session sections (HTTP guide, site EN and ES) also carry what the review found the first version left out. An idle session costs four goroutines once the ceiling exists, the fourth giving its slots back, and 56 to 110 KiB of resident set. A POST on protocol 2026-07-28 or later takes no slot, since the stateful transport answers it with the revisions it serves. The ceiling bounds descriptors, and memory only where the hard limit is small: the 114560 sessions a limit of 524288 allows come to six to twelve GiB idle, so the container's memory limit bounds them first, and whether a fixed cap should stand beside the derived figure is an open question. Filling the ceiling costs a caller nothing, since initialize spends no rate-limit token and an idle session holds no connection, and a session nobody deletes keeps its slot for up to --session-timeout, or with 0 until the pool evicts its credential, which startup now warns about. The rejection table, the startup line example (which carries the session figure only where sessions are kept), the troubleshooting rows, the --session-timeout flag rows in the CLI, configuration and environment references, the sizing, security and resource-consumption pages, the enterprise deployment guide and the specification's HLD-010 and HLD-011 paragraphs say the same in their own places. Part of #951.
The resident set this layer stated for an idle session, 56 to 110 KiB, joined the lowest sample of the build before the ceiling to the highest of the build after it, and left out the highest before it (119 KiB at 500 sessions under a hard limit of 1024). Each figure now describes one build, read at 500 to 4000 sessions, where a per-session figure is not dominated by what a few sessions share. Before the ceiling an idle session held 56 to 119 KiB of resident set, which is what the pages that describe that build now say. With it, and its fourth goroutine, an idle session holds 10 to 20 KiB of live heap and 88 to 110 KiB of resident set, from one credential and from a hundred, so the 114560 sessions a hard limit of 524288 allows come to about ten to twelve GiB idle rather than six to twelve. The server's comment and the register's HLD-010 comment say the same. Part of #951.
With --session-timeout=0 a session nobody deletes keeps its slot until the pool evicts its credential's entry. The comments and pages said that happens never while the credential stays active or with --pool-idle-timeout=0, and the second half is not what the pool does: with the idle sweep off, the size bound still evicts the least recently used entry to make room at --max-http-clients. They now name both ways an entry leaves: after --pool-idle-timeout without a request (never with 0), or to make room at --max-http-clients. The warning's own words were already right, since they say only that the slot is kept until the entry is evicted. Part of #951.
The sessions run built its two addresses on the server it started as "http://" strings, the shape SonarCloud reads as clear-text traffic (go:S5332). It now takes them from loopbackURL, which the held run added for the server it starts and which carries the reason plain HTTP is right there.
The session flood had its own copy of the held-call flood and the same failure on windows-latest, 475 of 600 initializes answered with 503. It now goes through flood, in waves on Windows and all at once elsewhere, and a failure names each ending that was not a 503.
695822e to
fb0a1d1
Compare
|
…es only Windows and macOS build (#1206) #1064 made the condition gate able to read `cmd/server` at all, by measuring it in a staged copy of the module without the files Linux does not build, and #1017 stayed open for what that left. Two things: the conditions and mutants of `cmd/server` that no test decided, and the files a Linux run leaves out, which nothing measured anywhere. This closes both. ## cmd/server on Linux At e19b579 gobco measured 2229 of 2270 conditions of `cmd/server` both ways. The 41 left are the list #1089 carried, less one closed since. Every one is answered here: the package now measures 2236 of 2236, the total lower because some conditions were removed rather than tested. The whole package was then put through gremlins, which had only ever run on the files a change touched. That run found 10 lived mutants and 1 timed out, all answered below. Its not-covered mutants sit where gremlins has no statement counter to read, and every one of them that compiles was applied by hand and fails the suite (see Measured). Most of the answer is tests, each for a case nothing exercised: a shorter block checked after a longer one (`longestAuthBlock`), the guard's two refusals with no rejected-token cache, a challenge with no scope or no metadata URL, a capture of a handler that wrote nothing, a pseudo-version whose patch overflows an int, request ids a decoder never hands over (an unterminated string, `+1`, `4-2`), both credentials on a terminal, a public URL that does not parse, an instance list entry with no host, the singular `gitlabURL` field, a prebuilt catalog on the dynamic and meta surfaces, a second `WriteHeader` and a second stop of the SSE heartbeat, a card preflight that names no header, a lone `-` among a peer's arguments, a probe target that does not parse, a peer that exited between being listed and having its command line read, `WithGroup("")`, `forget` among two shapes, a shape registry with no registration hook, a staging descriptor that names something other than a directory, closing a socket whose path is already gone, a listen carrying prompts or resource list-changed beside its resources, and the telemetry tool-name flag not registered at all. The Cygwin half of the terminal check is false on every platform but Windows, so it is reached through a seam, `isCygwinTerminal`, as `osExecutable` already is. Some conditions could never take their other value, and those are code changes rather than tests: - `transportBudget.window` tested `effectiveWindow <= 0`, which `newTransportBudget` has already resolved to the default, so a budget that exists answers its own window. That is the lived mutant #1017 named, and the window now falls back in two places rather than the three `AUB-002` described, so its comment in `internal/tenancy` says two (#958 owns the rest of that). - `requireExplicitInstance` tested for a nil request, which an HTTP handler is never given; `securityHeadersMiddleware` tested for a nil body, which net/http never hands a server; `newReadinessGate` tested for a nil context, which no caller passes; `listenUnix` tested `filepath.Dir` for an empty string, which it never returns; and both `Serve` loops tested their error for nil, which `Serve` never returns. - `isUnixSocketAddr` asked about the native separator and then about `/`, so on every platform but Windows the second question was reached only when the first had said no to the same byte. One `strings.ContainsAny` asks both. - `sdkLogHandler.WithGroup("")` now returns the receiver, as slog's contract for the method says. Passing the empty name on was not harmless: the JSON handler opens a group with no name for it, so the attributes after it were nested under `""` while this handler, seeing no group, still renamed them. The Logger never forwards an empty name, so nothing served changes. The rest are mutants that no input can tell from the original, which is the code saying it carries something it does not need: a capacity hint in `publicPaths`, a `servers` counter in `runProbe` that only ever meant "a failure was recorded", an `n > 0` guard before a scan that answers false for no bytes, a `len(commit) > 7` before a cut that `min` makes, an `IndexByte` boundary that `strings.Cut` replaces, a deadline comparison `context.WithTimeout` already makes, a `sort.Slice` `<` that `cmp.Compare` replaces, the two branches of `metricHostsFor` that answered the environment's instance the same way, and the check that skips the catalog text rewriter when there is nothing to substitute, which now lives in `catalogTextRewriter` with a test of its own. The SSE heartbeat's idleness boundary took a parameter instead: `writeKeepAlive` is handed the tick it answers, so a test holds the exact interval, which a call to `time.Since` cannot. The readiness gate's exemption for unmarked connections timed out rather than dying, because its mutant deadlocks registration and the first test to register waited for it without a bound; that helper now waits at most `testHTTPLivenessTimeout`, and the mutant is killed in 37 seconds. Eleven tagless switches had a case expression gremlins reports as not covered, since a case expression carries no statement counter of its own. They are rewritten as the chains of returns or ifs the repository already uses for that reason (`isRequestID`, `validateTLSFiles`, `metricHostsFor`, `prepareStdioCatalog` twice, `shutdownHTTPServer`, `parseProbeTarget`, `sanitizedInput.Read` and `readLine`, `logKeyringChoice`, `inferTransport`). `sanitizedInput.Read` gave its line handling to an `accept` method on the way, which golangci-lint's nesting rule asked for. ## The Windows and macOS halves, in CI The cross-platform job's Windows and macOS legs now run `scripts/coverage-conditions.sh` on `cmd/server` and `internal/toolutil` through bash, not make, with `GOBCO_GATE=beyond:linux/amd64`. **The step gates.** It fails the leg on a condition not evaluated both ways in a file that leg builds and linux/amd64 does not, and prints, counts and passes one in a file Linux builds too. Holding the shared files there as well would fail a leg over a test the platform skips, for a condition the Linux run decides both ways: measured on Windows, that is 20 conditions of `cmd/server` and 12 of `internal/toolutil`, every one of them both ways on Linux. That Linux run is the local one, `GOBCO_GATE=all make coverage-conditions`, which every touched package passes before it is committed. No CI step runs gobco on Linux, before this change or after it, so the shared files are held by that run and not by CI; the step's comment and the testing README say so. The script gained what the step needs. `GOBCO_GATE` makes the report a gate: unset it passes whatever it found, as before; `all` fails on any condition gobco names as never true, never false or never evaluated, which is the gate a touched package is held to locally; `beyond:GOOS/GOARCH` holds only the files the named platform does not build, asked of the go command under that platform and the same tags. A value it cannot read is refused before gobco runs. `GOBCO_TEST_FLAGS` hands gobco's `go test` flags of its own, here `-timeout=30m`, since gobco runs it with go test's ten-minute default and `cmd/server` is about three times slower on Windows. Windows builds four sources of `cmd/server` Linux does not (`conn_refused_windows.go`, `socket_mode_windows.go`, `probe_other.go`, `descriptor_limit_other.go`) and two of `internal/toolutil` (`download_output_other.go`, `file_utils_other.go`); macOS builds only `probe_other.go` beyond Linux. Two of them carry conditions. The connection-refused check's two operands were already decided by its own test. The Windows listen's failure was decided by nothing, so `socket_mode_windows_test.go` now binds under a directory that does not exist, and binds where it can. Running golangci-lint over the package with `GOOS=windows`, which nothing in CI does, reported two findings in those files, both corrected here: Winsock's refusal constant is renamed `errWSAEConnRefused`, as errname asks of a sentinel error, and a misspelling in `socket_mode_windows.go`'s file comment is fixed. ## Measured On the Windows Server 2025 test VM (Go 1.27.1, PortableGit 2.55.0's bash 5.3, 4 cores), the step as CI runs it: `cmd/server` 2180/2206, every condition of the four Windows-only files both ways, 20 shared conditions left to Linux; `internal/toolutil` 3836/3848, both Windows-only files both ways, 12 left to Linux; exit 0, 15 minutes for the two. The leg took 20 minutes before, so it stays well inside its 60. That run was on the tree before the last mutant fixes, which changed only files Linux builds too, before the rename above, which changes no condition, and before this branch moved onto the two layers below it in this stack, whose only change to these two packages is `internal/toolutil/errors.go` (the bound on a reflected message and Workhorse's Orbit query refusal), a file Linux builds too. That file adds 20 conditions to `internal/toolutil` which the Windows figure predates, all of them in the shared files the step does not gate; what it gates on Windows, the six files Linux does not build, is the same at the head. On linux/amd64 at the head: `cmd/server` 2236/2236 and `internal/toolutil` 3870/3870 with `GOBCO_GATE=all`, and `internal/tenancy`, whose comments changed, 444/444. The register row and the testing README carry the 3870. gremlins over the whole of `cmd/server`, `--workers 2`, in three groups of files run side by side on the same tree (main.go with the flag files; the gate, guard, probe, subscription, stdio and session-owner files; the rest): Killed 1249, Lived 10, Not covered 47, Timed out 1, of 1307 mutants. Every file a fix then changed was run again on the final tree: `main.go` Killed 354, Lived 0, Not covered 16; `probe.go` 73, 0, 2; `health.go` and `readiness.go` 50, 0, 0; and `stdio.go`, whose group had started before its line handling moved into `accept`, 58, 0, 0. None timed out. A gremlins dry run on the head lists the same 47 not-covered mutants (1242 runnable), every one of them in a place that has no statement counter of its own: a constant or package-level variable initializer, or the two Windows files, which gremlins mutates although Linux never compiles them. Not covered is not the same as asserted, so I applied the initializer mutants by hand and ran the whole `cmd/server` suite on each. The 23 that turn a `+` between strings into a `-` do not compile. The other 22, in 19 distinct forms (a timeout's `*` turned into `/`, a refusal code losing its sign, `maxUnixPathLen`'s `- 1` becoming `+ 1`), each fail the suite: the HTTP server timeouts through `TestHTTPServerTimeouts_AreDeadlinesRatherThanTheAbsenceOfOne`, all three forms of `idleTimeoutDisabled` through `TestIdleTimeoutDisabled_OutlastsEveryOtherDeadline`, the two refusal codes through `TestRefusalCodes_MirrorTheirStatusAndSitWhereTheSpecificationAllows`, the socket path limit through `TestMaxUnixPathLen_IsTheLongestPathAnAddressCanCarry`, `stdioRereadTimeout` through `TestRefreshStdioAuthority_FollowsTheInstanceUntilTheContextEnds`, `httpShutdownTimeout` through four tests, `TestHTTPShutdownTimeout_SitsInsideTheBracketItsCommentArgues` among them, the probe's timeout and budget and the telemetry, startup-drain, stale-socket, pprof and shutdown-grace timeouts each through tests of their own, and `sseKeepAliveInterval` at zero by panicking `time.NewTicker`. The script's tests (26, among them nine new for the gate, the test flags and the CI step) pass under bash 5 and under bash 3.2.57, the version macOS ships as `/bin/bash`, in a `bash:3.2` container. ## Gates This is the third layer of a stack: it sits on the Orbit query layer for #1031, which sits on the layer raising the bound on GitLab's reflected message, which sits on main at fbf81ff. Since the base this branch was first measured on, 862e393, nothing below it has changed `cmd/server`, and what main had gained before that, since the base the mutation runs measured, changes only string literals there (help text, a flag's usage, the first-run guidance), which no mutator touches, so the `cmd/server` mutation figures carry. Below it, #1199 changed IDN-013's text and added one of its declared sites in `internal/tenancy`, and the two layers changed `internal/toolutil/errors.go`; those two packages' figures were measured again. Everything below ran on the head of this branch; the one conflict of the rebase was `docs/development/testing/testing.md`, which was regenerated rather than merged. - `go build ./...`, and `go test -count=1 -cover`: `cmd/server`, `internal/tenancy` and `internal/toolutil` 100.0% of statements. - `GOBCO_GATE=all make coverage-conditions`: `cmd/server` 2236/2236, `internal/toolutil` 3870/3870, `internal/tenancy` 444/444. The `internal/toolutil` figure was taken again after the layer below began refusing a null message in Workhorse's query refusal, which added its four conditions. - `make coverage-mutants PKG=./internal/tenancy`: Killed 239, Lived 0, Not covered 67 (every one an `iota` or constant declaration), Timed out 0. `cmd/server` as above. - `golangci-lint run --build-tags e2e` on `./cmd/server/...` and `./internal/tenancy/...`: clean, and clean on `cmd/server` under `GOOS=windows` and `GOOS=darwin`. - `go vet` of `cmd/server` and `internal/toolutil` for windows and darwin on amd64 and arm64: clean. - `make check-test-subtests`, `check-test-goroutines`, `check-test-file-names`, `check-action-ids`, `check-md-escaping`, `check-doc-tool-names`, `check-surface-quality`, `check-tenancy`, `check-sdk-context`, `check-dead-consts`, `check-e2e-static`, `check-orbit-record`, `check-gateway-chars`, `audit-catalog-first`, `check-action-grants-derivation`, `audit-1to1`, `audit-1to1-sdk`, `audit-1to1-paths` and `audit-1to1-grants`: all pass. - `check-testing-docs`, `check-llms`, `check-tool-reference`, `check-lhm-manifest`, `check-footprint`, `check-site-stats` and `check-action-grants`: all pass. - `go run ./cmd/format_md_tables/ --check`, `node scripts/check-doc-links.mjs`, markdownlint-cli2 on the changed Markdown, `make check-em-dash`: pass. - `python3 -m unittest scripts/coverage_conditions_sh_test.py`: 26 tests pass under bash 5.2. The run under bash 3.2.57 in a `bash:3.2` container was before the rebase, which changed neither the script nor its tests. - actionlint v1.7.7 on `ci.yml`: no findings. That run had no shellcheck on its path, so shellcheck 0.9.0 was run on its own over the step's `run` block and over `scripts/coverage-conditions.sh`: no findings. Before the rebase, actionlint v1.7.7 with shellcheck 0.10.0 had found none either. ## Decisions I took here - The cross-platform step gates rather than reports, and gates only the files Linux does not build, for the reason above. The shared files stay with the local `GOBCO_GATE=all` run on Linux that a touched package passes before it is committed; this change adds no Linux gobco step to CI. - The step runs on Windows and macOS only. Linux builds none of the files it would hold. - A Go runtime crash of #467 fails this step the way it fails the unit suite's first run, and the answer is the same: rerun the failed job. The step does not reproduce the crash-aware runner's retry. - Conditions no input can reach were removed rather than tested through their impossible state, and equivalent mutants were answered by rewriting the code they sat in rather than by declaring them. - The two stale test comments the godoc audit reported in `main_test.go` were corrected on the way, since the package was being held to its gates anyway. ## Not done - The macOS leg was not run: no macOS machine was available. What it depends on was checked instead: the script's tests under bash 3.2.57, and its tools being the ones a macOS runner has (BSD awk, tar, mktemp and tr behave the way the tests exercise them). macOS builds only `probe_other.go` beyond Linux, which carries no condition, so the leg holds nothing today and is there for the next file only it builds. - gremlins does not honor build constraints either, so the two Windows files' mutants are reported not covered on Linux and measured nowhere. A gremlins run on Windows would answer them, and that is not this change. - The e2e harness and retiring the staged copy wait on gobco honoring build tags, register row 73, and stay with #1099. Closes #1017.



On
--stateless=falsethe SDK keeps every session a client opens until the client deletes it, the pool evicts its credential or it has sat idle for--session-timeout(30 minutes by default), andinitializeis metered to no bucket (RTC-004), so nothing bounded how many sessions a caller, or all of them, could open. Each session may also hold its standalone stream, aGETthe SDK serves for the session's life. Under a hard descriptor limit of 1024 the process kept every session it was offered until those streams had taken every descriptor, at about a thousand sessions, and from there answered nothing,/healthincluded. The register recorded this as finding F-31 onHLD-010andIDN-010.This is the third of issue 951's three layers. It makes
HLD-010a ceiling on the stateful sessions the process keeps, keyed on the process and set by no flag, and has each session reserve a held-call slot (HLD-011) for its standalone stream when it opens.What it is sized from. Half of
HLD-011's figure (SessionHeldDivisor, 2): 96 under a hard descriptor limit of 1024, 768 under 4096, 114560 under the 524288 a systemd service gets by default, 229248 under 1048576, and 96 on Windows, which is sized as 1024. ThePOSTthat opens a session takes the session's slot and one held-call slot for its stream together (mcpServerGate.takeSessionSlot), and the session keeps both until it ends, whether its stream is open or not. The sessions therefore hold at most half the held-call slots: the descriptor budgetHLD-011was derived from holds as it was sized, a call on an open session is still served when every session slot is taken, and a session the process keeps is never refused its stream. The standaloneGETtakes no slot of its own. The startup line announces the figure asstateful_sessions_per_processon a deployment that keeps sessions; on the default stateless transport nothing is counted and nothing is announced.What it counts. Every
POSTcarrying noMcp-Session-Idon a revision before 2026-07-28, which is what the SDK opens a session for. The gate takes the slots once the credential is admitted and before the SDK creates anything, the same orderHLD-011records. The first request the SDK dispatches on the new session keeps them, found through the per-POST carrier, and a goroutine waiting onServerSession.Waitgives them back when the session ends, however it ends (aDELETE,--session-timeout, the credential's eviction). APOSTwhose session did not survive it (not aninitialize, or one the SDK refused) gives them back as the gate returns. One flag guards both ends, so the slots are released once. APOSTon 2026-07-28 or later takes no slot: the stateful transport answers it,server/discoverincluded, with400,-32022and the revisions it serves, and keeps no session for it, so the client falls back.How it refuses. In the gate, before a session exists, with
HLD-011's answer word for word:503, JSON-RPC-50300,Retry-After: 30, the connection closed, andThis server is busy. Retry later.. It costs the credential no rate-limit token and is charged to nothing. The log linerequest refused: too many stateful sessions across the processnamesscope=processandlimit_stateful_sessions; aPOSTthat finds a session slot and no held slot for its stream is refused in the same words and logs the held-call ceiling's line.Why the stream is reserved at open. The first version of this layer counted the stream as a held call when its
GETarrived and refused it at a full held ceiling. The go-sdk client never asks again for a standalone stream it was refused: it logs a warning and gives the stream up, and the server then drops every message meant for that session outside a response, resource updates and everylist_changedamong them, while the session carries on and its subscribe still reports success.TestStatefulSessions_AGoSDKClientKeepsItsStreamWithEveryHeldSlotTakendrives the SDK's own client with every held slot taken and holds that its stream is served, and that the next client's connect fails in the gate.What it bounds, and what it costs. Descriptors, and memory only where the hard limit is small: an idle session costs four goroutines and 88 to 110 KiB of resident set, which the descriptor limit does not raise, so the 114560 sessions a hard limit of 524288 allows come to about ten to twelve GiB idle and the container's memory limit bounds them first. And filling the ceiling costs a caller nothing, since
initializespends no rate and an idle session holds no connection: one credential can open every session, and while it holds them every other tenant'sinitializeis refused for up to--session-timeout, half an hour by default and a day at most. Before this ceiling an idle session refused nobody. With--session-timeout=0(END-005, which keeps its meaning) a session nobody deletes keeps its slots until the pool evicts its credential, after--pool-idle-timeoutwithout a request or to make room at--max-http-clients, and startup now warns about that. Issue 951 decided that none of the three levers stands beside this row: no per-credential ceiling, no metering ofinitializeand no fixed cap beside the derived figure, for the reasons under Decided on issue 951 below.heldRequestsFailurebecomesprocessBusyFailure, the refusal both rows share, andheldRequestsbecomesprocessSlots, the count both ceilings use, whoseacquiretakes an optional hook between its read and its swap so a test drives the retry on purpose. The startup line's attributes move intohttpStartupAttrs, which keepsserveHTTPOnunder the maintainability threshold the new attribute pushed it past.Measured
Truenas, the new sessions mode of
cmd/bench_resources(make bench-sessions, first commit of this layer), against the stand-in GitLab, before this layer and at its tip. Per session over the idle process, read at 500 to 4000 sessions: before, an idle session held 3 goroutines, 10 to 17 KiB of live heap, no descriptor and 56 to 119 KiB of resident set, and one holding its stream 6 goroutines, 24 to 25 KiB, one descriptor and 95 to 124 KiB. After, from one credential and from a hundred, an idle session holds 4 goroutines (the fourth gives its slots back), 10 to 20 KiB and 88 to 110 KiB, and one holding its stream 7 goroutines, 25 to 28 KiB, one descriptor and 66 to 108 KiB./health"Open" is the bench's count of sessions open at the settled sample; "Live" is how many of them answered a ping, each waiting no longer than the
/healthprobe, stopping at the first that got no answer. In the first row the process accepted no connection at the sample, so none answered though 1013 were open. "Inherited" is the host's container hard limit, 1048576. The SDK's session map andsessionOwnershold one entry per open session and drop it when the session ends (TestSessionOwners_ADisconnectedSession_IsForgotten), so both stop at the ceiling; that is inferred from the test and the ceiling, not sampled by the bench.test/e2e/httpholds the same behaviour against the real binary underulimit -n 1024with--stateless=false: 96 announced and 192 held calls, 96 sessions opened each with its stream, the 97thinitializerefused in the gate with the connection closed and no session id, a call on an open session still served, 600 moreinitializes at once all refused while/healthis probed and answers throughout, and a session's slots back once it is deleted.Decided on issue 951
The seven questions this layer put are answered, each as proposed here, so nothing in this layer's row, pins, gate, e2e expectation or pages changes. The layer of this stack that records issue 951's decisions writes each answer and its reason where the tree still presented the question, and records the metering answer on
RTC-004itself.SessionHeldDivisor2, 96 under a hard limit of 1024), with the stream's slot reserved when the session opens, rather than a descriptor reservation of its own that would lowerHLD-011's figure on a stateful deployment. The cost stays as stated: a session holds a held slot while its stream is closed, so calls have half the held slots on a deployment whose sessions are all open.initializestays unmetered, for 951b's reason: the credential is a key a caller can mint (INV-003), so a per-credential number, or a price in the opener's own rate, multiplies with every token a caller mints and bounds nothing the process ceiling does not. One credential can therefore open every session at no rate cost and hold them for up to--session-timeout, and the operator pages say so, with what to do about it.HLD-011records from issue 951's request (PAT-003), accepted with its cost here: admission at the pool's bound can evict another credential's quiet entry, ending its sessions, before this row refuses the newcomer.HLD-011's does, and with no per-caller ceiling beside it any refusal says the process is full, the one bit the exception accepts;HLD-010stands under it besideHLD-011.sessionOwners' waiter and tying the two rows' sites together, accepted as implemented.RTC-007), 951b (HLD-011) and this layer the register carries F-03 and F-31 on no row, and the answers to 2 and 3 here and to 951b's per-credential and memory questions add no row, so this layer closes The process has no ceiling of its own on held requests, stateful sessions or catalog listings #951.Validation checklist (tenant-policy-spec.md)
KeyProcess, which no caller can mint. No new key.IDN-010now states the same.POSTtakes no slot): 503,-50300, fixed 30 sRetry-After, no challenge, connection closed; retry-later class, textThis server is busy. Retry later.. On stdio: none.Source: Derived, half ofHLD-011's derived figure through the pinnedSessionHeldDivisor(2), with a floor of one session; zero is not applicable; no flag or variable, so no malformed-value policy.--session-timeout=0keeps its meaning underEND-005, and startup warns what it costs here.StdioKeynone). The gate counts only on a stateful HTTP deployment, and the middleware finds no slot on any other request.make bench-fairnessanswers a claim this ceiling does not make.Gates
Measured on this layer before it was rebased onto main at 5da9699: the line numbers below are that tree's, and the one commit cited below (a0a03fe) is named as it sits in this stack. At its place in this stack on 50ef2cf, the layer built and vetted, the e2e gitlab, http and stdio packages compiled, the unit suites of
cmd/server,internal/tenancy,internal/oauthandcmd/audit_tenancypassed, andmake check-tenancyread the same 83 rows and 308 sites. The stack now sits on b44ab96, which adds #1083, #1084 and #1085; they change the upstream register, indirect module versions and the site's package pins, none of them a file this stack writes, and every commit of the stack applied unchanged.go test -shuffle=on: 100.0% of statements incmd/server,internal/tenancy,cmd/audit_tenancyandcmd/bench_resources;-raceon the session, gate and held tests ofcmd/serverand oncmd/bench_resources; thetest/e2e/httplimit, held, session and health cases against the binary built at the tip, without the Docker-only proxy and balancer cases, which need a daemon this lane did not start; the e2e gitlab packages compile.go vetand golangci-lint with thee2eandhttpe2etags: 0 issues. markdownlint,format_md_tables --checkandcheck-em-dash: clean.make check-tenancy: 83 rows, 21 failures and 308 declared sites over 202 packages (21 refusal returns, 81 refusals, 14 reasons and 31 settings read); 0 findings, 56 declarations exempted.check-dead-consts,check-test-file-names,check-test-subtests,check-test-goroutines,check-sdk-contextandcheck-doc-tool-namespass. The godoc audit finds nothing this layer wrote; the five findings in the packages it touches predate it (three test comments incmd/server/main_test.go, andcmd/bench_resources/fairness.go's package comment, twice).internal/tenancy408/408,cmd/bench_resources1420/1420.cmd/serverthrough a staged copy without its non-Linux twins (the issue 1017 approach): 2082/2124, every condition ofsessions.goandheld.goboth ways,acquire's retry now taken on purpose through its hook. The 42 open are the ones 951b's run left, none on a line this layer wrote:auth_gate.go3 (lines 728, 818, 916),bearer_guard.go5,etag.go2,first_run.go1,health.go1,jsonrpc_id.go3,listen.go2,main.go11,pprof.go1,probe.go3,readiness.go1,sdk_log.go2,shape.go2,socket_mode_unix.go2,subscriptions.go2,telemetry.go1.--workers 2 --timeout-coefficient 20,INVERT_LOGICALon):internal/tenancy, whole package: 226 killed, 0 lived, 0 timed out, 66 not covered, every one arithmetic in a constant declaration, the same 66 as the base.cmd/server, scoped to the four files this layer writes (sessions.go,auth_gate.go,held.go,main.go), at the review fix (a0a03fe, whose code the later commits leave as it is): 459 killed, 5 lived, 33 not covered, 0 timed out. Every mutant ofsessions.goandheld.gois killed. The 5 lived and 33 not covered are exactly 951b's on the same code, moved by the lines this layer inserted:auth_gate.go's unobservableeffectiveWindow <= 0(issue 958) and its 8 constant initializers, andmain.go's four recorded boundaries and 25 initializers,switchcases and the equivalent807:19.cmd/bench_resources, the four files this layer writes (held.go,main.go,sessions.go,target.go), on a tree whose package differs from the tip only in howsessions.gobuilds its one constant handshake message: 253 killed, 0 lived, 0 not covered, 0 timed out.sessions.goagain at the tip: 55 killed, 0 lived, 0 timed out, and 2 not covered, the two concatenations in that constant's initializer, which the compiler folds and no test can reach, the class ofinternal/tenancy's 66.Decisions I took here
HLD-010is half ofHLD-011(SessionHeldDivisor, 2, floor one), derived, keyed on the process and not configurable. Sessions get no descriptor reservation of their own, so 951b's derivation ofHLD-011is unchanged (decided on issue 951, item 1).GETtaking one when it arrives, which changes whatHLD-011counts on a--stateless=falsedeployment and leaves calls at least half the held slots there when every session is open (decided on issue 951, item 1).HLD-011's gate refusal is no longer limited to the 2026-07-28 era, since a legacyPOSTthat would open a session is refused there when no held slot is left for its stream.HLD-011's gate failure word for word, the log line alone naming the scope and the figure, andHLD-010is listed underINV-019's one-bit exception (decided on issue 951, item 5).initializewere added, and no fixed cap stands beside the derived figure (decided on issue 951, items 2 and 3).ServerSession.Wait, bounded by the ceiling, rather than bysessionOwners' waiter (decided on issue 951, item 6).HLD-010andIDN-010, joins the answered set in the specification and the tests, andIDN-010refuses the newcomer at capacity, decided in issue 951.serveHTTPOnmoved intohttpStartupAttrsso the new attribute does not push it past the maintainability threshold; behaviour is unchanged.Not done
sessionOwnerswere not sampled by the bench; that both stop at the ceiling is inferred fromTestSessionOwners_ADisconnectedSession_IsForgottenand the ceiling.make bench-fairnesswas not run: the ceiling protects the process's descriptors rather than one population from another, so the comparison of two populations it runs answers nothing this row claims.test/e2e/httpdid not run, and the Docker GitLab suites were compiled but not run.cmd/serverwas mutated over the four files this layer writes, not whole.Closes #951.