Repository navigation
Bound the OAuth identity cache and the round trips that verify a new token - #1087
Conversation
There was a problem hiding this comment.
Sorry @jmrplens, your pull request is larger than the review limit of 150,000 diff characters
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 29 minutes. View limit detailsLimit details: You’ve used the included review currently available. Review configuration: ⚙️ Run configurationConfiguration used: Repository: jmrplens/gitlab-mcp-server/.coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (32)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The fix for #945 (#948) removed the workflow's own `actions/cache` step for `~/.cache/golangci-lint` and left a second cache of that directory in place: `golangci-lint-action` keeps one itself. `install-only: true` returns after the install and nowhere else, so the action still restores that cache before installing (`run()` calls `restoreCache` first) and saves it again in its post step, under `golangci-lint.cache-<os>-<interval>-<go.mod hash>` keys. The comment in `ci.yml` said the cache was not restored and the static analysis page that it was not kept between runs, and neither was true. On 2026-09-30 the lint of #1088 to #1094, opened minutes before, reported fifty unused `//nolint:staticcheck` directives on each layer, in files the stack does not touch. Every one of those jobs restored, by exact key, the entry main's push run of b44ab96 had saved at 09:07 UTC: the key names only the OS, a seven-day interval and the hash of `go.mod`, which the stack shares with main, so the analysis of main's tree matched a different tree exactly. With that entry deleted, #1087's second run and a rerun of #1088's commit fell back through the restore key to main's entry of 2026-09-25 and linted clean, and a cold `make golangci-lint` of the stack's tip reports nothing. That is the failure #945 describes, a restored analysis disagreeing with the tree, coming from the cache #948 did not see, and this time from main's entry rather than the pull request's own. `skip-cache: true` turns off both the restore and the save. The comment above the step now says why it is needed alongside `install-only` and what the key leaves out, and `docs/development/static-analysis.md` says the same. The job's header comment no longer says the action caches the linter binary, which it downloads on every run. The Go build and module caches setup-go keeps are unaffected. What it costs, measured on the whole job as #945 asks: the eight pushes to main between 29 September 11:27 and 30 September 09:02 UTC each restored a cache and took 214 to 319 s, the `make golangci-lint` step 190 to 298 s of it; this pull request's run, the only one with the change so far, restored nothing and took 313 s, the step 287 s. All eight restored the entry main saved on 25 September, which the action does not save again while its key still matches, so the analysis they reused was of a tree days older than theirs: it was not buying time the spread between runs can show. Closes #945
In oauth mode two things on the admission path grew with whatever callers sent. The verified-identity cache was a map that lost an entry only to its TTL, a lazy read or the sweep, so it held one entry for every credential verified inside one TTL, and a valid credential is cheap to mint on GitLab. The verifier sent GET /api/v4/user and up to two introspection requests for every token the cache did not hold, with no ceiling anywhere: the pool's probe ceiling counts only the pool's own GET /user, which runs after these have been sent, and the per-address budgets bound one source at a time. An oauth deployment reachable from many addresses relayed up to three GitLab requests per invented token. The register recorded both as F-29 (ADM-005) and F-30 (ADM-002, POL-006). What I decided in issue 950, and what this change does: The identity cache holds at most 10000 identities (tenancy.OAuthCacheCapacity, the largest pool --max-http-clients allows) and, when full, drops the identity used least recently to hold the one GitLab has just verified. Refusing the newcomer would refuse a credential GitLab accepted. Every read counts as a use, so a credential in steady use is the last to go; the one dropped costs its credential a re-verification and, while its address is blocked, the exemption a cached identity gives it. The expiry is now judged under the same lock that removes the entry, so the separate stale re-check is gone. It is a take across keys, so ADM-005 records the decision (EvictAcrossKeys, Decided issue 950) and INV-005 now names it beside the pool entry. The verifier's round trips run under slots of its own, keyed on the process and not configurable: sixteen verifications at once (tenancy.OAuthVerifications), separate from the pool's sixteen probes, so neither kind of work can starve the other. The cache is read before a slot is asked for, so a cached credential never waits, and the order cache, then GET /user, then introspection is kept, with the slot held through introspection. A new token waits up to five seconds (tenancy.OAuthVerificationWait) and is then refused with oauth.ErrVerificationBusy, which the bearer guard answers 503 -50300 with Retry-After 30: not cached, charged to no budget, and worded "This token could not be verified right now" without saying other callers are the cause (INV-019). The operator reads the cause in one throttled log line naming neither the caller nor the token. The new register row is ADM-014, a process ceiling that waits then refuses; its refusal joins the failure table as classify's seventh return, uncharged. F-29 and F-30 stay in AllFindings as issue 950's and no row carries them; TestDecisions_AFindingNoRowCarries_IsAnsweredByItsIssue names them in the answered set. Two refactors ride along, both behaviour-neutral. The identity request moved out of the verifier closure into askIdentity, which brought the closure back under the cognitive complexity bound once it took a slot. And describeScopeShortfall's tagless switch became two conditions, because a case expression carries no statement counter and gremlins reported its mutants as uncovered while the tests drove all three shapes. Measured through the verifier against a stand-in GitLab: a hundred thousand distinct valid tokens held 100000 entries and 60.5 MB before and hold 10000 and 7.0 MB now (about 700 bytes an entry). Two thousand tokens at once put 2000 verification requests in flight and 6000 in all before, and 16 in flight now; at 50 ms a round trip, 528 were verified inside the five second wait and 1472 refused busy. test/e2e/http pins the ceiling on the wire: twenty tokens against an instance that holds each identity request, sixteen reach it and no more, four are refused 503 -50300 with Retry-After 30 and no challenge, a credential already cached is served while every slot is held, and a deployment that blocks an address after one charged failure still serves a new token afterwards. Closes #950.
The review of the first commit found its INV-019 argument wrong. A caller that presents one new token to an instance it knows to be healthy, and is refused after the five-second slot wait, learns that other callers are verifying, whatever the refusal says: ADR-0023 NEG-007 already says no wording closes that bit. The distinct sentence made it certain rather than inferred. So ADM-014 now answers from the return, and in the words, that ADM-002 gives a verification with no verdict, as RTC-007 answers in RTC-003's; the failure table is back to 21 returns, six in classify; and INV-019 and NEG-007 list ADM-014 beside the rows issue 951 decided. The comments, the tests and the docs that said no caller learns anything now say what the caller can infer and what the wording adds, which is nothing. The operator tells the causes apart in the log: saturation at WARN, a client that left while it waited at INFO, an unreadable answer at ERROR. The slot wait takes a free slot before looking at anything else. With the wait in one select, Go picked at random between a free slot and a context that had already ended, so half of those requests were refused, and logged as saturation, with a slot free. A wait the request's own end cut short is now an error of its own and never ErrVerificationBusy, whose text no longer mentions saturation either, since the SDK middleware writes a verifier error into its own 500 on the one path the guard does not word. The cache is read again when a wait ends, however it ended, and once a slot is held, so a request that waited behind another carrying the same new token finds the identity that one verified instead of repeating its round trips or being refused a credential the cache holds. A full identity cache drops an expired identity before the least recently used live one: a TTL shortened to the token's own expiry makes a recently used entry expire, and dropping a live identity to keep it cost a credential its entry and its exemption from an address block for nothing. It walks for one only once the earliest expiry it holds has passed, so a flood arriving at a cache of live identities still costs one comparison each. OAuthCacheCapacity stays its own literal and a values test holds it equal to PoolSizeMax, which every sentence justifying it names. The eviction guarantee was misstated. Every read of another cached credential moves ahead of the victim too, so what pushes a live identity out is the cache's whole capacity of other distinct credentials used since its last request, not more than that many verified; near a full pool, every new verification does. The cache, the guard's header, the guide and the security page say so, and the guard's header now states the two windows in which the SDK middleware's second verification can miss the identity the guard stored, and what it answers then. The docs say the ceiling bounds concurrency, not rate (about 320 requests a second at 50 ms, above GitLab.com's allowance for unauthenticated traffic from one address), state its cost to legitimate credentials presented for the first time during a flood, drop the before-and-after comparisons from the operator pages, and add the saturation and eviction causes to the troubleshooting tables (docs and site, EN and ES) and the capacity to the cache TTL reference rows. The unit test now holds all three authentication budgets uncharged across forwarded addresses behind a trusted proxy, and the httpe2e flood arrives the same way, one forwarded address per token, with the per-address and distinct-credential budgets both blocking on their first charge, then serves a new token from every one of those addresses. The flood helper reads the test context on the test goroutine. The tool-specific comment on describeScopeShortfall's two conditions is gone.
68729df to
7e64430
Compare
The arm with the rate limit off lists until the tools served exceed what the process's bucket could have granted, then asserted that again with the time measured after the drive returned. That later instant also counts the time the workers took to stop, and on windows-latest, which outlists the refill by a narrow margin, it failed a run 77 tools short of a crossing the drive had already made. The verdict is now kept from the one instant both figures were read at.
|



What was wrong
In oauth mode two things on the admission path grew with whatever callers sent. The verified-identity cache was a map that lost an entry only to its TTL, a lazy read or the sweep, so it held one entry for every credential verified inside one TTL, and a valid credential is cheap to mint on GitLab. The verifier sent
GET /api/v4/userand up to two introspection requests for every token the cache did not hold, with no ceiling anywhere: the pool's probe ceiling counts only the pool's ownGET /user, which runs after these have been sent, and the per-address budgets bound one source at a time. An oauth deployment reachable from many addresses relayed up to three GitLab requests per invented token. The register recorded both as F-29 (ADM-005) and F-30 (ADM-002,POL-006).What changed
The identity cache holds at most 10000 identities (
tenancy.OAuthCacheCapacity, the largest pool--max-http-clientsallows, held equal toPoolSizeMaxby the values test). When full it drops an expired identity if there is one, and otherwise the one used least recently, to hold the one GitLab has just verified: refusing the newcomer would refuse a credential GitLab accepted. It walks for an expired identity only once the earliest expiry it holds has passed, so a flood arriving at a cache of live identities costs one comparison each. Every read counts as a use, so what pushes a live identity out is 10000 other distinct credentials used since its last request, read or verified alike; on a deployment serving close to that many, every new verification does. The identity dropped costs its credential a re-verification and, while its address is blocked, the exemption a cached identity gives it. The expiry is judged under the same lock that removes the entry. It is a take across keys, soADM-005records the decision (EvictAcrossKeys, decided in issue 950) andINV-005names it beside the pool entry.The verifier's round trips run under slots of its own, keyed on the process and not configurable: sixteen verifications at once (
tenancy.OAuthVerifications), separate from the pool's sixteen probes, so neither kind of work occupies the other's slots. It bounds concurrency, not rate: at 50 ms a round trip, sixteen slots send about 320 requests a second, which is above GitLab.com's allowance for unauthenticated traffic from one address, so what it holds is the load relayed in proportion to the instance's speed rather than to how many tokens arrive at once. The cache is read before a slot is asked for, so a cached credential never waits, and read again when the wait ends and once a slot is held, so a request that waited behind another carrying the same new token finds the identity that one verified rather than repeating its round trips or being refused. The order cache, thenGET /user, then introspection is kept, with the slot held through introspection. A free slot is taken before anything else is looked at, so a request whose context has already ended is not refused with a slot free.A new token waits up to five seconds (
tenancy.OAuthVerificationWait) and is then refused withoauth.ErrVerificationBusy, which the bearer guard answers from the same return and in the same words as a verification with no verdict: 503,-50300,Retry-After: 30, "GitLab could not verify this token right now. Retry shortly. The token itself has not been rejected.", not cached and charged to no budget. The new register row isADM-014, a process ceiling that waits then refuses; its refusal isADM-002's, asPOL-006's isADM-001's, so the failure table keeps its 21 returns.What a refused caller can learn. One that presents a new token to an instance it knows to be healthy and is refused after the slot wait can infer that other callers are verifying. That is the one bit
INV-019accepts for a bound keyed on the process, andADM-014is listed there and in ADR-0023 NEG-007 besideHLD-002,HLD-004andRTC-007; the wording adds nothing to it. The operator tells the causes apart in the log:token verification refused: every verification slot stayed busy(WARN),token verification abandoned: the request ended before it was verified(INFO),token verification failed(ERROR); the first two are throttled and name neither the caller nor the token.What it costs. A legitimate credential presented for the first time while a flood holds every slot waits in the flood's queue and is served only if a slot frees within its five seconds, so under a sustained flood about the slots' share of new credentials is served and the rest are refused like the flood. The ceiling protects the instance's load at the cost of admitting new credentials during a flood. Credentials already cached never wait for a slot, but they share the listener with the waiting flood. This layer's pages (the HTTP server mode guide, the security concept page, the troubleshooting pages, the site in English and Spanish, and the specification) state the cost as a refusal for as long as the flood lasts; a later layer of this stack, the OAuth driver for
make bench-fairness, measures it (42 and 40 of 120 new credentials served at 100 ms under 400 invented tokens a second) and corrects them, with the descriptor and memory conditions on cached credentials.F-29 and F-30 stay in
AllFindingsas issue 950's and no row carries them;TestDecisions_AFindingNoRowCarries_IsAnsweredByItsIssuenames them in the answered set.Two refactors ride along, both behaviour-neutral. The identity request moved out of the verifier closure into
askIdentity, which brought the closure back under the cognitive complexity bound once it took a slot. AnddescribeScopeShortfall's tagless switch became two conditions: gremlins reads a case expression as outside every covered block, so its mutants were reported as not covered however the tests drove them.One test fix rides along too, in a test that was already on main:
TestLimit_ProcessBoundsListingAcrossCredentialsdecided its arm with the rate limit off by the time measured after the drive returned, which also counts the time the workers took to stop, and on windows-latest, which outlists the process's refill by a narrow margin, that failed a run 77 tools short of a crossing the drive had already made. The verdict is now kept from the one instant both figures were read at. It sits in the bottom layer so that every layer above it, each of which runs that suite on Windows, carries it.Figures
Measured through the verifier against a stand-in GitLab on truenas, 16 workers, distinct valid tokens, retained heap after GC:
About 605 bytes an entry before and 700 after (the recency list adds its element).
A flood of invented but valid tokens all at once, 50 ms a request at the stand-in:
test/e2e/httppins the ceiling on the wire: twenty tokens, each forwarded by a trusted proxy for an address of its own, against an instance that holds each identity request; sixteen reach it and no more, four are refused 503-50300withRetry-After: 30, no challenge andADM-002's words, a credential already cached is served while every slot is held, and with the per-address and distinct-credential budgets both blocking on their first charge a new token from every one of those twenty addresses is served afterwards.Validation checklist (tenant-policy-spec.md)
ADM-005stays onKeyVerified(a verified credential, cheap to mint: a personal access token is a form with no administrator);ADM-014is onKeyProcess, which no caller can mint. No new key.ADM-005class C,ADM-014class P; neither is class D, andADM-014's stated reason ("counting work rather than callers") is in the process's unit.ADM-014is a ceiling;ADM-005a lifetime with a capacity. No share on any key.ADM-014is itself keyed on the process, protects the process's upstream load, and is a constant because an operator who could raise it could undo it (INV-004). The cache capacity is a constant on the one process-wide cache.ADM-005evicts across keys, expired first, then least recently used (decided in issue 950: refusing would refuse a credential GitLab accepted);ADM-014waits five seconds, then refuses.ADM-014on the HTTP gate only (OAuth mode is HTTP-only;StdioKeynone): 503, JSON-RPC-50300,Retry-Afterfixed at 30 s, no challenge, retry-later class,ADM-002's text.ADM-005refuses nothing.verification-failedreturn is uncharged and G7 holds it; the unit test drives all three budgets (per-address, transport source, distinct credential) at a limit of one and the httpe2e test the two a flag reaches.OAuthCacheCapacity10000,OAuthVerifications16,OAuthVerificationWait5 s; constants, zero not applicable, no flag or variable, so no malformed-value policy; all three frozen inTestValues_HoldTheirPins.ADM-014does not claim to make a population better off than without it; it bounds what the process relays to GitLab. Its cost to the one population it reaches is measured in a later layer of this stack, which adds the OAuth driver tomake bench-fairness(BOUND=oauth-verification): against a stand-in answering in 100 ms and a flood of 400 invented tokens a second, once the flood has settled into its queue, 42 and 40 of 120 new credentials were served in two runs, after 5.5 s rather than 0.5 s, and the rest refused; no cached request was refused in any arm, and their medians lay between 9 and 16 ms with the ceiling and without it; and the instance received about 160 requests a second, sixteen slots over the round trip, with at most 20 at once, against about 420 a second with up to 56 at once without the ceiling. A cached credential is outside the ceiling by construction (the cache is read before a slot is asked for), which the unit and httpe2e tests pin and the measurement confirms, as long as the process has a descriptor and memory for its connection: the waiting flood holds a descriptor and about 50 KiB for each waiting request, and the measurement ran under a descriptor limit of 1048576 with no memory limit reached.Checks
internal/oauth,internal/tenancy,cmd/audit_tenancyandcmd/serverpass with 100.0% statement coverage each; the wholetest/e2e/httpsuite passes, and the flood test passed three runs in a row;internal/oauthpasses-raceand-shuffle=on -count=3, and the guard's tests pass-race.internal/oauth214/214,internal/tenancy408/408,cmd/audit_tenancy1002/1002.cmd/servercould not be instrumented in place when this was measured (The condition gate cannot instrument cmd/server: gobco panics on build-constrained duplicates #1017); the conditions this change adds there (logUnverified's two,describeScopeShortfall's two) are each driven both ways by a named test case.--workers 2 --timeout-coefficient 20):internal/oauthkilled 109, lived 0, not covered 0, timed out 0;internal/tenancykilled 226, lived 0, timed out 0, not covered 66, all arithmetic on constant initializers invalues.go,codes.goanddecision.go, which carry no statement counter (the known tool-artifact class).cmd/server:bearer_guard.gomutated against the whole package's suite with every other file excluded (the package's other files are unchanged here apart from one comment inmain.go): killed 65, lived 0, not covered 0, timed out 0.make check-tenancy: 82 rows, 21 failures and 280 declared sites over 202 packages (21 refusal returns, 76 refusals, 12 reasons and 31 settings read); 0 findings, 56 declarations exempted.check-test-subtests,check-test-goroutines,check-test-file-names,check-doc-tool-names,format_md_tables --checkand markdownlint on the changed pages: clean.cmd/server,internal/tenancy,internal/oauthandcmd/audit_tenancypassed, andmake check-tenancyread the same 82 rows and 280 sites. The stack now sits on b44ab96, which adds Record the upstream state of 2026-09-30 in the register #1083, Record the merge of gitlab-org/gitlab!255702 and the rebase of gitlab-org/gitlab!255704 #1084 and Update the dependencies the cooldown admits, and pin the site past five advisories published on 29 September #1085; they change the upstream register, indirect module versions and the site's package pins, none of them a file this stack writes, and every commit of the stack applied unchanged.Decisions I took here
PoolSizeMax, the largest--max-http-clients, so no configurable pool serves more credentials than the cache remembers, and at about 700 bytes an entry that is about 7 MB.POL-006's numbers (sixteen at once, a five-second wait, then refuse) on slots of its own, constant and not configurable (INV-004).ADM-014, rather than a change toADM-002, whose shape is a rule;ADM-002andPOL-006drop F-30.ADM-005records its eviction as a take across keys (EvictAcrossKeys), and the specification'sINV-005and ADR-0023 name it beside the pool entry. No new capacity kind was added.ADM-002's words and return, followingRTC-007reusingRTC-003's, andADM-014joinsINV-019's one-bit exceptions and ADR-0023 NEG-007. Both are decided: the listing because no wording closes the bit (NEG-007), and the words because what they tell the caller to do, retry shortly with its token intact, is right for a request GitLab never saw.ErrVerificationBusy, and is answered by the same uncharged 503 path rather than the generic verification failure, which logs at ERROR.make bench-fairnesslacked, and the specification cites what it measured.Not done
POL-006, the pool's probe ceiling, discloses the same one bit and is not listed inINV-019or NEG-007. It was so before this change, and listing it, with the reason its refusal gives nothing more away thanADM-014's, is a follow-up of its own rather than part of this stack.cmd/server: onlybearer_guard.gowas mutated, against the whole suite.Closes #950.