Repository navigation
Probe what GitLab 19.5 counts against an invalid token, and state the distinct-credential budget's reason from it - #1480
Conversation
… token Issue 954 asks whether a REST request carrying an invalid token counts against GitLab's per-IP limit on unauthenticated API requests, the question the distinct-credential budget's stated reason rests on. The new CE scenario turns that throttle on at four requests in two minutes, sends invalid-token GET /api/v4/user requests itself, in PRIVATE-TOKEN as the legacy verification sends a token and once as a Bearer token as the OAuth verification does, has the real binary in HTTP mode verify an invented token, reads what GitLab counted off the RateLimit headers it writes on every counted response, holds freshly started servers to serving a token GitLab accepts while the address is throttled, waits out the window to see whether a ban follows, and writes the instance's settings back. GitLab 19.5 counts that throttle with Labkit, whose window opens with the first request it counts and lasts the period, while the reset and Retry-After headers GitLab writes describe a window aligned to the epoch. So the probe starts from a request counted first in its window, waits for a window to end a full period after an answer counted in it rather than by those headers, and tells a ban apart from a count it did not expect. The throttle is an instance-wide setting, which issue 954 decided no complete run may change, so the probe runs only with E2E_AUTH_THROTTLE_PROBE=true, in Docker mode, on its own. A complete run refuses to start with the switch set, read wherever the harness reads it, and starts with the switch set to false in the environment otherwise: the script sources a dotenv file and the harness parses it with godotenv, which also reads KEY: value and goes on past a reference to a variable nobody set, and the environment outranks every file the harness reads. The probe then skips, and the skip is declared under a new category, opt-in-probe, since the skip gate fails a complete run on any skip nothing declares.
…swered it The budget said GitLab rate-limits failed authentication per source address, so a sprayer spends the deployment's standing and the throttle lands on everyone using the server. GitLab's failed authentication ban covers Git and the container registry, not the API, and the probe against GitLab 19.5.0 CE showed what does happen: with the unauthenticated API throttle on (off by default on a self-managed instance), each invalid token sent in PRIVATE-TOKEN, the server's own legacy verification of one included, is counted once against the address; past the limit every unauthenticated request from it is answered 429, an anonymous one too; a token GitLab accepts is still verified and served, counted by no unauthenticated limit; and no ban follows. GitLab.com counted an invalid Bearer token, as the OAuth verification sends one, the same way. The comment and the security page in both languages also say what lands on the server: its verification of a token GitLab does not accept is answered 429, no verdict, so the legacy gate admits such a token into a pool entry and the OAuth gate answers 503. They give GitLab.com's limits with their dates, the 500 unauthenticated requests a minute per address it answered with in October 2026 and the 60 an hour announced from 19 October 2026 and still listed as proposed, which the budgets' defaults do not hold a sprayer below. The configuration note and the comment say why the distinct count matters beside the failure lockout: a sprayer kept under ten failures a minute is never blocked by the lockout, and the distinct budget's blocks grow each time it is reached. The tenant policy specification records the answer to OQ-11 as observed.
The throttle probe's skip declaration puts this branch in cmd/audit_e2e_coverage, and a mutation run over the whole package left three mutants alive, on the parent as on the branch. report() of the fine-grained fold answered the same under || as under &&, because no test had fine-grained sessions without a call or calls without a session line, and either alone is still a measurement. enclosing's half-open span was never asked about at its edges, which no site the scanner reads can reach, since a site is an expression inside a body. Both are now asserted directly.
There was a problem hiding this comment.
jmrplens has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Summary
Merge Risk: ⚪ Minimal · up to
|
There was a problem hiding this comment.
jmrplens has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
|



Refs #954
The distinct-credential budget gave as its reason that GitLab rate-limits failed authentication per source address, so a token sprayer spends the deployment's standing with GitLab and the throttle GitLab eventually applies lands on everyone using the server. GitLab's documented failed-authentication ban covers Git and the container registry, not the API, and whether an invalid-token REST request counts against GitLab's per-IP limit on unauthenticated requests was an open question (OQ-11, finding F-25). This is the first layer of the issue: I measured the answer on GitLab 19.5 and rewrote the reason to it. The keying change of the budgets is the next layer.
The probe
I added
TestAuthThrottle_InvalidTokens_SpendTheAddressUnauthenticatedAllowancetotest/e2e/gitlab/ce. It reads the instance's unauthenticated API throttle, turns it on at four requests in two minutes, and waits the minute each GitLab process keeps its settings for. It then sends invalid-tokenGET /api/v4/userrequests itself, inPRIVATE-TOKENas the server's legacy verification sends a token, has the real binary in HTTP mode verify an invented token, and sends one more invented token as a Bearer token, as the OAuth verification sends one, reading what GitLab counted off theRateLimit-*headers GitLab 19.5 writes on every counted response. Once GitLab throttles the address, it sends an anonymous request and the run's own token, starts three servers in HTTP mode (one per surface) that verify the run's token and serveuser.current, and checks the address is still throttled afterwards. It then waits out the window to see whether a ban follows, writes the settings back, and checks nothing counts an invalid token any more.GitLab 19.5 counts that throttle with Labkit rather than Rack::Attack. Every Labkit cohort enforces by default (
rate_limiter_use_labkit_rack_cohort_{1,2,3}_enforce,default_enabled: trueat v19.5.0-ee), Rack::Attack then safelists every request aslabkit_fully_enforced, and every line of the run'sapi_json.logsays so. Labkit's counter is a Redis key that the first request it counts creates with the period as its lifetime (lib/labkit/rate_limit/evaluator.rbat labkit-ruby v5.5.0), so a window opens with the first counted request and ends a period later, while theRateLimit-ResetandRetry-AfterGitLab writes describe a window aligned to the epoch (RequestThrottleData.from_labkit_result). The probe therefore starts from a request counted as the first of its window, and waits for a window to end a full period after an answer counted in it (the answer that opened it, for the window it spends), never by those headers. Its last verdict tells a ban (a 429 or a 403 a full period after the window opened) apart from a count it did not expect (a 401 counted as anything but the window's first request).TestThrottleWindow_OpensWithItsFirstCountedRequestandTestNextWindowProblem_TellsABanFromAnUnexpectedCounthold that model with no GitLab, on the three calibrations of the recorded run.The throttle is an instance-wide setting, and the issue decided that no complete run may turn it on: a run cut short before the probe's cleanup would leave it on for every package after it. So the probe runs only when
E2E_AUTH_THROTTLE_PROBE=true, only in Docker mode, and on its own.run-docker-e2e.shnow refuses to start a complete run (E2E_GATE_SKIPS=true) with that switch set, read from the environment, the fileE2E_ENV_FILEnames or the repository.env, the same places it reads the external network switch from, through a newcomplete_run_valueinskip-gate.sh. That lookup sources the file, and the harness reads the same file with godotenv, which also takesKEY: valueandKEY = valueand reads on past a reference to a variable nobody set, so a switch written that way reached the probe without the script seeing it. A complete run the script does not refuse therefore starts withE2E_AUTH_THROTTLE_PROBE=falseexported, and the process environment outranks every file the harness reads.scripts/e2e_skip_gate_test.pycovers the lookup, the refusal, the export and its place in the script. A complete run skips the probe, and the skip gate fails a complete run on any skip nothing declares. So I declared the skip incmd/audit_e2e_coverage/skip_declarations.gounder a new category,opt-in-probe: a probe that changes an instance-wide setting, which a complete run must never run. It is not a setting a run left out, which the table still refuses to excuse.test/e2e/README.mdhas a new section on the probe, andcmd-utilities.md,testing.md, the header ofrun-docker-e2e.shand the test-file counts inCLAUDE.md, the repository map and the testing guide's tree follow the table, the refusal and the export (the testing guide's tree also had stale common and ee counts, now 175 and 48).What GitLab 19.5.0 answered
I ran the probe once, alone, on
gitlab/gitlab-ce:19.5.0-ce.0(revision 4c9e34e1faa) on the Docker stack, and it passed in 425 s. Every request from the test process and from the server children reached GitLab as the same address, according toapi_json.logand nginx's access log. That run was of the probe's first version, before it modelled Labkit's window and before it sent a Bearer token, so itswindow_leftcolumn is the epoch-alignedRateLimit-ResetminusDate, which the probe no longer reads. The probe as committed here has not run against GitLab. It needs one run on its own before this merges, with the command in the README's new section andGITLAB_IMAGE=gitlab/gitlab-ce:19.5.0-ce.0, and that run's lines take the place of the record below.With the throttle off (as a self-managed instance ships it), nothing counts an invalid token. With it on, the answer to OQ-11 is yes: every request carrying an invalid token in
PRIVATE-TOKENis counted once against the address, including the server's own legacy verification of one. The source agrees:Gitlab::Auth::RequestAuthenticatorrescues the invalid token's authentication error to nil, so the request is unauthenticated and the throttle keys on its address. Past the limit the address loses its whole unauthenticated allowance, so an anonymous request is answered 429 too. A token GitLab accepts is unaffected: it was answered 200 and counted by no unauthenticated limit (GitLab's authenticated API limit, which counts per user, was off, as it is by default), and freshly started servers verified and served it throughout. No ban followed: in the next window the invalid token was answered 401 and counted from 1 again.The first version's record has no Bearer token in it, and the OAuth verification sends one. GitLab 19.5 reads
PRIVATE-TOKENand a Bearer token through the same finder (find_user_from_any_authentication_method,lib/gitlab/auth/auth_finders.rbat v19.5.0-ee), whose refusalRequestAuthenticatorrescues the same way, and GitLab.com, which runs its throttle on, counted both alike when I asked it on 2026-10-21 at 01:34 UTC:So the comment, the security page and the specification say what the 19.5 run observed for
PRIVATE-TOKEN, and what GitLab.com and the source say for a Bearer token, until the run above records it on 19.5 too.Every count in the record is the probe's own. Calibration 2's count of 2 is calibration 1, sent 52 s earlier into the same Labkit window: the first version of the probe read the epoch-aligned reset as the window's end, waited it out and sent calibration 2 into the window calibration 1 had opened. The same reading made the run pass by timing. Calibration 3 opened the window the probe spent, which ended at 17:08:02.122, and the wait for its end, the throttled 429's
Retry-Afterplus a second counted from a later 429, sent the next-window request at 17:08:10.489, 8.4 s after it. Started from a calibration with between 60 s and about 110 s left by the epoch, the same arithmetic would have sent it inside the window and reported a ban that did not happen. That is what the window model above fixes. Its counts and verdicts are the first version's, and only the waits, the starting window and the one Bearer request moved; the Bearer request is the fourth counted one of the window, so the request past the limit is still the fifth.What lands on the server
The scenario cannot show what the server does with a token GitLab throttles, because
Env.RefusedAdmissionfails a test whose credential is admitted, and changing the harness is outside this layer. So I checked it by hand on the same stack after the run. I turned the throttle on again, spent the window with direct invalid tokens (429 from the fifth on), started the staged binary in legacy HTTP mode, and postedinitializewith an invented token. GitLab answered the verification 429, which is no verdict on the token, andverifyCredentialadmits a credential it got no verdict for, as it does when GitLab is unreachable. The gate answeredinitialize200 and the pool loggedcreated new entrywithscopes_detected: false. The build took 16 s and five more counted requests, because the entry's token and tier lookups retried against 429s. In OAuth mode the same 429 is an upstream error and the gate answers 503 without charging anything (TestOAuth_ThrottledUpstreamIsNotBlamedOnTheToken).So the rewritten reason says what was observed: a sprayer spends the deployment's unauthenticated allowance with GitLab, not its users' standing. What lands on the server once that allowance is spent is that its verification of every token GitLab does not accept, every invented one among them, gets no verdict: in legacy mode an invented token then gets a pool entry, and in OAuth mode a 503. A token GitLab accepts is still verified and served. The comment on
DistinctTokenBudgetand the security page in English and Spanish say this and name 19.5, and the note ininternal/configand the paragraph in the tenant policy specification beside F-25 say what GitLab answered. The comment no longer calls the distinct count the earliest signal of that spending, which it is not at the defaults: a sprayer at ten or more distinct tokens a minute meets the failure lockout at its tenth, long before the fiftieth. What the distinct count does is bound the sprayer the lockout cannot, one kept under ten a minute, and block for longer each time it is reached (a minute, ten, then an hour), where the lockout forgives after a minute. The security page and the comment also give GitLab.com's limits with their dates: in October 2026 it answered with 500 unauthenticated requests a minute per address, and GitLab announced 60 an hour for unauthenticated traffic from 19 October 2026, which its rate limits page still lists as proposed and not in effect. The budgets' defaults do not hold a sprayer below that 60, since the failure lockout alone lets ten through a minute. F-25 stays onAUB-001andAUB-003until the keying layer of this issue removes it.I think the legacy admission under a throttled GitLab deserves an issue of its own. While GitLab throttles the deployment's address, the legacy door is open to every invented token: each one builds a pool entry (at the pool's bound, evicting another credential's quiet entry,
POL-002) and holds a probe slot (POL-006) through the retries. The two doors answering the same state of GitLab differently is a finding by the specification's own rule, carried by the rows it concerns, andValidaterefuses a finding no issue carries, so the specification in this layer records only what GitLab answered and leaves the two doors' answers to that issue. Once it is filed, it becomes the next finding onADM-001andADM-002, in the specification's findings table and in the paragraph beside F-25, with the keying layer's register change or before it. This layer changes no server code.The package the skip declaration touches
The declaration puts this layer in
cmd/audit_e2e_coverage, so I measured the whole package. On the parent, exported in full (agit archiveleaves outtest/anddocs/, which the package's own tests read),make coverage-mutants PKG=./cmd/audit_e2e_coveragereported Killed 671, Lived 3, Not covered 88, the same as this branch before the third commit. The three that lived were real gaps, both in code this branch does not otherwise touch:report()of the fine-grained fold behaved the same under||as under&&, since no test had sessions without calls or calls without a session line, andenclosing's half-open span was never asked about at its edges, which no site the scanner reads can reach. The third commit asserts both directly, and the package now reports Killed 674, Lived 0, Not covered 88, on the final head too, with the same 88 positions. None of the 88 has a statement counter to be covered by: they sit inswitch { case ... }expressions and package-level initializers, the tool artifactdocs/development/testing/README.mddescribes, so I applied each one by hand to a copy of the tree and ran the package's tests against it. 53 die: 50 undergo testas it runs, and three more whose||turned&&makes a condition that cannot hold, which go vet refuses and the suite fails with vet off as well. The other 35 turn a+between two string literals into a-, which does not compile: 29 inportmap.go, four inruntime_record.goand two instatic.go. gobco reports 1318 of 1318 conditions.How I tested it
TestThrottleAnswer_ReadsTheRateLimitHeaders,TestThrottleWindow_OpensWithItsFirstCountedRequestandTestNextWindowProblem_TellsABanFromAnUnexpectedCount. Against the probe as it stood before the window model they do not build (answer.At undefined,undefined: opensWindow,undefined: windowMargin,undefined: nextWindowProblem); on this branch all three pass with their twelve subtests.scripts/e2e_skip_gate_test.pycovers the lookup, the refusal, a run of the lifecycle script with the switch in the environment and in the named file, and a complete run started with the switch written asKEY: value, asKEY = value, after a reference to an unset variable, and not at all, whose stand-indockerrecords the switch each call inherits. Against the parent's scripts the 33 tests end with 14 failures and 3 errors, those four cases among them, seeing the switch unset rather thanfalse, and the placement check finding no export; on this branch all 33 pass, among them a run not asked to be complete, which keeps the caller'strue. shellcheck is clean on both scripts.audit_e2e_coverage -check-skips -runtime ce: on the parent,1 skipped on ce, 0 declared, 1 undeclaredand exit 1; on this branch,declared (opt-in-probe),1 skipped on ce, 1 declared, 0 undeclared, 0 stale declarationsand exit 0.go test -tags e2e -cof./test/e2e/gitlab/...,go vetandgolangci-lint --build-tags e2eon the CE package,cmd/audit_e2e_coverage,internal/serverpoolandinternal/config, all clean.cmd/audit_e2e_coverage,internal/serverpoolandinternal/configis 100%. The last two change only in comments, and I measured them anyway: gobco 418 of 418 and 424 of 424; gremlins Killed 215, Lived 0, Not covered 6 oninternal/serverpooland Killed 205, Lived 0, Not covered 9 oninternal/config. Applied by hand, all nine ofinternal/config's not covered die and four ofinternal/serverpool's six, the other two being a+between string literals intoken.go.make check-tenancy: 87 rows, 0 findings.make tenancy-code-identity: against the plain parent, 7 sections differ besides the line table, the documented case of a change that inserts lines into the server's source, here comment lines indistinct_token_budget.go(30) andconfig.go(3). Against the parent with blank comment lines inserted at the same positions, whose non-comment lines are the parent's at this branch's line numbers, the two binaries are identical in every allocated section besides.gopclntab.make check-e2e-static,check-test-file-names,check-test-subtests,check-test-goroutines,check-dead-consts,check-md-tablesandcheck-doc-tool-namespass. markdownlint on the changed pages, the em dash check over the branch andscripts/check-doc-links.mjsare clean. The site'si18n:check,facts:check,chips:check,llms:check,buildandlintpass.gen_testing_docs --checkreports the generated counts intesting.mdbehind by the test functions and files this branch adds. That is the committed artifact this stack regenerates once at its top.mainata0e4e6cf6, with no conflict and every commit's patch unchanged. The figures above were measured on the branch as it stood on19746966465. Of the packages they measure, onlyinternal/serverpoolmoved onmainsince, in the place of a doc comment and in its tests (Give tests a dead endpoint that keeps its port, and move the sixteen closed-server sites onto it #1472);mainalso changed files undertest/e2eanddocs/that the tests ofcmd/audit_e2e_coverageread, which is why I ran those tests again here. At this headgo build ./...,go vet,golangci-lint --build-tags e2eandgo testoncmd/audit_e2e_coverage,internal/configandinternal/serverpoolpass, each at 100.0% of statements, the e2e suite compiles, the skip gate's 33 Python tests pass,make check-tenancyreports 87 rows and 0 findings, andmake check-closed-serversandmake check-godocs, the two gatesmaingained since (Wire the closed-server gate into make analyze and CI, and document netfixture #1474, Gate package and exported-symbol doc comments, and fix what the audit reports #1478), pass.