Skip to content

Bound the transport-source budget's record of the pairs it charged - #1482

Merged
jmrplens merged 4 commits into
issue-954-budget-keyfrom
issue-982-transport-bound
Oct 22, 2026
Merged

jmrplens merged 4 commits into
issue-954-budget-keyfrom
issue-982-transport-bound

Conversation

@jmrplens

@jmrplens jmrplens commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Closes #982.

This is the third layer of the authentication-budget stack, on top of the /64 key of #954. It bounds the record the transport-source budget keeps of the pairs it charged, as the decisions of 2026-09-27 and 2026-10-17 on the issue set out, and moves AUB-002 off F-35.

What was wrong

Behind --trusted-proxy-header the transport-source budget (AUB-002) charges a transport source once per distinct failing primary key inside its window, and to know which keys it already charged it keeps a map of (source, key) pairs, transportBudget.charged. Both halves are the caller's to choose, and nothing but the five-minute sweep removed a pair. A blocked source adds none, but the limiter under the budget records at most 4096 sources (AUB-004) and at that cap stops recording new ones. A source arriving then is never blocked by this budget, and the budget went on keeping a pair for every key it failed with.

What changed

AuthRateLimiter.RecordFailure now reports whether it counted the failure: true for a new key with room, for a live record counted into and for a lapsed record restarted in place, false for the one failure the cap leaves uncounted, a new key the full table has no room for. AuthRateLimiter.Limit returns the count the limiter blocks at. Nothing else reads either.

transportBudget.charge keeps a pair only when its limiter recorded the failure, so a source the full limiter never recorded keeps no pair at all. That also removes a quieter defect: the pair kept for an unrecorded failure stood in for a charge that never happened, so the same key failing again once the limiter had room was deduplicated against it and went uncounted. It is now charged as a first failure, which INV-007 allows, since the caller caused it. I moved the limiter call under the budget's own lock, so two concurrent failures of one fresh pair are still charged once, as they were when the pair was stored before the limiter was asked.

A source holds at most its limiter's limit in pairs (500 in production, tenancy.TransportSourceDistinctKeys). At that bound a fresh key takes the place of the source's own oldest pair and is charged as before, as decided on the issue on 2026-10-17. Stopping at the bound instead would let a source just released from its block fail with fresh keys uncharged while the pairs that blocked it are still held, and reclaiming only the source's own pair never takes another source's (INV-005). A key whose pair lapsed is re-opened in place and takes no room. I read the bound off the limiter rather than off the constant so that it is the count the limiter blocks at for every limiter, the tests' small ones included; in production the two are the same value.

Keeping a pair only for a recorded failure does not bound the sources by itself, although the decision of 2026-10-17 reads as though it did. The limiter's full table lets a lapsed record go at most an eighth of a window after it lapses and takes a new source in its place, while the budget forgot a pair only at its five-minute sweep (AUB-005), so a new table's worth of sources stayed for every window between two sweeps: four tables' worth at the default window, measured below, and more the shorter the window, since AUB-001 accepts any positive one. So once the budget indexes as many sources as the limiter's table can hold, a new source first makes it forget every source none of whose pairs is live, at most once every eighth of a window, as the limiter sweeps its own full table. Past that many, the sources it holds were all charged within one window and an eighth, each while the limiter held a record of it, and a record stays in that table for a window at least, so they are at most three times what the table holds (12288 sources of at most 500 pairs each), whatever the window. That is the decided bound times three rather than the decided bound itself: holding it to one table exactly would cost a pass over the index for every new source at the ceiling, which is the amplifier the limiter's own sweep is throttled against. The pass reads each source's last charge rather than its pairs, and the pairs of a source it forgets are deleted once.

The flat map keeps its key format, the source and the key joined by a NUL, so the tests that read it directly are untouched. Beside it a per-source index of the pairs each source holds and when it was last charged bounds the source's share, finds its oldest pair, and finds the sources with no live pair. Finding the oldest scans at most 500 entries and builds no string, and only a source at its bound pays for it: one never blocked whose lapsed pairs the sweep has not reached yet, one whose block has lifted while the pairs that earned it are still held, live or lapsed and not yet swept, or one charged by a request that passed the gates before the block. The sweep walks the index, drops each lapsed pair from both, and drops a source with its last pair.

The comments on transportFailureLimit, transportBudget, charge, maxTrackedAuthSources, AuthRateLimiter, RecordFailure and Limit now say what is bounded and what is not, instead of claiming the map cannot grow because a blocked source stops charging.

The register and the specification

AUB-002 records issue 982 among its decisions, declares AtCapacity: ReclaimOwnThenRefuse (the source's own oldest pair for a fresh key, and no place for a source the limiter did not record), names transportBudget.keepLocked, transportBudget.forgetLapsedSourcesLocked and transportBudget.reclaimOldestLocked among its enforcing sites, and carries F-16 alone. AUB-004 declares transportTrackedSources, the budget's alias of its value, which is the index size at which the budget forgets the sources with no live pair. F-35 stays in the list, filed as issue 982's, and joins the answered set TestDecisions_AFindingNoRowCarries_IsAnsweredByItsIssue names. The specification's F-35 paragraph now says how it was answered, both bounds and the factor of three included, and the count of answered findings is fifteen.

What this leaves alone is the saturation itself: a source arriving once the limiter's table is full is still blocked by neither table. As the issue's second decision says, that belongs to the /48 aggregate of #954 and the capacity rules of #958.

cmd/audit_tenancy/declarations.go needed no change: the one new package-level value is the declared alias, RecordFailure is still called only from the declared charge helpers, and make check-tenancy reports 0 findings over 88 rows and 370 declared sites.

Validation checklist for the change to AUB-002: VAL-001 to VAL-005, VAL-007, VAL-009 and VAL-011 are unchanged (Admit, keyed on the transport source, class A, a budget rather than a share, no process partner, the same 429 refusal, the constant 500 whose zero no operator sets, and nothing on stdio, where the budget does not exist). VAL-006 is the change: the row now states its capacity behavior and still marks the structure as a table on caller-chosen values, and the index it keeps is bounded across sources by AUB-004's value as above. VAL-008: what is charged is still a judged failure, a credential GitLab refused or one that was missing, and two such failures are now charged where the parent deduplicated them, both the caller's own doing (INV-007): the next failure of a key whose earlier failure the full limiter did not record, and a fresh failure of a key whose pair was reclaimed at the source's bound while it was still live. Forgetting a source drops no live pair, so it charges nothing new. VAL-010: telemetry is unchanged, the refusal is still counted under transport_source and nothing about a source or a key is recorded. VAL-012: the change claims a memory bound rather than protection of one population from another, so the measurements are the issue's own and the rotation the cross-source bound answers, below.

How it was tested

The tests came first. Eleven new tests in cmd/server/auth_gate_test.go pin the bounds, and nine of them read only the flat map, so they compile against the parent as it stood before the stack was rebased onto main (c1835d5b2, now 2dbd166fd) and fail there:

  • TestTransportBudget_Charge_KeepsNoPairForASourceTheFullLimiterDidNotRecord: 4096 sources fill the limiter, then one more fails with 100000 keys. Parent: 100000 pairs for it, 104096 in all. Now: none, 4096 in all, and the source is not blocked.
  • TestTransportBudget_Charge_BlocksAtTheLimitAndKeepsNoMoreThanItsPairs: blocked at 500 keys as before, then 500 more. Parent: 1000 pairs. Now: 500, and another source's pair is kept.
  • TestTransportBudget_Charge_AfterTheBlockLifts_StillChargesAFreshKey (synctest): after the block lifts with one of its two pairs still live, a fresh key reclaims the source's own oldest pair and the next one blocks it again. Parent: 3 and then 4 pairs.
  • TestTransportBudget_Charge_ALapsedPairReopensInPlace (synctest): re-opening a lapsed pair keeps both pairs, and the next fresh key takes the oldest. Parent: 3 pairs.
  • TestTransportBudget_Charge_ChargesAFailureOnceTheFullLimiterHasRoom (synctest): an unrecorded failure keeps no pair and blocks nothing, and the same key is charged once the limiter has room. Parent: the pair is kept and the later failure goes uncharged.
  • TestTransportBudget_Cleanup_KeepsTheRoomOfAPairItKeeps (synctest): a pair the sweep keeps still counts against its source's room. Parent: 3 pairs.
  • TestTransportBudget_Charge_TurningTheLimiterOverDoesNotStackSources (synctest): four generations of 4096 new sources, each arriving once the last one's records and pairs have lapsed. Parent: 8192 sources after the second generation. Now: 4096 after each.
  • TestTransportBudget_Charge_HoldsAtMostThreeTablesOfSources (synctest): six generations, each source charged its last key a second before its record lapses, so its pair outlives the record. Parent: 16384 sources after the fourth generation, past the 12288 of three tables. Now: at most 8192.
  • TestTransportBudget_Charge_ForgetsLapsedSourcesAnEighthOfAWindowApart (synctest): a new source at the ceiling forgets the lapsed sources, the next one an eighth of a window less a nanosecond later does not, the one exactly an eighth later does, and a source last charged exactly a window ago is kept. Parent: 4097 sources where the first sweep leaves 4.

The tenth, TestTransportBudget_Cleanup_ForgetsASourceWithNoPairLeft, reads the per-source index this change adds, so it cannot compile against the parent; it pins that the sweep leaves no empty source behind. The eleventh, TestTransportBudget_Charge_ChargesAConcurrentFreshPairOnce, pins what asking the limiter under the budget's lock buys: 32 concurrent failures of one fresh pair, 200 times, block no source whose limit is two. It passes on the parent, which stored the pair before it asked the limiter, and failed five runs out of five (14 to 26 sources of 200 blocked) on a copy of this branch that asks the limiter after releasing the lock. In internal/serverpool, TestAuthRateLimiter_RecordFailure_ReportsWhetherItRecorded and TestAuthRateLimiter_Limit_IsTheCountThatBlocks fail to compile against the parent (RecordFailure returned nothing and Limit did not exist). In internal/tenancy, TestDecisions_TransportSourcePairs_AreBoundedByIssue982 and the answered-set expectation of TestDecisions_AFindingNoRowCarries_IsAnsweredByItsIssue fail against the parent's register. All of them pass on this branch, and TestTransportBudget_ChargesOncePerKeyPerWindow and TestTransportBudget_Cleanup_ForgetsLapsedPairsOnly pass unchanged.

The issue's measurement, run with a throwaway test against the parent and against this branch (never committed). With the limiter's table full, one more source failing with N distinct keys:

N Map length, parent Map length, this branch
10 4106 4096
1000 5096 4096
10000 14096 4096
100000 104096 4096

With the table not full, the source is blocked after 500 distinct keys on both, and the map holds N pairs on the parent (1000, 10000 and 100000) and 500 on this branch.

The rotation the cross-source bound answers, measured the same way with a limit of 10 so that the pairs stay small, and inside one sweep interval, since the five-minute sweep resets both. Each generation is 4096 new sources failing with 10 keys each: in the first rotation a generation arrives a window and an eighth and a second after the last, in the second each source fails with its last key a second before its record lapses and the next generation arrives two seconds later. Keeping a pair only for a recorded failure, without forgetting sources, measures exactly as the parent does here.

Generation Rotation, parent Rotation, this branch Late last key, parent Late last key, this branch
0 40960 40960 40960 40960
1 81920 40960 81920 81920
2 122880 40960 122880 81920
3 163840 40960 163840 81920

Checks, all on this branch:

  • go test ./cmd/server ./internal/serverpool ./internal/tenancy ./cmd/audit_tenancy -count=1: pass, each at 100.0% of statements; go test -race over the budget, gate and bearer guard tests: pass; cmd/bench_resources and internal/finegrained, the other readers of the register and the pool: pass.
  • make coverage-conditions (gobco, GOBCO_GATE=all): cmd/server 2316/2316, internal/serverpool 418/418, internal/tenancy 450/450.
  • make coverage-mutants on the files this changes, the others excluded with GREMLINS_FLAGS=--exclude-files: cmd/server/auth_gate.go 112 killed, 0 lived, and 8 not covered, which are mutations in package-level constant initializers (the two negative error codes and the string concatenations of the two missing-token messages) that carry no statement counter, the tool artifact docs/development/testing/README.md describes, and the parent reports the same way; internal/serverpool/rate_limit.go 22 killed, 0 lived, 0 not covered; internal/tenancy/decisions_admit.go has no mutable expression.
  • make check-tenancy: 0 findings.
  • make test-e2e-http: 678 tests, 8 skipped for want of Docker, and TestGate_RotatingProxyHeader_IsBoundedByTheTransportSource and TestGate_BlockedAddressStillServesAnAdmittedCredential pass unedited.
  • golangci-lint run --build-tags e2e and golangci-lint fmt --diff on the four changed packages, the e2e compile, check-test-subtests, check-test-goroutines, check-test-file-names, markdownlint on the specification, format_md_tables --check, check-doc-links and make check-em-dash: clean.

I then rebased the stack onto main at a0e4e6cf6, with no conflict and every commit's patch unchanged, so the checks above were run on the branch as it stood on 19746966465. Of the packages this layer touches, main moved cmd/server in two test files and internal/serverpool in its tests and the place of one doc comment (#1472) and cmd/audit_tenancy in its scaffolding (#1479). At this head go build ./..., go vet, golangci-lint --build-tags e2e and go test on the four packages pass, each still at 100.0% of statements; make check-tenancy still reports 0 findings over 88 rows and 370 declared sites; and make check-closed-servers and make check-godocs, the two gates main gained since (#1474, #1478), pass. The gobco figures count the conditions of code main did not change, so they stand; I did not repeat the mutation runs.

…4 left

TestRunMain_TheTree_PassesTheGate and
TestProductionRegister_IsTheRegisterAsItIs pin what the gate reads off
the real register, and both had fallen behind it since the /64 key of
issue 954: AUB-006 refuses with the two blocked refusals every budget
before admission carries, so the rules read 95 refusals rather than 93,
and its network key names a derivation of its own between the transport
source and the refused credential. Both tests failed before this
commit; they pass with the two figures as the register has them.
…limit that blocks

RecordFailure now answers whether it counted the failure. It does for
a new key while the table has room, for a live record counted into and
for a lapsed record restarted in place, and it does not for a new key
the full table has no room for, which is the one failure the cap of
AUB-004 leaves uncounted. Limit returns the count the limiter was built
to block at.

Nothing reads either yet. The transport-source budget will keep a pair
only for a failure its limiter recorded and bound each source's pairs at
the limiter's own limit (issue 982), and both need the limiter to say
so. The comments on maxTrackedAuthSources and AuthRateLimiter now say
what the transport-source budget's block leaves alone: a source
arriving once its table is full is never recorded, so it is blocked by
neither table.
Behind --trusted-proxy-header the transport-source budget (AUB-002)
charges a source once per distinct failing primary key in a window, and
remembers each (source, key) pair it charged so as not to charge it
again. Both halves are the caller's to choose, and only the sweep
removed a pair. A blocked source adds none, but once as many sources as
the limiter tracks had failed inside a window, a source arriving then
was never recorded and so never blocked, and the budget kept a pair for
every key it failed with: 100000 keys left 100000 pairs.

Issue 982 decided two bounds, and this is them.

- A pair is kept only when the limiter recorded the failure. A source
  the full limiter did not record keeps none. The pair kept for an
  unrecorded failure also stood in for a charge that never happened:
  the same key failing again once the limiter had room went uncounted.
  It is now charged as a first failure. The limiter is asked under the
  budget's lock, so two concurrent failures of one fresh pair are still
  charged once.
- A source holds at most the limiter's own limit in pairs. At that
  bound a fresh key takes the place of the source's own oldest pair and
  is charged as before, so a source released from its block is charged
  for the keys it fails with while the pairs that blocked it are still
  live, and no other source's pair is ever taken (INV-005). A lapsed
  pair is re-opened in place and takes no room.

Keeping a pair only for a recorded failure does not bound the sources
by itself. The limiter's full table lets a lapsed record go at most an
eighth of a window after it lapses and takes a new source in its place,
while the sweep forgets a pair only every five minutes (AUB-005), so a
new table's worth of sources stayed for every window between two
sweeps: about five at the default window, and more the shorter the
window. Once the budget indexes as many sources as the limiter's table
can hold (AUB-004), a new source now first makes it forget every source
none of whose pairs is live, at most once every eighth of a window, as
the limiter sweeps its own. Past that many, the sources it holds were
all charged within one window and an eighth, while the limiter held a
record of each, so they are at most three times what that table holds,
whatever the window.

The flat map keeps its key, the source and the key joined by a NUL, and
an index beside it of each source's pairs and last charge bounds the
source's share, finds its oldest pair, and finds the sources with no
live pair without reading their pairs; the sweep drops a source from it
with its last pair. Nine of the tests that pin the bounds read only the
flat map, so they compile against the parent and fail there, and
TestTransportBudget_ChargesOncePerKeyPerWindow and
TestTransportBudget_Cleanup_ForgetsLapsedPairsOnly pass unchanged.
TestTransportBudget_Charge_ChargesAConcurrentFreshPairOnce passes on the
parent too, which stored the pair before it asked the limiter, and
fails when the limiter is asked outside the lock.

The register follows. AUB-002 records issue 982 among its decisions,
declares that at capacity it reclaims the source's own holding
(ReclaimOwnThenRefuse: the source's oldest pair for a fresh key, and no
place at all for a source the limiter did not record), names
keepLocked, forgetLapsedSourcesLocked and reclaimOldestLocked among its
enforcing sites, and carries F-16 alone. AUB-004 declares the budget's
alias of its value. F-35 stays in the list, filed as issue 982's, and
joins the answered set
TestDecisions_AFindingNoRowCarries_IsAnsweredByItsIssue names.
The specification's F-35 paragraph said only the sweep bounded the
transport-source budget's map of charged pairs. It now says how issue
982 answered it: a source's own oldest pair reclaimed at its bound, a
pair kept only for a failure the limiter recorded, and the sources with
no live pair forgotten once the budget indexes as many as the limiter's
table can hold, which keeps them to three times that table whatever the
window. It also says what was left alone: a source arriving once the
limiter's table is full is still blocked by neither table, which is the
/48 aggregate's and issue 958's. F-35 joins the answered findings,
which are fifteen now.
@sourcery-ai

sourcery-ai Bot commented Oct 21, 2026

Copy link
Copy Markdown

Sorry @jmrplens, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 4 days and 20 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

jmrplens has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository: jmrplens/gitlab-mcp-server/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 7697e8b5-0b4b-463e-a9a6-b67265887d89
📥 Commits

Reviewing files that changed from the base of the PR and between 2dbd166 and 5db3a17.

📒 Files selected for processing: 8
  • cmd/audit_tenancy/main_test.go
  • cmd/server/auth_gate.go
  • cmd/server/auth_gate_test.go
  • docs/development/tenant-policy-spec.md
  • internal/serverpool/rate_limit.go
  • internal/serverpool/rate_limit_test.go
  • internal/tenancy/decisions_admit.go
  • internal/tenancy/decisions_test.go
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@jmrplens
jmrplens added this pull request to stack #1486 October 21, 2026 23:21
@jmrplens jmrplens added this to the 3.4.0 milestone Oct 21, 2026
@jmrplens jmrplens changed the title issue 982 transport bound Bound the transport-source budget's record of the pairs it charged Oct 21, 2026
@github-actions github-actions Bot added security Security-related issue transport stdio and HTTP transports, the server process, and the transport e2e modules tooling The audit and generator commands under cmd/, and the Makefile targets that run them labels Oct 21, 2026
@sonarqubecloud

Copy link
Copy Markdown

@jmrplens
jmrplens merged commit fcf6ea0 into main Oct 22, 2026
39 checks passed
@jmrplens
jmrplens deleted the issue-982-transport-bound branch October 22, 2026 03:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

security Security-related issue tooling The audit and generator commands under cmd/, and the Makefile targets that run them transport stdio and HTTP transports, the server process, and the transport e2e modules v3.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The transport-source budget remembers every (source, key) pair it charges, and only its sweep bounds that record

1 participant