Repository navigation
Bound the transport-source budget's record of the pairs it charged - #1482
Merged
Merged
Conversation
…4 left TestRunMain_TheTree_PassesTheGate and TestProductionRegister_IsTheRegisterAsItIs pin what the gate reads off the real register, and both had fallen behind it since the /64 key of issue 954: AUB-006 refuses with the two blocked refusals every budget before admission carries, so the rules read 95 refusals rather than 93, and its network key names a derivation of its own between the transport source and the refused credential. Both tests failed before this commit; they pass with the two figures as the register has them.
…limit that blocks RecordFailure now answers whether it counted the failure. It does for a new key while the table has room, for a live record counted into and for a lapsed record restarted in place, and it does not for a new key the full table has no room for, which is the one failure the cap of AUB-004 leaves uncounted. Limit returns the count the limiter was built to block at. Nothing reads either yet. The transport-source budget will keep a pair only for a failure its limiter recorded and bound each source's pairs at the limiter's own limit (issue 982), and both need the limiter to say so. The comments on maxTrackedAuthSources and AuthRateLimiter now say what the transport-source budget's block leaves alone: a source arriving once its table is full is never recorded, so it is blocked by neither table.
Behind --trusted-proxy-header the transport-source budget (AUB-002) charges a source once per distinct failing primary key in a window, and remembers each (source, key) pair it charged so as not to charge it again. Both halves are the caller's to choose, and only the sweep removed a pair. A blocked source adds none, but once as many sources as the limiter tracks had failed inside a window, a source arriving then was never recorded and so never blocked, and the budget kept a pair for every key it failed with: 100000 keys left 100000 pairs. Issue 982 decided two bounds, and this is them. - A pair is kept only when the limiter recorded the failure. A source the full limiter did not record keeps none. The pair kept for an unrecorded failure also stood in for a charge that never happened: the same key failing again once the limiter had room went uncounted. It is now charged as a first failure. The limiter is asked under the budget's lock, so two concurrent failures of one fresh pair are still charged once. - A source holds at most the limiter's own limit in pairs. At that bound a fresh key takes the place of the source's own oldest pair and is charged as before, so a source released from its block is charged for the keys it fails with while the pairs that blocked it are still live, and no other source's pair is ever taken (INV-005). A lapsed pair is re-opened in place and takes no room. Keeping a pair only for a recorded failure does not bound the sources by itself. The limiter's full table lets a lapsed record go at most an eighth of a window after it lapses and takes a new source in its place, while the sweep forgets a pair only every five minutes (AUB-005), so a new table's worth of sources stayed for every window between two sweeps: about five at the default window, and more the shorter the window. Once the budget indexes as many sources as the limiter's table can hold (AUB-004), a new source now first makes it forget every source none of whose pairs is live, at most once every eighth of a window, as the limiter sweeps its own. Past that many, the sources it holds were all charged within one window and an eighth, while the limiter held a record of each, so they are at most three times what that table holds, whatever the window. The flat map keeps its key, the source and the key joined by a NUL, and an index beside it of each source's pairs and last charge bounds the source's share, finds its oldest pair, and finds the sources with no live pair without reading their pairs; the sweep drops a source from it with its last pair. Nine of the tests that pin the bounds read only the flat map, so they compile against the parent and fail there, and TestTransportBudget_ChargesOncePerKeyPerWindow and TestTransportBudget_Cleanup_ForgetsLapsedPairsOnly pass unchanged. TestTransportBudget_Charge_ChargesAConcurrentFreshPairOnce passes on the parent too, which stored the pair before it asked the limiter, and fails when the limiter is asked outside the lock. The register follows. AUB-002 records issue 982 among its decisions, declares that at capacity it reclaims the source's own holding (ReclaimOwnThenRefuse: the source's oldest pair for a fresh key, and no place at all for a source the limiter did not record), names keepLocked, forgetLapsedSourcesLocked and reclaimOldestLocked among its enforcing sites, and carries F-16 alone. AUB-004 declares the budget's alias of its value. F-35 stays in the list, filed as issue 982's, and joins the answered set TestDecisions_AFindingNoRowCarries_IsAnsweredByItsIssue names.
The specification's F-35 paragraph said only the sweep bounded the transport-source budget's map of charged pairs. It now says how issue 982 answered it: a source's own oldest pair reclaimed at its bound, a pair kept only for a failure the limiter recorded, and the sources with no live pair forgotten once the budget indexes as many as the limiter's table can hold, which keeps them to three times that table whatever the window. It also says what was left alone: a source arriving once the limiter's table is full is still blocked by neither table, which is the /48 aggregate's and issue 958's. F-35 joins the answered findings, which are fifteen now.
There was a problem hiding this comment.
jmrplens has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 59 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Closes #982.
This is the third layer of the authentication-budget stack, on top of the /64 key of #954. It bounds the record the transport-source budget keeps of the pairs it charged, as the decisions of 2026-09-27 and 2026-10-17 on the issue set out, and moves
AUB-002off F-35.What was wrong
Behind
--trusted-proxy-headerthe transport-source budget (AUB-002) charges a transport source once per distinct failing primary key inside its window, and to know which keys it already charged it keeps a map of (source, key) pairs,transportBudget.charged. Both halves are the caller's to choose, and nothing but the five-minute sweep removed a pair. A blocked source adds none, but the limiter under the budget records at most 4096 sources (AUB-004) and at that cap stops recording new ones. A source arriving then is never blocked by this budget, and the budget went on keeping a pair for every key it failed with.What changed
AuthRateLimiter.RecordFailurenow reports whether it counted the failure: true for a new key with room, for a live record counted into and for a lapsed record restarted in place, false for the one failure the cap leaves uncounted, a new key the full table has no room for.AuthRateLimiter.Limitreturns the count the limiter blocks at. Nothing else reads either.transportBudget.chargekeeps a pair only when its limiter recorded the failure, so a source the full limiter never recorded keeps no pair at all. That also removes a quieter defect: the pair kept for an unrecorded failure stood in for a charge that never happened, so the same key failing again once the limiter had room was deduplicated against it and went uncounted. It is now charged as a first failure, whichINV-007allows, since the caller caused it. I moved the limiter call under the budget's own lock, so two concurrent failures of one fresh pair are still charged once, as they were when the pair was stored before the limiter was asked.A source holds at most its limiter's limit in pairs (500 in production,
tenancy.TransportSourceDistinctKeys). At that bound a fresh key takes the place of the source's own oldest pair and is charged as before, as decided on the issue on 2026-10-17. Stopping at the bound instead would let a source just released from its block fail with fresh keys uncharged while the pairs that blocked it are still held, and reclaiming only the source's own pair never takes another source's (INV-005). A key whose pair lapsed is re-opened in place and takes no room. I read the bound off the limiter rather than off the constant so that it is the count the limiter blocks at for every limiter, the tests' small ones included; in production the two are the same value.Keeping a pair only for a recorded failure does not bound the sources by itself, although the decision of 2026-10-17 reads as though it did. The limiter's full table lets a lapsed record go at most an eighth of a window after it lapses and takes a new source in its place, while the budget forgot a pair only at its five-minute sweep (
AUB-005), so a new table's worth of sources stayed for every window between two sweeps: four tables' worth at the default window, measured below, and more the shorter the window, sinceAUB-001accepts any positive one. So once the budget indexes as many sources as the limiter's table can hold, a new source first makes it forget every source none of whose pairs is live, at most once every eighth of a window, as the limiter sweeps its own full table. Past that many, the sources it holds were all charged within one window and an eighth, each while the limiter held a record of it, and a record stays in that table for a window at least, so they are at most three times what the table holds (12288 sources of at most 500 pairs each), whatever the window. That is the decided bound times three rather than the decided bound itself: holding it to one table exactly would cost a pass over the index for every new source at the ceiling, which is the amplifier the limiter's own sweep is throttled against. The pass reads each source's last charge rather than its pairs, and the pairs of a source it forgets are deleted once.The flat map keeps its key format, the source and the key joined by a NUL, so the tests that read it directly are untouched. Beside it a per-source index of the pairs each source holds and when it was last charged bounds the source's share, finds its oldest pair, and finds the sources with no live pair. Finding the oldest scans at most 500 entries and builds no string, and only a source at its bound pays for it: one never blocked whose lapsed pairs the sweep has not reached yet, one whose block has lifted while the pairs that earned it are still held, live or lapsed and not yet swept, or one charged by a request that passed the gates before the block. The sweep walks the index, drops each lapsed pair from both, and drops a source with its last pair.
The comments on
transportFailureLimit,transportBudget,charge,maxTrackedAuthSources,AuthRateLimiter,RecordFailureandLimitnow say what is bounded and what is not, instead of claiming the map cannot grow because a blocked source stops charging.The register and the specification
AUB-002records issue 982 among its decisions, declaresAtCapacity: ReclaimOwnThenRefuse(the source's own oldest pair for a fresh key, and no place for a source the limiter did not record), namestransportBudget.keepLocked,transportBudget.forgetLapsedSourcesLockedandtransportBudget.reclaimOldestLockedamong its enforcing sites, and carries F-16 alone.AUB-004declarestransportTrackedSources, the budget's alias of its value, which is the index size at which the budget forgets the sources with no live pair. F-35 stays in the list, filed as issue 982's, and joins the answered setTestDecisions_AFindingNoRowCarries_IsAnsweredByItsIssuenames. The specification's F-35 paragraph now says how it was answered, both bounds and the factor of three included, and the count of answered findings is fifteen.What this leaves alone is the saturation itself: a source arriving once the limiter's table is full is still blocked by neither table. As the issue's second decision says, that belongs to the /48 aggregate of #954 and the capacity rules of #958.
cmd/audit_tenancy/declarations.goneeded no change: the one new package-level value is the declared alias,RecordFailureis still called only from the declared charge helpers, andmake check-tenancyreports 0 findings over 88 rows and 370 declared sites.Validation checklist for the change to
AUB-002: VAL-001 to VAL-005, VAL-007, VAL-009 and VAL-011 are unchanged (Admit, keyed on the transport source, class A, a budget rather than a share, no process partner, the same 429 refusal, the constant 500 whose zero no operator sets, and nothing on stdio, where the budget does not exist). VAL-006 is the change: the row now states its capacity behavior and still marks the structure as a table on caller-chosen values, and the index it keeps is bounded across sources byAUB-004's value as above. VAL-008: what is charged is still a judged failure, a credential GitLab refused or one that was missing, and two such failures are now charged where the parent deduplicated them, both the caller's own doing (INV-007): the next failure of a key whose earlier failure the full limiter did not record, and a fresh failure of a key whose pair was reclaimed at the source's bound while it was still live. Forgetting a source drops no live pair, so it charges nothing new. VAL-010: telemetry is unchanged, the refusal is still counted undertransport_sourceand nothing about a source or a key is recorded. VAL-012: the change claims a memory bound rather than protection of one population from another, so the measurements are the issue's own and the rotation the cross-source bound answers, below.How it was tested
The tests came first. Eleven new tests in
cmd/server/auth_gate_test.gopin the bounds, and nine of them read only the flat map, so they compile against the parent as it stood before the stack was rebased ontomain(c1835d5b2, now2dbd166fd) and fail there:TestTransportBudget_Charge_KeepsNoPairForASourceTheFullLimiterDidNotRecord: 4096 sources fill the limiter, then one more fails with 100000 keys. Parent: 100000 pairs for it, 104096 in all. Now: none, 4096 in all, and the source is not blocked.TestTransportBudget_Charge_BlocksAtTheLimitAndKeepsNoMoreThanItsPairs: blocked at 500 keys as before, then 500 more. Parent: 1000 pairs. Now: 500, and another source's pair is kept.TestTransportBudget_Charge_AfterTheBlockLifts_StillChargesAFreshKey(synctest): after the block lifts with one of its two pairs still live, a fresh key reclaims the source's own oldest pair and the next one blocks it again. Parent: 3 and then 4 pairs.TestTransportBudget_Charge_ALapsedPairReopensInPlace(synctest): re-opening a lapsed pair keeps both pairs, and the next fresh key takes the oldest. Parent: 3 pairs.TestTransportBudget_Charge_ChargesAFailureOnceTheFullLimiterHasRoom(synctest): an unrecorded failure keeps no pair and blocks nothing, and the same key is charged once the limiter has room. Parent: the pair is kept and the later failure goes uncharged.TestTransportBudget_Cleanup_KeepsTheRoomOfAPairItKeeps(synctest): a pair the sweep keeps still counts against its source's room. Parent: 3 pairs.TestTransportBudget_Charge_TurningTheLimiterOverDoesNotStackSources(synctest): four generations of 4096 new sources, each arriving once the last one's records and pairs have lapsed. Parent: 8192 sources after the second generation. Now: 4096 after each.TestTransportBudget_Charge_HoldsAtMostThreeTablesOfSources(synctest): six generations, each source charged its last key a second before its record lapses, so its pair outlives the record. Parent: 16384 sources after the fourth generation, past the 12288 of three tables. Now: at most 8192.TestTransportBudget_Charge_ForgetsLapsedSourcesAnEighthOfAWindowApart(synctest): a new source at the ceiling forgets the lapsed sources, the next one an eighth of a window less a nanosecond later does not, the one exactly an eighth later does, and a source last charged exactly a window ago is kept. Parent: 4097 sources where the first sweep leaves 4.The tenth,
TestTransportBudget_Cleanup_ForgetsASourceWithNoPairLeft, reads the per-source index this change adds, so it cannot compile against the parent; it pins that the sweep leaves no empty source behind. The eleventh,TestTransportBudget_Charge_ChargesAConcurrentFreshPairOnce, pins what asking the limiter under the budget's lock buys: 32 concurrent failures of one fresh pair, 200 times, block no source whose limit is two. It passes on the parent, which stored the pair before it asked the limiter, and failed five runs out of five (14 to 26 sources of 200 blocked) on a copy of this branch that asks the limiter after releasing the lock. Ininternal/serverpool,TestAuthRateLimiter_RecordFailure_ReportsWhetherItRecordedandTestAuthRateLimiter_Limit_IsTheCountThatBlocksfail to compile against the parent (RecordFailurereturned nothing andLimitdid not exist). Ininternal/tenancy,TestDecisions_TransportSourcePairs_AreBoundedByIssue982and the answered-set expectation ofTestDecisions_AFindingNoRowCarries_IsAnsweredByItsIssuefail against the parent's register. All of them pass on this branch, andTestTransportBudget_ChargesOncePerKeyPerWindowandTestTransportBudget_Cleanup_ForgetsLapsedPairsOnlypass unchanged.The issue's measurement, run with a throwaway test against the parent and against this branch (never committed). With the limiter's table full, one more source failing with N distinct keys:
With the table not full, the source is blocked after 500 distinct keys on both, and the map holds N pairs on the parent (1000, 10000 and 100000) and 500 on this branch.
The rotation the cross-source bound answers, measured the same way with a limit of 10 so that the pairs stay small, and inside one sweep interval, since the five-minute sweep resets both. Each generation is 4096 new sources failing with 10 keys each: in the first rotation a generation arrives a window and an eighth and a second after the last, in the second each source fails with its last key a second before its record lapses and the next generation arrives two seconds later. Keeping a pair only for a recorded failure, without forgetting sources, measures exactly as the parent does here.
Checks, all on this branch:
go test ./cmd/server ./internal/serverpool ./internal/tenancy ./cmd/audit_tenancy -count=1: pass, each at 100.0% of statements;go test -raceover the budget, gate and bearer guard tests: pass;cmd/bench_resourcesandinternal/finegrained, the other readers of the register and the pool: pass.make coverage-conditions(gobco,GOBCO_GATE=all):cmd/server2316/2316,internal/serverpool418/418,internal/tenancy450/450.make coverage-mutantson the files this changes, the others excluded withGREMLINS_FLAGS=--exclude-files:cmd/server/auth_gate.go112 killed, 0 lived, and 8 not covered, which are mutations in package-level constant initializers (the two negative error codes and the string concatenations of the two missing-token messages) that carry no statement counter, the tool artifactdocs/development/testing/README.mddescribes, and the parent reports the same way;internal/serverpool/rate_limit.go22 killed, 0 lived, 0 not covered;internal/tenancy/decisions_admit.gohas no mutable expression.make check-tenancy: 0 findings.make test-e2e-http: 678 tests, 8 skipped for want of Docker, andTestGate_RotatingProxyHeader_IsBoundedByTheTransportSourceandTestGate_BlockedAddressStillServesAnAdmittedCredentialpass unedited.golangci-lint run --build-tags e2eandgolangci-lint fmt --diffon the four changed packages, the e2e compile,check-test-subtests,check-test-goroutines,check-test-file-names, markdownlint on the specification,format_md_tables --check,check-doc-linksandmake check-em-dash: clean.I then rebased the stack onto
mainata0e4e6cf6, with no conflict and every commit's patch unchanged, so the checks above were run on the branch as it stood on19746966465. Of the packages this layer touches,mainmovedcmd/serverin two test files andinternal/serverpoolin its tests and the place of one doc comment (#1472) andcmd/audit_tenancyin its scaffolding (#1479). At this headgo build ./...,go vet,golangci-lint --build-tags e2eandgo teston the four packages pass, each still at 100.0% of statements;make check-tenancystill reports 0 findings over 88 rows and 370 declared sites; andmake check-closed-serversandmake check-godocs, the two gatesmaingained since (#1474, #1478), pass. The gobco figures count the conditions of codemaindid not change, so they stand; I did not repeat the mutation runs.