Repository navigation
Measure the OAuth verification ceiling with an OAuth driver for bench-fairness - #1091
Conversation
There was a problem hiding this comment.
Sorry @jmrplens, your pull request is larger than the review limit of 150,000 diff characters
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 59 minutes. View limit detailsLimit details: You’ve used the included review currently available. Review configuration: ⚙️ Run configurationConfiguration used: Repository: jmrplens/gitlab-mcp-server/.coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (38)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
d849aa4 to
7afa8c0
Compare
bench-fairness could not measure ADM-014, the sixteen verification slots issue 950 put in front of GitLab, so its cost to legitimate credentials was stated and not measured. The mode now knows a bound named oauth-verification. The ceiling has no switch and must not grow one, since a flag that turns off a security bound is a flag somebody sets in production. So both arms start with the same flags and the arm without the ceiling runs a second build of the same checkout, made with the go command's -overlay, in which the one declaration that sizes the slots is replaced. The tree on disk is never touched, the record names the line that arm ran, and a checkout that no longer declares it, or declares it twice, stops the run rather than comparing the binary with itself. The stand-in GitLab answers the identity request for every token but an invented one, which it answers 401, answers both scope introspection endpoints 404, holds each of those requests for a configurable round trip (-fairness-upstream-delay), and counts how many it answered and how many it held at once. The flood presents a new invented token on every request, forwarded for an address of its own, from as many transport sources in 127.2.0.0/16 as keep each one under half the transport-source failure budget, which from one source would block the flood and the quiet population behind it. The quiet population presents a new valid token in one request of four and reuses it once it is served, so its two rows are the onboarding the ceiling can cost and the cached credentials it must not. A 401 for an invented token is counted as refused otherwise, a fifth outcome beside served, refused, failed and timed out. The bound brings its own plan defaults, and the plan is refused when the flood could not hold every slot, when the lead-in or the deadline is shorter than the wait, or when the quiet population's onboarding would fill the slots on its own. It protects the instance rather than the quiet tenant, so the verdict says what it cost the tenant and the saturation gate does not apply.
The specification, the security page, the HTTP server mode guide and the site stated the price of ADM-014 and said bench-fairness could not measure it. They now cite `make bench-fairness BOUND=oauth-verification` with its numbers. With a stand-in GitLab answering in 100 ms and a flood of 400 invented tokens a second, the ceiling served 50 of 120 new credentials, after 5.5 s where they took 0.52 s without it, and refused the other 70, in both repetitions; cached credentials were served at 14 ms at the median in both arms. The instance received 5,638 requests, at most 19 at once, against 12,600 with up to 76 at once without the ceiling, and 5,612 against 30,600 under a flood of 1,000 a second. The share served follows the round trip: 104 and 100 of 120 at 50 ms, 26 and 29 at 200 ms. The runs also found what the pages did not say: the queue moves into the process. Each waiting request holds its connection for up to the five seconds, the resident set peaked at 273 MiB against 172 at 400 a second and at 434 against 194 at 1,000, and nothing but the arrival rate bounds how many requests wait. The specification and the guide say so. cmd-utilities documents the new bound, the -fairness-upstream-delay flag, the bound defaults and the fifth outcome; ADR-0023 NEG-004 and CLAUDE.md say how a bound with no switch is compared.
The gauge raised its peak with a compare-and-swap loop, whose retry is taken only when another request moves the peak between the load and the swap. No test can arrange that on purpose, so gobco reported the swap's false branch as never taken. A mutex moves the count, the total and the peak together with no branch to leave untaken, and a stand-in answering a few hundred requests a second does not notice it. A test now steps the gauge through a window: the peak keeps its high as requests leave, and a reset starts from what is still in flight.
The mutation run over the files this layer writes left nine survivors and a handful of uncovered mutants on its own lines. Each now dies: - the flood-source limit is tested on its edge, a flood filling the 127.2.0.0/16 range exactly accepted and one source more refused; - the source count is tested for a flood that invents a token on every other request, where the share it multiplies by is not one; - a bound with no instance round trip and no invented tokens is shown to name neither in its description; - the deadline floor is tested a millisecond under it, where it used to be tested only on it and well below it, so its round-trip count was never pinned; - the quiet onboarding refusal is held to the share it names; - a rendered row's dispatched count is held to the sum of its five outcomes, each a distinct power of two; - the variant replaces a declaration on a file's first line. Two survivors were comparisons nothing could observe, and the code no longer makes them: refusalsExpected cuts a bucket bound's excess with max, as its slot branch already did, and pause lets a timer of zero fire instead of guarding it.
cmd-utilities said the plan refuses a deadline shorter than the slot wait. The floor it checks is the wait, a new credential's round trips and a second of margin, which is what the plan's error names.
…lone The OAuth bound's run expects the 401 an invented token earns and files it as refused otherwise. The shape names no method, and the tally asked it of every row, so a quiet population's call on its own or on a new credential that the server answered 401 was filed there too rather than as a failure. The arm then stayed comparable, and its served percentiles were computed over the requests that survived: a broken server counted as a fair one, which is the over-matching the classifier exists to refuse. The tally now consults those shapes for a request presenting an invented credential and for no other, and a test holds a cached and a new credential's 401 as failures beside the flood's as refused otherwise.
Every phase of the OAuth verification bound opened on an empty queue. The lead-in's drive returned only once its last request had landed, so the flood stopped, every waiter drained for up to the five-second wait, and the phase started from nothing: its first seconds served every new credential, the flood's earliest waiters included. The records show it. In all eight runs with the ceiling, the flood's 401 count equals the stand-in's invented requests exactly, which holds only if no lead-in waiter reached the instance during the phase, and at fifty milliseconds the arm refused 1,421 of 12,000 flood requests where a queue already formed refuses about 2,400. The published shares were that transient, biased toward the ceiling. The phase's schedule now begins where the lead-in's clock puts it, whether or not the lead-in's requests have landed: the lead-in runs in its own goroutine into its own discarded tally, and the phase opens at start plus the lead-in, where the peak, the stand-in's counters and the processor samples are reset. The instance's count is read once the lead-in's stragglers have landed too, so the window's load is what reached the instance inside it. A bucket bound gains the same thing, since the pause used to refill its bucket. The plan's floor for a slot bound's lead-in was the wait, on the grounds that the queue had formed by then. It is now the time the flood takes to settle into its queue from empty: the wait times the flood over its excess above what the slots finish, which is when the first request that will not be served is refused and the queue holds a wait's worth of arrivals. That is 8.3 seconds at the default hundred milliseconds, inside the ten-second default lead-in, and 25 seconds at fifty, which the plan now refuses without a longer lead-in.
…a flood The comment on the verification slots still said a new credential waits with the flood and is refused like it for as long as the flood lasts. The pages that cite the measurement say something else, and the measurement agrees with them: a new credential waits in the flood's queue and is served when a slot frees before its wait runs out, which under a sustained flood is about the slots' share of the demand. The comment now says that, and names the command that measures it.
…tled The figures the pages cited were taken with every phase opening on an empty queue, so a new credential's share and median included the seconds in which the slots serve everyone. Re-run with the phase on the lead-in's clock, and at fifty milliseconds with a thirty-second lead-in the settled queue needs, the ceiling served 42 and 40 of 120 new credentials at a hundred milliseconds rather than 50, 80 and 82 at fifty rather than 104 and 100, 17 at two hundred and 17 under the larger flood, after 5.5, 5.3, 6.0 and 5.5 seconds at the median. The share tracks the slots' share of what arrives and stays below it; the instance's load is where it was. Cached credentials are quoted from the record rather than called the same in both arms: every one was served in every arm, at a median of 14 ms with the ceiling and 13 to 15 ms without it at a hundred milliseconds. The spec's "twelve requests on credentials" becomes twelve requests a second, since the count was 360. The claim that cached credentials are unaffected now carries its condition. They never wait for a slot, but they share the listener with the waiting flood, which holds one descriptor per waiting request, so under a hard descriptor limit below the arrival rate times the wait a cached credential opening a connection is delayed or refused with it. The measurement ran under a limit of 1048576 and says so. cmd-utilities.md states the lead-in rule the plan now enforces and that the run's own 401 is expected of an invented credential alone.
The whole-run test of the OAuth verification bound ran captureStdout inside finishWithin, and the build seams it installed wrote their wrapped stand-ins when runFairness called them. Both are on the goroutine finishWithin starts: captureStdout calls t.Fatalf when the pipe cannot be made, and wrappedStandin calls t.TempDir and t.Fatalf. An abort there ends that goroutine alone, and the test goes on against a half-run. The two scripts are now written on the test goroutine before the run, and captureStdout wraps finishWithin, so only runFairness runs on the goroutine finishWithin starts. finishWithin's own t.Fatalf now runs on the test goroutine, inside captureStdout's inner function, whose deferred restore of os.Stdout still runs. The two scripts sit in t.TempDir directories of their own, so the harness removing a binary's directory after the run still works.
This layer gave httpStatusError an Unwrap that returns the JSON-RPC error a refusal carried in its body, so a refusal shape naming a code reads one carried at 429 as it reads one carried at 200. coldList asks a warm-up listing again whenever errors.As finds rateLimitCode, and the authentication gate's per-address lockout is HTTP 429 carrying that same code, -42900. Since the Unwrap, a credential admitted from a locked-out address was asked again every 500 ms until the five-minute warm-up timeout, where its comment promises that any other answer stands. coldList now asks again only when the refusal also came at HTTP 200, the bucket's in-band refusal, and its table carries the lockout case. The fairness matcher was never affected, since refusalSpec.matches checks the status first.
…as a condition The instance totals compared two windows of different length. With the ceiling the stand-in counted from the phase's first request to the answer of its last waiter, 35 seconds, and without it over 30, so 5,640 requests read as 188 a second beside the 160 the same paragraph derives from sixteen slots at 100 ms. The spec, the HTTP server guide, the security concept and the site's security page in both languages now quote the rate over the counted window, which matches the slots exactly (about 160 a second under either flood, against about 420 and 1,020 without the ceiling), and give the counts with their windows. The condition on cached credentials named descriptors alone. The same measurement shows the waiting flood holding memory too, about 50 KiB a waiting request (100 MiB more for some 2,000 waiting, 250 MiB for some 5,000), with nothing bounding the waiters, so a memory limit below the arrival rate times five seconds times that ends the process and every cached credential with it. The same pages say so. The new credentials' share sat a few points under the slots' share in every configuration, which the pages left unexplained. A queue model of the gate puts it down to the driver's schedule, the flood on an even grid and the new credentials in clumps on fixed ticks: with both arriving at random the two shares match. The spec and the guide say so. The troubleshooting rows, in the guide and on the site in both languages, still said credentials the server holds are served during a flood and a new one gets through once it stops. They now say a held credential never waits for a slot while the process has descriptors and memory left for the waiting requests, and a new one is served only if a slot frees within its five seconds (about a third of them under the measured flood), so it may take several retries. The site's HTTP server step on OAuth carries the same listener caveat and links to the security page's account of the price.
gremlins reported one mutant of coldList's new status condition as timed out: turning its second || into && makes coldList ask again a listing that was served and a refusal it should return, every backoff. The table ran those cases on t.Context(), which never ends, so the mutant hung the test until the binary's own timeout rather than failing it. Every case the context does not end now runs under a ten-second deadline no correct answer comes near; against that mutant, applied by hand, the test fails in ten seconds with twenty calls where it wants one. The deadline is released with t.Cleanup, since the cases run in parallel after the parent returns.
SonarCloud flagged endpoint(), the address every HTTP scenario posts to, as a clear-text "http://" string (go:S5332) on #1091. The same target's health probe and the concurrency series' profiler built theirs the same way, so all three take loopbackURL now.
drive and driveContinuously took eight parameters each, four of them the same plan, connections, call and presenter every window of a run shares. They travel together now, as paceInput already does one level down. The flood-range test compares its refusal inline rather than through a variable that existed only for the condition.
Both builders wrote the binary as server, and os/exec on Windows runs a file only by an extension PATHEXT lists, so the variant build test could not start what it built: executable file not found in %PATH%. The name now comes from serverExecutable, server.exe on Windows and server elsewhere.
7afa8c0 to
e1e1669
Compare
|



Follows #950.
#950 put sixteen verification slots and a five-second wait in front of GitLab (register row
ADM-014) and stated what that costs a legitimate credential presented for the first time during a flood, becausemake bench-fairnesshad no OAuth driver and could not measure it. The decision on #950 was to build the driver rather than accept the statement. This layer is that driver and what it measured.What the driver does
make bench-fairness BOUND=oauth-verification(orgo run ./cmd/bench_resources/ -fairness oauth-verification) runs the two arms every bound gets, with the ceiling in force and without it, and reports the quiet population's served and refused counts apart, with latency percentiles over served requests only.The arm without the ceiling is a build of its own. The ceiling is a constant no operator can move, and it must not grow a switch: a flag that turns off a security bound is a flag somebody sets in production. So both arms start with the same flags (
--auth-mode=oauth, a--public-url,--trusted-proxies=127.0.0.0/8with--trusted-proxy-header=X-Forwarded-For, the rate limit off) and the arm without the ceiling runs a second build of the same checkout in whichconst verificationSlots =ininternal/oauth/verifier.goreads1 << 20, made through the go command's-overlay. The tree on disk is never touched, the two builds differ in that line alone, the record names the line (variant_off), and a checkout that no longer declares it, or declares it twice, stops the run rather than comparing the binary with itself. The bound refuses-binary, since one binary cannot be both arms.The stand-in GitLab answers
GET /api/v4/userfor every token except one startingbench-invented-, which it answers 401, answers/api/v4/personal_access_tokens/selfand/oauth/token/info404 the way an instance that will not describe a token does, holds each of those requests for-fairness-upstream-delay(100 ms unless given), and counts how many it answered and how many it held at once, for every token and for invented ones apart. Those counts include the pool's own two requests to the same endpoints when it admits a new credential, which run under the pool's probe slots rather than the verification slots.The two populations. The noisy one lists tools with a new invented token on every request, each forwarded for an address of its own. It sends from as many transport sources in 127.2.0.0/16 as keep each one under half of the transport-source failure budget (
AUB-002, 500 distinct failing credentials a source a minute), because from one source that budget blocks the flood, and the quiet population behind it, long before the slots fill. Linux answers every address in 127.0.0.0/8 on its loopback; elsewhere every flood request fails and says why. The quiet population calls tools with its own cached credential three times in four and presents a new valid token the fourth, using it once it is served, so its two rows are the onboarding the ceiling can cost and the cached credentials it must not.The stand-in server the package's whole-run tests start (
testdata/standin) gains an OAuth gate that mirrors the bearer guard: a token it has verified before is admitted at once, a new one after the instance answersGET /api/v4/userfor it, underSTANDIN_VERIFY_SLOTSslots with aSTANDIN_VERIFY_WAITwait (100 ms unless set), refused in the guard's codes and words, 503 for a slot that never freed and 401 for a token the instance refuses.The phase follows the lead-in on one clock. The lead-in runs into a tally of its own, and the phase's schedule begins where the lead-in's clock puts it, whether or not the lead-in's requests have landed; the resident set peak, the stand-in's counters and the processor samples are reset at that instant. Run one after the other, as the first version of this layer did, the flood stopped at the end of the lead-in, every waiter drained, and each phase opened on an empty queue whose first seconds served every new credential.
What the record and the verdict say. A 401 for an invented token is counted as refused otherwise, a fifth outcome beside served, refused, failed and timed out, and only on the flood's row: the same 401 to a cached or a new valid credential is a failure. The per-arm summary says what the instance answered. The bound carries its own plan defaults (eight noisy credentials at fifty tokens a second, a thirty-second phase after a ten-second lead-in, a fifteen-second deadline, 100 ms at the instance), and the plan is refused when the flood could not hold every slot at the round trip given, when the lead-in is shorter than the time the flood takes to settle into its queue from empty (the wait times the flood over its excess above what the slots finish: 8.3 seconds at the defaults, 25 at 50 ms), when the deadline is shorter than the wait plus a new credential's round trips and a second, or when the quiet population's own new credentials would keep half the slots busy. The ceiling protects the instance rather than the quiet tenant, so the verdict says what it cost the tenant instead of calling the tenant the one it was meant to protect, and the saturation gate, which asks whether a per-credential bound reached the quiet tenant through the processor, does not apply to a bound whose contended resource is its slots.
Measured
On truenas through
rgo: Intel i5-14400, sixteen threads, the container held to five cores, Go 1.27.1, a descriptor limit of 1048576. Two repetitions per configuration, arms alternated, thirty-second phases, each opening on the lead-in's clock after the flood has settled into its queue: a ten-second lead-in, and thirty seconds at 50 ms, where the queue takes twenty-five to settle. Eight quiet credentials present four new tokens a second between them (120 in a phase) beside twelve requests a second on cached credentials (360). "Instance" is every request the stand-in answered at the three endpoints from the instant the phase opened until the last request of either window landed: the lead-in's waiters that a slot served inside the window are in it, since that is the window's load, and so are the pool's own two requests for each new credential it admits. Each pair is the two repetitions.Every new credential the ceiling did not serve was refused 503 with the rest of the flood; none failed and none timed out, and no cached request was refused in any arm. Without the ceiling all 120 new credentials were served in every run. The verdict of every run is
worse, in the first configuration's words: "the bound refused 78 of the quiet population's 120 tools/call (new credential) requests. This bound protects the GitLab instance the verifications reach rather than the quiet tenant, and that is what it cost the tenant".What the numbers say:
HLD-011counts, since the wait comes before the gate admits a request. These runs had a descriptor limit of 1048576 and did not reach it; a process under a low hard limit would, cached credentials' connections included, and I did not measure what it does then. The waiters hold memory as well: the resident set rose by about 100 MiB for some 2,000 waiting and 250 MiB for some 5,000, about 50 KiB a waiting request, so a process whose memory limit is below the arrival rate times five seconds times that is ended by the flood, cached credentials with it (1 GiB is some 21,000 waiters, a flood of about 4,200 invented tokens a second). The pages now say both wherever they say a cached credential never waits for a slot, the troubleshooting rows and the site's HTTP server step among them.What changed from the first measurement. The first version of this layer ran the lead-in and the phase one after the other, so every phase opened on an empty queue. Its figures were biased toward the ceiling: 50 of 120 at 100 ms against 42 and 40 now, 104 and 100 at 50 ms against 80 and 82 at a median of 4.2 s against 5.3, 26 and 29 at 200 ms against 17, and 24 and 27 under the larger flood against 17. The records showed it: in all eight runs with the ceiling the flood's 401 count equalled the stand-in's invented requests exactly, which holds only if no lead-in waiter reached the instance during the phase. The instance's load, which the slots saturate from the first second either way, barely moved.
To reproduce:
make bench-fairness BOUND=oauth-verificationfor the first row, andgo run ./cmd/bench_resources/ -fairness oauth-verificationwith-fairness-upstream-delay 50ms -fairness-lead-in 30s,-fairness-upstream-delay 200msor-fairness-noisy 20for the others. Each run took three to four and a half minutes on that host. The documents arebench/fairness.jsonand are not committed.Documentation
docs/development/tenant-policy-spec.md: the findings paragraph on issue 950 cites the measurement and the command instead of saying the driver speaks no OAuth, and states the unbounded queue.docs/guides/http-server-mode.md,docs/concepts/security.mdand the site's security page in English and Spanish: the price of the ceiling now carries the measured figures, and the guide says why the descriptor limit matters to it. Where they say a cached credential never waits for a slot, they also say it shares the listener with the waiting flood, so under a hard descriptor limit below the arrival rate times five seconds its connection is delayed or refused with the flood, and that the measurement ran under a limit of 1048576; and that the waiting flood holds about 50 KiB a request, so under a memory limit below the arrival rate times five seconds times that the flood ends the process and every cached credential with it. The instance's load is quoted as a rate over the window each arm counted, about 160 requests a second against about 420 and 1,020, with the counts and their windows beside it, since 5,640 requests over 35 seconds and 12,600 over 30 are not comparable totals. The spec and the guide say the new credentials' share sits a few points under the slots' share because of the driver's evenly spaced schedule.docs/guides/troubleshooting.mdand the site's troubleshooting page in English and Spanish: the row for a 503 on a new token said credentials the server holds are still served and a new one gets through once the flood stops; it now says a held credential never waits for a slot while the process has descriptors and memory left for the waiting requests, and that a new one is served only if a slot frees within its five seconds (about a third of them under the measured flood), so it may take several retries. The site's HTTP server page in both languages, whose OAuth step said a cached token never waits, now says it shares the listener with the requests waiting for a slot and links to the price of the ceiling on the security page.internal/oauth/verifier.go: the comment on the verification slots said a new credential is refused like the flood for as long as the flood lasts; it now says what the measurement shows, that a new credential is served when a slot frees within its wait, about the slots' share under a sustained flood, and names the command.docs/development/cmd-utilities.md: the new bound,-fairness-upstream-delay, the bound defaults, the fifth outcome and the row it applies to, the lead-in on the phase's clock and the settled-queue floor, and how the arm without the ceiling is built;Makefilehelp lists theBOUNDspellings and whatoauth-verificationneeds.CLAUDE.mdsays the same in one clause.Checks
All through
rgoon truenas unless said otherwise, on this layer before the stack was rebased onto main at 50ef2cf, which changes no file this layer writes. The stack now sits on b44ab96, which adds #1083, #1084 and #1085; they change the upstream register, indirect module versions and the site's package pins, none of them a file this stack writes, and every commit of the stack applied unchanged.go test ./cmd/bench_resources/: passes with 100.0% statement coverage;-race -shuffle=onpasses.go vet,go build ./...and golangci-lint with thee2etag on the package: clean.make coverage-conditions PKG=./cmd/bench_resources): 1558/1558. The first reading was 1565/1566: the stand-in's gauge raised its peak with a compare-and-swap loop whose retry only contention takes, which no test can arrange, so the gauge now counts under a mutex.--workers 2 --timeout-coefficient 20, staged bymake coverage-mutants), over the ten files of the package this layer writes (fairness,fairness_result,fairness_run,main,presenter,rpc,run,stubs,target,variant); the stand-in server undertestdata/standin, a separate program the script's default exclusion of files below the package leaves out, as it did for The process has no ceiling of its own on held requests, stateful sessions or catalog listings #951's layers, is exercised through theoauth-verificationbound byTestRunFairness_OAuthVerification_MeasuresTheCeilingAgainstABuildWithout, which drives its gate with and without slots. First run, at the driver's commit: 627 killed, 16 lived, 27 not covered, 0 timed out. After the fixes: 632 killed, 5 lived, 27 not covered, 0 timed out.above <= 0inrefusalsExpected, nowmax(0, above)as the slot branch already was,pause's guard on a zero delay, which a timer of zero makes redundant, and the gauge'snow <= peakshort cut, which went with its compare-and-swap loop.fairness.go:360(at share zero the division is+Infandmincaps it at the same default),fairness_result.go:610(a floor of zero),fairness_result.go:777(reached only once the claim exceeds a spread of zero or more) andfairness_result.go:796twice (the tie-break shape).switchcase expressions, which carry no statement counter. 18 are on lines this layer does not change and are the ones The process has no ceiling of its own on held requests, stateful sessions or catalog listings #951's layer read. The 9 on lines it writes were applied by hand: the four bound defaults (fairness.go:606,:607) die inTestFairnessPlanFor_TakesTheBoundsDefaultsWhereNoFlagWasTyped, which states each literal;case err == nilinclassifyExpectingdies inTestClassifyExpecting_FilesTheRunsOwnRefusalsApartandTestClassifyOutcome_KeepsRefusalsApartFromSuccessAndFailure;maxFloodSources' two die inTestSourceAddress_AndForwardedAddress_AreDistinctPerIndex; the stand-in'scase body == ""dies inTestStubGitLab_Verification_AnswersAfterItsRoundTripAndCountsWhatIsInFlight; and the+between the proxy header flag and its value does not compile as a-.go test ./cmd/bench_resources/at 100.0% of statements,-race -shuffle=onpassing, golangci-lint clean oncmd/bench_resourcesandinternal/oauth; conditions 1560/1560, the two new ones among them; gremlins over the two files the fixes touch (fairness,fairness_run), 256 killed, 1 lived, 19 not covered, 0 timed out. Every mutant of the new code died. The one lived isfairness.go:363, the share-zero shape recorded above at its old line, and the 19 not covered are the constant initializers, composite-literal fields,switchcase expressions and thegostatement's argument the first run classified, at the lines the fixes moved them to.captureStdoutinsidefinishWithin, sot.TempDir,t.Fatalfandt.Cleanupran on the goroutinefinishWithinstarts (make check-test-goroutinesdoes not follow calls through helpers or function-valued seams); both scripts are now written on the test goroutine andcaptureStdoutwrapsfinishWithin, leaving onlyrunFairnesson the spawned goroutine. TheUnwrapthis layer gavehttpStatusErrorletcoldList, which asks a warm-up listing again while a rate bound refuses it, match the authentication gate's per-address lockout, which carries the same-42900at HTTP 429, so a credential admitted from a locked-out address would have been asked again every 500 ms for the five-minute warm-up;coldListnow asks again only for the refusal answered at 200, and its table carries the lockout case.go test ./cmd/bench_resources/passes at 100.0% of statements,-race -shuffle=onpasses, golangci-lint is clean; conditions 1562/1562, the two parts of the new status test among them; gremlins overseries.go, the file the fix touches, 61 killed, 0 lived, 7 not covered, 0 timed out. The first run of it reported one timed out, the new condition's second||turned into&&, which asks again a listing that was served; the table ran those cases ont.Context(), so the mutant hung the test rather than failing it, and every case the context does not end now runs under a ten-second deadline, against which that mutant, applied by hand, fails in ten seconds. The 7 not covered are two variable initializers and a constant initializer (series.go:57,:265,:371) and fourswitchcase expressions (:534,:536twice,:538), which carry no statement counter; none is on a line this layer writes.make check-test-goroutines,make check-test-subtests,make check-test-file-names,make check-doc-tool-namesandmake check-tenancy(83 rows, 0 findings): pass.gofumpt,go run ./cmd/format_md_tables/ --checkand markdownlint on every changed page: clean, and no em or en dash in anything this layer adds.Decisions I took here
1 << 20slots rather than no gate at all, so the arm without the ceiling runs the same code path, a slot taken and given back per verification, with a count no flood here can fill.AUB-002's budget, because that budget is not what this measures and from one source it answers first. The per-address budgets are kept out of the way the same way, by a new forwarded address per request.worse, and says in the same sentence that the ceiling protects the instance and that this is its price. Calling it anything else would hide the one thing the measurement is for.api, which is the path the verifier takes against an instance that will not describe a token.Not done
INV-021, a change of its own. It is drafted as an issue of its own and not yet filed.-held-nofile, so what a process under a low hard limit does while a flood waits is derived above and not measured.