Skip to content

Measure the OAuth verification ceiling with an OAuth driver for bench-fairness - #1091

Merged
jmrplens merged 16 commits into
issue-1045-stdio-http-flagsfrom
issue-950-fairness-oauth
Sep 30, 2026
Merged

jmrplens merged 16 commits into
issue-1045-stdio-http-flagsfrom
issue-950-fairness-oauth

Conversation

@jmrplens

@jmrplens jmrplens commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

Follows #950.

#950 put sixteen verification slots and a five-second wait in front of GitLab (register row ADM-014) and stated what that costs a legitimate credential presented for the first time during a flood, because make bench-fairness had no OAuth driver and could not measure it. The decision on #950 was to build the driver rather than accept the statement. This layer is that driver and what it measured.

What the driver does

make bench-fairness BOUND=oauth-verification (or go run ./cmd/bench_resources/ -fairness oauth-verification) runs the two arms every bound gets, with the ceiling in force and without it, and reports the quiet population's served and refused counts apart, with latency percentiles over served requests only.

The arm without the ceiling is a build of its own. The ceiling is a constant no operator can move, and it must not grow a switch: a flag that turns off a security bound is a flag somebody sets in production. So both arms start with the same flags (--auth-mode=oauth, a --public-url, --trusted-proxies=127.0.0.0/8 with --trusted-proxy-header=X-Forwarded-For, the rate limit off) and the arm without the ceiling runs a second build of the same checkout in which const verificationSlots = in internal/oauth/verifier.go reads 1 << 20, made through the go command's -overlay. The tree on disk is never touched, the two builds differ in that line alone, the record names the line (variant_off), and a checkout that no longer declares it, or declares it twice, stops the run rather than comparing the binary with itself. The bound refuses -binary, since one binary cannot be both arms.

The stand-in GitLab answers GET /api/v4/user for every token except one starting bench-invented-, which it answers 401, answers /api/v4/personal_access_tokens/self and /oauth/token/info 404 the way an instance that will not describe a token does, holds each of those requests for -fairness-upstream-delay (100 ms unless given), and counts how many it answered and how many it held at once, for every token and for invented ones apart. Those counts include the pool's own two requests to the same endpoints when it admits a new credential, which run under the pool's probe slots rather than the verification slots.

The two populations. The noisy one lists tools with a new invented token on every request, each forwarded for an address of its own. It sends from as many transport sources in 127.2.0.0/16 as keep each one under half of the transport-source failure budget (AUB-002, 500 distinct failing credentials a source a minute), because from one source that budget blocks the flood, and the quiet population behind it, long before the slots fill. Linux answers every address in 127.0.0.0/8 on its loopback; elsewhere every flood request fails and says why. The quiet population calls tools with its own cached credential three times in four and presents a new valid token the fourth, using it once it is served, so its two rows are the onboarding the ceiling can cost and the cached credentials it must not.

The stand-in server the package's whole-run tests start (testdata/standin) gains an OAuth gate that mirrors the bearer guard: a token it has verified before is admitted at once, a new one after the instance answers GET /api/v4/user for it, under STANDIN_VERIFY_SLOTS slots with a STANDIN_VERIFY_WAIT wait (100 ms unless set), refused in the guard's codes and words, 503 for a slot that never freed and 401 for a token the instance refuses.

The phase follows the lead-in on one clock. The lead-in runs into a tally of its own, and the phase's schedule begins where the lead-in's clock puts it, whether or not the lead-in's requests have landed; the resident set peak, the stand-in's counters and the processor samples are reset at that instant. Run one after the other, as the first version of this layer did, the flood stopped at the end of the lead-in, every waiter drained, and each phase opened on an empty queue whose first seconds served every new credential.

What the record and the verdict say. A 401 for an invented token is counted as refused otherwise, a fifth outcome beside served, refused, failed and timed out, and only on the flood's row: the same 401 to a cached or a new valid credential is a failure. The per-arm summary says what the instance answered. The bound carries its own plan defaults (eight noisy credentials at fifty tokens a second, a thirty-second phase after a ten-second lead-in, a fifteen-second deadline, 100 ms at the instance), and the plan is refused when the flood could not hold every slot at the round trip given, when the lead-in is shorter than the time the flood takes to settle into its queue from empty (the wait times the flood over its excess above what the slots finish: 8.3 seconds at the defaults, 25 at 50 ms), when the deadline is shorter than the wait plus a new credential's round trips and a second, or when the quiet population's own new credentials would keep half the slots busy. The ceiling protects the instance rather than the quiet tenant, so the verdict says what it cost the tenant instead of calling the tenant the one it was meant to protect, and the saturation gate, which asks whether a per-credential bound reached the quiet tenant through the processor, does not apply to a bound whose contended resource is its slots.

Measured

On truenas through rgo: Intel i5-14400, sixteen threads, the container held to five cores, Go 1.27.1, a descriptor limit of 1048576. Two repetitions per configuration, arms alternated, thirty-second phases, each opening on the lead-in's clock after the flood has settled into its queue: a ten-second lead-in, and thirty seconds at 50 ms, where the queue takes twenty-five to settle. Eight quiet credentials present four new tokens a second between them (120 in a phase) beside twelve requests a second on cached credentials (360). "Instance" is every request the stand-in answered at the three endpoints from the instant the phase opened until the last request of either window landed: the lead-in's waiters that a slot served inside the window are in it, since that is the window's load, and so are the pool's own two requests for each new credential it admits. Each pair is the two repetitions.

Round trip Flood New served with the ceiling New median with / without Cached median with / without Instance requests with / without At once with / without Resident set peak with / without
100 ms 400 a second 42, 40 of 120 5.52, 5.51 s / 0.52 s 14.0, 13.6 / 14.6, 12.5 ms 5,642, 5,644 / 12,600, 12,600 20, 19 / 56, 56 277, 273 / 175, 178 MiB
50 ms 400 a second 80, 82 of 120 5.27, 5.26 s / 0.27 s 14.3, 15.6 / 15.5, 12.0 ms 11,054, 11,149 / 12,600, 12,600 18, 18 / 35, 41 280, 280 / 172, 171 MiB
200 ms 400 a second 17, 17 of 120 6.01, 6.01 s / 1.01 s 9.9, 9.6 / 11.6, 10.3 ms 2,845, 2,841 / 12,600, 12,600 19, 19 / 92, 90 265, 269 / 175, 176 MiB
100 ms 1,000 a second 17, 17 of 120 5.51, 5.51 s / 0.51 s 9.1, 9.1 / 9.5, 9.5 ms 5,606, 5,599 / 30,600, 30,600 17, 18 / 110, 110 438, 436 / 188, 188 MiB

Every new credential the ceiling did not serve was refused 503 with the rest of the flood; none failed and none timed out, and no cached request was refused in any arm. Without the ceiling all 120 new credentials were served in every run. The verdict of every run is worse, in the first configuration's words: "the bound refused 78 of the quiet population's 120 tools/call (new credential) requests. This bound protects the GitLab instance the verifications reach rather than the quiet tenant, and that is what it cost the tenant".

What the numbers say:

  • The cost falls where OAuth verification is unbounded: the identity cache has no size limit and the verifier's round trips to GitLab have no concurrency ceiling #950 said, and it tracks the slots' share of what arrives, a little below it. A new credential waits in the flood's queue and is served only when a slot frees before its five seconds do, which is why the ones served took the whole wait and their round trips, 5.5 s rather than 0.5 s, and why the share served moves with the round trip: 35% and 33% at 100 ms, where the slots finish two in five of the flood, 67% and 68% at 50 ms (four in five), 14% at 200 ms (one in five), and 14% at 100 ms under the larger flood (about one in six). The flood's own refusals match the slots' share closely (61%, 24%, 80% and 84% refused), and the newcomers' shortfall below it comes from the driver's schedule rather than from the ceiling: in a queue whose waiters all give up at the same five seconds, a freed slot serves the oldest waiter still inside its wait, so a request's chance grows with the gap before it, and the flood arrives on an even grid (one request every 2.5 ms) while the new credentials arrive in clumps on fixed ticks. A queue model of the gate reproduces the shortfall with that schedule (36% against 39% at 100 ms, 67% to 71% against 78% at 50 ms) and removes it when both streams arrive at random (39.5% against 38.9%, 78.6% against 77.8%, 19.8% against 19.4% at 200 ms).
  • Cached credentials never wait for a slot, as the code says: no cached request was refused in any arm of any configuration, and their medians lay between 9 and 16 ms with the ceiling and without it.
  • What it buys is the instance's load, and that is independent of the flood. At 100 ms the instance received about 160 requests a second under 400 invented tokens a second and under 1,000, sixteen slots over the round trip, never more than 20 at once, against about 420 and 1,020 a second and up to 110 at once without the ceiling. Over the window each arm counted that is 5,640 and 5,600 requests in the 35 seconds from the phase's first request to the answer of its last waiter, against 12,600 and 30,600 in 30, and the rates match the slot arithmetic in every configuration: 160.8 a second at 100 ms, 315 and 318 at 50 ms, 80.8 at 200 ms and 160.0 under the larger flood. Of the 12,000 invented tokens of the first configuration's first run, 4,630 reached the instance and 7,370 were refused 503 without reaching it.
  • The queue moves into the process, and nothing bounds it. Each waiting request holds its connection for up to the five seconds, so the server holds the arrival rate times the wait, about 2,000 requests at 400 a second, and its resident set peaked at 273 and 277 MiB against 175 and 178, and at 436 and 438 against 188 at 1,000 a second. Nothing but the arrival rate bounds how many wait, and their connections are descriptors outside the held calls HLD-011 counts, since the wait comes before the gate admits a request. These runs had a descriptor limit of 1048576 and did not reach it; a process under a low hard limit would, cached credentials' connections included, and I did not measure what it does then. The waiters hold memory as well: the resident set rose by about 100 MiB for some 2,000 waiting and 250 MiB for some 5,000, about 50 KiB a waiting request, so a process whose memory limit is below the arrival rate times five seconds times that is ended by the flood, cached credentials with it (1 GiB is some 21,000 waiters, a flood of about 4,200 invented tokens a second). The pages now say both wherever they say a cached credential never waits for a slot, the troubleshooting rows and the site's HTTP server step among them.

What changed from the first measurement. The first version of this layer ran the lead-in and the phase one after the other, so every phase opened on an empty queue. Its figures were biased toward the ceiling: 50 of 120 at 100 ms against 42 and 40 now, 104 and 100 at 50 ms against 80 and 82 at a median of 4.2 s against 5.3, 26 and 29 at 200 ms against 17, and 24 and 27 under the larger flood against 17. The records showed it: in all eight runs with the ceiling the flood's 401 count equalled the stand-in's invented requests exactly, which holds only if no lead-in waiter reached the instance during the phase. The instance's load, which the slots saturate from the first second either way, barely moved.

To reproduce: make bench-fairness BOUND=oauth-verification for the first row, and go run ./cmd/bench_resources/ -fairness oauth-verification with -fairness-upstream-delay 50ms -fairness-lead-in 30s, -fairness-upstream-delay 200ms or -fairness-noisy 20 for the others. Each run took three to four and a half minutes on that host. The documents are bench/fairness.json and are not committed.

Documentation

  • docs/development/tenant-policy-spec.md: the findings paragraph on issue 950 cites the measurement and the command instead of saying the driver speaks no OAuth, and states the unbounded queue.
  • docs/guides/http-server-mode.md, docs/concepts/security.md and the site's security page in English and Spanish: the price of the ceiling now carries the measured figures, and the guide says why the descriptor limit matters to it. Where they say a cached credential never waits for a slot, they also say it shares the listener with the waiting flood, so under a hard descriptor limit below the arrival rate times five seconds its connection is delayed or refused with the flood, and that the measurement ran under a limit of 1048576; and that the waiting flood holds about 50 KiB a request, so under a memory limit below the arrival rate times five seconds times that the flood ends the process and every cached credential with it. The instance's load is quoted as a rate over the window each arm counted, about 160 requests a second against about 420 and 1,020, with the counts and their windows beside it, since 5,640 requests over 35 seconds and 12,600 over 30 are not comparable totals. The spec and the guide say the new credentials' share sits a few points under the slots' share because of the driver's evenly spaced schedule.
  • docs/guides/troubleshooting.md and the site's troubleshooting page in English and Spanish: the row for a 503 on a new token said credentials the server holds are still served and a new one gets through once the flood stops; it now says a held credential never waits for a slot while the process has descriptors and memory left for the waiting requests, and that a new one is served only if a slot frees within its five seconds (about a third of them under the measured flood), so it may take several retries. The site's HTTP server page in both languages, whose OAuth step said a cached token never waits, now says it shares the listener with the requests waiting for a slot and links to the price of the ceiling on the security page.
  • internal/oauth/verifier.go: the comment on the verification slots said a new credential is refused like the flood for as long as the flood lasts; it now says what the measurement shows, that a new credential is served when a slot frees within its wait, about the slots' share under a sustained flood, and names the command.
  • docs/development/cmd-utilities.md: the new bound, -fairness-upstream-delay, the bound defaults, the fifth outcome and the row it applies to, the lead-in on the phase's clock and the settled-queue floor, and how the arm without the ceiling is built; Makefile help lists the BOUND spellings and what oauth-verification needs.
  • ADR-0023 NEG-004 said the benchmark drives two bounds and that no switch could exist for a bound no operator can turn off; it now names the four it drives and says how the one with no switch is compared. CLAUDE.md says the same in one clause.

Checks

All through rgo on truenas unless said otherwise, on this layer before the stack was rebased onto main at 50ef2cf, which changes no file this layer writes. The stack now sits on b44ab96, which adds #1083, #1084 and #1085; they change the upstream register, indirect module versions and the site's package pins, none of them a file this stack writes, and every commit of the stack applied unchanged.

  • go test ./cmd/bench_resources/: passes with 100.0% statement coverage; -race -shuffle=on passes. go vet, go build ./... and golangci-lint with the e2e tag on the package: clean.
  • Conditions (gobco, make coverage-conditions PKG=./cmd/bench_resources): 1558/1558. The first reading was 1565/1566: the stand-in's gauge raised its peak with a compare-and-swap loop whose retry only contention takes, which no test can arrange, so the gauge now counts under a mutex.
  • Mutants (gremlins, --workers 2 --timeout-coefficient 20, staged by make coverage-mutants), over the ten files of the package this layer writes (fairness, fairness_result, fairness_run, main, presenter, rpc, run, stubs, target, variant); the stand-in server under testdata/standin, a separate program the script's default exclusion of files below the package leaves out, as it did for The process has no ceiling of its own on held requests, stateful sessions or catalog listings #951's layers, is exercised through the oauth-verification bound by TestRunFairness_OAuthVerification_MeasuresTheCeilingAgainstABuildWithout, which drives its gate with and without slots. First run, at the driver's commit: 627 killed, 16 lived, 27 not covered, 0 timed out. After the fixes: 632 killed, 5 lived, 27 not covered, 0 timed out.
    • Killed by a new test: the flood-source limit on its edge, the source count for a flood inventing every other token, a bucket bound's description naming no instance and no sources, the deadline floor a millisecond under it (the old boundary cases sat on it and well below it, so its round-trip count was never pinned), the onboarding refusal's share, a row's dispatched sum, and a declaration on a file's first line.
    • Removed rather than recorded: a bucket bound's above <= 0 in refusalsExpected, now max(0, above) as the slot branch already was, pause's guard on a zero delay, which a timer of zero makes redundant, and the gauge's now <= peak short cut, which went with its compare-and-swap loop.
    • The 5 lived are the shapes The process has no ceiling of its own on held requests, stateful sessions or catalog listings #951's layer recorded on the same, unchanged code, at lines this layer moved: fairness.go:360 (at share zero the division is +Inf and min caps it at the same default), fairness_result.go:610 (a floor of zero), fairness_result.go:777 (reached only once the claim exceeds a spread of zero or more) and fairness_result.go:796 twice (the tie-break shape).
    • The 27 not covered are constant and variable initializers and switch case expressions, which carry no statement counter. 18 are on lines this layer does not change and are the ones The process has no ceiling of its own on held requests, stateful sessions or catalog listings #951's layer read. The 9 on lines it writes were applied by hand: the four bound defaults (fairness.go:606, :607) die in TestFairnessPlanFor_TakesTheBoundsDefaultsWhereNoFlagWasTyped, which states each literal; case err == nil in classifyExpecting dies in TestClassifyExpecting_FilesTheRunsOwnRefusalsApart and TestClassifyOutcome_KeepsRefusalsApartFromSuccessAndFailure; maxFloodSources' two die in TestSourceAddress_AndForwardedAddress_AreDistinctPerIndex; the stand-in's case body == "" dies in TestStubGitLab_Verification_AnswersAfterItsRoundTripAndCountsWhatIsInFlight; and the + between the proxy header flag and its value does not compile as a -.
  • After the review fixes (the invented-credential classification, the lead-in and phase on one clock, the settled-queue floor), at the fixes' commit: go test ./cmd/bench_resources/ at 100.0% of statements, -race -shuffle=on passing, golangci-lint clean on cmd/bench_resources and internal/oauth; conditions 1560/1560, the two new ones among them; gremlins over the two files the fixes touch (fairness, fairness_run), 256 killed, 1 lived, 19 not covered, 0 timed out. Every mutant of the new code died. The one lived is fairness.go:363, the share-zero shape recorded above at its old line, and the 19 not covered are the constant initializers, composite-literal fields, switch case expressions and the go statement's argument the first run classified, at the lines the fixes moved them to.
  • After the second review, at the layer's head: the whole-run test of the verification bound wrote its two wrapped stand-ins inside the build seams and ran captureStdout inside finishWithin, so t.TempDir, t.Fatalf and t.Cleanup ran on the goroutine finishWithin starts (make check-test-goroutines does not follow calls through helpers or function-valued seams); both scripts are now written on the test goroutine and captureStdout wraps finishWithin, leaving only runFairness on the spawned goroutine. The Unwrap this layer gave httpStatusError let coldList, which asks a warm-up listing again while a rate bound refuses it, match the authentication gate's per-address lockout, which carries the same -42900 at HTTP 429, so a credential admitted from a locked-out address would have been asked again every 500 ms for the five-minute warm-up; coldList now asks again only for the refusal answered at 200, and its table carries the lockout case. go test ./cmd/bench_resources/ passes at 100.0% of statements, -race -shuffle=on passes, golangci-lint is clean; conditions 1562/1562, the two parts of the new status test among them; gremlins over series.go, the file the fix touches, 61 killed, 0 lived, 7 not covered, 0 timed out. The first run of it reported one timed out, the new condition's second || turned into &&, which asks again a listing that was served; the table ran those cases on t.Context(), so the mutant hung the test rather than failing it, and every case the context does not end now runs under a ten-second deadline, against which that mutant, applied by hand, fails in ten seconds. The 7 not covered are two variable initializers and a constant initializer (series.go:57, :265, :371) and four switch case expressions (:534, :536 twice, :538), which carry no statement counter; none is on a line this layer writes.
  • make check-test-goroutines, make check-test-subtests, make check-test-file-names, make check-doc-tool-names and make check-tenancy (83 rows, 0 findings): pass.
  • Locally: gofumpt, go run ./cmd/format_md_tables/ --check and markdownlint on every changed page: clean, and no em or en dash in anything this layer adds.
  • The four measurement runs above, plus a ten-second smoke run of the default before them.

Decisions I took here

  • The arm without the ceiling is an overlay build rather than a flag, an environment variable or a build tag. A flag or a variable is a switch somebody sets in production; a build tag would put a second value of the ceiling in the tree for the register gate and every reader to account for. The overlay leaves the tree as it is and makes the two binaries differ in one line.
  • The replacement is 1 << 20 slots rather than no gate at all, so the arm without the ceiling runs the same code path, a slot taken and given back per verification, with a count no flood here can fill.
  • The flood comes from many transport sources, each under half of AUB-002's budget, because that budget is not what this measures and from one source it answers first. The per-address budgets are kept out of the way the same way, by a new forwarded address per request.
  • The bound's verdict keeps the benchmark's rule that a quiet request refused is worse, and says in the same sentence that the ceiling protects the instance and that this is its price. Calling it anything else would hide the one thing the measurement is for.
  • The stand-in answers introspection 404 for valid tokens, so a new credential costs its slot three requests and the server assumes api, which is the path the verifier takes against an instance that will not describe a token.
  • The lead-in and the phase run as two schedules on one clock, the lead-in in a goroutine of its own, rather than as one schedule whose observations are split by their intended instant. Each window keeps exactly its own tick count, which the whole-run tests hold, and a lead-in whose length is not a whole number of periods cannot move ticks from one window to the other.
  • The plan's floor for a slot bound's lead-in is the time the queue takes to settle, the wait times the flood over its excess above what the slots finish, rather than the time it takes to reach the wait, the wait times capacity over the excess. The second is when a request arriving would wait the whole wait; the first is a wait later, when that request is refused and the queue holds as many waiters as it ever will, so the resident set the phase reports is the sustained one too. It is never shorter than the wait, and at the defaults it is 8.3 seconds, inside the ten-second lead-in.
  • The run's own 401 is asked of the invented-credential row alone, in the tally, rather than through a credential field on the refusal shape: the tally already knows the row, and a shape that named a credential would need every other bound's shapes to say none.

Not done

  • The queue in the process is a finding and not fixed here: nothing bounds how many requests wait for a verification slot. A bound on the waiters (or an immediate refusal once the queue is deeper than the slots can drain within the wait) is a change of policy and, by INV-021, a change of its own. It is drafted as an issue of its own and not yet filed.
  • The fairness mode has no descriptor limit flag like -held-nofile, so what a process under a low hard limit does while a flood waits is derived above and not measured.
  • The stand-in's round trip is fixed; a real instance's varies, and GitLab.com would also rate-limit the flood itself, which the stand-in does not.
  • The flood needs Linux, for the loopback addresses it sends from.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @jmrplens, your pull request is larger than the review limit of 150,000 diff characters

@coderabbitai

coderabbitai Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: jmrplens/gitlab-mcp-server/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 29ed0c6b-9efc-43dd-a75b-427b02626000

📥 Commits

Reviewing files that changed from the base of the PR and between 8649b06 and e1e1669.

📒 Files selected for processing (38)
  • CLAUDE.md
  • Makefile
  • cmd/bench_resources/fairness.go
  • cmd/bench_resources/fairness_result.go
  • cmd/bench_resources/fairness_result_test.go
  • cmd/bench_resources/fairness_run.go
  • cmd/bench_resources/fairness_run_test.go
  • cmd/bench_resources/fairness_test.go
  • cmd/bench_resources/host.go
  • cmd/bench_resources/host_test.go
  • cmd/bench_resources/main.go
  • cmd/bench_resources/presenter.go
  • cmd/bench_resources/presenter_test.go
  • cmd/bench_resources/rpc.go
  • cmd/bench_resources/rpc_test.go
  • cmd/bench_resources/run.go
  • cmd/bench_resources/series.go
  • cmd/bench_resources/series_test.go
  • cmd/bench_resources/stubs.go
  • cmd/bench_resources/stubs_test.go
  • cmd/bench_resources/target.go
  • cmd/bench_resources/target_test.go
  • cmd/bench_resources/testdata/standin/main.go
  • cmd/bench_resources/variant.go
  • cmd/bench_resources/variant_test.go
  • docs/concepts/security.md
  • docs/development/adr/adr-0023-tenant-policy-is-declared-once.md
  • docs/development/cmd-utilities.md
  • docs/development/tenant-policy-spec.md
  • docs/guides/http-server-mode.md
  • docs/guides/troubleshooting.md
  • internal/oauth/verifier.go
  • site/src/content/docs/es/operations/http-server.mdx
  • site/src/content/docs/es/operations/security.mdx
  • site/src/content/docs/es/operations/troubleshooting.mdx
  • site/src/content/docs/operations/http-server.mdx
  • site/src/content/docs/operations/security.mdx
  • site/src/content/docs/operations/troubleshooting.mdx

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@jmrplens
jmrplens added this pull request to stack #1095 September 30, 2026 10:18
@jmrplens jmrplens added the v3.1.0 Targeted at the 3.1.0 release label Sep 30, 2026
@jmrplens jmrplens added this to the 3.1.0 milestone Sep 30, 2026
@jmrplens jmrplens changed the title issue 950 fairness oauth Measure the OAuth verification ceiling with an OAuth driver for bench-fairness Sep 30, 2026
@github-actions github-actions Bot added security Security-related issue tooling The audit and generator commands under cmd/, and the Makefile targets that run them labels Sep 30, 2026
@jmrplens
jmrplens force-pushed the issue-950-fairness-oauth branch from d849aa4 to 7afa8c0 Compare September 30, 2026 12:05
bench-fairness could not measure ADM-014, the sixteen verification slots
issue 950 put in front of GitLab, so its cost to legitimate credentials
was stated and not measured. The mode now knows a bound named
oauth-verification.

The ceiling has no switch and must not grow one, since a flag that turns
off a security bound is a flag somebody sets in production. So both arms
start with the same flags and the arm without the ceiling runs a second
build of the same checkout, made with the go command's -overlay, in
which the one declaration that sizes the slots is replaced. The tree on
disk is never touched, the record names the line that arm ran, and a
checkout that no longer declares it, or declares it twice, stops the run
rather than comparing the binary with itself.

The stand-in GitLab answers the identity request for every token but an
invented one, which it answers 401, answers both scope introspection
endpoints 404, holds each of those requests for a configurable round
trip (-fairness-upstream-delay), and counts how many it answered and how
many it held at once. The flood presents a new invented token on every
request, forwarded for an address of its own, from as many transport
sources in 127.2.0.0/16 as keep each one under half the transport-source
failure budget, which from one source would block the flood and the
quiet population behind it. The quiet population presents a new valid
token in one request of four and reuses it once it is served, so its
two rows are the onboarding the ceiling can cost and the cached
credentials it must not.

A 401 for an invented token is counted as refused otherwise, a fifth
outcome beside served, refused, failed and timed out. The bound brings
its own plan defaults, and the plan is refused when the flood could not
hold every slot, when the lead-in or the deadline is shorter than the
wait, or when the quiet population's onboarding would fill the slots on
its own. It protects the instance rather than the quiet tenant, so the
verdict says what it cost the tenant and the saturation gate does not
apply.
The specification, the security page, the HTTP server mode guide and
the site stated the price of ADM-014 and said bench-fairness could not
measure it. They now cite `make bench-fairness BOUND=oauth-verification`
with its numbers.

With a stand-in GitLab answering in 100 ms and a flood of 400 invented
tokens a second, the ceiling served 50 of 120 new credentials, after
5.5 s where they took 0.52 s without it, and refused the other 70, in
both repetitions; cached credentials were served at 14 ms at the median
in both arms. The instance received 5,638 requests, at most 19 at once,
against 12,600 with up to 76 at once without the ceiling, and 5,612
against 30,600 under a flood of 1,000 a second. The share served
follows the round trip: 104 and 100 of 120 at 50 ms, 26 and 29 at
200 ms.

The runs also found what the pages did not say: the queue moves into
the process. Each waiting request holds its connection for up to the
five seconds, the resident set peaked at 273 MiB against 172 at 400 a
second and at 434 against 194 at 1,000, and nothing but the arrival
rate bounds how many requests wait. The specification and the guide say
so.

cmd-utilities documents the new bound, the -fairness-upstream-delay
flag, the bound defaults and the fifth outcome; ADR-0023 NEG-004 and
CLAUDE.md say how a bound with no switch is compared.
The gauge raised its peak with a compare-and-swap loop, whose retry is
taken only when another request moves the peak between the load and the
swap. No test can arrange that on purpose, so gobco reported the swap's
false branch as never taken. A mutex moves the count, the total and the
peak together with no branch to leave untaken, and a stand-in answering
a few hundred requests a second does not notice it. A test now steps the
gauge through a window: the peak keeps its high as requests leave, and a
reset starts from what is still in flight.
The mutation run over the files this layer writes left nine survivors
and a handful of uncovered mutants on its own lines. Each now dies:

- the flood-source limit is tested on its edge, a flood filling the
  127.2.0.0/16 range exactly accepted and one source more refused;
- the source count is tested for a flood that invents a token on every
  other request, where the share it multiplies by is not one;
- a bound with no instance round trip and no invented tokens is shown
  to name neither in its description;
- the deadline floor is tested a millisecond under it, where it used to
  be tested only on it and well below it, so its round-trip count was
  never pinned;
- the quiet onboarding refusal is held to the share it names;
- a rendered row's dispatched count is held to the sum of its five
  outcomes, each a distinct power of two;
- the variant replaces a declaration on a file's first line.

Two survivors were comparisons nothing could observe, and the code no
longer makes them: refusalsExpected cuts a bucket bound's excess with
max, as its slot branch already did, and pause lets a timer of zero
fire instead of guarding it.
cmd-utilities said the plan refuses a deadline shorter than the slot
wait. The floor it checks is the wait, a new credential's round trips
and a second of margin, which is what the plan's error names.
…lone

The OAuth bound's run expects the 401 an invented token earns and files
it as refused otherwise. The shape names no method, and the tally asked
it of every row, so a quiet population's call on its own or on a new
credential that the server answered 401 was filed there too rather than
as a failure. The arm then stayed comparable, and its served
percentiles were computed over the requests that survived: a broken
server counted as a fair one, which is the over-matching the classifier
exists to refuse.

The tally now consults those shapes for a request presenting an
invented credential and for no other, and a test holds a cached and a
new credential's 401 as failures beside the flood's as refused
otherwise.
Every phase of the OAuth verification bound opened on an empty queue.
The lead-in's drive returned only once its last request had landed, so
the flood stopped, every waiter drained for up to the five-second wait,
and the phase started from nothing: its first seconds served every new
credential, the flood's earliest waiters included. The records show
it. In all eight runs with the ceiling, the flood's 401 count equals
the stand-in's invented requests exactly, which holds only if no
lead-in waiter reached the instance during the phase, and at fifty
milliseconds the arm refused 1,421 of 12,000 flood requests where a
queue already formed refuses about 2,400. The published shares were
that transient, biased toward the ceiling.

The phase's schedule now begins where the lead-in's clock puts it,
whether or not the lead-in's requests have landed: the lead-in runs in
its own goroutine into its own discarded tally, and the phase opens at
start plus the lead-in, where the peak, the stand-in's counters and the
processor samples are reset. The instance's count is read once the
lead-in's stragglers have landed too, so the window's load is what
reached the instance inside it. A bucket bound gains the same thing,
since the pause used to refill its bucket.

The plan's floor for a slot bound's lead-in was the wait, on the
grounds that the queue had formed by then. It is now the time the
flood takes to settle into its queue from empty: the wait times the
flood over its excess above what the slots finish, which is when the
first request that will not be served is refused and the queue holds a
wait's worth of arrivals. That is 8.3 seconds at the default hundred
milliseconds, inside the ten-second default lead-in, and 25 seconds at
fifty, which the plan now refuses without a longer lead-in.
…a flood

The comment on the verification slots still said a new credential waits
with the flood and is refused like it for as long as the flood lasts.
The pages that cite the measurement say something else, and the
measurement agrees with them: a new credential waits in the flood's
queue and is served when a slot frees before its wait runs out, which
under a sustained flood is about the slots' share of the demand. The
comment now says that, and names the command that measures it.
…tled

The figures the pages cited were taken with every phase opening on an
empty queue, so a new credential's share and median included the
seconds in which the slots serve everyone. Re-run with the phase on the
lead-in's clock, and at fifty milliseconds with a thirty-second lead-in
the settled queue needs, the ceiling served 42 and 40 of 120 new
credentials at a hundred milliseconds rather than 50, 80 and 82 at
fifty rather than 104 and 100, 17 at two hundred and 17 under the
larger flood, after 5.5, 5.3, 6.0 and 5.5 seconds at the median. The
share tracks the slots' share of what arrives and stays below it; the
instance's load is where it was.

Cached credentials are quoted from the record rather than called the
same in both arms: every one was served in every arm, at a median of
14 ms with the ceiling and 13 to 15 ms without it at a hundred
milliseconds. The spec's "twelve requests on credentials" becomes
twelve requests a second, since the count was 360.

The claim that cached credentials are unaffected now carries its
condition. They never wait for a slot, but they share the listener with
the waiting flood, which holds one descriptor per waiting request, so
under a hard descriptor limit below the arrival rate times the wait a
cached credential opening a connection is delayed or refused with it.
The measurement ran under a limit of 1048576 and says so.

cmd-utilities.md states the lead-in rule the plan now enforces and that
the run's own 401 is expected of an invented credential alone.
The whole-run test of the OAuth verification bound ran captureStdout
inside finishWithin, and the build seams it installed wrote their
wrapped stand-ins when runFairness called them. Both are on the
goroutine finishWithin starts: captureStdout calls t.Fatalf when the
pipe cannot be made, and wrappedStandin calls t.TempDir and t.Fatalf.
An abort there ends that goroutine alone, and the test goes on against
a half-run.

The two scripts are now written on the test goroutine before the run,
and captureStdout wraps finishWithin, so only runFairness runs on the
goroutine finishWithin starts. finishWithin's own t.Fatalf now runs on
the test goroutine, inside captureStdout's inner function, whose
deferred restore of os.Stdout still runs. The two scripts sit in
t.TempDir directories of their own, so the harness removing a binary's
directory after the run still works.
This layer gave httpStatusError an Unwrap that returns the JSON-RPC
error a refusal carried in its body, so a refusal shape naming a code
reads one carried at 429 as it reads one carried at 200. coldList asks
a warm-up listing again whenever errors.As finds rateLimitCode, and the
authentication gate's per-address lockout is HTTP 429 carrying that
same code, -42900. Since the Unwrap, a credential admitted from a
locked-out address was asked again every 500 ms until the five-minute
warm-up timeout, where its comment promises that any other answer
stands.

coldList now asks again only when the refusal also came at HTTP 200,
the bucket's in-band refusal, and its table carries the lockout case.
The fairness matcher was never affected, since refusalSpec.matches
checks the status first.
…as a condition

The instance totals compared two windows of different length. With the
ceiling the stand-in counted from the phase's first request to the
answer of its last waiter, 35 seconds, and without it over 30, so 5,640
requests read as 188 a second beside the 160 the same paragraph derives
from sixteen slots at 100 ms. The spec, the HTTP server guide, the
security concept and the site's security page in both languages now
quote the rate over the counted window, which matches the slots
exactly (about 160 a second under either flood, against about 420 and
1,020 without the ceiling), and give the counts with their windows.

The condition on cached credentials named descriptors alone. The same
measurement shows the waiting flood holding memory too, about 50 KiB a
waiting request (100 MiB more for some 2,000 waiting, 250 MiB for some
5,000), with nothing bounding the waiters, so a memory limit below the
arrival rate times five seconds times that ends the process and every
cached credential with it. The same pages say so.

The new credentials' share sat a few points under the slots' share in
every configuration, which the pages left unexplained. A queue model of
the gate puts it down to the driver's schedule, the flood on an even
grid and the new credentials in clumps on fixed ticks: with both
arriving at random the two shares match. The spec and the guide say so.

The troubleshooting rows, in the guide and on the site in both
languages, still said credentials the server holds are served during a
flood and a new one gets through once it stops. They now say a held
credential never waits for a slot while the process has descriptors
and memory left for the waiting requests, and a new one is served only
if a slot frees within its five seconds (about a third of them under
the measured flood), so it may take several retries. The site's HTTP
server step on OAuth carries the same listener caveat and links to the
security page's account of the price.
gremlins reported one mutant of coldList's new status condition as timed
out: turning its second || into && makes coldList ask again a listing
that was served and a refusal it should return, every backoff. The
table ran those cases on t.Context(), which never ends, so the mutant
hung the test until the binary's own timeout rather than failing it.
Every case the context does not end now runs under a ten-second
deadline no correct answer comes near; against that mutant, applied by
hand, the test fails in ten seconds with twenty calls where it wants
one. The deadline is released with t.Cleanup, since the cases run in
parallel after the parent returns.
SonarCloud flagged endpoint(), the address every HTTP scenario posts to,
as a clear-text "http://" string (go:S5332) on #1091. The same target's
health probe and the concurrency series' profiler built theirs the same
way, so all three take loopbackURL now.
drive and driveContinuously took eight parameters each, four of them the
same plan, connections, call and presenter every window of a run shares.
They travel together now, as paceInput already does one level down. The
flood-range test compares its refusal inline rather than through a
variable that existed only for the condition.
Both builders wrote the binary as server, and os/exec on Windows runs a
file only by an extension PATHEXT lists, so the variant build test could
not start what it built: executable file not found in %PATH%. The name
now comes from serverExecutable, server.exe on Windows and server
elsewhere.
@jmrplens
jmrplens force-pushed the issue-950-fairness-oauth branch from 7afa8c0 to e1e1669 Compare September 30, 2026 13:04
@sonarqubecloud

Copy link
Copy Markdown

@jmrplens
jmrplens merged commit 6dabbc7 into main Sep 30, 2026
6 checks passed
@jmrplens
jmrplens deleted the issue-950-fairness-oauth branch September 30, 2026 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

security Security-related issue tooling The audit and generator commands under cmd/, and the Makefile targets that run them v3.1.0 Targeted at the 3.1.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant