Repository navigation
List the registered tools once per registration, bound a registration under the race detector by its own budget, and dial closed listeners only outside the parallel phase - #1258
Conversation
…tion under the race detector by its own budget Registration listed its own tools twice over an in-memory session: once only to log how many there were, and again to build the gitlab://tools manifest, with nothing registering a tool in between. A listing marshals every tool's schemas, so on the individual surface the first listing was a fifth of a registration, and under the race detector, which runs every phase of it ten to fourteen times slower on one core, it was what took the individual surface past the 30-second bounds two tests put on it. The release rehearsal's race gate failed on both, on a runner about a third slower per core than the one that passed the same gate three days earlier: TestCreateServer_FineGrained_EveryListingNarrowedAndFinalized, whose bound on register was the HTTP liveness one, and the stdio TestTokenScope_FineGrainedTokenWithoutMetadataRead_StartsWhole, whose call budget was a hard-coded 30 seconds. The listing is now made once and its answer is both the count and the manifest's tools; countRegisteredTools is gone. A listing failure is one warning, which says the manifest is not served. Measured on the same five-core machine against main, three cold runs each: an instrumented cold registration of the individual surface goes from 16.0 to 13.1 seconds, a warm one from 6.3 to 3.4, and a fine-grained stdio start's first tools/list from 24.1 to 20.1; an ordinary build goes from 1.25 to 1.03 seconds cold and 0.47 to 0.26 warm, and the stdio answer from 1.83 to 1.56 (medians of five). That alone leaves a slow runner at the edge, so a wait on a whole registration gets a bound of its own, 30 seconds in an ordinary run as before (60 where a stdio test already allowed that) and three minutes under the detector, still a bound so a registration that never returns is reported by the test that waited for it. It is chosen through a build-tag seam on the race tag. cmd/server had none, so it gains main_race_test.go and main_norace_test.go with testCatalogBuildTimeout, following the pattern of the stdio harness, whose seam gains callTimeout for call and registrationTimeout for the tests that read the answer themselves. In cmd/server the registration bound is used for every wait on a registration: the fine-grained shell, the settling of shape registrations, the catalog wait, the HTTP test client, a stdio start's held listing and its catalog-ready line, and the server card builds. In the stdio module it bounds the first listing of an individual-surface start, held or read through call, and its catalog-ready line. TestCreateServer_StartupSelfInspection_ListsTheToolsOnce pins one listing and holds the logged count to what a client is listed. TestCreateServer_StartupInspectionIsNotChargedToTheCatalogBucket relied on the second listing being refused to notice an unmarked inspection, so it now asks for the client's first tools/list, which a charged startup would have spent. TestCreateServer_ToolManifestInspectionError now requires the warning a failed listing logs, and the tools, a resource and the prompts still served beside the missing manifest.
… can be starting a process The unit suite under the race detector failed once in cmd/server, on TestClearStaleSocket_DeadSocketIsRemoved: clearStaleSocket found the socket the test had just closed "already served by another process", and the package passed when run again on its own. clearStaleSocket was right. Starting a process copies every descriptor into the child, and close-on-exec closes the copy only when the child reaches exec, so for the moments between a sibling's fork and its exec a listener a test has closed is still listening in the child and a connect succeeds into its backlog. TestCountAlive_CountsOnlyProcessesStillRunning and TestPeerStdinIsNull_ReadsTheProcessFileDescriptor start processes in the parallel phase, and the dead-socket test ran beside them. Measured on the five-core build box: a loop of listen, keep the file, close and dial connected after the close in 0 of 20000 iterations with nothing else running, with and without the detector, and in 5184 of 20000 (6135 under the detector) with four goroutines starting /bin/sleep 0; every one of those had a Start in flight, and every accepted connection was reset a moment later. Holding syscall.ForkLock across the close and the dial still left 24 of 20000, since the lock is released when the child leaves the shared address space, before exec closes its copies, so it is not a fix. The unchanged test run 3000 times beside only those two siblings failed in 242 runs (269 under the detector), and in none with them left out. The probe test asserting that a closed httptest server is refused has the same shape: TestRunProbe_ExplicitTarget's "nothing listening" case failed once in 3000 runs, its connect reset instead of refused. Both now run outside the parallel phase. os/exec's Start returns only once the child's exec has closed those copies, since that is how it learns the exec succeeded, and no test of the package starts a process from a goroutine that outlives it, so a test outside the parallel phase dials while no copy can exist. The dead-socket fixture is one helper, deadUnixSocket, which says so, used by both dead-socket tests (the second was already sequential for its seam), and the refusal case is TestRunProbe_ATargetNothingListensOn_IsReportedRefused. Run 3000 times beside the same siblings they fail in none, with or without the detector; beside a scratch sibling that starts processes for the first 150 ms of every parallel phase, 500 runs fail in none where the unchanged tests failed 49 times (47 under the detector), plus two probe refusals. The rest of the module was swept for a test that closes a listener and then expects it refused or its port free. In cmd/bench_resources, TestFreePorts_ReturnsAPortThatCanBeBound and TestFreePorts_SuccessiveCalls_DoNotCollide both bind a port the moment freePorts releases it and both were sequential without saying why; no parallel test of that package starts a process today, and the first now says why both have to stay sequential, the second pointing at it. Every other such test either accepts any failure, which a held copy also produces when the child's exec resets the connection, or sits in a package whose tests start no process, or reserves its port outside the parallel phase, where no process is being started: test/e2e/http's freePort callers, test/e2e/collector, the harness's HTTP sessions in test/e2e/gitlab and cmd/server's pprof test. Timing alone would not protect them. A child's copy usually goes within a millisecond, but one was measured holding a closed listener for 11 ms under the detector on an idle machine and for 40 ms with its CPUs loaded.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (17)
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 SummarySummary by CodeRabbit
WalkthroughServer registration now uses one tools listing for the count and manifest, and skips manifest registration if listing fails. Tests use catalog-registration timeout budgets, including longer limits for race builds, and isolate port and socket probes from parallel execution. ChangesStartup registration and waits
Test isolation for socket and port probes
Priority: ⬆️ High Estimated code review effort: 3 (Moderate) | ~20 minutes Change: Bug fix Merge Risk: ⚪ Minimal · up to The change makes server startup list its tools once instead of twice and gives test waits longer, race-aware time limits. Socket and port tests that were flaky are now isolated from parallel runs. No merge-blocking risk was identified. Security Architecture ReviewSecurity architecture risk: ⚪ Minimal · up to No new privileges or externally reachable operations were identified. Manifest discovery remains permission-filtered, and the longer timeout budgets apply only to tests. Retained concerns Security review detailsSecurity Blast Radius
Trust Boundaries and Controls
Resilience and Maintainability Implications
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
…closed-server sites onto it (#1472) This is the first layer of #1132: the helper and the sites. The gate that keeps the pattern out comes in the next layer, and the Makefile, CI and documentation wiring in the one after it. A test that needed an instance that is gone started an `httptest` server and closed it, keeping the URL. Closing it gives the port back to the kernel at once, and whatever is given that port next can answer the request the test exists to see fail: a parallel test of the same package starting a server of its own, or a test binary of another package running beside it in the same coverage run. #1131 fixed one such flake in `cmd/bench_resources`; this moves every other site. `internal/testutil/netfixture` has two things. `DeadEndpoint(t)` holds a loopback port for the whole test and closes every connection it accepts before reading a byte, and its cleanup waits for the accept loop to end. An accept that fails for any reason but the listener being closed, such as a process out of file descriptors while several test binaries run at once, is retried after a pause, as `net/http`'s server does: a loop that ended there would keep the port with nothing accepting, and the kernel would queue every connection made to it, so a request would wait out its deadline instead of failing at once. `NewStoppableServer(t, handler)` answers through its handler until `Stop`, then hangs up on every request, the ones on a connection a client kept from before included, closes the connections already open, and keeps its port. Each gives its port back in the cleanup it registers, which runs after every cleanup registered later and before any registered earlier, so a cleanup registered before either is asked for must not use its address; their doc comments say so. The bench package's `closedEndpoint` moves onto `DeadEndpoint`. The issue asked for the helper in `internal/testutil`, but the code does not allow it: `internal/testutil` imports `internal/gitlab`, so `internal/gitlab`'s own tests, which hold four of the sites, cannot import it (`import cycle not allowed in test`). The package sits beside `testutil` instead, the way `internal/testutil/hints` does for the same reason, imports nothing of this module, and is added to the test support `TestDependencies_TestSupport_NeverReachesTheServerBinary` keeps out of the server binary. The sites are the sixteen the gate in the next layer finds on this branch's parent, in seven packages. Fourteen are the issue's own rows that still exist, at the line numbers they have moved to: the issue lists fifteen, and `TestCredentialRejected_OnlyAnExplicitRefusalCountsAsOne` went away with #1139, so the issue's three credential tables in `internal/gitlab` are two now (`TestCheckCredential_ThreeAnswers_KeepsNoVerdictApart` is `TestCheckCredential_FiveAnswers_KeepsEachApart`). The other two arrived with #1146, after the issue was opened: `TestAdmitToken_AssumedScopes_SaySoInTheAdmission` in `internal/oauth` and `TestDetectToken_NoAnswer_IsNothingKnown` in `internal/gitlab/scopes_test.go`. One more of the issue's rows changed shape before this: `TestRunProbe_ExplicitTarget`'s "nothing listening" case had been moved into a sequential test of its own (#1258) because the reset it could meet broke its `refused` assertion. I read each site before changing it. As decided on the issue on 2026-10-17, the probe stops asserting `refused`: an accepted and closed connection ends with an EOF or a reset. The case goes back into the parallel table on a dead endpoint and holds what the probe is for, an unhealthy exit that names the target that gave no answer, and the sequential test is gone. The comments that described a refused connection in `internal/oauth`, `internal/gitlab` and `cmd/internal/apidocs` now say what happens. `TestNewGitLabVerifier_NetworkErrorWithCache` is the one site that needs an instance to answer first and then stop, with the verifier and the cache bound to its URL, so it uses `NewStoppableServer`. Its comment claimed a failed re-validation removes the cached entry, but the test deletes the entry itself, standing in for its expiry, since a cached identity is answered without asking the instance. The comment now says what the test holds: a re-verification against an instance that stopped answering admits nothing and caches nothing. The two credential tables in `internal/gitlab` built their client on a live server and closed it for the unanswered row; building a client sends nothing, so that row now builds it on the dead address from the start, through a small `credentialEndpoint` helper, and `internal/serverpool` does the same for its unreachable row. Three packages whose tests this layer touches were below the bar on `main`, and five commits close them. `cmd/audit_graphql_documents` (97.0% of statements) ran `main` only in a child process, whose coverage counters are not the test's: `main` now hands its arguments, the environment and both streams to `runMain` and ends the process through `exitProcess`, and two tests in the test process cover both; the child-process tests stay. `test/e2e/modeleval/internal/provider` (96.4%) had failure paths held for one adapter and not the others, a few cases no test built, and four branches no run can reach: those are reached through the catalog builder handed to `buildFakeCatalog`, the `encodeCall` variable and the cached catalog error, or removed where the code could not fail (a second decode in `producedValue`, an encoding error in `normalizeSchema`), and its unknown-provider test now holds the refusal it is named for. At 100% of statements it still had twenty conditions evaluated one way only and 28 mutants lived and 21 uncovered, all in code this layer did not otherwise change; a commit of its own answers each of them, with tests for what nothing held (the echo checks of all three adapters, a call named for its position, the wrong-args variant planting one fact, an individual tool name a sibling took, every side of the HTTP status classification and of the price window, the latency in milliseconds, the default client's timeout), four switches with no tag rewritten as ifs because gremlins cannot see into their case expressions, and two checks removed that could never decide anything. `cmd/internal/apidocs` had a loop condition that was never false, since the loop always left through the break after its last attempt, a status classification in that loop written as a switch with no tag, whose six case mutants gremlins reported uncovered although each dies when applied by hand, and a const group whose one exported name go doc read as undocumented, so the godoc audit reported it. One commit fixes the loop and rewrites the classification as a chain of ifs gremlins can measure, and another the const group. The testing README's account of those six mutants now says so. The godoc audit also reported, in packages this layer touches, three comments go doc does not read where they stand, none of them new here: a second package comment in `cmd/bench_resources`, because `fairness.go`'s file comment sat directly on its package clause; `StartIdleEviction`'s comment in `internal/serverpool`, run into `recoverSweep`'s two functions above it, which left the method undocumented; and, under `--include-tests`, eleven tests of `cmd/internal/apidocs`, the one this layer moves among them, and three of `internal/serverpool` with no doc comment at all. The last commit puts each comment on what it documents. How I tested it. The netfixture tests were written first and failed to build on the parent (`undefined: DeadEndpoint`, `undefined: NewStoppableServer`), and the accept-loop test failed on the loop that ended at the first failed accept (`read pipe: i/o timeout, want the EOF of a hang-up`). They pass now, 50 times in a row under `-race`. They hold that every request to a dead endpoint ends without an answer and promptly, that its port cannot be listened on while the test runs and is free once it ends, that the accept loop survives a failed accept and still ends once the listener is closed, that a stopped server hangs up on a kept connection and on a new one and is never reached again, that `Stop` cuts a response being streamed, and that a listener that cannot be opened fails the test naming why. The package is at 100% of statements, 10/10 conditions under gobco, and 4 mutants killed with none lived or uncovered. The new `runMain` tests failed to build on their parent (`undefined: runMain`, `undefined: exitProcess`); `cmd/audit_graphql_documents` is now at 100% of statements, 154/154 conditions, and 88 mutants killed with none lived or uncovered. `test/e2e/modeleval/internal/provider` is at 100% of statements, 406/406 conditions, and 235 mutants killed with none lived; the two it reports uncovered are the operators of the constant `180 * 24 * time.Hour` that sizes the price window, which carries no statement counter, and the price tests hold that window to 180 days at its edge and past it. `cmd/internal/apidocs` is at 100% of statements and 110/110 conditions, and with the switch gone its mutants went from 62 killed, 4 lived and 10 uncovered to 67 killed, 4 lived and 4 uncovered, every survivor of a kind `docs/development/testing/README.md` records under "Reading a survivor": the four lived are the three boundaries whose sides agree and the clock boundary that page names for this package, and the four uncovered are operators in constant initializers. The packages this layer touches only in their tests are at the bar too. `internal/oauth` is at 224/224 conditions and 114 mutants killed with none lived or uncovered; `internal/gitlab` at 708/708 and 359 killed, none lived and 8 uncovered; `internal/serverpool` at 418/418 and 215 killed, none lived and 6 uncovered. The fourteen uncovered are operators in package-level constant and variable initializers, such as `30 * time.Second` and `config.EnvPrefix + "ALLOW_PRIVATE_INSTANCES"`, which carry no statement counter: the tool artifact the same page records. `cmd/server` and `cmd/bench_resources` are at 2270/2270 and 1544/1544 conditions. This layer changes only their tests (and a blank line in `fairness.go`), and a mutation run of either whole package means thousands of mutants each running the package's whole suite, so I measured the files those tests exercise: `cmd/server`'s `probe.go` has 83 mutants killed, none lived and 2 uncovered, the operators of its two timeout constants, and `cmd/bench_resources`' `sessions.go` and `held.go` have 134 killed with none lived or uncovered. Every moved test passes under `-race`, every touched package passes in full under `-race` and is at 100% of statements, and `golangci-lint` with every e2e tag is clean on all of them. The gate in the next layer, run on this branch's parent, reports exactly these sixteen sites and exits 1; on this branch it exits 0. Refs #1132



The release rehearsal of 2026-10-08 (run 37707835685, on main at 7519408) failed its race gate in two jobs. In the unit suite,
TestCreateServer_FineGrained_EveryListingNarrowedAndFinalizedstopped atregister did not return within 30s. In the stdio transport module, thea_paid_namespace_plancase ofTestTokenScope_FineGrainedTokenWithoutMetadataRead_StartsWholestopped atthe server did not answer within 30s, and the server's own log shows why: 24 seconds between detecting the tier and finishing the registration of 1088 individual tools. The scheduled race run of 2026-10-05 and the rehearsal of 2026-10-01 passed the same gate.Nothing got slower between those runs. I timed main at both ends and in between, with and without the detector, and they agree within noise. What changed is that the bound on
registerin the unit test is new (#1206 added it on 2026-10-06, and the race workflow runs only on Mondays and in rehearsals, so this rehearsal was the first run under the detector to meet it), the stdio budget had only about five seconds of margin over the start it waits for even on a fast machine, and the runner of 2026-10-08 was about 1.3 times slower per core than the one of 2026-10-05. Under the race detector a registration of the individual surface runs ten to fourteen times slower than an ordinary one, on one core, so a fixed 30 seconds was a bet on the runner.Two things were wrong, and this fixes both.
Registration listed its own tools twice over an in-memory session: once only to log how many there were, and again to build the
gitlab://toolsmanifest, with nothing registering a tool in between. A listing marshals every tool's schemas, so the first of the two was about a fifth of a registration, with or without the detector. It is now made once and its answer is both the count and the manifest's tools;countRegisteredToolsis gone, and a failed listing is one warning that says the manifest is not served.Measured on the same five-core machine against main at 7519408, cold processes, median of three runs under the detector and of five without:
registerinside the failing unit test, race, 4 CPUs, the test alone (two runs of its whole package)That alone does not get a slow runner comfortably under 30 seconds: 20 seconds on this machine is about 27 on the runner that failed. So a wait on a whole registration now has a bound of its own, chosen by a build-tag seam on the race tag: 30 seconds in an ordinary run, as before (60 where a stdio test already allowed that), and three minutes under the detector, which is four times the worst registration I measured (43 seconds on half a core, before this change) and still a bound, so a registration that never returns is reported by the test that waited for it.
cmd/serverhad no such seam, so it gainsmain_race_test.goandmain_norace_test.gowithtestCatalogBuildTimeout, following the pattern of the stdio harness, whoseharness_race_test.goandharness_norace_test.gogaincallTimeoutforcallandregistrationTimeoutfor the tests that read the answer themselves. Incmd/serverthe new bound is used for every wait on a registration: the fine-grained shell, the settling of shape registrations, the catalog wait, the HTTP test client (whose comment already said its budget was about the first request building a catalog under the detector), a stdio start's held listing and its catalog-ready line, and the server card builds. In the stdio module it bounds the first listing of an individual-surface start, held or read throughcall, and its catalog-ready line. Every other liveness wait keeps the bound it had.TestCreateServer_StartupSelfInspection_ListsTheToolsOncepins the single listing (it fails on main, with two) and holds the logged count to what a client is listed.TestCreateServer_StartupInspectionIsNotChargedToTheCatalogBucketused to notice an unmarked self-inspection because the second listing was refused on a bucket that holds one; with one listing that refusal no longer happens, so it now asks for the client's firsttools/list, which a charged startup would have spent, and I checked that it fails with the inspection mark removed.TestCreateServer_ToolManifestInspectionErrornow requires the one warning a failed listing logs, and that the tools, a workflow guide and the prompts are still served beside the missing manifest, which gives the negative check above a positive twin.Running the whole unit suite under the detector on this branch turned up one more failure, unrelated to registration:
TestClearStaleSocket_DeadSocketIsRemovedfound the socket it had just closedalready served by another process, and passed when the package ran again.clearStaleSocketwas right. A child process gets a copy of every descriptor at fork and loses the close-on-exec ones only at exec, so while a sibling test is between the two, a listener a test has closed is still listening in the child and the probe's connect succeeds. Beside only the package's two tests that start processes during the parallel phase, the unchanged test failed in 242 of 3000 runs (269 under the detector) and in none without them, and the "nothing listening" case ofTestRunProbe_ExplicitTargetfailed once in 3000 the same way, its connect reset instead of refused. Both now run outside the parallel phase, where no test can be starting a process:deadUnixSocketsays why, once, for the two dead-socket tests, and the refusal case isTestRunProbe_ATargetNothingListensOn_IsReportedRefused. They pass all 3000 runs beside the same siblings, with and without the detector, and all 500 beside a scratch sibling that starts processes for the first 150 ms of every parallel phase, where the unchanged tests failed 49 times. I swept the module for the same shape. Incmd/bench_resources,TestFreePorts_ReturnsAPortThatCanBeBoundandTestFreePorts_SuccessiveCalls_DoNotCollideboth bind a port the momentfreePortsreleases it and were already sequential; no parallel test of that package starts a process today, and the comments now say why both have to stay sequential. Every other test that closes a listener and then expects it gone either accepts any failure, sits in a package whose tests start no process, or reserves its port outside the parallel phase (test/e2e/http'sfreePortcallers,test/e2e/collector, the harness's HTTP sessions intest/e2e/gitlab,cmd/server's pprof test), where no process is being started. Timing alone would not protect them: a child's copy usually goes within a millisecond, but one was measured holding a closed listener for 11 ms under the detector and for 40 ms with the CPUs loaded.Gates
On the tree with both commits, through
rgoon four CPUs to match the hosted runner, with the race workflow's own commands: the unit job (go test -race -count=1 -timeout 60m ./cmd/... ./internal/...with the request inventory recording) passed 288 packages in 1128 s with no data race,TestCreateServer_FineGrained_EveryListingNarrowedAndFinalizedincluded; the stdio transport job passed 57 of 57 in 761 s,TestTokenScope_FineGrainedTokenWithoutMetadataRead_StartsWholein 39.75 s across both its cases; the HTTP transport job passed 193 with 8 skips (seven need Docker, one skips itself under the detector), and the collector job passed what it can without Docker. Without the detector the unit suite and the stdio module pass.go build,go vet(with the stdio and HTTP tags too), the e2e compile, golangci-lint on the touched packages,make check-test-file-names,check-test-goroutines,check-test-subtests,check-dead-consts,check-tenancyandgen_testing_docs --checkall pass.main.gois the only production file whose code changed. Its mutation run (gremlins scoped to it) killed 349 with none living, both mutants on the changed lines among them; the 16 not covered are package-level constant expressions that are not covered on main either, at the same lines. Condition coverage ofcmd/serveris 2236 of 2236 and statement coverage 100 %. After the review fixes, which touched only tests and comments, the wholecmd/serverpackage under the detector, the stdio tests whose bounds changed, the bench tests, both packages without the detector, the stdio module, coverage and lint were run again and pass.This unblocks the release rehearsal that #1232 is waiting on.