Observed
In two cloud e2e runs on main, every cold session build that had to wait for a slot at apps/cloud/src/mcp/session-build-semaphore.ts behaved the same way (read from the exported spans of cloud/mcp-session-cap-eviction.test.ts, which opened 34 sessions and then DELETEd them at concurrency 8):
- Inits that queued waited the full
MAX_QUEUE_WAIT_MS (~10.0 s in every queued init's span, mcp.init.build_queue_timeout: true) and then built uncapped.
- Inits that were handed a slot by a release did not finish. The object was reset at the platform's 30 s
blockConcurrencyWhile limit (concurrency_reset, 32 per run), and the slot they held was never released, so the queue behind them stayed wedged.
Runs:
Both show cloud/mcp-session-cap-eviction.test.ts timing out at 180 s.
Most likely cause
releaseBuildSlot shifts the next waiter off the FIFO and calls next.grant() synchronously, from the request context of whichever Durable Object is releasing. grant resolves a promise that was created inside the waiting Durable Object's own request. In workerd, continuations and I/O are bound to the IoContext that created them, and a promise settled from another object's context appears not to resume the waiting fiber. It is the same class of problem the session Durable Object already routes around for eviction (requestCapEviction: a candidate's teardown must run in the candidate's own context, not the evictor's).
This is diagnosed from span timings only. It has not been reproduced in isolation.
Not fixed by #2060
#2060 removes one source of extra cold builds (an RPC restore followed by a second onStart under the input gate) and keeps the cap-eviction scenario's own request concurrency at MAX_CONCURRENT_BUILDS, so nothing that scenario opens ever waits in the queue. The hand-off itself is unchanged.
Possible directions
- Confirm the mechanism with a unit test under
@cloudflare/vitest-pool-workers that grants a waiter from a second Durable Object's request.
- Let a waiter re-check the counter on its own timer, in its own context, instead of being resolved from the releaser's context.
- Move the queue behind a single-context owner (a Durable Object, or a per-isolate scheduler that runs waiters through
ctx.waitUntil of their own request).
Observed
In two cloud e2e runs on main, every cold session build that had to wait for a slot at
apps/cloud/src/mcp/session-build-semaphore.tsbehaved the same way (read from the exported spans ofcloud/mcp-session-cap-eviction.test.ts, which opened 34 sessions and then DELETEd them at concurrency 8):MAX_QUEUE_WAIT_MS(~10.0 s in every queued init's span,mcp.init.build_queue_timeout: true) and then built uncapped.blockConcurrencyWhilelimit (concurrency_reset, 32 per run), and the slot they held was never released, so the queue behind them stayed wedged.Runs:
Both show
cloud/mcp-session-cap-eviction.test.tstiming out at 180 s.Most likely cause
releaseBuildSlotshifts the next waiter off the FIFO and callsnext.grant()synchronously, from the request context of whichever Durable Object is releasing.grantresolves a promise that was created inside the waiting Durable Object's own request. In workerd, continuations and I/O are bound to the IoContext that created them, and a promise settled from another object's context appears not to resume the waiting fiber. It is the same class of problem the session Durable Object already routes around for eviction (requestCapEviction: a candidate's teardown must run in the candidate's own context, not the evictor's).This is diagnosed from span timings only. It has not been reproduced in isolation.
Not fixed by #2060
#2060 removes one source of extra cold builds (an RPC restore followed by a second
onStartunder the input gate) and keeps the cap-eviction scenario's own request concurrency atMAX_CONCURRENT_BUILDS, so nothing that scenario opens ever waits in the queue. The hand-off itself is unchanged.Possible directions
@cloudflare/vitest-pool-workersthat grants a waiter from a second Durable Object's request.ctx.waitUntilof their own request).