Skip to content

MCP session build semaphore: waiter granted from another DO request context never resumes #2063

Description

@RhysSullivan

Observed

In two cloud e2e runs on main, every cold session build that had to wait for a slot at apps/cloud/src/mcp/session-build-semaphore.ts behaved the same way (read from the exported spans of cloud/mcp-session-cap-eviction.test.ts, which opened 34 sessions and then DELETEd them at concurrency 8):

  • Inits that queued waited the full MAX_QUEUE_WAIT_MS (~10.0 s in every queued init's span, mcp.init.build_queue_timeout: true) and then built uncapped.
  • Inits that were handed a slot by a release did not finish. The object was reset at the platform's 30 s blockConcurrencyWhile limit (concurrency_reset, 32 per run), and the slot they held was never released, so the queue behind them stayed wedged.

Runs:

Both show cloud/mcp-session-cap-eviction.test.ts timing out at 180 s.

Most likely cause

releaseBuildSlot shifts the next waiter off the FIFO and calls next.grant() synchronously, from the request context of whichever Durable Object is releasing. grant resolves a promise that was created inside the waiting Durable Object's own request. In workerd, continuations and I/O are bound to the IoContext that created them, and a promise settled from another object's context appears not to resume the waiting fiber. It is the same class of problem the session Durable Object already routes around for eviction (requestCapEviction: a candidate's teardown must run in the candidate's own context, not the evictor's).

This is diagnosed from span timings only. It has not been reproduced in isolation.

Not fixed by #2060

#2060 removes one source of extra cold builds (an RPC restore followed by a second onStart under the input gate) and keeps the cap-eviction scenario's own request concurrency at MAX_CONCURRENT_BUILDS, so nothing that scenario opens ever waits in the queue. The hand-off itself is unchanged.

Possible directions

  • Confirm the mechanism with a unit test under @cloudflare/vitest-pool-workers that grants a waiter from a second Durable Object's request.
  • Let a waiter re-check the counter on its own timer, in its own context, instead of being resolved from the releaser's context.
  • Move the queue behind a single-context owner (a Durable Object, or a per-isolate scheduler that runs waiters through ctx.waitUntil of their own request).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions