RPC calls from a regionally placed Worker have long delays before the API method starts #7707

Our frontend calls a separate API Worker through private service-binding RPC. The API uses PostgreSQL in Frankfurt.

When we set placement.region: "aws:eu-central-1" on the frontend, RPC calls to the API sometimes took hundreds of milliseconds before the API method span began. This also happened when the method returned cached data and issued no SQL.

RPC timings

Two catalog RPC traces showed:

Both calls hit the cache. Some slow API calls came from frontend instances that had already handled requests.

In a separate 12-call comparison, the median wait for the same catalog RPC was 11 ms from an edge caller and 304 ms from a Frankfurt-placed caller. We measured these waits inside the caller, excluding browser latency and forwarding to the Frankfurt caller. This batch used 100% trace sampling. The deployment reset the API's cache, though the two traces above were cache hits.

Effect on page requests

With unchanged application code, we compared 32 sequential page requests per placement. Requests alternated between guest and authenticated sessions.

The 11 first-observed requests in Frankfurt instances had a median TTFB of 1,452 ms. The other 21 requests had a median of 282 ms. The edge instances had received earlier probes, so those batches do not measure fresh-instance startup.

A small control without a database or application dependencies also reached more instances with regional HTTP placement. Across 16 calls, the placed Worker reached 12 instances and the edge Worker reached one. After we swapped their placement settings, the counts were 10 and one.

Placement and RPC

Setting the region hint only on the API did not move its custom RPC calls to Frankfurt. Placing the frontend moved the downstream RPC execution there. Separate outbound probes confirmed this location change.

This matches the documented placement limitation, but differs from the page's example of an edge frontend calling a placed backend through RPC. cloudflare-docs#30840 reports that contradiction.

Expected behavior

Is this expected behavior with regional placement? In these comparisons, placement increased RPC latency and made page performance worse. The long delays before API methods began and the higher instance count look like a possible bug, or a performance cost that the placement documentation does not explain clearly.

We would like to understand what causes these delays and whether there is a way to diagnose them.

We ran the comparisons with Milan ingress at low concurrency.

AI assisted with the testing and wrote this issue report. I validated and double-checked the measurements and the report myself.