Skip to content

rs-dapi: coalesce concurrent getStatus cache misses into one upstream refresh #5900

Open

Description

@romchornyi

Expected Behavior

When the getStatus cache entry is missing, concurrent callers share one upstream refresh: one caller builds the status from Drive getStatus and Tenderdash status / net_info, the others wait for that result and read it from the cache. The generation check that keeps a status read across a block from being cached stays as it is.

Current Behavior

get_status_impl caches only completed responses and has no in-flight sharing. Every caller that misses the cache calls build_status_response_with_health, which issues three upstream RPCs:

trace!(?key, "get_status cache miss; building response");
// Build fresh response and cache it, noting the cache's generation first: a block
// clearing the cache while the status is read leaves it a block behind, and caching it
// would serve that height until the next block
let generation = self.platform_cache.generation();
match self.build_status_response_with_health().await {
Ok((response, health)) => {
trace!(
drive_error = ?health.drive_error,
tenderdash_status_error = ?health.tenderdash_status_error,
tenderdash_netinfo_error = ?health.tenderdash_netinfo_error,
"get_status upstream fetch completed"
);
self.platform_cache.put_from_generation(
key,
&response,
response.encoded_len(),
generation,
);
Ok(Response::new(response))
}
Err(status) => Err(status),
}
}

The cache is cleared on every new block, so right after each block all concurrent getStatus requests refresh the status independently until the first response is stored.

This already applies to the public getStatus endpoint. With #5659, broadcastStateTransition also reads the status (to check the size against the active protocol version, with a 2 s timeout), so concurrent broadcasts after a block add to the same duplicate upstream work. The timeout bounds each caller's wait, not the number of refreshes.

Possible Solution

Single-flight the refresh in get_status_impl: on a miss, take a per-service tokio::sync::Mutex<()> (or an equivalent in-flight future), check the cache again, and build only if it is still empty. Keep put_from_generation so a status built across a block is not cached.

PlatformServiceCache::get_or_try_insert already coalesces through quick_cache, but it has neither the 10 s TTL nor the generation check getStatus relies on, so it does not fit as is.

A concurrent cold-cache test needs a seam for the upstream calls; get_status_impl has no async tests with mocked upstream today.

Alternatives Considered

  • Keep the last known Drive protocol version separately (e.g. an atomic updated whenever a status is built) and let broadcasts read it without a lookup. Removes broadcasts from the refresh path, but the value can go stale when nobody calls getStatus, so it needs a background refresh.

Additional Context

Raised by the automated review on #5659 (#5659 (comment)), deferred there because the behavior predates the PR and the fix changes the public getStatus path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions