Overlap store reads and sends when tier1 streams cached execution output - #922
Merged
Conversation
Production-mode requests opened, decompressed and sent each 1000-block segment before asking the store for the next one, so every segment paid a full round trip on the critical path. The walker now downloads the next segments concurrently in the background, bounded per request by a segment depth and a decompressed byte budget, sized from the last download instead of any store lookup.
The buffer each item is read into is a fresh allocation that nothing else reads, so the item can alias it instead of copying the payload.
The walker built a batch, sent it, then decoded the next one, so the compress-and-write of every batch sat on the critical path. Sends now run on a goroutine behind an unbuffered channel and the walker drains it before a segment is reported done, so order is unchanged.
The spkg and last_used writes ran before the pipeline started, so their object store round trips delayed the first block sent. They now run in the background on a detached context and the request waits for them on its way out.
🔍 Vulnerabilities of
|
| digest | sha256:3236bc5596d85bfa33d6315cf5d6694a0e1e4eacbd81db082c2e68581becb111 |
| vulnerabilities |     |
| platform | linux/amd64 |
| size | 124 MB |
| packages | 380 |
📦 Base Image ubuntu:24.04
| also known as |
|
| digest | sha256:1e0a86e57d247923571b75e0aaf48a1449cf8c543d51fb3e07a4a7d7bfa79316 |
| vulnerabilities |   |
The walker's download goroutine recovers panics and turns them into a quit message. Sends now run on their own goroutine, so a panic inside a send has to be recovered there and reported as a send error.
A hanging object store could hold a cancelled request and its slot in the active requests manager for the full write timeout. The writes are now fire and forget, and the integration tests poll for the spkg marker instead of expecting it the moment the request returns.
Each download could read whatever was unreserved at launch while only reserving the estimate, so four downloads could hold four budgets when file sizes jumped. Concurrency is now budget divided by the estimate, capped at depth, and each in-flight download reserves an even share as its read limit. A file over its share goes to the walker and doubles the estimate; a file over the whole budget still turns prefetching off.
Starting the goroutine does not block and the download only takes the lock in complete, so the unlock around it did nothing.
A one-slot notification channel does the same job with cancellation as a select case, which removes the AfterFunc and the wait loop that sync.Cond needs.
sduchesneau
commented
Sep 3, 2026
sduchesneau
left a comment
sduchesneau
left a comment
Contributor
Author
There was a problem hiding this comment.
Gives me good +20% improvements for precached stuff
UlysseCorbeil
approved these changes
Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four tier1 changes to the production-mode streaming path.
Prefetch cached execution output files. Each 1000-block segment was opened, decompressed and sent before the next one was requested from the object store, so every segment paid a full store round trip on the critical path. Tier1 now downloads the next segments while streaming the current one, bounded per request by
Tier1Config.ExecOutPrefetch: at most 4 segments ahead holding at most 64 MiB of decompressed data, and that budget is a hard cap. No size is asked of the store: the decompressed size of the last downloaded segment is the estimate for the next ones, as many download at once as estimate-sized files fit in the budget, and each in-flight download reserves an even share of the budget as its read limit, so in-flight reads never add up to more than the budget. A file over its share is left to the walker and doubles the estimate, so a chain going from quiet to busy shrinks the concurrency instead of overshooting. A file over the whole budget turns prefetching off for that request. Missing files are left to the walker's existing retry loop. Off in noop mode.Send batches on their own goroutine. The walker built a batch, sent it, then decoded the next one, so gRPC compression and the socket write sat on the critical path of every batch. Sends now go through an unbuffered channel to a sender goroutine, so one batch is compressed and written while the next is decoded. A failed send is sticky and nothing goes out after it. A panic inside a send is recovered into a send error, as the walker's own goroutine did before. The walker drains the sender before a segment is reported done, so the linear pipeline can never send ahead of cached data and message order is unchanged. Measured on a synthetic 10 MiB segment: decode 1.3 ms, marshal plus zstd 5.9 ms, serial 7.2 ms, overlapped 6.5 ms. The send side dominates, and most of it is zstd.
Stop copying payloads while decoding.
ReadNextItemreads each item into a fresh buffer nothing else touches, so the item now aliases it instead of copying the payload out.Write cache markers off the critical path. The
substreams.spkgandlast_usedwrites ran synchronously before the pipeline started. They now run fire and forget on a detached context with a 30 s timeout. The request neither starts nor exits waiting for them, so a hanging store cannot hold a cancelled request's slot. The integration tests poll for the spkg marker instead of expecting it the moment the request returns.Ordering is preserved at every step: the walker still asks for one segment at a time and only advances on a completed send, the prefetcher only answers for the segment asked, and the segmenter is immutable. Files are created atomically and never overwritten, so reading them earlier cannot pick up different content.
Tests: prefetch tests cover ordering, depth and budget bounds, the share overflow path with a held-bytes probe that never exceeds the budget, overflow of the whole budget, a missing segment and cancellation. The walker had no tests before; new ones cover block order under a slow sink in buffered and unbuffered modes, a send error stopping the segment, a panic in a send, and the exclusive end block. All pass under the race detector, 20 repetitions for the prefetch suite.
One measured non-change: egress accounting calls
proto.SizebeforeSendMsg, which looked like a double marshal. On a 100-block, 10 MiB batch it costs 11 µs against 812 µs for the marshal, so it was left alone.