Repository navigation
Fix/streaming bulk reads - #2
Open
ryandamour wants to merge 2 commits into
Open
ryandamour wants to merge 2 commits into
ryandamour wants to merge 2 commits into
Conversation
A filtered get() read the entire cached export into memory and filtered afterwards, and Arrow's allocator kept the freed memory, so a worker backfilling a large non-templated export (liquid_universe_prices) window by window ratcheted up until it was OOM-killed. - Push start/end/symbols into a batch-by-batch pyarrow.dataset scan for feather (Arrow IPC) and parquet exports; memory now scales with the rows returned. Feather V1 (FX per-pair exports) and unsupported column types fall back to the previous full-read-then-filter path. - Return freed Arrow memory to the OS after every read. Year-by-year loop over a 5.26M-row synthetic export: peak RSS 1,224 MB -> 604 MB, identical rows. Tests pin the memory bound (metered Arrow pool, with an unfiltered control), that filtered reads never load a whole export, result equivalence with the old path across formats and column types, the V1 fallback, memory release, and that results are never shared between get() calls (live seam values must not leak into later bulk reads; mutating a result must not affect the next one) -- the two failure modes of an in-memory frame cache. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The scanner's background read-ahead can release buffers to the test's proxy pool after to_table() returns. When the pool was destroyed first, a later, unrelated test crashed the interpreter (about 1 run in 10). Keep the metered pools for the life of the process. The library is not affected: it always uses Arrow's process-wide default pool. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.