Skip to content

list and info no longer load every grain to count its chunks - #191

Merged
torstei merged 1 commit into
mainfrom
bugfix/list-grain-coverage
Oct 2, 2026
Merged

torstei merged 1 commit into
mainfrom
bugfix/list-grain-coverage

Conversation

@torstei

@torstei torstei commented Oct 2, 2026

Copy link
Copy Markdown
Owner

Problem

summarize_store (behind list, info and list --json) called grain::load on each store's grain to read chunk_count() and discard the rest: the whole file, plus a copy of every filter into its own Vec. list handles stores one after another, so its peak was about twice the largest grain, not the sum. On a host with a multi-GB grain that is several GB for a command that is documented as read-only and cheap.

Change

grain::coverage(dir, name) returns the grain's size and chunk count from the commit (#189) when it still describes the file, otherwise from one buffered forward walk. It opens nothing for writing and never creates or updates a commit, so list keeps working on a store it can only read.

Measured

Three stores, grains of 68, 17 and 17 MB:

peak RSS wall
before 145 MB 0.17 s
after, warm (commit present) 7 MB 0.00 s
after, cold (no commit, read-only walk) 7 MB 0.20 s

Cold, list still reads each grain once, with a bounded buffer. It created no commit files, and it works on a read-only directory.

Tests

coverage agrees with loading the grain, with and without a commit, and does not write one; it ignores a torn tail; it is None for a missing or non-grain file.

Not in this PR

query --has and serve still call grain::load and hold about twice the grain. They need the filters, so streaming them is a separate change.

🤖 Generated with Claude Code

summarize_store called grain::load on each store's grain to read
chunk_count() and discard the rest: the whole file, plus a copy of every
filter. list walks the stores one after another, so its peak was about
twice the largest grain.

grain::coverage returns the grain's size and chunk count from the commit
when it still describes the file, otherwise from one buffered walk. It
opens nothing for writing, so list stays usable on a store it can only
read.

Three stores with grains of 68, 17 and 17 MB: list peaked at 145 MB
before, 7 MB after.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@torstei
torstei merged commit ba23451 into main Oct 2, 2026
7 checks passed
@torstei
torstei deleted the bugfix/list-grain-coverage branch October 2, 2026 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant