Repository navigation
feat(claude,codex): seed only the transcripts newer than the sweep mark - #26
Merged
Merged
Conversation
A re-add, and `funes update` on a new release, run `setup add`, whose seed read every transcript of the agent to tell which ones the memory already held: the last lines through the converter, then jq, then a grep, for each file. On a history of 26,000 transcripts that took hours, almost all of it confirming sessions the memory holds (#24). The hooks' sweeps already keep a mark: every transcript older than it was converted as it stands. The seed now stamps and trusts the same mark. It reads only the transcripts newer than it, and applies the last-turn check to those alone. Everything is read when there is no mark, or when a session the rows of the last add list is gone from the memory, as after a rebuild. Like a sweep, the seed keeps its mark only when every conversion succeeded. Both setups keep the rows beside their scripts now, to tell a session gone from the memory. The mark is renamed from `swept` to `converted`. A `swept` left by the hooks before #25 could claim transcripts whose last turns were never converted, so no mark is trusted while it is there, and it goes once a walk has read every transcript. The first add after this release walks the history once more, and repairs those gaps. On 979 transcripts, a re-add on an unchanged history lists the memory in 1.3 s and finds what changed in 25 ms, where the check took 18.6 s. Fixes #24. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
funes add claude, andfunes updateon a new integration release, runsetup add, whose seed read every transcript to tell which ones the memory already held (#19's check: the last lines through the converter, then jq, then a grep, per file). On the reporter's 26,036 transcripts that is about 3.5 hours, nearly all of it confirming sessions the memory holds (#24).The hooks' sweeps already keep a mark: every transcript older than it was converted as it stands. The seed now shares that mark. It stamps it before reading anything, reads only the transcripts newer than it, applies the last-turn check to those alone, and keeps the mark only when every conversion succeeded, as a sweep does. Everything is read when there is no mark, or when a session the rows of the last add list is gone from the memory (a rebuilt memory, even one the hooks refilled since). Both setups keep those rows beside their scripts now, codex too.
Migration. The mark is renamed from
swepttoconverted. Asweptleft by the hooks before #25 could move past a transcript whose last turns were never converted, so no mark is trusted while it is there, and it goes once a walk has read every transcript. The first add after this release walks the history once more, which repairs those gaps. A hook publishingconvertedthrough itsspoolfallback while that add runs does not shortcut the walk (the setup test covers this).Measured on this host, 979 transcripts and 1,221 sessions, a re-add on an unchanged history: the listing walk 1.3 s and the
find25 ms, where the per-file check took 18.6 s. The listing walk is onefunes sessionspage per 200 sessions, so about 131 pages for the reporter.Limits
spoolrecord the add wrote, so later adds read the failed transcript again only once it changes. The hook log names it.rsync -t) is older than the mark, so no add or sweep reads it. The sweeps have that limit today.funes removereads every transcript, as today:removedeletes the mark along with the spool.Fixes #24.
🤖 Generated with Claude Code