Skip to content

feat(claude,codex): seed only the transcripts newer than the sweep mark - #26

Merged
dacorvo merged 1 commit into
mainfrom
feat/seed-since-mark
Oct 30, 2026
Merged

dacorvo merged 1 commit into
mainfrom
feat/seed-since-mark

Conversation

@dacorvo

@dacorvo dacorvo commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

funes add claude, and funes update on a new integration release, run setup add, whose seed read every transcript to tell which ones the memory already held (#19's check: the last lines through the converter, then jq, then a grep, per file). On the reporter's 26,036 transcripts that is about 3.5 hours, nearly all of it confirming sessions the memory holds (#24).

The hooks' sweeps already keep a mark: every transcript older than it was converted as it stands. The seed now shares that mark. It stamps it before reading anything, reads only the transcripts newer than it, applies the last-turn check to those alone, and keeps the mark only when every conversion succeeded, as a sweep does. Everything is read when there is no mark, or when a session the rows of the last add list is gone from the memory (a rebuilt memory, even one the hooks refilled since). Both setups keep those rows beside their scripts now, codex too.

Migration. The mark is renamed from swept to converted. A swept left by the hooks before #25 could move past a transcript whose last turns were never converted, so no mark is trusted while it is there, and it goes once a walk has read every transcript. The first add after this release walks the history once more, which repairs those gaps. A hook publishing converted through its spool fallback while that add runs does not shortcut the walk (the setup test covers this).

Measured on this host, 979 transcripts and 1,221 sessions, a re-add on an unchanged history: the listing walk 1.3 s and the find 25 ms, where the per-file check took 18.6 s. The listing walk is one funes sessions page per 200 sessions, so about 131 pages for the reporter.

Limits

  • A conversion that fails during a full walk leaves no mark. The next hook run publishes one, swept from the spool record the add wrote, so later adds read the failed transcript again only once it changes. The hook log names it.
  • A transcript restored with an old mtime (a backup, rsync -t) is older than the mark, so no add or sweep reads it. The sweeps have that limit today.
  • An add after funes remove reads every transcript, as today: remove deletes the mark along with the spool.

Fixes #24.

🤖 Generated with Claude Code

A re-add, and `funes update` on a new release, run `setup add`, whose
seed read every transcript of the agent to tell which ones the memory
already held: the last lines through the converter, then jq, then a
grep, for each file. On a history of 26,000 transcripts that took
hours, almost all of it confirming sessions the memory holds (#24).

The hooks' sweeps already keep a mark: every transcript older than it
was converted as it stands. The seed now stamps and trusts the same
mark. It reads only the transcripts newer than it, and applies the
last-turn check to those alone. Everything is read when there is no
mark, or when a session the rows of the last add list is gone from the
memory, as after a rebuild. Like a sweep, the seed keeps its mark only
when every conversion succeeded. Both setups keep the rows beside
their scripts now, to tell a session gone from the memory.

The mark is renamed from `swept` to `converted`. A `swept` left by the
hooks before #25 could claim transcripts whose last turns were never
converted, so no mark is trusted while it is there, and it goes once a
walk has read every transcript. The first add after this release walks
the history once more, and repairs those gaps.

On 979 transcripts, a re-add on an unchanged history lists the memory
in 1.3 s and finds what changed in 25 ms, where the check took 18.6 s.

Fixes #24.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@dacorvo
dacorvo merged commit 6de42f7 into main Oct 30, 2026
5 checks passed
@dacorvo dacorvo mentioned this pull request Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

claude: seed() takes hours on a large history even when nothing changed

1 participant