Repository navigation
Rules-aware skip memo: re-decode only changed records, invalidate on any rule change #154
Description
Prompt
Add to crawlkit a small helper that lets a crawler skip re-decoding source records whose bytes have not changed since the last sync, and that invalidates every skip automatically when the crawler's decoding rules change. API sketch:
state.RulesSignature(versions map[string]int, rules any) []bytehashes the declared decoder and derivation versions plus a canonical, path-tagged rendering of the rule tables (strings quoted, maps and sets in key order, compiled regular expressions by pattern; panic on an unknown kind so a new table cannot be skipped silently), and a per-record memo compares(source key, byte digest, signature)before the crawler decodes. Keep record storage in the app; crawlkit owns only the signature and the comparison. Reference implementation: https://github.com/ourostack/teamscrawl/blob/main/internal/teamsdesktop/memo.go (MemoSignature,hashValue).
Why
A crawler that re-reads a large local cache on every sync pays to decode records that have not changed. teamscrawl skips them with a per-record byte digest, which made a re-sync of a real Teams cache (about 62,000 messages) do work in proportion to what changed. The subtle part is invalidation: a new credential-scrub key, deny term or regular expression must re-read everything, and hashing the rule tables themselves (rather than relying on someone remembering to bump a version) is what makes that safe.
Why an issue, not a PR
Only teamscrawl uses this today, and crawlkit's boundary asks for at least two apps. If another crawler (slacrawl, discrawl, imsgcrawl) would adopt it, we can send a PR from the proven code; say the word.
Activity
clawsweeper commentedon Oct 6, 2026
Codex review: this still needs some work.
Summary
This issue remains unimplemented, and repository policy permits automatic closure only for implemented requests; shared extraction still needs a second adopter and a maintainer scope decision.
Reproducibility: This is a feature proposal with inspected downstream source; the reported Teams cache performance was not independently exercised.
Maintainer decision needed
Next step
Decide whether to wait for a second demonstrated adopter or sponsor one before defining the shared memo API.
Review details
Best possible solution:
Keep the helper downstream until two crawlers establish a compatible signature and comparison contract with app-owned storage.
Do we have a high-confidence way to reproduce the issue?
This is a feature proposal with inspected downstream source; the reported Teams cache performance was not independently exercised.
Is this the best way to solve the issue?
The downstream helper is a useful starting point after shared adoption is established, but table hashing must preserve explicit version bumps for decoder and rule-application code changes.
AGENTS.md: found and applied where relevant.
Remaining risk / open question:
Codex review notes: model internal, reasoning medium; reviewed against b7cf3a6de3dc.
Label changes
Label changes:
No label changes.
Label justifications:
P3: This optional optimization establishes no broken crawlkit contract and awaits demonstrated shared adoption.Evidence reviewed
What I checked:
docs/boundary.md:3, b7cf3a6de3dc)state/state.go:95, b7cf3a6de3dc)worker/worker.go:25, b7cf3a6de3dc)How this review workflow works
@clawsweeper re-reviewcomments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.@clawsweeper re-reviewor@clawsweeper re-runon an open PR or issue to request a fresh review only.@clawsweeper reviewto request a fresh review only.@clawsweeper explainto ask for more context, or@clawsweeper stopto stop active automation.Reviewed October 30, 2026, 2:48 AM ET / 06:48 UTC.
260 remaining items