Repository navigation
perf(index): leave secret redaction to push - #296
Merged
Merged
Conversation
bench_backends timed every compiled backend side by side, with ONNX as the reference. ONNX is no speed target, but it is a sound accuracy reference. So bench_models times the embedder and the reranker on one backend chosen with --backend (blas, or metal in a macOS build), and score_models scores blas and metal against ONNX, which it requires. Both now cover the Metal embedder, and neither runs bge-reranker-base, the speed baseline of the switch to gte. The workloads move to benchmark/workloads.rs, which both include. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
funes reranks with gte-reranker-modernbert-base on every backend, and bench_backends was the last thing that ran bge-reranker-base, as the speed baseline of the switch to gte. The BERT encoder's XLM-R position offsets and roberta. weight prefix go with it, since only that model used them. The embedders still match ONNX as before in score_models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An index run writes rows, embeds them, then compacts the memory, and the embedding takes long enough to hide the rest. index::write_rows runs everything but the embedding, and bench_index times it in a scratch memory, over generated sessions or a directory of turns files. It reports the rows written and their rate, the fragments the writes leave for the compaction, and the time the end of the run takes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
record rewrote state.json and index-coverage.json after every unit, and each write holds every unit the run has seen, so a run over many small sessions paid a cost that grew with the square of their count. The state is now persisted once per batch, after a record, and the batch's spool files are drained after that write. A run that fails mid-batch re-reads that batch on its next run, where the stored chunk ids keep its rows from being written twice. bench_index, rows written before and after: - 5,000 day-zero sessions: 181.6 s, then 167.6 s - 2,000 generated sessions: 64.9 s, then 64.1 s Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The push gate held back every block that still held a secret, and its rows waited for funes scrub. It now redacts each such block as scrub does: the secret excised from the block's text, the text re-split by the same chunker, the result scanned again and re-embedded. The pieces are published under their rows' ids, so the remote holds the ids the local memory does, and the local memory keeps its rows as they are. A block that can't be redacted in place is still held back whole: a secret in an encoded region, a redaction that still scans dirty, or one that re-splits into a different number of pieces. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Indexing spawned TruffleHog for every batch of 32 sessions, and so for every agent turn the hook indexed: about 1 s of CPU and 160 MB each, for a few kilobytes of text. Push redacts what it publishes now, so indexing stores sessions as written and only elides data URIs. The local memory is no publication boundary, and push and scrub still refuse to run without a scanner. bench_index, the two builds alternated, rows written before and after: - 5,000 day-zero sessions: 179.4 s, then 20.7 s (294.6 s, then 34.6 s in a busier first round) - 2,000 generated sessions: 65.4 s, then 4.9 s The end of the run, merging and indexing, takes the same time in both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…written SECURITY.md still said credentials are redacted at index time, and that the gate withholds every row holding a secret. Indexing now stores sessions as written, and the gate redacts what it can and withholds the rest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The GPU embedder embeds a repeated text once, and four workloads repeated theirs: 30×~500tok is one text, 100×real and 256×real hold 32, 256×mixed 17. bench_models --backend metal timed those few texts while it reported the whole workload, up to 30x too fast. Each document of those workloads now starts with its number. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
workloads.rs opened by naming bench_models and score_models, its two users. It also called 256 documents the production batch size, which holds on the CPU only: an indexing call on the GPU takes 256 to 2,048 texts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its name and doc said a directory is indexed a batch at a time, which no assertion has checked since indexing stopped spawning the scanner once per batch. take_batch's unit test covers how a batch fills. The test now says what it asserts, that indexing runs no secret scanner, and its file is named for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bench_index's docs and RowsWritten listed scanning for secrets among the work of writing rows, which indexing no longer does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gate's comments said it redacts the secrets the rows "still" hold, from when indexing redacted first. Skipped said rows are held back for holding a secret, when they are held back only where their block can't be redacted in place. The escaped-key test told the story of the leak it once caught, and the gate's doc kept an em dash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Several described their callers instead: write_rows named the bench that times it, record and persist said how often write_units writes the state, and write_unit said what write_units does with its spool file. Why the state is written once a batch now sits where it is written. write_unit no longer calls its turns sanitized, since only their data URIs are elided. BATCH_UNITS gives no reason for 32: the reason was the TruffleHog spawn, which indexing no longer runs. The two lines the branch touched with em dashes lose them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The field was called drained, but it holds the spool files of a batch that are not drained yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gate's result named them held, as the push coverage's field does, and run_push renamed them skipped on the way out. One name now runs through. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dropping bge-reranker-base from it left a short line mid-paragraph. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Indexing deduplicated chunk by chunk, so a block read again with a longer text kept its stored pieces and took the extra ones. funes 1.12 and older stored a block with its secrets cut out, so this branch re-reading such a session stored the raw tail of the secret behind the redacted head. The push gate rebuilds the block from both, finds no secret without the head, and would publish the tail. A block is now deduplicated by its first piece: one already in the memory, or met earlier in the unit, takes none of the new pieces. An edit re-emitted under its turn_uuid is then deduped away whole, as docs/funes-jsonl.md says. Reproduced with the v1.12.0 release and a 4096-bit RSA key: 33 of its 47 lines were stored after the re-read, and scrub found nothing. With this commit, none are stored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A block's text renders its tool name, which is also a column of its own. The push gate cut a secret found there out of the text, then re-split the text under the original tool name, so the redacted rows uploaded the secret in that column, out of the rescan's sight. Scrub re-splits the same way, so a scrubbed block carried such a secret to the next push too. resplit now takes the tool name, and both callers pass it with the same findings cut out as from the text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 of 9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Indexing spawned TruffleHog for every batch of 32 sessions, so for every agent turn a hook indexed: about 1 s of CPU and 160 MB each, for a few kilobytes of text. It also rewrote the whole index state after every session. This PR leaves secret redaction to push, where a session leaves the machine, and writes the state once per batch.
Behavior
funes indexstores sessions as written and only elidesdata:URIs. It no longer needs TruffleHog.funes pushredacts the secrets its gate finds, the wayfunes scrubdoes, where it held their blocks back. The redacted chunks are published under the stored rows' ids, and the local memory keeps its rows as they are. A block it can't redact in place is held back whole: a secret in an encoded region, a redaction that still scans dirty, or one that re-chunks into a different number of chunks.tool_namecolumn. Scrub on main redacted the name in the text only, so a secret in a tool name reached the next push in that column.state.json,index-coverage.json) is written once per batch of up to 32 sessions instead of after every session. Each write holds every session the run has seen, so the cost grew with the square of their count. A run that stops mid-batch re-reads that batch, and the stored chunk ids keep its rows from being written twice.The local memory now holds credentials as written, like the transcripts it reads. SECURITY.md, docs/push.md and docs/automation.md say so.
Benchmarks
bench_index(new) times the part offunes indexthat writes rows, without the embedding.bench_backendsis split intobench_models, which times the embedder and the reranker on one backend, andscore_models, which scoresblasandmetalagainst ONNX. Both cover the Metal embedder.roberta.weight prefix that only it used.bench_backendswas the last thing that ran it.bench_indexrow-writing time on an M2 MacBook Air, each change against the commit before it:🤖 Generated with Claude Code