Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
8b581c3
refactor(bench): split bench_backends into bench_models and score_models
dacorvo Nov 12, 2026
35b8610
refactor(inference): drop bge-reranker-base
dacorvo Nov 12, 2026
19a8423
feat(bench): time an index's row writing with bench_index
dacorvo Nov 12, 2026
6f0a1d4
perf(index): persist the index state once per batch
dacorvo Nov 13, 2026
df922a6
feat(push): redact the secrets a push finds instead of holding them back
dacorvo Nov 13, 2026
d018c19
perf(index): leave secret redaction to push
dacorvo Nov 13, 2026
73b2858
review: say in SECURITY.md that push redacts, and indexing stores as …
dacorvo Nov 13, 2026
f7164d6
review: give every workload document its own text
dacorvo Nov 13, 2026
77bf690
review: describe the workloads, not the programs that run them
dacorvo Nov 13, 2026
0657faf
review: claim in the scanner test only what it asserts
dacorvo Nov 13, 2026
2bc8c42
review: leave the secret scan out of what bench_index times
dacorvo Nov 13, 2026
269820b
review: say why push holds a block back, without an earlier redaction
dacorvo Nov 13, 2026
8494cc2
review: keep the index's doc comments to what each thing is
dacorvo Nov 13, 2026
3ec1a76
review: name the spool files left to drain for what they are
dacorvo Nov 13, 2026
2fbb324
review: call the rows a push holds back held, as the coverage does
dacorvo Nov 13, 2026
953b9b2
review: re-wrap the backends paragraph of AGENTS.md
dacorvo Nov 13, 2026
12e8f4a
review: keep a stored block to the pieces it was first stored with
dacorvo Nov 13, 2026
632814f
review: redact a block's tool name along with its text
dacorvo Nov 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,9 +65,10 @@ default `blas` (src/inference/blas.rs, hand-written forwards on Accelerate/faer)
(fastembed/ort). On macOS the default build also carries `metal` (src/inference/metal.rs): the same
encoders as MPSGraphs in fp16, which the embedder hands calls of 32+ texts to and the reranker calls
of 8+ pairs, falling back to `blas` where no GPU starts. `blas` and `metal` each keep one module per
encoder architecture: `bert.rs` (the embedders, and bge-reranker-base on the CPU) and `modernbert.rs`
(gte-reranker-modernbert-base, the reranker). `onnx` runs the same reranker, from the ONNX export in
gte's Hub repo. CI lints and unit-tests the default build on Linux and on macOS 15, each with the
other's seam compiled out, and `onnx` on Linux: so also run
`cargo clippy --all-targets --no-default-features --features onnx` before calling work done;
`cargo run --release --features onnx --example bench_backends` A/Bs them (latency + agreement).
encoder architecture: `bert.rs` (the embedders) and `modernbert.rs` (gte-reranker-modernbert-base,
the reranker). `onnx` runs the same reranker, from the ONNX export in gte's Hub repo. CI lints and
unit-tests the default build on Linux and on macOS 15, each with the other's seam compiled out, and
`onnx` on Linux: so also run
`cargo clippy --all-targets --no-default-features --features onnx` before calling work done.
`cargo run --release --features onnx --example score_models` scores `blas` and `metal` against
`onnx`, and `cargo run --release --example bench_models -- --backend <blas|metal>` times one of them.
24 changes: 19 additions & 5 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -76,19 +76,33 @@ metal = [
"dep:half",
]
# ONNX inference backend (fastembed/ort). When both backends are compiled in, `blas` is the one
# funes runs; the ONNX types are the reference backend for bench_backends.
# funes runs, and the ONNX types are the reference score_models measures it against.
onnx = ["dep:fastembed"]

# Benchmarks live under benchmark/ alongside their README; run via `cargo run --example <name>`.
[[example]]
name = "bench_recall"
path = "benchmark/bench_recall.rs"

# A/B the inference backends (BLAS vs ONNX as reference). Needs both compiled in:
# cargo run --release --features onnx --example bench_backends
# Time the part of `funes index` that writes rows, without the embedding:
# cargo run --release --example bench_index -- --sessions 2000 2>/dev/null
[[example]]
name = "bench_backends"
path = "benchmark/bench_backends.rs"
name = "bench_index"
path = "benchmark/bench_index.rs"

# Time the embedder and the reranker on one backend, blas or, in a macOS build, metal:
# cargo run --release --example bench_models -- --backend metal
[[example]]
name = "bench_models"
path = "benchmark/bench_models.rs"
required-features = ["blas"]

# Score the backends against ONNX Runtime, which it needs compiled in:
# cargo run --release --features onnx --example score_models
[[example]]
name = "score_models"
path = "benchmark/score_models.rs"
required-features = ["onnx"]

# A default `funes recall` scored against a labeled set of questions.
[[example]]
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,9 +106,9 @@ funes recall "..." --memory <user|org>/funes-memory # read any remote memory f
```

That second form is how you read **someone else's** published memory on a topic, without touching
your own setup. Publishing is guarded: funes redacts credentials at index time, and a separate,
always-on gate refuses to push any chunk that still contains a secret. And because a published memory
is just a dataset, you can try recall **right now**, before indexing anything of your own:
your own setup. Publishing is guarded: an always-on gate redacts credentials from what it publishes,
and holds back what it can't redact. And because a published memory is just a dataset, you can try
recall **right now**, before indexing anything of your own:

```bash
funes recall "why is funes append-only" --memory huggingface/funes-memory
Expand Down Expand Up @@ -185,7 +185,7 @@ back to the CPU where no GPU starts. An ONNX Runtime backend is available as an

```bash
cargo build --release --no-default-features --features onnx # ONNX backend instead
cargo run --release --features onnx --example bench_backends # A/B both backends
cargo run --release --features onnx --example score_models # score the built-in backend against it
```

## Notes
Expand Down
15 changes: 8 additions & 7 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,15 +27,16 @@ machine. The design keeps you in control of it:

## Secrets

Three layers keep credentials out of a memory and off the Hub:
The local memory keeps sessions as written, credentials included: it is not a publication
boundary. Two layers keep credentials off the Hub:

- **At index time**, credentials are redacted from each session before it is saved.
- **At publish time**, an always-on gate scans every outgoing chunk (via
[trufflehog](https://github.com/trufflesecurity/trufflehog)) and *withholds* any row that
still contains a secret, exiting non-zero rather than uploading it. The gate is fail-closed:
if the scanner is missing or crashes, nothing is published.
- **`funes scrub`** removes secrets from rows already in your local memory, so a subsequent
push goes through clean.
[trufflehog](https://github.com/trufflesecurity/trufflehog)) and redacts the secrets it finds
before uploading. A block it can't redact in place is *withheld*, and the push exits non-zero
when that leaves nothing to publish. The gate is fail-closed: if the scanner is missing or
crashes, nothing is published.
- **`funes scrub`** redacts secrets from the rows already in your local memory, including the
blocks a push withholds, so the next push publishes them.

### If a secret reached a remote memory anyway

Expand Down
82 changes: 60 additions & 22 deletions benchmark/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
# Benchmarks

Four runnable examples, each `cargo run --release --example <name>`:
Six runnable examples, each `cargo run --release --example <name>`:

- **`bench_recall`** — `recall()` latency, local vs remote, cold vs warm.
- **`bench_backends`** — latency and output agreement between the inference backends.
- **`bench_index`** — the part of `funes index` that writes rows, without the embedding.
- **`bench_models`** — embedding and reranking time on one inference backend.
- **`score_models`** — how closely the inference backends reproduce ONNX Runtime's embeddings and
reranker scores.
- **`score_recall`** — `funes recall` scored against a labeled set of questions.
- **`sweep_rerank`** — how much reranking helps at each pool size, on a `dacorvo/funes-recall-benchmark`
set.
Expand Down Expand Up @@ -89,34 +92,69 @@ that floor moves with the CPU and the inference backend. What the benchmark meas
- For a private dataset, a token must be available (`HF_TOKEN` or the cached login) — the same one
recall uses.

## `bench_backends` — inference backend comparison
## `bench_index` — writing an index's rows

`bench_backends.rs` compares every compiled `Embedder`/`Reranker` implementation. Build with both
the default BLAS backend and the optional ONNX reference:
`bench_index.rs` times the part of `funes index` that writes rows, in a scratch memory. It leaves
the embedding out, since it takes the longest and does not depend on how the rows are written. It
times writing the rows (reading each turns file, chunking it, appending its rows, recording its
state), then the end of the run (the language counts and the compaction). Both grow with the number
of sessions as well as with their size, so a history of many short sessions is where they show.

```sh
cargo run --release --features onnx --example bench_backends
cargo run --release --features onnx --example bench_backends -- intfloat/multilingual-e5-small
cargo run --release --example bench_index -- --sessions 2000 2>/dev/null
cargo run --release --example bench_index -- --dir path/to/turns --limit 5000 2>/dev/null
```

It runs seven fixed workloads: 16 short documents to expose per-call overhead; 30 documents near
the 512-token truncation limit to approximate recall's rerank worst case; a ragged batch of 2 capped
documents among 14 short ones; 32 and 100 documents whose token lengths are the quantiles of real
chunks (100 is recall's pool at `--candidates 100`); and 256-document mixed and real batches, the
production batch size, which show how a backend scales and what it spends on padding.
The corpus is generated, `--sessions` sessions (2000 by default) of `--turns` turns (12) that are
the same on every run, or a directory of turns files given with `--dir`, of which `--limit` keeps
the most recent sessions. The indexer reports each session and the compaction on stderr. The bench
prints the sessions and chunks written with their time and rate, the fragments the writes left for
the compaction to merge, then the time the end of the run took.

Rerankers are compared within a model: gte-reranker-modernbert-base on ONNX, BLAS and, on macOS,
Metal, and bge-reranker-base on BLAS as the latency baseline. Each reranker's startup comes first: how
long it took to load, then its first call on the 32 real documents, which a one-shot `funes recall`
pays in full. Then each workload's latency, and agreement with the first backend of the same model:
## `bench_models` — embedding and reranking time

- `cos↔ref` is the minimum cosine similarity between corresponding embedding vectors.
- `Δ↔ref` is the maximum absolute difference between corresponding reranker scores.
`bench_models.rs` times funes's embedder and reranker (gte-reranker-modernbert-base) on one
backend: `--backend blas`, the default, runs the CPU forwards every build carries, and
`--backend metal` the GPU forwards of a macOS build. `--model` names the embedding model,
`BAAI/bge-small-en-v1.5` when omitted.

ONNX is the reference when that feature is present, BLAS otherwise. The one argument names the
embedding model, `BAAI/bge-small-en-v1.5` when omitted. Edit the fixed query or workloads when investigating a
particular regression. Building with only one backend still prints its timing, along with a note
that there is nothing to compare.
```sh
cargo run --release --example bench_models -- --backend metal
cargo run --release --example bench_models -- --model intfloat/multilingual-e5-small
```

It times the forwards themselves. funes runs them behind a router that keeps a call on the CPU
until it holds 32 texts or 8 pairs, and the two backends' startup and first calls are what those
thresholds weigh. Each model's startup comes first: how long it took to load, then its first call
on the 32 real documents, which a one-shot process such as `funes recall` pays in full. Then each
workload's time per call and documents per second, after one warm-up call.

The seven workloads live in `workloads.rs`, which `score_models` shares:

- 16 short documents, to expose per-call overhead.
- 30 documents near the 512-token truncation limit, recall's rerank worst case.
- A ragged batch of 2 capped documents among 14 short ones.
- 32 and 100 documents whose token lengths are the quantiles of real chunks. 100 is recall's pool
at `--candidates 100`.
- 256-document mixed and real batches, an indexing call's size on the CPU, which show how a backend
scales and what it spends on padding.

Edit the query or the workloads when investigating a particular regression.

## `score_models` — the backends against ONNX

`score_models.rs` measures how closely funes's backends reproduce ONNX Runtime running each model's
ONNX export: the CPU forwards (`blas`) and, in a macOS build, the GPU forwards (`metal`), over the
same seven workloads. ONNX is the reference, so the example needs it compiled in:

```sh
cargo run --release --features onnx --example score_models
cargo run --release --features onnx --example score_models -- --model intfloat/multilingual-e5-small
```

- `cos↔onnx` is the lowest cosine similarity between a document's embedding on the backend and on
ONNX.
- `Δ↔onnx` is the largest absolute difference between their reranker scores.

## `score_recall` — recall against labeled questions

Expand Down
Loading
Loading