Skip to content

Commit 602d6ee

Browse files
committed
docs(s031): G1 plan with FTS5 and the porter limit measured, and the slice opens
ROADMAP row 031 to in_progress. NOTES.md records that exqlite 0.40.0's SQLite 3.53.4 carries FTS5 and that porter stems running to runs and not to ran, then the six-line build plan, the two-item manual queue and three deviations stated before any code. plan_check: PASS. Signed-off-by: Ayla Croft <aylacroft@proton.me>
1 parent d9ee12d commit 602d6ee

2 files changed

Lines changed: 56 additions & 1 deletion

File tree

‎ROADMAP.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ standards register names the rows that ask for them.
4545
| 025 | Encryption at rest, and the key-custody seam | 2 Tools | M | 010, 024 | planned |
4646
| 026 | Store-and-forward receipts for disconnected operation | 2 Tools | L | 024 | planned |
4747
| 030 | Persona (SOUL) + always-on memory tier | 3 Memory | M | 012 | planned |
48-
| 031 | Session search (SQLite FTS5) | 3 Memory | S | 010 | planned |
48+
| 031 | Session search (SQLite FTS5) | 3 Memory | S | 010 | in_progress |
4949
| 032 | Embeddings + semantic memory + hybrid retrieval | 3 Memory | L | 031 | planned |
5050
| 033 | Project context: AGENTS.md | 3 Memory | S | 030, 022 | planned |
5151
| 034 | Export, import, restore | 3 Memory | S | 030, 031 | planned |
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# Slice 031: NOTES
2+
3+
## Two facts measured 2026-09-20 before any code
4+
5+
**FTS5 is in the bundled SQLite.** exqlite 0.40.0 (the lock) carries SQLite 3.53.4 with `ENABLE_FTS3`,
6+
`ENABLE_FTS4` and `ENABLE_FTS5` in `PRAGMA compile_options`; a `fts5(content, tokenize = 'porter unicode61')`
7+
table creates, `snippet()` and `bm25()` answer. Whether the same holds in the Burrito binary is slice 100's
8+
question, as SLICE.md's risk line says; this is the development and gate build.
9+
10+
**Porter does not reach "ran".** In that table, `MATCH 'running'` returns "running late" and "it runs" (porter
11+
stems both to `run`) and not "we ran the tests": an irregular past tense is not a suffix, and no stemmer maps it.
12+
`MATCH 'ran'` finds "ran" alone. AC2 says "running" finds "ran"/"runs": the test proves "runs" and asserts that
13+
"ran" is not found, stating the tokenizer's limit rather than claiming it away. On Postgres, `to_tsvector('english')`
14+
(Snowball) stems the same way, and the test documents it the same way. Deviation (a), stated before code.
15+
16+
## G1 plan, 2026-09-20
17+
18+
Tree at `d9ee12d` on `main` (024 approved, M2 reached); branch `slice/031-session-search-fts`; ROADMAP row 031
19+
to `in_progress` in this commit. Each line names its test.
20+
21+
1. Migration `create_messages_fts`: on SQLite (branching on `repo().__adapter__()`), the virtual table
22+
`messages_fts(content, session_id UNINDEXED, message_id UNINDEXED, tokenize = 'porter unicode61')` and the
23+
three triggers (insert, update of `content`, delete) on `messages`, plus a backfill of the rows present; on
24+
Postgres, a generated `content_tsv tsvector` column on `messages` (`to_tsvector('english', coalesce(content,
25+
''))`) with a GIN index. Test on both adapters (the postgres job): a message inserted through the Store is
26+
found at once (AC1); an updated content is re-found; a deleted one is gone.
27+
2. `Trinity.Memory.Search.messages(query, opts)`: SQLite through `MATCH` with `bm25()` order and `snippet()`;
28+
Postgres through `plainto_tsquery` with `ts_rank` and `ts_headline`; hits as `%{message_id, session_id,
29+
session_title, seq, role, snippet, inserted_at, rank}`; options `limit:` (20, capped at 100), `role:`,
30+
`persona_id:`, `since:`, `until:`. The query text is bound as a parameter and never spliced; FTS5 syntax
31+
characters are quoted so a user's `"` or `*` is text, not an operator. Tests: stemming (AC2, both halves),
32+
filters, the empty query, an injection-shaped query.
33+
3. `mix trinity.search.reindex` (SQLite: `INSERT INTO messages_fts(messages_fts) VALUES('rebuild')` after a
34+
delete-all and a refill from `messages`; Postgres: a no-op that says the column is generated) and
35+
`Trinity.Memory.Search.reindex/0` behind it. Test AC3: 10,000 generated messages, the incremental index's hit
36+
counts for ten queries equal the rebuilt index's; the time printed.
37+
4. Tool `Trinity.Tools.SessionSearch` (`session_search`, risk `:read`, effect `:none`, schema `query` and
38+
`limit`), registered in `config :trinity, :tools` for every environment; the result is the hits as text the
39+
model reads (`session`, `when`, `snippet`), capped by `limit`. Test: the tool through the runner returns at
40+
most `limit` hits with snippets and is receipted as a read.
41+
5. `/search` LiveView: a query box, results with the session title, the time, the role and the snippet, each
42+
linking to `/s/:id#message-<id>` (the chat's stream dom ids are `message-<id>`; SLICE.md's `#m-<seq>` is
43+
read as that anchor, deviation (b)); the chat's bar and the index page link to it. LiveView test.
44+
6. docs/05 synced (the table as built), docs/01's Memory row mentions Search.
45+
46+
Manual verification queue (two items, for the owner at G4):
47+
- **AC4**: the tool returns at most `limit` hits with snippets, and the agent answers "what did we decide about
48+
X last week" using it. I record a GIF under `proof/` from a real-provider run through
49+
`scripts/dev_chat_on_test_registry.sh`; the owner watches it.
50+
- **AC5**: the search page renders results and deep-links. A screenshot under `proof/` from the same run; the
51+
owner opens the page.
52+
53+
Deviations stated before any code: (a) AC2's "ran" is asserted as not found, with the reason (above); (b) the
54+
deep link anchors on the chat's existing dom id `message-<id>` rather than a new `m-<seq>`; (c) reindex on
55+
Postgres is a documented no-op, because a generated column cannot be stale.

0 commit comments

Comments
 (0)