Skip to content

fix(server): thread search finds phrases inside quotes and their comments - #22553

Open
only21mil wants to merge 3 commits into
pingdotgg:mainfrom
only21mil:r11/h7-h1
Open

only21mil wants to merge 3 commits into
pingdotgg:mainfrom
only21mil:r11/h7-h1

Conversation

@only21mil

@only21mil only21mil commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Thread search can't find a phrase inside a quote or its comment (#22304). A quote is stored in the message text as an [Assistant quote](t3-citation://v1/…) link, and URLSearchParams encodes the quote and comment inside it: what are you talking about is stored as what+are+you+talking+about. Thread search matches that stored text, both through the message index and the fallback scan, so phrases with spaces never match, and accented, CJK and apostrophe words in a quote never match through the index. Single words do, including words from the quote's hidden prefix and suffix selection context, so a thread can show up for a word its reader never saw in it.

Change

  • When ProjectionStore projects a message, it now stores search_text on orchestration_v2_projection_messages: assistantCitationsToPlainText(text), the helper the queued-message and stash previews already use. A quote becomes its text plus Comment: …, without the link or its hidden context. The column is NULL when that equals the stored text, so plain messages aren't stored twice. Every message.updated upsert rewrites it, so edits and projection rebuilds keep it current.
  • Migration 077 adds the column, recreates the message search triggers, and backfills existing history. The backfill pages through messages whose payload contains t3-citation://, 500 at a time by message_id, and writes search_text only where the readable text differs; the update trigger re-indexes each message it changes.
  • The search triggers come from the shared createMessageSearchTriggers, which now indexes search_text once the column exists. Migrations 069 and 074 are unchanged and, running before 077, keep indexing the stored text; a later migration that recreates the triggers through the helper keeps quote indexing instead of reverting it. Migration 077 recreates them through a new recreateMessageSearchTriggers. rebuildMessageSearch (t3 doctor, migrate-dev-db) indexes the same text.
  • Thread search matches COALESCE(search_text, text) and builds the snippet from the same value, through one messageSearchText helper, on the same reader-worker query.
  • Migration id: open PRs feat(server): recover limited Codex threads with banked resets #28341, fix(server): a stored boolean settledOverride no longer stops startup #28314, perf(server): prune thread event history older than a day #28144, perf(server): index live pull request polling #27898 and perf(server): index historical usage-limit recovery lookups #27896 also add migration 077; whichever lands second renumbers above the other.

Scope and approval

Refs #22304. This fixes its matching half. @EtanHey reported it, traced both halves and suggested a derived search-text column written at projection time; this builds that.

Snippets come from the matched text, so a quoted match also shows readable text instead of the encoded link.

Verification

Linux (Ubuntu 26.04), in apps/server:

  • New ThreadSearch.test.ts cases project a quote with hidden prefix and suffix words, find its accented, CJK and apostrophe words through the index, rebuild the index, and replay the quote from events through ProjectionMaintenance.rebuild. Quote and comment phrases match and the hidden words don't. After a second message.updated drops the quote, the comment phrase stops matching.
  • 077_ProjectionMessageSearchText.test.ts migrates to 076, inserts a quoted message, a plain one, an invalid citation link, a non-string text, an accented quote and 501 quoted rows to cross a page, then runs 077 and checks search_text and the index. A second case recreates the triggers through recreateMessageSearchTriggers, as a later migration would, and checks that a new quote is still indexed as it reads.
  • On main 54c5f25249, the three new ThreadSearch cases fail (3 failed, 26 passed). On this branch, ThreadSearch, 077, 069, Migrations and reconcileV2PreviewMigration pass, 49 of 49.
  • With the trigger helper forced back to the stored text, both 077 cases fail.
  • vp run check:changed: format, lint, knip, typecheck and tests pass.

Real app, on every client: each run starts on main with an isolated state of test threads, where I selected a line of an agent reply (from the dev Replay provider), chose Cite, added the comment what are you talking about, moved the quote into a new thread with quiet silver lantern, and sent it. Then I stopped main and started this branch on the same state. It ran only migration 077, which filled search_text for that quoted message. Searching for the comment finds nothing on main; after the upgrade it finds the quoting thread with a readable snippet.

Web (dev servers, Chromium):

main this branch
https://gist.githubusercontent.com/only21mil/9900a9292b7d0ac07e57580b0abdd405/raw/cb5ed48448bac8b79beee211fcf74e22cf06c5df/pr22553r3-main-sidebar-comment.png https://gist.githubusercontent.com/only21mil/9900a9292b7d0ac07e57580b0abdd405/raw/cb5ed48448bac8b79beee211fcf74e22cf06c5df/pr22553r3-branch-sidebar-comment.png
https://gist.githubusercontent.com/only21mil/9900a9292b7d0ac07e57580b0abdd405/raw/cb5ed48448bac8b79beee211fcf74e22cf06c5df/pr22553r3-main-palette-comment.png https://gist.githubusercontent.com/only21mil/9900a9292b7d0ac07e57580b0abdd405/raw/cb5ed48448bac8b79beee211fcf74e22cf06c5df/pr22553r3-branch-palette-comment.png

A quote sent after the upgrade goes through the projection path and is found by its comment.

https://gist.githubusercontent.com/only21mil/9900a9292b7d0ac07e57580b0abdd405/raw/cb5ed48448bac8b79beee211fcf74e22cf06c5df/pr22553r3-branch-new-quote.png

Desktop (Linux Electron under Xvfb, vp run dev:desktop of main and of this branch):

main this branch
https://gist.githubusercontent.com/only21mil/9900a9292b7d0ac07e57580b0abdd405/raw/cb5ed48448bac8b79beee211fcf74e22cf06c5df/pr22553r3-desktop-main-comment.png https://gist.githubusercontent.com/only21mil/9900a9292b7d0ac07e57580b0abdd405/raw/cb5ed48448bac8b79beee211fcf74e22cf06c5df/pr22553r3-desktop-branch-comment.png

Mobile (T3 Code Dev client on an Android 16 emulator and an iOS 26.5 simulator, against main's and this branch's dev servers):

main this branch
Android https://gist.githubusercontent.com/only21mil/9900a9292b7d0ac07e57580b0abdd405/raw/cb5ed48448bac8b79beee211fcf74e22cf06c5df/pr22553r3-android-main-comment.png https://gist.githubusercontent.com/only21mil/9900a9292b7d0ac07e57580b0abdd405/raw/cb5ed48448bac8b79beee211fcf74e22cf06c5df/pr22553r3-android-branch-comment.png
iOS https://gist.githubusercontent.com/only21mil/9900a9292b7d0ac07e57580b0abdd405/raw/cb5ed48448bac8b79beee211fcf74e22cf06c5df/pr22553r3-ios-main-comment.png https://gist.githubusercontent.com/only21mil/9900a9292b7d0ac07e57580b0abdd405/raw/cb5ed48448bac8b79beee211fcf74e22cf06c5df/pr22553r3-ios-branch-comment.png

Not checked:

  • Desktop on macOS and Windows. They run the same server and web UI.
  • Quoted-phrase and hidden-context searches in the app: the Replay provider gives every thread the same reply, so those words also match the replies there. The unit tests cover both.
  • Migration time on a large database. The backfill test crosses one 500-row page.

Prior PRs: #22391 (open) fixes the snippet only. No open or closed PR changes the matching. #16858 (open) edits the same query for qualifiers and may need a small rebase.

Generated with Claude Opus 5.5 in T3 Code Nightly

@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Oct 20, 2026
@macroscopeapp

macroscopeapp Bot commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 694931a

Macroscope's review found this PR approvable — This is a focused server bug fix that makes thread search use readable citation text while preserving existing behavior for ordinary messages. The additive migration, fallback logic, and targeted projection/replay tests keep the change bounded and backward-compatible.

You can add or adjust custom eligibility rules. Learn more.

@coderabbitai

coderabbitai Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 1cd68f2 and 7c15aaf.

Files selected for processing: 8
  • apps/server/src/orchestration-v2/ProjectionStore.ts
  • apps/server/src/orchestration-v2/ThreadSearch.test.ts
  • apps/server/src/orchestration-v2/ThreadSearch.ts
  • apps/server/src/orchestration-v2/messageSearchIndex.ts
  • apps/server/src/persistence/Migrations.ts
  • apps/server/src/persistence/Migrations/077_ProjectionMessageSearchText.test.ts
  • apps/server/src/persistence/Migrations/077_ProjectionMessageSearchText.ts
  • apps/server/src/persistence/reconcileV2PreviewMigration.test.ts

Limit details: You’ve used all 10 included reviews currently available.

Walkthrough

Migration 77 adds and backfills readable citation text for message search. Projection writes maintain this text, and thread search uses it in indexed and non-indexed paths.

Changes: Citation-aware thread search

Layer / File(s) Summary
Add and index projected search text
apps/server/src/orchestration-v2/messageSearchIndex.ts, apps/server/src/persistence/Migrations/077_ProjectionMessageSearchText.ts, apps/server/src/persistence/Migrations/077_ProjectionMessageSearchText.test.ts, apps/server/src/persistence/Migrations.ts, apps/server/src/persistence/reconcileV2PreviewMigration.test.ts
Migration 77 adds search_text and backfills eligible citation messages in batches. Search triggers and index rebuilds use the schema-appropriate text expression. Migration tests cover backfill and indexed search.
Maintain search text on projected messages
apps/server/src/orchestration-v2/ProjectionStore.ts
Projection inserts and conflict updates store readable search text derived from message text.
Search readable citation text
apps/server/src/orchestration-v2/ThreadSearch.ts, apps/server/src/orchestration-v2/ThreadSearch.test.ts
Thread search uses the shared message-search text expression for scans, indexed queries, and substring filtering. Tests cover citation matches, edits, index rebuilds, and projection replay.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Possibly related issues

Suggested reviewers: maria-rcks, juliusmarminge

Merge Risk: Minimal · up to 7c15a

No identified issue remains that should block merging after normal checks.

Architecture risk: Low · up to bb269

The change affects 1 system.

Changed systems: apps/server

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — apps/server (service) was modified; 7 changed files map to changed impact.

Before / after behavior

  • observed — Modified behavior in apps/server/src/orchestration-v2/ProjectionStore.ts: Added the assistantCitationsToPlainText import used to derive searchable message text.
  • observed — Modified behavior in apps/server/src/orchestration-v2/ProjectionStore.ts: message.updated now derives citation-stripped plain text from the message text for search indexing.
  • observed — Modified behavior in apps/server/src/orchestration-v2/ProjectionStore.ts: The projected messages insert now includes the search_text column.
  • observed — Modified behavior in apps/server/src/orchestration-v2/ProjectionStore.ts: Message inserts store normalized citation text, using NULL when it matches the original message text.
Pre-merge checks: Passed 3 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Description check Warning The description includes Problem, Change, Scope and approval, and Verification sections. It gives detailed behavior, implementation scope, test results, manual checks, limitations, screenshots, and ag… Add a link to the triaged issue or discussion with the maintainer's explicit approval of the direction and scope, including the approval comment. If no approval is required, explain why this qualifies as a focused fix of an obvious bug unde…
Passed Passed checks: 3 passed
Check name Status Explanation
Title check Passed The title "fix(server): thread search finds phrases inside quotes and their comments" matches the main change. The PR stores readable citation text in a new search_text column, backfills it via migrat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title "fix(server): thread search finds phrases inside quotes and their comments" matches the main change. The PR stores readable citation text in a new search_text column, backfills it via migration 071, and makes ThreadSearch match COALESCE(search_text, text). It is a conventional commit title in plain language, concise, and specific. The 73-character length is within normal limits.

Full details: Description check

Explanation

The description includes Problem, Change, Scope and approval, and Verification sections. It gives detailed behavior, implementation scope, test results, manual checks, limitations, screenshots, and agent attribution. However, the Scope and approval section does not include an explicit maintainer approval comment or explain why the focused-fix exemption applies.

Resolution

Add a link to the triaged issue or discussion with the maintainer's explicit approval of the direction and scope, including the approval comment. If no approval is required, explain why this qualifies as a focused fix of an obvious bug under the repository requirements.

Finishing Touches
Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@github-actions github-actions Bot added size:L 100-499 changed lines (additions + deletions). and removed size:M 30-99 changed lines (additions + deletions). labels Oct 24, 2026
@juliusmarminge juliusmarminge added bugfix Fix PR opened by an issue lead; enters the PR loop pr:gate-failed A land gate failed labels Oct 27, 2026
only21mil and others added 3 commits October 31, 2026 23:11
…ents

A quote is stored in message text as a serialized t3-citation link, so its
text and comment are percent-encoded: a phrase with spaces never matched,
and accented, CJK and apostrophe words in a quote never matched through the
index.

Projected messages now keep a readable search_text, made with
assistantCitationsToPlainText, whenever it differs from the stored text.
Migration 077 adds the column, recreates the search triggers and backfills
messages that hold a citation link; the update trigger re-indexes each one
it changes. createMessageSearchTriggers and the doctor rebuild index
search_text once the column exists, so migrations 69 and 74, which run
before it, keep indexing the stored text, and a later migration that
recreates the triggers keeps quote indexing. Thread search matches and
builds snippets from the same text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… rule

- recreateMessageSearchTriggers replaces the drop-and-recreate block that
  migrations 74 and 77 each wrote out.
- readableSearchText is the one rule for what search_text holds, used by
  the projection and the backfill.
- The update trigger compares the text the index stores, so it has one
  shape before and after migration 77 and still re-indexes the backfill.
- The backfill checks the raw payload for a citation link before parsing
  its JSON.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration 074 called the trigger helper directly; only migration 77 needs
the shared recreateMessageSearchTriggers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Fix PR opened by an issue lead; enters the PR loop pr:gate-failed A land gate failed size:L 100-499 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants