Skip to content

perf(query): tokenize a query once per completion, hover and check - #53

Merged
ragnorc merged 3 commits into
mainfrom
fix/studio-sweep-4-20261031
Nov 1, 2026
Merged

ragnorc merged 3 commits into
mainfrom
fix/studio-sweep-4-20261031

Conversation

@ragnorc

@ragnorc ragnorc commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

One efficiency fix that the wider review (#50, #51, #52) left as minor; it's cheap to take.

Completion tokenized the whole query text twice per keystroke (to rule out a comment or string, then for the outline), and hover twice. The check's underlines tokenized the whole text again for every diagnostic, so a long query with many problems cost its length times their number. Now:

  • outline_of and in_comment_or_string take the tokens their caller already has. outline stays as the one-call form for its other callers.
  • problem_range takes the text's tokens, made once for all its problems, and finds the token under a problem by halving instead of a linear scan.

Fixes from a review of this branch

  • A declaration checked on its own still lexed the whole text up to its end, and counted lines from the start, once per problem. It now takes those tokens from the ones the analysis already made (tokens_in, by halving), and finds line starts once per check.
  • The diff view (results.rs) and the gateway's branch rule (held_off_the_branch) lexed their text twice, once for tokens and once for the outline. Both now lex once.

Behaviour is unchanged. The lexer, completion, hover and analysis tests cover it.

Testing

  • cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace (1,166 passed) and the repo checks pass locally, on main 10b7786.
  • analysis::tests::a_broken_declaration_keeps_its_siblings_known gains a broken declaration on line 4 after an accented line, and pins the problem's byte, line, column (counted in characters) and underline. It fails if the column is counted in bytes, or if the problem isn't placed on the declaration's last word (both checked).

Settled, not changed

The review's other open item, coalescing settings writes, isn't worth the change. The busiest source writes the conversation list once per prompt sent, the workspace checks for changes every 250 ms, and the rest write when a setting changes. Nothing queues writes to one file fast enough to save an fsync, and the write thread's strict order is what read-edit-write jobs and the quit-time flush rely on.

Completion tokenized the whole text twice per keystroke (to rule out a
comment or string, then for the outline), and hover twice. The check's
underlines tokenized the whole text again for every diagnostic, so a
long query with many problems cost its length times their number. Now
`outline_of` and `in_comment_or_string` take the tokens their caller
already has, and `problem_range` takes the text's tokens, made once for
all its problems, and finds the one under a problem by halving.
A review found the check still tokenized the text twice: once for its
outline and once more for the underlines, so a valid query, which used
to tokenize none for underlines, now did one more. The outline is now
made from the same tokens (`outline_of`).
A problem a declaration's own check found lexed the whole text up to the
declaration's end, and counted lines from the start, once per problem: work that
grew with the text times the problems. It now takes those tokens from the ones
the analysis already made and finds lines once per check. The diff view and the
gateway rule also lex their text once for both the tokens and the outline.
@ragnorc
ragnorc merged commit e903eb2 into main Nov 1, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant