Repository navigation
perf(query): tokenize a query once per completion, hover and check - #53
Merged
Merged
Conversation
Completion tokenized the whole text twice per keystroke (to rule out a comment or string, then for the outline), and hover twice. The check's underlines tokenized the whole text again for every diagnostic, so a long query with many problems cost its length times their number. Now `outline_of` and `in_comment_or_string` take the tokens their caller already has, and `problem_range` takes the text's tokens, made once for all its problems, and finds the one under a problem by halving.
A review found the check still tokenized the text twice: once for its outline and once more for the underlines, so a valid query, which used to tokenize none for underlines, now did one more. The outline is now made from the same tokens (`outline_of`).
A problem a declaration's own check found lexed the whole text up to the declaration's end, and counted lines from the start, once per problem: work that grew with the text times the problems. It now takes those tokens from the ones the analysis already made and finds lines once per check. The diff view and the gateway rule also lex their text once for both the tokens and the outline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One efficiency fix that the wider review (#50, #51, #52) left as minor; it's cheap to take.
Completion tokenized the whole query text twice per keystroke (to rule out a comment or string, then for the outline), and hover twice. The check's underlines tokenized the whole text again for every diagnostic, so a long query with many problems cost its length times their number. Now:
outline_ofandin_comment_or_stringtake the tokens their caller already has.outlinestays as the one-call form for its other callers.problem_rangetakes the text's tokens, made once for all its problems, and finds the token under a problem by halving instead of a linear scan.Fixes from a review of this branch
tokens_in, by halving), and finds line starts once per check.results.rs) and the gateway's branch rule (held_off_the_branch) lexed their text twice, once for tokens and once for the outline. Both now lex once.Behaviour is unchanged. The lexer, completion, hover and analysis tests cover it.
Testing
cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warnings,cargo test --workspace(1,166 passed) and the repo checks pass locally, on main 10b7786.analysis::tests::a_broken_declaration_keeps_its_siblings_knowngains a broken declaration on line 4 after an accented line, and pins the problem's byte, line, column (counted in characters) and underline. It fails if the column is counted in bytes, or if the problem isn't placed on the declaration's last word (both checked).Settled, not changed
The review's other open item, coalescing settings writes, isn't worth the change. The busiest source writes the conversation list once per prompt sent, the workspace checks for changes every 250 ms, and the rest write when a setting changes. Nothing queues writes to one file fast enough to save an fsync, and the write thread's strict order is what read-edit-write jobs and the quit-time flush rely on.