Repository navigation
perf: faster cold tokenization for slow catalog languages, plus engine compatibility fixes - #15
Merged
Merged
Conversation
Toolchain and tooling:
- Raise the MSRV and pinned toolchain from Rust 1.88 to 1.98 and fix the
clippy lints it enables.
- Refresh Cargo lockfiles (toml 1.1.6 plus transitive patch releases),
GitHub Action pins, and flake inputs; migrate hk.pkl to hk 2.2.
Grammars and oracle:
- Vendor @shikijs/langs 4.4.3 (from 3.23.0). 52 grammars change, coq now
uses source.rocq, and seven new Shiki grammars are private assets.
- Move the oracle to vscode-textmate 9.3.2, matching current VS Code.
vscode-oniguruma stays at 1.7.0 because VS Code still ships it.
- Regenerate goldens, provenance, and parity reports, and read the oracle
version from the pinned packages instead of hardcoding it.
Engine parity fixes found by the refresh:
- Support \p{XIDS}/\p{XIDC} with Oniguruma loose property names.
- Stop the start-byte shortcut from skipping nullable patterns whose empty
branch is guarded by an assertion, such as x|(?<=T).
- Follow vscode-textmate for captures outside a match: skip empty ones,
stop at the first capture after the match, and resume scanning at the
match end.
- Try anchor alternations such as (^|\G) only at their anchors and reuse
one matcher scratch per search, recovering the scan's allocation cost.
C++ allocation ceilings are raised for the larger upstream grammar.
The multi-pattern scanner evaluated every character class 256 times (128 ASCII probes, both case modes) each time a candidate set compiled, and candidate sets recompile shared patterns per tokenizer. Reuse the bytecode VM's exact per-atom ASCII masks, give Perl classes constant masks, and walk only set bits when bucketing start bytes. Masks are checked against class_contains in debug builds and by new unit tests covering Perl, POSIX, Unicode, negated, intersected, and non-ASCII case-folding classes.
Start-byte analysis merged one set per alternation branch by probing all 256 bytes. Large literal inventories (thousands of branches per pattern) made that loop a leading cost of first-use regex compilation. OR the four bitmap words and recount instead; the set contents are unchanged.
The required-literal Aho-Corasick finder allocated an edge vector per trie state and cloned it again while computing failure links. Huge keyword alternations produce hundreds of thousands of states, so construction and teardown were dominated by allocator traffic. Insert through first-child/next-sibling links, renumber states breadth-first so each state's byte-sorted edges are one contiguous run of shared arrays, and compute failure links in that order. The automaton and its leftmost-start answers are unchanged; a randomized test compares the finder against a naive leftmost search.
Case-insensitive class ranges and Unicode literal-trie edges re-derived full Unicode case mappings for every comparison: six binary searches per range atom and four per trie edge. BSL/SDBL keyword alternations probe every Cyrillic scalar against dozens of edges and a lookbehind class, so case mapping dominated their matching time. CaseFoldKey stores a scalar's NUL-padded full lowercase and uppercase mappings and compares exactly like unicode_case_eq (including the non-Turkic dotless-i exclusion). Trie edges keep their keys, and classes used by case-insensitive instructions get a FoldedClass with prepared range bounds and char keys, so each input scalar is mapped at most once per probe. Tests check both against the existing evaluators over ASCII, Latin, Greek, Cyrillic, letterlike, and supplementary scalars.
The ordered candidate scan skipped patterns by previous/current word class only when both neighbors were ASCII. In Cyrillic BSL/SDBL text most positions have a non-ASCII neighbor, so every keyword rule of the form (?i:(?<=[^.а-яё\w]|^)(...)) ran its lookbehind and trie mid-word. Masks now carry a second nibble that is sound for any scalar: it drops ASCII-only claims (negated ASCII sets excluding \w, [[:alpha:]] and [[:upper:]]/[[:lower:]] implying \w) and keeps case-insensitive non-ASCII scalars conservative, since not every Unicode case pair shares word-ness across the tables in use. Positions with a non-ASCII neighbor classify both neighbors with the Unicode \w predicate and consult that nibble. Positive lookbehinds also treat a ^ or \A branch as a non-word previous character instead of an unconstrained nullable branch, which lets (?<=X|^) keyword guards prune mid-word positions everywhere. Differential tests compare gated and ungated selection per pattern over Unicode texts; an exhaustive test pins the ASCII case-partner word-ness assumption.
Deeply nested documents (for example a grammar whose begin/end rules never close, so the stack grows by a fixed amount per repeated block) made every new scope stack pay for its full depth several times: - The result scope table walked the whole parent chain of every emitted stack and did one hash lookup per scope. It now walks only to the nearest already-emitted ancestor, copies that ancestor's atoms, and maps scope IDs through a dense vector. Atom numbering keeps first-appearance order. - Candidate lookup resolved the full scope stack into shared strings for each new state just to evaluate injection selectors. Grammar closures without injection selectors now skip that resolution; the outcome is the same empty injection set.
An ASCII case-insensitive required-literal prefilter searched each literal separately and took the minimum, so a keyword rule such as ABAP's (?i)(?<=^|\s)(abap-source|abstract|...) or Common Lisp's symbol lists ran hundreds of memchr2 scans over the rest of the line for every viability query. Large sets now use the existing failure-linked multi-literal finder over lowercased literals, lowercasing each input byte while scanning. For ASCII literals that accepts exactly the windows eq_ignore_ascii_case accepts, and the finder already reports the leftmost start; the long-s and Kelvin-sign candidates are still merged in unchanged. Small sets keep per-literal search. A differential test compares both paths from every start position.
The multi-pattern Thompson scanner expanded every branch of an ordered alternation, and every unrestricted start entry, at every position. A thread whose first consuming instruction cannot accept the next byte dies on the next step without producing anything, so omitting it is unobservable. Keyword lists such as \b(and|band|...|while)\b paid for all of their branches at every word start. - Compute a conservative first-byte set and nullability for any compiled closure. Anchors pass, case-insensitive ASCII keeps every non-ASCII lead byte (Unicode folds such as U+212A and U+017F), classes keep all non-ASCII bytes, and unpatched loop targets are treated as unrestricted. - Alternations with at least eight branches compile to a byte-class dispatch that lists, in branch order, the branches that can consume the byte or reach Accept without consuming. The epsilon closure pushes that list in the same LIFO order as the replaced Split chain. - Start entries use the same byte-class table, intersected with the existing non-nullable start-byte hints, which also removes the per-position merge of restricted and unrestricted entries. - Once an Accept is queued no later start can outrank it, so stop injecting start threads for the rest of that search. Differential tests against the fallback interpreter cover branch priority, nullable, anchored and looping branches, cross-width case folds, start-table priority at line end, and queued accepts.
Keyword inventories parse thousands of single-literal branches. Each one grew its literal string a scalar at a time, allocated a sequence vector for a single node, and was rewrapped in another vector by option-change normalization. Measure each literal run before allocating it, hold a lone branch node inline until a second node arrives, and leave branches without an option-change marker in place. The resulting AST is identical.
Required-literal sets are already sorted and deduplicated, so each literal shares exactly its longest common prefix with the previous literal's trie path. Build the finder in one pass without edge searches, lay out the byte-sorted edge runs by counting, and compute failure links breadth-first over the flat arrays. Unsorted input is sorted through a borrowed view. The automaton's answers are unchanged.
RegexAnalysis now derives effective flags, referenced groups, capture bytecode support, ordered fanout, and the bytecode instruction-capacity hint in one AST traversal. Lazy bytecode compilation reads the hint from the analysis instead of re-walking a cold AST. StartByteSet unions use word ORs, POSIX class names resolve to a predicate once per class, and alternations without bare option changes skip the flag rewrite. All derived values are unchanged; a unit test pins the fused fanout and capacity results against the standalone walks.
A greedy single-character repeat such as [A-Z_a-z][$0-9A-Z_a-z]* before
(?![$0-9A-Z_a-z]), '::', or '(' pushed one backtrack frame per consumed
character, and every one of those frames failed at the continuation. The
SystemVerilog port-declaration rule chains several such identifiers and
optional groups, so failed attempts spent most steps unwinding them.
The bytecode compiler now emits the existing possessive ScanRepeat for a
greedy repeat whose body consumes only known ASCII characters
(case-sensitive single ASCII literals or classes of ASCII-only atoms)
when a bounded walk of the continuation proves that every path fails
before consuming anything if the current character is one of them.
Giving a character back always leaves such a character at the current
position, so no dropped path could have matched. The walk looks through
captures, anchors, cuts, splits, zero-count repeat entries, and
lookarounds (using single-class lookahead guards when present), and
answers no on Accept, backreferences, calls, repeat loops, or anything it
does not model. Case-insensitive bodies are excluded because ASCII
letters fold to non-ASCII scalars.
Tests pin which shapes convert and compare positions and captures with
the recursive engine.
Bundled construction decoded and walked a root's whole external-include closure before tokenizing. Roots that embed Markdown fenced code (AsciiDoc, Haml, LaTeX, MDC, Nim, reST, Typst, ...) carry about 75 grammars, so construction took ~11 ms even when a document used none of them. - The bundle (format 3) now records, per grammar, its dependency closure (computed by the same dependency walk, moved to engine/grammar_closure.rs so the builder shares it) and per-member traits: whether the member can reach a rule-local repository, contains or reaches $base, and registers injectTo injections. - GrammarSet slots decode bundled members on first access. Construction decodes the root, injectors, and only the members the repository-context walk must visit: a member that cannot reach a rule-local repository keeps empty contexts everywhere, and skipping its subtree cannot reorder the walk's first visits (its $base edge targets the already-visited root or, under a capture base, the member is walked when it can reach $base). - Include-availability checks for an undecoded member use a recorded top-level proof (first-reference chain ending at a match rule) and insert the same cache entries the full search would, so embedded grammars are not decoded just to admit their fenced-block containers. Tests compare recorded closures with the walk, lazy vs eager repository contexts and injection selectors for every bundled language, availability proofs vs the full search with identical cache contents, and lazy vs eager tokens over every multi-grammar bundled fixture.
Building a byte table rebuilt and hashed each byte's full target list, which cost 256 passes over every branch of a large keyword alternation whenever a candidate set was compiled. Transpose the first-byte sets into one item bitset per byte and compare those few words to find each byte's class; only new classes materialize their target lists. The resulting tables are unchanged.
Compiling a class evaluated every non-ASCII scalar or range atom against all 128 ASCII scalars twice, running Unicode case conversion on each probe. Classes listing many symbols (APL glyph sets, for example) spent most of their bytecode compile time there. A non-ASCII scalar never equals an ASCII one and folds to one only through a single-scalar ASCII lower- or uppercase mapping, so derive the case-insensitive mask from its two case maps. Ranges compute their folded bounds once. A test checks the masks against direct evaluation for Kelvin, long s, dotted/dotless i, and spanning ranges; debug builds already assert that agreement for every compiled class.
Every branching node of an ordered literal trie owned an edge vector, so keyword inventories with thousands of branches paid an allocation per branch point to build the trie and again to drop it. Insert through first-child/next-sibling links with a dense root table, then lay each node's children out as one byte-sorted run of shared edge arrays. Lookups keep the linear/binary search split, terminal order is unchanged, and the Unicode case-folding trie is untouched. A test compares wide, unsorted, prefix-sharing inventories against the recursive engine with and without case folding.
A candidate set selected its winner with the multi-pattern scanner and then re-ran that pattern with the backtracking matcher at the selected start just to recover the end, even when no captures were requested. For identifier patterns such as \w+ that replay was a sixth of the tokenizer's time. For patterns in the scanner's regular subset (no captures needed, no lookaround, backreferences, atomic groups, or possessive repeats), the priority-ordered Thompson simulation keeps only the first thread per instruction and position; a discarded duplicate has exactly the future of the thread that was kept, so the first accepted end is the backtracking end. Two cases keep the authoritative replay: any repeat whose body can match empty, because backtracking's empty-iteration check depends on history, and case-insensitive patterns, to keep fold handling on one path. The frontier and whole-set scanner paths return the scanner end directly for such patterns. Tests pin the classification, compare exact patterns with the fallback interpreter at every start, and keep the forced replay-failure fallback covered with a case-insensitive pattern.
Patterns mixing case-sensitive and case-insensitive scopes, such as (?<!@)@@(?i)\b(...) or the tagged-template (#?(?i)sql|sqlFragment(?-i)), got neither start bytes nor a required-literal prefilter, so the candidate scan attempted them at every position of every line. Both gates now fold the whole pattern case-insensitively when scopes are mixed. Case-insensitive start-byte expansion and case-insensitive literal search only add candidates relative to a case-sensitive reading (including the conservative non-ASCII lead bytes and long-s/Kelvin candidates), so they remain rejection gates without false negatives.
Constructing a tokenizer walked every rule reachable from the root, including all embedded grammars, to record each rule's first repository context. A grammar that embeds dozens of languages paid for the whole closure although a document usually touches a few grammars. Contexts only rename a rule's own repository includes, never cross grammar boundaries, and are extended only by rules declaring a local repository. A lookup for a rule that no local repository scope can reach, or whose includes use no locally bound name, is answered as unbound (which every consumer treats like the empty context) without walking. The first lookup that could observe a context runs the unchanged walk. Prepared languages keep the eager, bounded walk. The walk itself now names contexts by index instead of cloning an Arc per work item and per bound rule. A new test checks, across bundled grammars with local repositories, that every rule answered without the walk has an inert walk-assigned context.
When a greedy, variable-width repeat of a single-character consumer is followed by a continuation whose first consumed character can never be one the repeat accepts, handing characters back cannot produce a match. Oniguruma applies the same automatic possessification. The bytecode compiler now proves this from a bounded walk of the already-compiled continuation (positive lookaheads intersect with what follows; open loops contribute their exit and body) and emits ScanRepeat, removing a backtrack frame per iteration and the quadratic retry of identifier prefixes before failing lookaheads common in TypeScript-derived rules. First-character sets are exact per ASCII byte and split non-ASCII into whitespace and other scalars. Tests pin which shapes convert, compare spans with the recursive matcher, and check every scalar against the class-kind table.
decompress_to_vec_zlib starts from twice the compressed size and doubles, zero-filling and copying the IR two or three times per grammar. The blob records its raw length, so inflate once into an exact buffer; a stream longer or shorter than recorded is still rejected. Uncompressed blobs are borrowed instead of cloned.
Prefilters and start-byte sets are built on first use in every new tokenizer, so their construction is part of process-cold throughput for keyword-heavy grammars such as Stata. The multi-literal finder now keeps single-child transitions inline instead of allocating a Vec per trie node, walks transitions by index instead of cloning them during failure-link construction, uses the dense root table while building, and lowercases case-insensitive literals in place instead of allocating lowered copies. StartByteSet::extend unions the bitmaps word-wise instead of probing all 256 bytes per alternation branch. Outputs are unchanged.
Bundled construction still decoded every closure member the repository- context walk must visit (about 17 grammars, 370 KB of IR, for roots that embed Markdown and HTML) only to read their include structure. The bundle now also stores, for each grammar some closure walks, a skeleton IR that keeps what the walk reads: rules and their pattern, capture, and repository references, rule-local repositories, and the scope names of external includes. Regexes, other scope names, and metadata are blanked while pattern and scope tables keep their lengths, so every ID and rule number matches the full grammar. The unbounded walk reads an undecoded member through its skeleton; bounded preparation still reads full grammars. Skeletons add 175 KB (uncompressed) to the bundle. Construction now decodes only the root: AsciiDoc/Nim/Typst and the other Markdown-embedding roots construct in about 1 ms instead of 2-2.5 ms. The lazy-versus-eager context, availability, and fixture token tests cover the skeleton path.
ASCII masks for non-ASCII class chars and ranges re-ran Unicode case conversion of the atom's operands for every one of the 128 probe characters. Fold the operands once and compare against each probe's single-character ASCII case map, which is exactly what class evaluation does. Literal interning used a linear scan per literal, making captured keyword alternations (HTML tag lists) quadratic to compile; index them by content instead. A test compares the new masks with full class evaluation for Kelvin, long s, dotted and dotless i, and mixed-width ranges.
Every group allocated a branch vector even without an alternation, and each punctuation escape such as \. produced its own one-scalar literal that push_concat_node then merged into the neighbouring run. Return a lone branch directly unless it carries an option-change marker, and let a literal run absorb ASCII punctuation escapes that are not quantified (every special escape is a letter or digit). A differential check against the previous parser over all 35,652 catalog patterns and edge cases produced identical ASTs, features, and diagnostics; unit tests cover quantified and extended-mode escapes.
Required-literal extraction cloned every literal it inspected, including the exact-literal probes of each sequence node, and the finder-backed prefilter then retained a String per alternative that no query reads. Borrow literals as Cow<str>, append sequence runs into one buffer, and copy literals only into prefilters that search them directly. Sets searched by the compiled finder keep no literal copies. Extraction results and prefilter answers match the previous implementation for all catalog patterns.
The diagnostic Rust-regex spelling re-encoded every pattern scalar by scalar although only \h, \H, \R, and \Z are rewritten. Scan escape pairs with memchr and copy patterns without those escapes directly.
The ambiguity check recomputed case variants for every sibling on each insertion, adding 8% of instructions to bsl's cold start, whose keyword sets build large Cyrillic tries. A scalar with the same full case mappings as an existing edge has the same variants, so it joins that edge directly; only a new edge at a node that already has edges is checked, by testing each sibling's case key against the new scalar's variants.
Captures starting at the match end were emitted flat from the rule's stack, so an inner capture lost its enclosing capture's scope: `x(?=((a)b))` gave `outer` instead of `outer inner` over `a`. Like vscode-textmate, capture handling now only stops at the first capture that starts after the match and lets captures extend past it, using the same nesting as captures inside the match, including after an empty match. Pre-existing on main; covered by new oracle engine regressions.
`Return` replayed every repeat undo entry since its call and logged each replay, so an outer return also replayed its callees' replays and the log doubled per nesting level: `(?<n>\((?:[^()]|\g<n>)*\))` on 26 nested parentheses took 1.6GB. Returns now restore each repeat and capture slot at most once, from its oldest entry, using per-slot generation stamps. This also replaces the quadratic duplicate check in the capture restore.
Under the class-level fold, `(?i)[[:ascii:]]` matches the Kelvin sign and long s, but its first-character summary still claimed no non-ASCII starts, so `Split` guards rejected `(?i)(?:[[:ascii:]]x|yz)` on `Kx` in position selection. A case-insensitive bracketed class now always admits other (non-space) non-ASCII starts.
Start-byte sets reaching `expand_case_insensitive_start_bytes` hold only ASCII bytes (non-ASCII class atoms disable the gate), and the expansion already admits every non-ASCII lead byte.
The previous budget made long rejected lines report a possible match, so the matcher then ran until its step budget: `<[^>]*>(?!x)` on 2,000 `<` became a degraded line instead of a clean miss. Factor items now cache their last maximal byte run, so later candidates inside it reuse its end instead of rescanning, and the search returns the exact answer in linear time. The budget remains only as a guard.
The tool read grammars from a sibling project's @shikijs/langs 3.23.0 while recording the 4.4.3 pin; read them from tools/golden-oracle, which installs the pinned version.
The per-rule charge shrank to a u32 context ID, but each overlay context also costs an Arc header and a slot in the context table, which the old per-rule pointer size used to cover.
…s iteratively A begin rule whose begin matches empty and includes itself pushed a new frame on every step until the per-line step budget, about 20,000 frames per line, and dropping those frames recursed through their parent `Arc`s until the stack overflowed. Pre-existing on main. - Like vscode-textmate's `hasSameRuleAs`, a push that did not advance past the scan position and re-enters a rule already entered at that position on the line keeps the earlier stack and stops the line, in both line and capture tokenization. - `FrameNode` unlinks its parent chain in a loop when dropped, so legitimately deep states (one frame per nested line) drop safely. Covered by new oracle engine regressions and a small-stack drop test.
When a match rule wins without consuming input, vscode-textmate stops the line and `safePop`s the enclosing rule for the rest of the line and for later lines. We stopped the line but kept the rule, so the bundled GraphQL grammar stayed inside a type block: on the catalog GraphQL corpus 3,282 of 3,653 lines disagreed with vscode-textmate, and now none do. Pre-existing on main. The engine-regression harness now merges adjacent oracle tokens with the same scopes, as the golden harness does, since vscode-textmate emits one token per produce call.
Keeping cached item runs from the first candidate cost 4% of instructions on objective-cpp's cold start, whose prefilters see many short candidates. Candidates use the plain greedy check until rescanning exceeds the line length, then unbounded items reuse their last maximal run.
Once the byte-run prefilter switched to cached runs, bounded items still
rescanned up to `max` bytes per candidate, so `<[^>]{0,20}>` on 3,000
`<` exhausted the budget and the matcher ran to its step limit. A cached
maximal run serves any `max`, so bounded items take at most `max` bytes
of it.
A return scans the undo entries logged since its call, and backtracking into a returned routine can repeat that scan, so time grew quadratically without using steps: `\g<n>z|(?<n>(?:ac?)*)` on 64k `a` took 0.7s with only 640k steps. Returns now charge one step per scanned entry.
`(\g<1>)?` re-entered group 1 without consuming input until the process overflowed its stack; Oniguruma rejects such patterns. The evaluator now fails a call nested deeper than 128, the bytecode VM's limit. Pre-existing on main.
The derived `Debug` for frame stacks recursed through every parent, so formatting a state 20,000 frames deep overflowed the stack. Frame stacks now list their frames iteratively. With `FrameNode` unlinking its chain on drop, the separate iterative drop helper and `FrameStack` drop are redundant and removed.
Charging every return for all entries since its call made enclosing returns pay again for entries their callees had already scanned, so the cost grew with nesting depth: 70-deep C++ template arguments exhausted the capture-replay budget and lost their scopes. Frames now count entries scanned by their callees' returns and charge only the rest.
Backtracking restored the current call frame but not a frame's credit for entries its callees' returns had scanned, so a failed branch could leave credit that made every later return of that frame free: `\g<n>z|(?<n>(?:\g<k>)*b|(?:ac?)*)|(?<k>a)` on 64k `a` took 0.7s on 2.1M steps. Credit changes are now logged in the repeat undo log, which backtracking already unwinds, under a reserved slot.
Credit for entries a callee's return had scanned was never used up, so a caller that backtracked and returned again rescanned those entries for free each time: `\A\g<n>z|(?<n>\g<k>a*)|(?<k>(?>(?:bc?)*))` on 100k characters took 2.9s within a 1M-step budget. A returned call's repeat entries and its own restores leave every repeat slot as it was at the call, so a return now logs a jump entry back to its call mark, and an enclosing return scanning newest first skips that range. Every visited entry is charged; deep nesting stays linear and repeated returns pay for each rescan. Capture restores keep their full, charged scan. Call frames drop the credit field.
Capture restores still rescanned each callee's whole capture log at every enclosing return, so their charge grew with the square of the depth: `(?<n>\((?:[^()]|\g<n>)*\))+` over 35 40-deep nests ran out of budget in capture replay. A return now logs, after its restores, a capture jump entry to its call's mark and one summary entry per slot it saw with the slot's value at the call. Enclosing returns scan newest first, read the summaries and skip the range; undoing either entry writes nothing.
grammar-stats reads regex constructs from pattern text and took the `^` and `*+` inside the new class-folding conformance case for an anchor and a possessive star. Escaping them keeps the class identical.
Source-compiled grammars record include-walk facts that the IR does not carry, so `syntaxmate-bundle --check` rejected every grammar after 086d1b1. The check now compares against the compiled grammar without the summary, as the IR round-trip test does. The bundle is regenerated because its provenance hash covers the bundle tool.
phongndo
added a commit
that referenced
this pull request
Sep 27, 2026
…d API, leaner output (#16) ## Summary Makes Syntaxmate a better library to depend on: smaller and more configurable builds, a cleaner 0.2 public API, and the pieces that editors, servers and static-site generators need. This is a breaking 0.2 change; the migration table is in `CHANGELOG.md` → *Migrating from 0.1*. - **Choose your languages.** `syntaxmate-bundle --languages rust,toml,json` builds a subset bundle with the dependency closures and only the licenses it needs. Load it with `Catalog::from_static(include_bytes!(…))`; this works with `default-features = false`. - **Shareable `Highlighter`.** It is cheap to clone and `Send + Sync`, with `&self` methods. Clones share a bounded cache of prepared languages and idle tokenizers, configured through `HighlighterOptions`. Sessions reuse the shared preparation. - **Catalog.** Language queries return borrowed data instead of allocating. New `LanguageInfo` exposes each language's aliases, extensions, basenames and root scope. Detection now combines path, shebang, modeline and `firstLineMatch` in one entry point. - **Faster cold start.** The bundle's string and scope tables are read directly from the embedded bytes, with no per-string allocation or decompression. The bundle format is still v4 (it was never released), and grammar data is unchanged. - **Leaner output.** - HTML puts the theme's default colours on `<pre>` and merges adjacent runs with the same style. - ANSI no longer paints the theme background on every token. - New `render_html_to` / `render_ansi_to` write directly to any `fmt::Write`. - **Theme-independent CSS classes.** `HtmlOptions::class_prefix` emits nested scope classes, and `html_stylesheet(theme, prefix)` produces ranked CSS for them. Render the HTML once and switch themes by swapping stylesheets. - **API cleanup.** - Token types: `TokenizedDocument → TokenizedLine → Token` and `HighlightedDocument → HighlightedLine → HighlightedToken`, with one `.scopes()` view in both document and incremental mode. - `Theme` is the only theme type and gains `default_style`, `color` and `from_rules`; theme-matching introspection requires the `diagnostics` feature. - `Style`, `FontModifiers` and `ScopeStackId` are the canonical names; scope storage and instrumentation are private. - `#![warn(missing_docs)]` is on. - **Editor support.** `TokenizerState: PartialEq + Eq + Hash`, compared in constant time, so an editor can stop re-highlighting once the state after an edited line matches its previous state. - **Structured errors.** Matchable kinds with context (grammar scope, include target, regex position, JSON line/column, limits, render mismatch vs writer failure) and `source()` chaining. Payloads are boxed so `Error` stays at 32 bytes. - **Version 0.2.0.** `Cargo.toml`, both lockfiles and the README dependency line are bumped for the breaking changes; the changelog section stays *Unreleased* until the release date is set. - **Packaging.** Raw grammar JSON and checkout-only targets are left out of the crate. docs.rs documents only the user-facing features, with feature labels. `rust-version = "1.88"` is verified with a real 1.88 toolchain. ## Compatibility impact - **Breaking public API (0.2).** Every removed or renamed item has a row in the migration table. The top-level `canonical_language`, `detect_language_from_path` and `available_languages` move to `Catalog`. - **Behaviour fix.** Static frame identities are now keyed by embedded base-grammar context and scope prefix. Previously a cached identity could be reused across different `$base` contexts. There is a regression test. - **HTML/ANSI bytes change.** They're smaller, and the resolved styles are unchanged. Wrapper-free HTML keeps full colours on every span. - **Bundle format.** v4 is redefined in place (unreleased). The minimal dependency tree includes `miniz_oxide` again, because custom catalogs need to decompress grammars without default features. - **MSRV.** 1.88. PR #15's unreleased raise to 1.98 is reverted, because nothing requires it; the development toolchain stays at 1.98. ## Validation Run locally on the final branch head (NixOS x86_64, Rust 1.98.1): - `cargo fmt --all --check`: pass - `cargo clippy --all-targets --all-features --locked -- -D warnings`: pass - `cargo test --all-features --locked`: 575 passed, 0 failed - `cargo run --locked --bin syntaxmate-bundle --features bundle-tools -- --check`: current - `cargo check --lib --no-default-features --locked`, plus each of `bundled-grammars`, `bundled-themes`, `html`, `ansi`, `diagnostics` and `bundle-tools` on its own: pass - `cargo test --locked --no-default-features --test custom_bundle` and `--features bundled-grammars --test custom_theme_session`: pass - `cargo check --manifest-path fuzz/Cargo.toml --locked`: pass - `RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features --locked`, `python3 tools/check-docs.py` and `python3 tools/check-language-docs.py --check`: pass - `cargo package --locked` (verified): pass, 2.28 MB `.crate` (was 3.58 MB) - Rust 1.88.0 `cargo check --lib --locked`, with and without default features: pass - `cargo semver-checks check-release` (0.50.0) against published 0.1.3: 0.1.3 → 0.2.0 is a major change for 0.x; no further update required Not run locally: `cargo hack` (not installed; each feature was checked on its own instead), the fuzz campaigns, and the oracle/performance CI jobs. ## Output evidence Measured locally; A/B comparisons were interleaved. | | `main` | This PR | |---|---:|---:| | Stripped default consumer, same `highlight_html` program (fat LTO) | 4.06 MB | 4.14 MB (+71 KB: catalog, CSS classes, structured errors) | | Consumer with a rust+toml+json subset bundle, `default-features = false` | — | 1.60 MB | | Cold setup + first highlight, rust / json / tsx (median of 21 processes) | 2.22 / 1.85 / 4.93 ms | 1.30 / 0.96 / 3.92 ms | | Retained heap after construction (`profile-alloc`, rust) | 3.95 MB | 2.01 MB | | HTML output, `src/highlighter.rs`, github-dark | 321 KB | 81 KB | | Steady-state throughput (`profile-product`, 6 languages) | — | within ±1.5% | - **Scopes:** unchanged. All golden scope and theme tests pass, and the throughput runs produced identical output digests. - **CSS classes:** checked in headless Chromium. Computed styles matched inline styles for every character across the 4 bundled themes × 6 stress fixtures (rust, tsx, html, markdown, php, css), with 0 mismatches. A negative control with the stylesheet removed mismatched on every character.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Speeds up cold tokenization across the catalog and fixes a set of Oniguruma and vscode-textmate compatibility bugs found while reviewing that work.
Performance
Process-cold throughput on the
catalog-repeatedcorpus, median of 10 interleaved sweeps against the previous integration build:Earlier commits on this branch (grammar refresh, allocation guardrails, first-use compile work) are in
CHANGELOG.md.benchmarks/textmate/catalog-performance.jsonanddocs/language-status.mdare refreshed.The main changes, grouped by cause:
Fixes from review
Eight rounds of review covered the whole diff, then each round's fixes. Every change below is pinned by a regression test or an oracle-generated case.
Regex engine:
\p{..}stays unfolded.(\g<1>)?, no longer overflows the stack.\k<name>and(?(<name>)…)consider every group with that name.a{1,2}+is parsed as a repeated interval, not a possessive one.Tokenizer:
safePop). The bundled GraphQL grammar no longer stays inside a type block: on the catalog GraphQL corpus, 3,282 of 3,653 lines used to disagree with vscode-textmate, and none do now.begincrashed the process.Verification
cargo test --all-features.Known differences not addressed here
These are older than this branch and are left for follow-up:
contentName;ϑ/ϴ,ß/ss); ours is not.\Kisn't supported, malformed[:…brackets are parsed differently, and a few lazy or empty-iteration capture cases differ.