The matcher retains every rule's token slice after New. The current corpus
contains about 6.45 million rule tokens, so these slices account for roughly
26 MB plus one allocation per rule. Aho-Corasick matching needs each rule's
length, while the whole-text hash path uses the tokens to verify collisions.
Change the runtime representation so full rule token slices can be released
after the engine is built without weakening collision checks. One possible
route is to verify a hash candidate against the automaton's terminal outputs.
Flattening token decoding into an arena may also reduce cold-start allocations
if tokens still need to exist during construction.
Record cold New time, allocated bytes, retained memory, and allocations before
and after the change. Matching output and the conformance baseline must remain
unchanged.
The matcher retains every rule's token slice after
New. The current corpuscontains about 6.45 million rule tokens, so these slices account for roughly
26 MB plus one allocation per rule. Aho-Corasick matching needs each rule's
length, while the whole-text hash path uses the tokens to verify collisions.
Change the runtime representation so full rule token slices can be released
after the engine is built without weakening collision checks. One possible
route is to verify a hash candidate against the automaton's terminal outputs.
Flattening token decoding into an arena may also reduce cold-start allocations
if tokens still need to exist during construction.
Record cold
Newtime, allocated bytes, retained memory, and allocations beforeand after the change. Matching output and the conformance baseline must remain
unchanged.