Skip to content

fix(expr): give every evaluator its own ExprTk parser (#257) - #279

Merged
wshlavacek merged 1 commit into
mainfrom
fix/257-evaluator-owns-its-exprtk-parser
Aug 10, 2026
Merged

wshlavacek merged 1 commit into
mainfrom
fix/257-evaluator-owns-its-exprtk-parser

Conversation

@wshlavacek

Copy link
Copy Markdown
Collaborator

Closes #257.

What the residual race actually was

#201 found two threads compiling through one exprtk::parser and serialized compile(). Its regression test then went on failing about 10% of the time on main (measured: 18 of 20), with a signature #201 was never calibrated against — SIGTRAP, the macOS malloc freelist check, raised in NetworkModel::clone() while another thread sat in ExprTkEvaluator::evaluate() under CVode with the GIL released. That thread was not compiling, so no lock on compile() was ever going to cover it.

The mechanism is one line of ExprTk. parser::compile() ends with

symtab_store_.symtab_list_ = expr.get_symbol_table_list();

and never clears it, so a parser retains a strong handle on the symbol table of the last expression compiled through it — and ExprTk refcounts symbol tables with a plain std::size_t, not an atomic. One parser behind two evaluators therefore means thread B's compile dropping thread A's symbol table while thread A is churning that same counter with no lock at all: in register_symbol_table, in the growth of its expression vector, in its own destructor. A lost update runs clear() on a symbol table whose variable addresses are already baked into A's live compiled nodes. The result is a corrupt heap, so the crash lands wherever the next allocation looks — which is exactly the clone-side operator new in the issue's crash report.

Reproduced directly, not inferred. A standalone harness mimicking the old design (shared parser, mutex around compile(), 8 threads cloning/compiling/evaluating) dies with Trace/BPT trap: 5 in 19 of 20 runs, plus SIGSEGV and SIGBUS. Two variants isolate the cause: giving each evaluator its own parser is clean 20 of 20, and keeping the shared parser but scrubbing symtab_store_ after each compile is also clean 20 of 20. The retained symbol-table handle is the whole of it.

The fix

No lock could close this, because the counter is touched on paths no evaluator API sees. So the sharing goes instead, which is what the issue asked for — "deciding which of the evaluator's parts can legitimately be shared at all". The answer is none:

The audit the issue asked for came back clean otherwise: every other Impl member is per-evaluator, and the three file-scope reserved-name sets are function-local statics that are read-only after initialization.

New ExprTkEvaluator::parser_identity() makes the invariant assertable rather than merely documented.

Cost: none, and negative where the clone compiles nothing

The performance argument for sharing was illusory. It existed to avoid re-constructing the ~100 KB exprtk::parser<double> (51 µs, measured) per model clone — but NetworkModel copy; default-constructs an evaluator that clone() replaces one line later, so that construction was already being paid and thrown away on every clone. Lazy construction makes the discarded evaluator genuinely free. NetworkModel::clone(), before → after:

model before after
BIOMD0000000701.xml (71 params, events) 521 µs 527 µs
func_composition.net 92 µs 94 µs
simple_decay.net (no expressions) 67 µs 34 µs

Tests

Two new cases in tests/test_bngsim.cpp, both run against the pre-fix design before being trusted:

  • test_evaluator_clone_shares_no_exprtk_state — the invariant, deterministically: a clone's parser is not its source's, and a fresh evaluator has built none yet.
  • test_concurrent_clone_compile_evaluate — the failure it prevents: 8 threads each clone, rebuild from the cached preprocessed strings, integrate, and fire a lazy compile mid-integration, with every evaluation checked against the serial answer.

Rebuilt on the old shared-parser evaluator, the test binary died in 8 of 8 runs, always inside that second case, always on the SIGTRAP. On the fix it passes, and ctest is 8/8 green.

tests/CMakeLists.txt links Threads::Threads for it — needed on glibc, harmless on Apple.

Verification

  • python/tests/test_expression_parser_thread_safety.py: 25 of 25 clean (18 of 20 on main). Its docstring and failure message now distinguish the two defects a signal can mean.
  • Full Python suite: 4180 passed, 30 skipped, 0 failed.
  • ctest: 8/8.
  • clang-format (v17.0.6) and ruff (v0.14.0) clean on every touched file.

test_parallel_workers_clone_the_model.py's docstring said clones "genuinely share" the parser; corrected, since they no longer share anything.

`compile()`. The regression test it added then failed ~10% of the time on main
(2 of 20), with a fourth signature it was never calibrated against: SIGTRAP —
the macOS malloc freelist check — raised in `NetworkModel::clone()` while
another thread sat in `ExprTkEvaluator::evaluate()` under CVode, GIL released.
That thread was not compiling, so no lock on `compile()` was going to cover it.

The mechanism is one line of ExprTk. `parser::compile()` ends with

    symtab_store_.symtab_list_ = expr.get_symbol_table_list();

and never clears it, so a parser retains a strong handle on the symbol table of
the last expression compiled through it — and ExprTk refcounts symbol tables
with a plain `std::size_t`. One parser behind two evaluators means thread B's
compile dropping thread A's symbol table while thread A churns that same counter
with no lock at all: `register_symbol_table`, growth of its expression vector,
its own destructor. A lost update runs `clear()` on a symbol table whose
variable addresses are already baked into A's live compiled nodes, and the crash
lands wherever the corrupted heap is next touched.

No lock could close that, because the counter is touched on paths no evaluator
API sees. So the sharing goes instead: an ExprTkEvaluator now owns all of its
ExprTk state and shares none of it, `clone_empty()` inherits nothing, and the

The performance argument for sharing was illusory. It avoided re-constructing
the ~100KB parser per clone, but `NetworkModel copy;` default-constructs an
evaluator that clone() replaces one line later, so that construction was being
paid and discarded every clone anyway. The parser is now built lazily on first
compile, which makes the discarded one free. Measured clone(), before -> after:
BIOMD0000000701 521 -> 527 us, func_composition.net 92 -> 94 us,
simple_decay.net (no expressions) 67 -> 34 us.
@wshlavacek
wshlavacek force-pushed the fix/257-evaluator-owns-its-exprtk-parser branch from c64797c to 28cf0d4 Compare August 10, 2026 19:39
@wshlavacek
wshlavacek merged commit c80126a into main Aug 10, 2026
5 checks passed
@wshlavacek
wshlavacek deleted the fix/257-evaluator-owns-its-exprtk-parser branch August 10, 2026 20:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The #201 regression test is ~10% flaky on main: a second ExprTk race between clone() and a GIL-free evaluate(), not compile()

1 participant