[Montana]

Closes #1217 (with #1348, which fixes its #1212 group).

What was wrong

Running the whole suite with every model forced through the MIR JIT (BNGSIM_CODEGEN_JIT=mir BNGSIM_CODEGEN_THRESHOLD=1) is the check that found #1208 and #1212. It is only a signal if it passes. On main 9a26f5b with #1348 merged in, 53 tests fail under it. That matches #1217's accounting once #1212's 38 are gone:

group	tests	why it cannot pass under a forced JIT
cc's compiled artifact	32	the test is about the .so, its cache key, the redirected cache dir, or the output-sensitivity and sens-RHS notes written beside it. The JIT compiles in process and writes none of them.
cc by name	7	the test asserts codegen_backend == "cc", and the JIT reports "mir"
a forced threshold	13	the test needs a 2-species model to run interpreted (exprtk, analytical, no .so, the interpreter's RHS counters), and THRESHOLD=1 compiles it. They fail under forced cc too.
engine-dependent answers	2	below

The 32 artifact tests are in test_output_sens_support_note.py (22), test_sens_rhs_decline_note.py (5), test_hpc_contract.py (2), and one each in test_codegen_cache_key.py, test_ar_species_output_sensitivities.py and test_sens_numerically_zero_base.py. The 7 cc-by-name tests are in test_engine_choice_accessors.py (4), test_tfun_analytical_jacobian.py (2) and test_lu_fill_routing.py (1). The 13 threshold tests are in test_engine_choice_accessors.py (3), test_rhs_obs_func_gate.py (3), test_steady_state_polish_jacobian.py (3), test_jacobian_discontinuous_fallback.py, test_lazy_jacobian.py, test_model_codegen_sensitivity.py, and test_steady_state_scaled_criterion.py's MODEL1712050008 retry test. That last one pins the arm64 interpreter's route (solver_jacobian_retried); compiled, the march reaches the same root, within 1.6e-6, without a retry.

The change

python/tests/_codegen_backend.py. A new helper, imported the way _fd_sens.py and _ci_workflow.py are. It provides three skipifs and one xfail:

• under_jit_no_cc_artifact and under_jit_engine_name read the JIT selection through bngsim._codegen._codegen_jit_backend(), the library's own reading of BNGSIM_CODEGEN_JIT.
• threshold_forced applies when BNGSIM_CODEGEN_THRESHOLD is set below the default of 256.
• xfail_when_threshold_forced(issue, why) is a strict xfail under the same condition.

Each skip reason is declared in conftest.py's _DECLARED_SKIPS (tier ANYWHERE), so BNGSIM_SKIP_AUDIT=strict accepts them.

+    @threshold_forced
     def test_small_model_uses_exprtk(self):
 ...
+    @under_jit_engine_name
     def test_explicit_codegen_uses_cc(self):

The two engine-dependent tests are not skipped. A forced run shows each gives a different answer on the compiled path. Both findings are filed, as #1357 and #1358. Each test carries a strict xfail under a lowered threshold, naming its issue, so the mark has to go when the fix lands:

1. Steady-state refusal (steady_state(method='integration') on the compiled RHS and Jacobian refuses a state at residual 2e-39 and does not retry on difference quotients #1357). steady_state(method="integration") on the compiled RHS refuses glioblastoma_egfrviii_seed.net's steady state at residual 2.1e-39, with continued_run_distance = inf, because its run test is never retried on difference quotients. That affects test_a_march_whose_integrator_fails_returns_the_state_it_passed[integration]. Fixed in steady state: the run test retries a NaN f at the clamped state, as the time course does (#1357) #1370.
2. Stale get_param (get_param on an assignment-rule parameter after a compiled run returns its load-time value, not the rule's #1358). After a compiled run, get_param on a parameter an assignment rule sets returns its load-time value: q reads 1.0, where an interpreted run reads the rule's end-of-run 0.0183. The trajectory is unaffected. This is test_a_write_after_a_run_does_not_move_it_to_the_rules_current_value, whose precondition reads the slot.

Verified

On this branch with #1348 merged in, macOS arm64 MIR build, --max-worker-restart=500, BNGSIM_SKIP_AUDIT=strict:

run	passed	skipped	xfailed	failed
BNGSIM_CODEGEN_JIT=mir BNGSIM_CODEGEN_THRESHOLD=1	16,476	185	4	0 (53 before)
BNGSIM_CODEGEN_THRESHOLD=1 (forced cc)	16,512	149	4	0 (15 before)
default	16,527	137	1	0, the same skips as main

The suite's existing xfail is the 1 in each count; the forced runs added three. That was measured when MODEL1712050008's retry test still carried an xfail. It is now a forced-threshold skip, so a forced run reports 3 xfailed and one more skip. Rerun after that change, test_steady_state_scaled_criterion.py under BNGSIM_CODEGEN_THRESHOLD=1 with the strict audit gives 222 passed, 3 skipped and 1 xfailed (glioblastoma, #1357), and 224 passed by default. The strict audit accepts every new skip.

Negative control: with the artifact reason removed from _DECLARED_SKIPS, the strict audit reports 1 undeclared skip reason(s).

Not in this PR

• The scheduled forced-MIR job Make the full suite a clean signal under BNGSIM_CODEGEN_JIT=mir: 93 forced-MIR failures hide new ones (cc-artifact tests, engine-name asserts, #1212, #1213) #1217 proposes. It is a workflow edit, for a maintainer to add once this and codegen: write C for what ExprTk means, or decline before compiling (#1212) #1348 are in. A sketch, on the MIR build mir.yml already makes:

  - name: Whole suite, every model through the JIT (issue #1217)
    env: {BNGSIM_CODEGEN_JIT: mir, BNGSIM_CODEGEN_THRESHOLD: "1", BNGSIM_SKIP_AUDIT: strict}
    run: python -m pytest python/tests -n 3 --max-worker-restart=500

• Fixes for the two engine-dependent findings: steady_state(method='integration') on the compiled RHS and Jacobian refuses a state at residual 2e-39 and does not retry on difference quotients #1357 and get_param on an assignment-rule parameter after a compiled run returns its load-time value, not the rule's #1358.

[elyfmiller]

Thanks. We reviewed this independently, Opus 5.5 and Fable 5.1, each on our own tree: main c0877fa with #1348 and 263700b merged in (both clean), on macOS arm64 with MIR on. The marks don't hide a wrong number, and the before and after counts are the body's. A default run doesn't change.

Three medium findings follow, then three low:

• two are about the two strict xfails;
• one is a skip the forced run still treats as a broken environment.

What we checked comes first, so it needn't be redone.

What we checked

check	result
Before, unmarked on main + #1348	Forced MIR (BNGSIM_CODEGEN_JIT=mir BNGSIM_CODEGEN_THRESHOLD=1): 53 failed. Forced cc (BNGSIM_CODEGEN_THRESHOLD=1): 15 failed (13 threshold, 2 engine-dependent). Both match the body.
After, full suite, BNGSIM_SKIP_AUDIT=strict	Fable: • forced MIR: 16,465 passed, 215 skipped, 2 xfailed, 0 failed, exit 1 via the audit (item 3) • forced cc: 16,501 passed, 179 skipped, 2 xfailed, 0 failed, exit 0 (+13 skips over default) • default: 16,516 passed, 166 skipped, 0 failed, exit 0, the same as main + #1348 Opus: forced MIR 16,028 passed, 505 skipped, 2 xfailed; forced cc 16,064 passed, 469 skipped, 2 xfailed. Opus's one failure, and the higher skip count, come from a local run_tests.sh probe for packages missing there (jax, diffrax, roadrunner). The suite's existing xfail, #1254's, was dropped by #1295, merged since this PR's base, so 2 xfailed is the two added here. The new skips are +49 and +13: the body's +48 and +12, plus MODEL1712050008's move from an xfail to a skip.
Skips by reason, forced MIR	32 artifact, 7 cc-by-name, 13 forced threshold, as the body groups them. The strict audit accepts them.
Each mark can fail	We ran the 54 tests this PR marks (52 skips, 2 xfails) unmarked on main + #1348. Forced MIR: 53 failed and 1 passed (item 4). Forced cc: all 15 threshold and engine-dependent tests failed. Setting under_jit_no_cc_artifact's condition to False gives 27 failures in the two artifact files under forced MIR.
The two strict xfails under forced MIR and forced cc	both xfail, with #1358 and #1357 as the reasons
Default run	the 16 changed files give 534 passed and 7 skipped, with and without this PR
Audit negative control	with the artifact reason removed from _DECLARED_SKIPS, the audit reports "1 undeclared skip reason(s)" and test_every_hand_written_reason_is_declared fails; restored, it passes. As the body says.
Gates	ruff clean on all 18 files; the changelog check passes (tests only, so no fragment)
The helper	_jit_selected() is the library's own _codegen_jit_backend(), which reads an unknown value (cc, a typo) as no JIT, so a mistyped selection can't skip. On a build without MIR, JIT=mir makes codegen=True fail, so the JIT skips lose nothing there.
Merge order (git merge-tree)	this PR onto #1370: clean. Onto #1359: a conflict in test_sbml_initial_assignment_param_lift.py.

1. The forced-threshold marks don't look at the model's size

_threshold_lowered() asks whether BNGSIM_CODEGEN_THRESHOLD is below 256. The Simulator compiles when n_species >= threshold, so a test's model compiles only once the threshold is at or below its size. Between the two, the marks apply, but the model still runs interpreted:

threshold	[integration] (#1357, glioblastoma, 10 species)	test_a_write_after_a_run… (#1358, 2 species)	the 13 threshold_forced tests, unmarked
1 or 2	XFAIL	XFAIL	at 1: 13 fail
3 to 10 (we ran 3, 4, 10)	XFAIL	FAILED [XPASS(strict)]
11 to 255 (we ran 11, 50)	FAILED [XPASS(strict)]	FAILED [XPASS(strict)]	at 50: 10 pass, 3 fail (test_steady_state_polish_jacobian.py)

So a run with the threshold tuned rather than forced fails on one or both strict xfails, and at 50 it skips ten tests that would pass. We'd suggest one of two fixes:

• Pin the threshold inside the 13 tests. monkeypatch.delenv("BNGSIM_CODEGEN_THRESHOLD"), or a fixture doing that, so they run interpreted under any setting. With an autouse fixture doing this, all 13 passed under JIT=mir THRESHOLD=1. That removes the threshold_forced skip and its audit entry.
• Or make the conditions size-aware, e.g. xfail_when_threshold_forced(issue, why, n_species=10), true when threshold <= n_species.

2. The strict xfails go stale when #1357 and #1358 are fixed

Strict is the right choice, but nothing here runs it. #1370 fixes #1357, and its diff doesn't touch _COMPILED_MARCH_REFUSES. It merges clean with this PR, so neither rebase will make anyone remove the mark. After both land, a forced run fails on an XPASS. #1359 fixes #1358 and conflicts here, so that one should be caught in the merge.

No CI job sets the forced variables, so the first to see it is whoever next runs the forced suite by hand. That is the run this PR exists to keep clean. Two ways to close it:

• add the mir.yml step this PR sketches;
• or have steady state: the run test retries a NaN f at the clamped state, as the time course does (#1357) #1370 and ode: leave a rule-set parameter at its end-of-run value after a compiled warm run (#1358) #1359 each drop their mark.

3. A forced-MIR run still exits 1 under the strict audit

test_stops_on_the_evaluators_flip.py::test_the_compiled_right_hand_side_flips_where_the_interpreter_does skips its two compiled cases at line 133:

if codegen and sim.codegen_backend != "cc":
    pytest.skip("no C compiler: the compiled right-hand side is not built with cc")

Under the JIT the backend is "mir", so the skip fires. Its reason is declared at tier _LOCAL_ONLY ("a CI leg without cc has broken, not adapted"), so under BNGSIM_SKIP_AUDIT=strict the audit lists the skips as an incomplete environment:

!!   2  no C compiler: the compiled right-hand side is not built with cc

Then pytest_sessionfinish sets exit status 1, with no FAILED line. The job this PR sketches sets BNGSIM_SKIP_AUDIT: strict, so it would fail on this.

It's on the base too; the 53 failures hid it. It doesn't fire under forced cc or by default. It's the PR's under_jit_engine_name case, but skipping isn't needed: with the condition changed to sim.codegen_backend not in ("cc", "mir"), both cases pass under forced MIR. The JIT's compiled right-hand side flips where the interpreter does, which is what a forced run is for.

Low

4. test_no_sens_does_not_trigger_codegen can't fail under the JIT. It asserts only sim._codegen_so_path == "". MIR writes no .so, so the test passes under forced MIR while the 2-species model is compiled. That's on main already. It's also why the body's group rows (32 + 7 + 13 + 2) add to 54 against its 53. Asserting sim.codegen_backend == "exprtk" would let it see codegen through either backend.
5. The default threshold is copied. _DEFAULT_CODEGEN_THRESHOLD = 256 repeats _simulator.py's literal (int(os.environ.get("BNGSIM_CODEGEN_THRESHOLD", "256"))). If the default moves, the helper drifts without notice; a shared constant avoids that. Pinning the threshold, as in item 1, removes the need for it.
6. The xfails take no raises=. The suite's other strict xfails pass one (test_invariants.py, test_codegen_sens_fd_oracle.py). Without it, any error in these two tests under a forced run counts as the known failure. raises=AssertionError keeps an unrelated error visible.

Not checked: Linux and Windows forced runs (CI runs none), and #1370 and #1359 built in. The stale-xfail point follows from strict=True and the threshold runs in item 1.