Skip to content

differential: the stable check also reruns from a nudged start (BIOMD0000000908) - #1427

Open
wshlavacek wants to merge 6 commits into
mainfrom
fix/stable-start-nudge
Open

wshlavacek wants to merge 6 commits into
mainfrom
fix/stable-start-nudge

Conversation

@wshlavacek

@wshlavacek wshlavacek commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Explains the loose end #1423 left: BIOMD0000000908's reorder row, recorded as a DIFF of 1.09e+134 of the peak with no reason or issue.

What 908 is

Not a bngsim defect. The model's clearance of sunitinib is the reaction Sunitinib_not_working, which consumes S at the rate compartment * (-n * S). A negative rate on a reaction that consumes S makes S' = +0.277·S. With no dose (vs = 0), the model's answer is S = 0 for all time, and an unstable one.

On main 15ba102, at rtol 1e-6 over 0 to 300:

S at t = 10 outcome
species as written 0 exactly runs; same at rtol 1e-8 and 1e-10
species reversed −3.6e-19 grows 16× per 10 days (e^0.277·10); CV_CONV_FAILURE near t = 152

Reversing the species reorders the linear solves, and their rounding leaves S a speck off 0. libRoadRunner from the same nudged start moves 2.6e14, as bngsim does. stable passed the model: no jittered grid or tolerance moves S off 0.

The change (parity_checks/differential/_differential.py)

  • check_stable reruns the model once more, from a start nudged by START_NUDGE = 1e-3 of what a step may err by, rtol·|x| + atol. A departure past the bound marks the model contested, as a jittered grid or tolerance does; so does a run that fails from the nudged start.
  • Only the species the equations hold still at the start are nudged (held_at_start): a derivative of exactly 0 there, a row of the Jacobian's pattern that can move it, not fixed, and not read by any event trigger. A value held exactly is what rounding alone can break. The others are not a rounding's to move, and the first version of this PR, which nudged every species, flagged 9 models for them:
    • a species already moving at the start, which the nudge only shifts in time;
    • an input nothing in the equations moves: BIOMD0000000265's r31switch and Flag, which only its events write (0.37 nudged; 3.4e-5 without them);
    • a species a trigger reads, carried across its threshold at the start: BIOMD0000000285's PIdeath > 0 and p38death > 0, both true at t = 0, fire neither (a Level 2 trigger is initialValue = true, which bngsim follows and all 494 Level 3 twins of Level 2 events in the SBML test suite say), so the cell they kill at once lives on.
  • Only a departure the nudge grew into counts: one already past the bound at the first row after the start is the model's own laws telling the starts apart, as the equality-dwell shape's if(Bobs==0, kb, 0) does (A rate law that branches on an equality the state sits on (if(B==0, ...) with B held at 0) runs on the analytic sensitivity path and returns 0 for the columns that move the switch #1228).

What it flags

A stable-only sweep of the corpus on this branch (macOS, 1,349 models, 600 s a model), extracted with verdicts.py, moves 6 stable rows PASS → DIFF, recorded in baselines/differential.json (those 6 lines and the tally). Each was checked against libRoadRunner from the same nudged start, with a fresh RoadRunner per run:

model nudged departure what it is
BIOMD0000000908 2.7e+14 the unstable S = 0 above; RoadRunner 2.6e14
MODEL2003180003 5.0e+148 blows up in both engines, at rtol 1e-6 and 1e-11
MODEL1012110001 71 an unstable balance, M_329 = 7.5: both engines climb to 3,539 from the nudged start, and RoadRunner at rtol 1e-11 leaves the balance unnudged
MODEL1812040005 the run fails fails in both engines at both tolerances (#1340's model)
MODEL1108260014 1.2e-3 (bound 1e-3) the rate-rule parameter_3; both engines agree to 9 digits at every tolerance
BIOMD0000000739 1.5e+4 (bound 1.7) bngsim at the SED-ML tolerances, not the model: see below

BIOMD0000000739 and BIOMD0000000457 (which the first version flagged) are the same thing, and it is not this PR's to fix. At the SED-ML tolerances (rtol 1e-6, atol 1e-9) bngsim's answer for their small species is off by much more than the tolerance: 457's species_42, about 1.6e-7, is 13% off at t = 5 against a run at rtol 1e-10, atol 1e-16, where bngsim and RoadRunner agree to five digits. libRoadRunner's answer at the same nominal tolerances is right because its absolute_tolerance is multiplied by each species' value; bngsim's is absolute. This is #1420's question, for an explicit scalar atol (here the SED-ML file's), which AtolScaleWarning does not second-guess. The RoadRunner parity suite forces rtol 1e-9, atol 1e-12 on both engines, so it does not see it.

The sweep's other differences from the baseline (TIMEOUT → PASS and the like) are its 600 s cap against CI's, and are left as recorded. This PR's own Linux run compares against these rows.

Also

ci/test_floors.env: the full suite's collected floor to 17360. All three legs collected 17370 here, 502 above the floor, past its slack of 500.

Tests (python/tests/test_differential_shapes.py)

  • An unstable zero (S → 2S at rate k from S = 0) is contested; its decaying twin passes.
  • The nudge moves a held start by its share of a step's allowance, and nothing else.
  • Only a species the equations hold still is nudged: not one a trigger reads, not an input, not one already moving.
  • A run that fails from the nudged start is contested.
  • All 514 shape tests pass, equality-dwell's stable among them.

BIOMD0000000908's `reorder` row reads DIFF, 1.09e+134 of the peak, with
no explanation in the baseline. It is not a bngsim defect. The model's
sunitinib clearance law is -n*S on a reaction that consumes S, so
S' = +0.277*S, and with no dose its answer S = 0 is an unstable one.
The file's species order keeps S at exactly 0; reversed, the rounding of
the linear solves leaves 3.6e-19 at t = 10, which grows until the run
breaks near t = 152. `stable` passed it, since no jittered grid or
tolerance moves S off 0.

check_stable now also runs the model from a start moved up by
START_NUDGE = 1e-3 of what a step may err by, rtol*|x| + atol, a start
the solver cannot tell from the base's. A departure past the bound
marks the model contested, as a jittered grid or tolerance does, and so
does a run that fails from it. Only a departure the nudge grew into
counts: one already past the bound at the first row after the start is
the model reading its start exactly, `if(Bobs==0, kb, 0)` on a B its
law holds at 0 (#1228), which no rounding breaks and which the
equality-dwell shape keeps passing.

908 now reads contested (2.7e+14 from the nudged start), so its reorder
DIFF is a fragile model's, not an unexplained one. run() takes the
nudge; the tests plant an unstable zero beside a decaying twin, check
the nudge's size, and a run that fails from it.
All three legs collected 17370 on 34a8899, 502 above the floor of 16868,
past its slack of 500. Main at cec9b79 collected 17367 there.
A stable-only sweep of the differential corpus on this branch (macOS,
1349 models), extracted with verdicts.py: 14 models whose answer leaves
its bound from a start nudged by 1e-3 of a step's allowance, BIOMD0000000908
among them (2.7e+14), and MODEL1812040005 (#1340) and BIOMD0000000457,
whose runs fail from it. Each goes PASS -> DIFF, contested; no other row
changes. This PR's own Linux run compares against them.
…start

Checking the 14 models the first nudge flagged against libRoadRunner from
the same start: 5 are the models' own (both engines move the same way,
or fail, at rtol 1e-11 too), and the other 9 were the nudge moving what
no rounding moves.

- A species already moving at the start: the nudge only shifts it in
  time.
- An input nothing in the equations moves: BIOMD0000000265's r31switch
  and Flag, which only its events write. Nudged, its answer moved 0.37;
  without them, 3.4e-5.
- A species an event trigger reads, carried across the threshold at the
  start: BIOMD0000000285's PIdeath > 0 and p38death > 0, both true at
  t = 0, fire neither (initialValue true, as a Level 2 trigger is, which
  bngsim follows and the SBML test suite's 494 Level 3 twins of Level 2
  events all say), and the cell those events kill at once lives on.

held_at_start keeps a species whose derivative is exactly 0 at the
start, that a row of the Jacobian's pattern can move, that is not
fixed, and that no event trigger reads (through the functions it calls).
A model whose triggers read observables that are not plain species is
not nudged.

A stable-only sweep of the corpus (macOS, 1349 models) under the rule
moves 6 stable rows PASS -> DIFF, recorded here in place of the 14:
BIOMD0000000908, MODEL1012110001 (an unstable balance libRoadRunner
leaves by itself at rtol 1e-11), MODEL1108260014, MODEL1812040005,
MODEL2003180003, and BIOMD0000000739, whose nudged run makes 0.0149 of
PT at the nightly's atol where both engines at rtol 1e-11 make 0.
# Conflicts:
#	ci/test_floors.env
Between two nights, a change into the verdict the committed baseline has
re-recorded since the earlier night's commit was reviewed when it was
recorded. compare.sh fetches the baseline as of that night's commit and
verdicts.py diff reports such rows under changed instead of regressed.
Without it, the 6 stable rows this PR moves to DIFF alert on its own run
and would open the tracking issue on the first night after it merges.
A fix that regresses into a stale baseline still alerts.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant