Repository navigation
differential: the stable check also reruns from a nudged start (BIOMD0000000908) - #1427
Open
wshlavacek wants to merge 6 commits into
Open
wshlavacek wants to merge 6 commits into
wshlavacek wants to merge 6 commits into
Conversation
BIOMD0000000908's `reorder` row reads DIFF, 1.09e+134 of the peak, with no explanation in the baseline. It is not a bngsim defect. The model's sunitinib clearance law is -n*S on a reaction that consumes S, so S' = +0.277*S, and with no dose its answer S = 0 is an unstable one. The file's species order keeps S at exactly 0; reversed, the rounding of the linear solves leaves 3.6e-19 at t = 10, which grows until the run breaks near t = 152. `stable` passed it, since no jittered grid or tolerance moves S off 0. check_stable now also runs the model from a start moved up by START_NUDGE = 1e-3 of what a step may err by, rtol*|x| + atol, a start the solver cannot tell from the base's. A departure past the bound marks the model contested, as a jittered grid or tolerance does, and so does a run that fails from it. Only a departure the nudge grew into counts: one already past the bound at the first row after the start is the model reading its start exactly, `if(Bobs==0, kb, 0)` on a B its law holds at 0 (#1228), which no rounding breaks and which the equality-dwell shape keeps passing. 908 now reads contested (2.7e+14 from the nudged start), so its reorder DIFF is a fragile model's, not an unexplained one. run() takes the nudge; the tests plant an unstable zero beside a decaying twin, check the nudge's size, and a run that fails from it.
A stable-only sweep of the differential corpus on this branch (macOS, 1349 models), extracted with verdicts.py: 14 models whose answer leaves its bound from a start nudged by 1e-3 of a step's allowance, BIOMD0000000908 among them (2.7e+14), and MODEL1812040005 (#1340) and BIOMD0000000457, whose runs fail from it. Each goes PASS -> DIFF, contested; no other row changes. This PR's own Linux run compares against them.
…start Checking the 14 models the first nudge flagged against libRoadRunner from the same start: 5 are the models' own (both engines move the same way, or fail, at rtol 1e-11 too), and the other 9 were the nudge moving what no rounding moves. - A species already moving at the start: the nudge only shifts it in time. - An input nothing in the equations moves: BIOMD0000000265's r31switch and Flag, which only its events write. Nudged, its answer moved 0.37; without them, 3.4e-5. - A species an event trigger reads, carried across the threshold at the start: BIOMD0000000285's PIdeath > 0 and p38death > 0, both true at t = 0, fire neither (initialValue true, as a Level 2 trigger is, which bngsim follows and the SBML test suite's 494 Level 3 twins of Level 2 events all say), and the cell those events kill at once lives on. held_at_start keeps a species whose derivative is exactly 0 at the start, that a row of the Jacobian's pattern can move, that is not fixed, and that no event trigger reads (through the functions it calls). A model whose triggers read observables that are not plain species is not nudged. A stable-only sweep of the corpus (macOS, 1349 models) under the rule moves 6 stable rows PASS -> DIFF, recorded here in place of the 14: BIOMD0000000908, MODEL1012110001 (an unstable balance libRoadRunner leaves by itself at rtol 1e-11), MODEL1108260014, MODEL1812040005, MODEL2003180003, and BIOMD0000000739, whose nudged run makes 0.0149 of PT at the nightly's atol where both engines at rtol 1e-11 make 0.
# Conflicts: # ci/test_floors.env
Between two nights, a change into the verdict the committed baseline has re-recorded since the earlier night's commit was reviewed when it was recorded. compare.sh fetches the baseline as of that night's commit and verdicts.py diff reports such rows under changed instead of regressed. Without it, the 6 stable rows this PR moves to DIFF alert on its own run and would open the tracking issue on the first night after it merges. A fix that regresses into a stale baseline still alerts.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Explains the loose end #1423 left: BIOMD0000000908's
reorderrow, recorded as a DIFF of 1.09e+134 of the peak with no reason or issue.What 908 is
Not a bngsim defect. The model's clearance of sunitinib is the reaction
Sunitinib_not_working, which consumes S at the ratecompartment * (-n * S). A negative rate on a reaction that consumes S makes S' = +0.277·S. With no dose (vs = 0), the model's answer is S = 0 for all time, and an unstable one.On main 15ba102, at rtol 1e-6 over 0 to 300:
Reversing the species reorders the linear solves, and their rounding leaves S a speck off 0. libRoadRunner from the same nudged start moves 2.6e14, as bngsim does.
stablepassed the model: no jittered grid or tolerance moves S off 0.The change (
parity_checks/differential/_differential.py)check_stablereruns the model once more, from a start nudged bySTART_NUDGE= 1e-3 of what a step may err by,rtol·|x| + atol. A departure past the bound marks the model contested, as a jittered grid or tolerance does; so does a run that fails from the nudged start.held_at_start): a derivative of exactly 0 there, a row of the Jacobian's pattern that can move it, not fixed, and not read by any event trigger. A value held exactly is what rounding alone can break. The others are not a rounding's to move, and the first version of this PR, which nudged every species, flagged 9 models for them:r31switchandFlag, which only its events write (0.37 nudged; 3.4e-5 without them);PIdeath > 0andp38death > 0, both true at t = 0, fire neither (a Level 2 trigger isinitialValue = true, which bngsim follows and all 494 Level 3 twins of Level 2 events in the SBML test suite say), so the cell they kill at once lives on.equality-dwellshape'sif(Bobs==0, kb, 0)does (A rate law that branches on an equality the state sits on (if(B==0, ...) with B held at 0) runs on the analytic sensitivity path and returns 0 for the columns that move the switch #1228).What it flags
A
stable-only sweep of the corpus on this branch (macOS, 1,349 models, 600 s a model), extracted withverdicts.py, moves 6stablerows PASS → DIFF, recorded inbaselines/differential.json(those 6 lines and the tally). Each was checked against libRoadRunner from the same nudged start, with a fresh RoadRunner per run:parameter_3; both engines agree to 9 digits at every toleranceBIOMD0000000739 and BIOMD0000000457 (which the first version flagged) are the same thing, and it is not this PR's to fix. At the SED-ML tolerances (rtol 1e-6, atol 1e-9) bngsim's answer for their small species is off by much more than the tolerance: 457's species_42, about 1.6e-7, is 13% off at t = 5 against a run at rtol 1e-10, atol 1e-16, where bngsim and RoadRunner agree to five digits. libRoadRunner's answer at the same nominal tolerances is right because its
absolute_toleranceis multiplied by each species' value; bngsim's is absolute. This is #1420's question, for an explicit scalaratol(here the SED-ML file's), whichAtolScaleWarningdoes not second-guess. The RoadRunner parity suite forces rtol 1e-9, atol 1e-12 on both engines, so it does not see it.The sweep's other differences from the baseline (TIMEOUT → PASS and the like) are its 600 s cap against CI's, and are left as recorded. This PR's own Linux run compares against these rows.
Also
ci/test_floors.env: the full suite's collected floor to 17360. All three legs collected 17370 here, 502 above the floor, past its slack of 500.Tests (
python/tests/test_differential_shapes.py)equality-dwell'sstableamong them.