Skip to content

nightly: re-record differential.json so #1384's twelve new check kinds can alert #1395

Closed

Description

@wshlavacek

#1384 (merged as 29bfc85) landed seven differential pull requests. Five of them add check kinds that parity_checks/nightly/baselines/differential.json does not have yet. A row missing from the baseline is reported as new, and a new row never alerts. Until the baseline records these rows, a regression in any of the new checks goes unreported.

On #1384's own nightly (run 40904665252) the new kinds add 14,066 rows: 28,949 in all, against the baseline's 14,883, with none missing.

PR issue kinds rows each
#1316 #1305 rename, reorder, rescale_time, split 1,353
#1318 #1307 restart, reuse, batch 1,353
#1319 #1309 stable 1,353
#1320 #1311 steady_vs_run, newton_vs_run 1,353
#1322 #1310 ssa_cc_vs_interp, ssa_backend_as_asked 268

#1317's generated models are not part of the nightly. #1323's NF arm has its own baseline, differential_nf.json, committed in that PR.

Gated on: a scheduled nightly on main at or after 29bfc85. The next one is 2026-10-30 08:17 UTC. A workflow_dispatch of nightly-parity on main gives one sooner.

To do:

  1. Take that run's nightly-differential artifact (verdicts.json) and diff it against the committed baseline with parity_checks/nightly/verdicts.py. Expect the twelve kinds as new, and no alert.
  2. Before recording them, look at the new kinds' DIFF and EXCEPTION rows. Recording one makes it a known failure, so each needs a reason in the pull request, or an issue of its own. On run 40904665252 there were 17:
    • reorder: BIOMD0000000908 DIFF, 1.09e+134 of the trajectory's peak with species, reactions and parameters reversed. MODEL1812040005 EXCEPTION (CV_REPTD_RHSFUNC_ERR at t = 1).
    • restart: BIOMD0000000827 EXCEPTION (CV_CONV_FAILURE at t = 106).
    • stable: 10 DIFF, each "contested" by differential: attribute a disagreement with a third arm, and flag answers that flip with the grid or tolerance #1309's design (BIOMD0000000620, 621, 755, 1043, 1060; MODEL1109160000, MODEL1911100005, MODEL1912090001, MODEL2310250001, MODEL7908934508). 4 EXCEPTION (BIOMD0000000336, 763, 810; MODEL1006230034).
  3. Copy verdicts.json over the baseline in a re-baseline pull request. Baseline-only pull requests skip the nightly sweep by the paths filter.

Overlap with #1371. #1371 re-records the same file for the four klu_vs_dense rows #1295 fixed, and is gated on the same kind of run. One re-baseline pull request from one nightly can close both.

Done when the re-baseline pull request is merged and the next nightly reads the twelve kinds as unchanged rather than new.

Activity

  1. added
    gatedBlocked on an external event; no action possible here until it lands
    testingTest suites, parity checks, nightlies and benchmarks: how bngsim is checked, not what it computes
    on Oct 29, 2026
  2. added a commit that references this issue on Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    gatedBlocked on an external event; no action possible here until it landstestingTest suites, parity checks, nightlies and benchmarks: how bngsim is checked, not what it computes

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions