Repository navigation
differential: seven reviewed PRs as one merge (#1316-#1320, #1322, #1323) - #1384
Merged
Merged
Conversation
Every path the differential suite compares is built from one loaded model, so a bug in the loader, the model build or the parser makes them agree on the wrong answer. Four checks transform the file instead, where the answer is known to follow, and run it again (_transforms.py; libsbml for SBML, the .net blocks rewritten as text): * reorder: species and reactions (and SBML's parameters and species references) in reverse. Unchanged by name. * rename: parameters, observables, functions and species (SBML: species, parameters, compartments) to fresh names. Unchanged by the map back. * split: each reaction into two at half the rate (a parameter rate through a new parameter, a function's through a new function, BNG's MM/Sat/Hill through their first argument). Unchanged. * rescale_time: every rate and rate rule times 3 and the window divided by 3, where the model reads no clock, delay, rateOf, event or reaction rate. x'(t) = x(3t). Each is judged with Base.compare, aligned by name; one that does not keep the answer for a model is a SKIP that says why. Base keeps its run's species names: a Result leaves out a parameter an event promotes. Over the corpus (5,412 rows): 4,413 PASS, 997 SKIP, 0 DIFF, 2 EXCEPTION, both reorder. BIOMD0000000908 blows up past t = 141 on a reordered sum in both bngsim and RoadRunner. MODEL1812040005 is a finding: with its species reversed bngsim lets P go below 0, where a*P^z is NaN, and fails at t = 0.106, where in the order written it holds P at 0; RoadRunner runs the reversed file to 3.6e-8 of bngsim's original answer. Controls: each transform keeps an autonomous SBML model's answer (it has a rate rule, so the rescale's is exercised), and done wrong on a .net and on that model it reads DIFF (values left in the old order, two names crossed, the whole rate on each half, the rates scaled and not the window). Each fix the first corpus run asked for has a control that fails without it: a rate rule reading a reaction rate refused (BIOMD0000000556 came out 9x), MM and a spaced expression in a .net, an id bngsim respells as _ant_<id>.
…er (#1306) The differential checks run fixed models, so they are oracles only over the expressions those models hold. differential_generate.py draws models instead, seeded, from the grammar bngsim accepts, runs each through check_all, and shrinks every DIFF or EXCEPTION to a ready reproducer. _generate.py draws a network of 2-4 species and 1-4 reactions whose laws are random trees over arithmetic, guarded powers and quotients, exp/log/ln/ log10/log2/sqrt, abs/floor/ceil/round/rint/trunc/sign, tanh/sin/cos/atan/erf, min/max/avg/clamp, if() on every comparison and on &&/||, the clock, and in SBML piecewise and rateOf of a driver species; renders it as a .net or as SBML; and shrinks a model by dropping reactions and replacing subexpressions by their children or constants while the same check finds the same thing. Laws stay finite by construction (logs and roots of 1 + x^2, bases of at least 1, guarded quotients), with squares, not abs(), as guards so the sensitivity checks keep the analytic path; each enters its reaction as k*g^2/(1 + g^2). Each seed runs in its own process under a wall cap and a step budget (20,000: a law that switches on the clock and on rateOf can crawl). A model whose base does not run is no finding. First run, seeds 0-199 in both formats (400 models, 430 s): 389 run, 3 no base, 8 timeouts, 2 findings, both grid and both genuine. net 72 (shrunk 20 -> 10, floor(2*O1) and round(O2)): at rtol 1e-8 161 output points read 4.9e-5 of the peak off the converged answer where 21 read 4.4e-8. net 174 (35 -> 13, trunc(O2) as O2 slides to 1): 8.4e-3 at 161 points, 3.1e-8 at 21; its rtol 1e-12 run fails (CV_CONV_FAILURE at t = 8.5) on 21 points and runs on 161. Tests: a seed is its model; every grammar entry, alone in a law with a negative first argument, loads and integrates finitely in each format it has; generated models stay finite; the shrinker cuts a model with a floor() to one reaction of size <= 3; a planted cc_vs_interp defect is found and shrunk to a one-reaction reproducer, and none is reported where every check passes. Mutations caught: a shrinker that cuts nothing, DIFF not counted as a finding, an unguarded log or sqrt.
…1306) The SBML half skipped the nine .net-only entries (rint, erf, avg...) with an undeclared reason, which the suite's skip audit fails on every shard: a .net-only entry is never drawn for SBML, so there is nothing of it to test there. 65 cases now, none skipped.
Every other differential check runs a fresh model, by design; callers reuse one, and "the solver leaves scratch state in the user's Model, or drops state it should carry" is a family of its own (#786; #748, #705, #693). Three checks, each against the base run with Base.compare: * restart: run_until to the middle sample and on to the end, on one Simulator (which carries the events' trigger state across the stop, #693); and, for a model without events, a fresh model given the middle's saved state. Joined, each is the base. * reuse: a second run() on one Simulator, and a first run on a second Simulator over one Model, each after the first is undone with snapshot()/restore() (species, parameters and event carry state; Model.reset() restores species only, so an event that writes a parameter would leave it written). * batch: run_batch on 4 threads against the same four points one at a time (the first sensitivity parameter at x1, 1.1, 0.9, 1.05), and the x1 point against the base. A point that does not integrate one at a time is the model's, not the batch's: SKIP, named. Base keeps its run's species names (the same lines as #1305's PR). Over the corpus (1,353 models each): restart 1,260 PASS, 92 SKIP, 1 EXCEPTION; reuse 1,260 PASS, 93 SKIP; batch 1,259 PASS, 94 SKIP. No DIFF. The exception, BIOMD0000000827, is no leak: stopped anywhere (t = 50, 100, 150) and continued, by run_until or by a fresh Simulator from the saved state (which equals run()'s), it fails at the next NaN in a rate law where the straight run, robust at 0.5x to 2x its tolerance, steps around it. Controls, each mutating the model where a leak would: a parameter doubled between run_until's legs, after set_state in the fresh restart, inside restore(), and for one threaded batch point; each reads DIFF, on a .net and on an SBML model with an event. A point failing both ways is SKIP; one failing only on threads is still a finding. Mutations caught: restart without its run_until comparison, reuse without restore, batch against itself, a combiner that passes everything, the serial failure uncaught.
# Conflicts: # python/tests/test_differential_shapes.py
# Conflicts: # python/tests/test_differential_shapes.py
… models (#1309) A DIFF says two paths disagree, not which is wrong, and a small own drift does not make an answer uncontested (#1304: KLU on BIOMD0000000338 moved 1e-7 at a tenfold tighter tolerance while dense and libRoadRunner sat 3.5% away). Attribution: every DIFF whose comparison kept its two trajectories (Verdict now carries them, never written out) is put to third arms, run once per model: the base's own route at 100x tighter tolerances, the base on the other linear solver (dense for a KLU base, KLU otherwise), and libRoadRunner for SBML (concentrations as [id], as rr_parity selects). Each arm sides with the comparison's reference, its path, both or neither; the row's extra["attribution"] says which, and its verdict is the one side every arm took, or split. Branch stability: a stable check reruns the base at n_points -1 and +1 (read at the last point, which every grid shares), twice the points (read at the base's), and rtol x0.5 and x2. A jump past the bound is a DIFF marked contested. The report's meta lists every contested model: a stable DIFF, or a DIFF whose arms split (D.contested). #1304's two models read contested with each arm's side named: 338 split (KLU tighter with KLU; dense and libRoadRunner with dense), 339 split (libRoadRunner gives up there and is left out). Over the corpus with every check: 13 contested; of 7 attributed DIFFs, 4 split and 3 with the reference; stable reads 10 DIFF, among them BIOMD0000000620 and 621, whose answer at t = 10 depends on the grid (Anakinrasc 14,222 at 101 points, the largest species 107 at 100 or 102: an event that fires on a sample). Controls: arms that split mark the model contested and arms that agree attribute the DIFF; a planted cc_vs_interp DIFF is attributed by real arms to the reference; an answer that jumps at rtol x2, or at n_points+1, is contested; #1304's model reads split (skip-if-absent). Mutations caught: attribution always the reference, none in check_all, stable without either jitter, contested ignoring a split, no other-solver arm.
"grammar_net_if<.net" is no file name on Windows (Errno 22), so the eight comparison entries failed the MIR codegen windows-latest shard.
An SSA job runs the same seeds once with the default Simulator (which compiles the propensities of an eligible model) and once with codegen=False. On an all-elementary model the two must agree draw for draw; otherwise the ensembles must agree at ensemble_k_for(n_rep). A second check holds ssa_diagnostics["propensity_backend"] to what was asked for. differential_run.py reads rr_parity/ssa_jobs.json under --ssa-jobs, capped at --max-reps replicates.
…ng (#1310) Most of rr_ssa's 268 models run interpreted whatever is asked, so both SSA checks are decided by one replicate of each. On the overloaded first corpus run 31 of the first 98 models timed out at 600 s with full ensembles both ways; the nightly's job has a 180-minute cap.
steady_vs_run and newton_vs_run compare each steady_state() method with a run to t=1e6 from the same state, judged only where the solve says it converged and the run's last quarter moves it by at most 1e3 rtol. Where they differ, a state a run leaves, one off the initial state's conserved totals, or an unstable root is a DIFF; another state a run rests at is SKIP, both named, for the baseline to hold.
… no finding (#1311) The corpus run's 2 DIFFs and 6 EXCEPTIONs were the reference's. Four models' runs to t=1e6 fail (CVODE) far past their own window: SKIP, no reference. BIOMD0000000518 starts on an unstable root (an eigenvalue of 5.6e-4): the solve stays on it, as the trajectory does in exact arithmetic, and the run leaves it on rounding alone.
An NF job is one of bng_nf's models from its reference archive: the BNG-XML BNG2.pl wrote and the run it was recorded for, so the nightly's differential job needs no BNG2.pl. nf_reject_vs_exact runs n_rep replicates on each engine at the same seeds and options and judges the observables' ensembles with differ.ensemble_verdict, escalating a DIFF as bng_parity's NF track does (_run_escalation), within a 300 s budget.
…#1309) The PR's nightly read 33 regressions, every one a TIMEOUT: the pool kills a model's worker at its 900 s cap, and every row of the model with it. BIOMD0000000810's checks take 8 s, but a jittered stable rerun ground on past the cap; MODEL1006230049, 0053 and 0077 took 716-751 s before stable reran them five times. check_all now takes a deadline, 0.9 of the cap: the existing checks run first, then the attribution's arms, then stable, each run bounded by what is left (run(timeout=)); none starts past it. A stable that does not finish is SKIP, said why, and the model keeps its other rows.
…fusal by name (#1312) Over bng_nf's 234 models the two engines' ensembles cost 9,793 CPU-s on an M-series machine (48 models, 92% of it), which the path-agreement job, already 52 of its 180 minutes, could not also hold. The arm runs in its own job, "Network-free engine agreement", under its own baseline (differential_nf); the path-agreement job is as it was. "RuleMonkey cannot honor this BNG XML" (ft_total_rate's TotalRate over a repeated component) is a refusal by name: UNSUPPORTED, not EXCEPTION.
…ty library (#1310) One replicate each way was still too much: 18 of the first 101 rr_ssa models took a single replicate past the 900 s cap. The Simulator decides before any run whether a model is eligible for compiled propensities, and builds no library where it is not. Without one there is nothing compiled to tell apart, and #1213's class (codegen=False ignored) needs a default that compiles: both checks are SKIP, with no replicate run.
restart, reuse and batch run after every check main runs, each run and run_batch with what is left of the model's cap; run_until takes no timeout, and a kill at the cap still leaves the model's other rows.
steady_vs_run and newton_vs_run run after every check main runs, the runs to equilibrium bounded by what is left of the cap. A reference run the cap stops is this check's TIMEOUT, not a missing reference; steady_state() takes no timeout, and a kill still leaves the other rows.
#1321 bounds every run by the model's cap, which replaces the deadline 6357188 threaded through stable and the arms alone. The order stays: main's checks, then the attribution's arms, then stable, each starting only with time left. A stable the cap stops is TIMEOUT, as any check's is under #1321, where it was SKIP.
… fix/1307-state-leaks
… fix/1309-attribution
… fix/1311-steady-vs-run
check_all keeps #1321's deadline and its verdicts-as-reached; an SSA job is routed to check_ssa inside it, so its verdicts are kept the same way.
…onent of a number alone (#1305) A model's checks share its cap (#1321), and every check added since runs after main's. The transforms ran in main's loop, ahead of locale: four more runs that could tip a slow model's locale row (MODEL1006230049, 0053, 0077 already take 689-894 s), which the baseline holds as PASS, to TIMEOUT. _subst matched names inside numeric literals: with a parameter named e1, 2e1*e1 became 2mt_p1*mt_p1, which does not load. The branch also merges lanl/main, which it conflicted with (check_all's deadline, #1321).
…runk; the shrinker takes a number for a named leaf (#1306) run() reached its status only with res None, which happens only when the queue was empty, so 'dead' was unreachable: a worker gone without a result (a segfault in bngsim) read 'timeout', at once. Now dead, with its exit code. The seed's cap covers the shrink (up to its budget of check runs), and the reproducer was written only after it: a seed killed mid-shrink left none. The model as found is written first, its note saying it is not shrunk. A parameter, species or time() leaf replaced by a number is no fewer nodes, so shrink's size test refused every such cut and a reproducer kept them. Fewer named leaves now breaks a tie in size. The branch also merges lanl/main, which it conflicted with (tests).
…review/differential-group-2
…ew/differential-group-2
…fferential-group-2
…ew/differential-group-2 # Conflicts: # parity_checks/differential/_differential.py # parity_checks/differential/differential_run.py # python/tests/test_differential_shapes.py
…differential-group-2 # Conflicts: # python/tests/test_differential_shapes.py
…ew/differential-group-2 # Conflicts: # parity_checks/differential/_differential.py # parity_checks/differential/differential_run.py # python/tests/test_differential_shapes.py
…ew/differential-group-2 # Conflicts: # parity_checks/differential/_differential.py # python/tests/test_differential_shapes.py
…view/differential-group-2 # Conflicts: # parity_checks/differential/_differential.py # parity_checks/differential/differential_run.py # python/tests/test_differential_shapes.py
…uests collect together Every leg collected 16878 on 6495801 (ubuntu, ubuntu-lapack and macos-14 alike), 516 above the floor of 16362, past the 500 of slack. Skips stay under their ceiling of 225 (217, 191, 179).
This was referenced Oct 29, 2026
Montana
pushed a commit
to Montana/bngsim
that referenced
this pull request
Oct 30, 2026
Run 40899940910 recorded reference-data-v3 at e425546, but its publish job was refused (HTTP 403) twice creating the release's tag at that commit. lanl#1384 had changed this workflow on main mid-run, and the job's token may not create a ref whose workflow files differ from the default branch's. reference-data-v2 published because nothing changed a workflow during its run. Tag the default branch's head instead. The notes and the manifest still name the recorded commit, and a replay reads only the assets and their sha256s.
wshlavacek
pushed a commit
that referenced
this pull request
Oct 31, 2026
…, #1371) From run 41076909610 on main de59f5b, after #1384: the twelve check kinds it landed get their 14,066 rows, so a regression in any of them alerts. No regressions against the old baseline; three klu_vs_dense rows #1295 fixed are recorded as PASS (#1371). Each new failing row's reason is in the pull request.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands seven of the eight open differential pull requests by Michael Mendy (Montana) in one merge, each at its current head. The eighth, #1315, merged on its own first (09a045e): it carries the baseline, and its CI was already running.
Why one PR. The eight touch the same three files (
_differential.py,differential_run.py,test_differential_shapes.py), so each merge makes the next one conflict. Landing the seven one at a time would take seven CI cycles and seven nightly-parity runs (1¼ to 3 hours each), one after another. Here they get one of each.How. The branch holds merge commits plus one formatting commit: main, then each PR's head (#1315's too, now on main), then main again, then
ruff formatadding the blank lines where merged sections meet, then main once more (only #1315's merge, which leaves the tree unchanged). Every PR's commits land unchanged, and each PR's head is an ancestor of this branch, so GitHub should mark each one merged. Any it does not mark gets closed with a link here.Review. The eight were reviewed as a group on the combined tree. Each finding was fixed in its own PR before this branch was cut:
dead, nottimeout; a finding is written before it is shrunk; the shrinker takes a number for a named leaf.stableis SKIP where nothing could jump.Model.jacobian()refuses is SKIP, not EXCEPTION.--max-reps 0.differential_nf.json) is committed, from that PR's own nightly (run 40539506449).Conflicts. Code: resolved on the review branch by keeping each PR's definitions verbatim. None of the seven touches
parity_checks/nightly/baselines/differential.json; it is main's, from #1315. #1323 addsdifferential_nf.json.Baseline. Rows from the new check kinds in #1316–#1323 are not in the baseline yet. A case missing from the baseline is reported as new, never alerted. After this merges,
differential.jsongets re-recorded once from main's first scheduled nightly.Testing. These were run locally on macOS arm64, with the extension built from this branch. Main now has #1354 and #1295, which none of the eight PRs was tested with.
test_cache_cli.py::TestPrune::test_an_entry_is_never_selected_twice. It passes alone and with its whole file under xdist (3 of 3 each), and this branch touches no cache code.test_differential_shapes.pyandtest_expression_backend_equivalence.py: 909 passed, 1 skipped. The skip is the BIOMD0000000338 corpus test, which passes with the corpus model present.Closes #1305
Closes #1306
Closes #1307
Closes #1309
Closes #1310
Closes #1311
Closes #1312