Skip to content

nightly: re-record differential.json from the 2026-10-30 nightly (#1395, #1371) - #1423

Merged
wshlavacek merged 1 commit into
lanl:mainfrom
Montana:nightly/1395-rebaseline-differential
Oct 31, 2026
Merged

wshlavacek merged 1 commit into
lanl:mainfrom
Montana:nightly/1395-rebaseline-differential

Conversation

@Montana

@Montana Montana commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

Fixes #1395. Also fixes #1371: the klu_vs_dense rows #1295 fixed are recorded here from a CI run, so this supersedes #1392, which patched them by hand.

What it records

parity_checks/nightly/baselines/differential.json is replaced by verdicts.json from the scheduled nightly on main, run 41076909610 (2026-10-30, de59f5b, after 29bfc85). All 14 jobs succeeded. Its own comparison against the committed baseline reads no regressions:

outcome baseline tonight
DIFF 10 19
EXCEPTION 8 12
PASS 12,505 22,999
SKIP 2,285 5,076
TIMEOUT 22 72
UNSUPPORTED 53 771
cases 14,883 28,949

The diff (row changes, abridged)

The file is 14,128 lines longer. Every row that existed before keeps its outcome but these six:

 # old check kinds whose outcome changes
-MODEL1912090001|klu_vs_dense     DIFF
+MODEL1912090001|klu_vs_dense     PASS        # #1295 (#1371)
-MODEL2002030001|klu_vs_dense     EXCEPTION
+MODEL2002030001|klu_vs_dense     PASS        # #1295 (#1371)
-MODEL7743528808|klu_vs_dense     DIFF
+MODEL7743528808|klu_vs_dense     PASS        # #1295 (#1371)
-BIOMD0000000827|klu_vs_dense     EXCEPTION
+BIOMD0000000827|klu_vs_dense     SKIP        # no reference: the base moves by 2.5e-01 at a tenfold tighter tolerance
-MODEL1006230053|jacobian_values  TIMEOUT
+MODEL1006230053|jacobian_values  SKIP        # no analytic Jacobian: Model.jacobian differences the RHS
-MODEL1006230077|jacobian_values  TIMEOUT
+MODEL1006230077|jacobian_values  SKIP        # the same
 # plus 14,066 new rows (the twelve kinds); per-model timing and work of the rest is refreshed

The new kinds' failing rows, each with its reason

Recording a row makes it a known failure, so each one is accounted for here. There are 17 DIFF or EXCEPTION rows, the same 17 the issue listed from #1384's own nightly, plus 57 TIMEOUTs.

rows reason
stable DIFF ×10 (BIOMD0000000620, 621, 755, 1043, 1060; MODEL1109160000, MODEL1911100005, MODEL1912090001, MODEL2310250001, MODEL7908934508) Contested, by #1309's design: the answer jumps past its bound under the check's own perturbations (rtol ×2 or ×0.5, n_points ±1). The row reports the model's sensitivity to its tolerance, not a defect.
stable EXCEPTION ×4 (BIOMD0000000336, 763, 810; MODEL1006230034) Under a perturbed tolerance or grid, the run fails with a non-finite right-hand side (CV_CONV_FAILURE, CV_RHSFUNC_FAIL, CV_REPTD_RHSFUNC_ERR). 810 and MODEL1006230034 already fail old kinds in the baseline (810: ajac_vs_fdjac, grid, klu_vs_dense; MODEL1006230034: klu_vs_dense).
restart EXCEPTION, BIOMD0000000827 CV_CONV_FAILURE at t = 106, a non-finite right-hand side. The same model's klu_vs_dense row has no reference, because the base itself moves by 2.5e-1 at a tighter tolerance.
reorder EXCEPTION, MODEL1812040005 Fragility at the edge of the model's domain, measured: a_hat = a*P^0.5, and P reaches exactly 0.0 in the original order. Reversing the species re-rounds P just below 0, and a_hat is NaN at t = 0.106 (CV_REPTD_RHSFUNC_ERR). The original passes by its rounding.
reorder DIFF, BIOMD0000000908, 1.09e+134 Filed as its own issue (see below). Reversing the species alone reproduces it on macOS (CV_CONV_FAILURE at t = 149.7); reversing the reactions, the parameters, the rules or the reactant and product lists is bit-identical.
TIMEOUT ×57 Six models exhaust their per-model wall cap ("not finished within the model's cap: the model's cap is spent"): BIOMD0000000385 (10), MODEL1006230077 (10), MODEL1006230090 (10), MODEL1006230049 (9), MODEL1006230053 (9), BIOMD0000000876 (4). Four of them (385, 053, 077, 090) already time out on old kinds in the baseline. Against the baseline, PASS → TIMEOUT still alerts.

Checks

  • Pre-commit: pre-commit run --all-files is clean.
  • Changelog: ci/changelog.py required passes; a baseline-only change needs no fragment.
  • Sweeps: test_pr_legs.py passes: a baseline-only PR re-runs no sweeps, by the paths filter.
  • Parity tests: 1,015 passed. The 2 failures (test_bngsim_provenance and the reference-archive replay) also fail on main here: their subprocess can't import bngsim without my local import shim.

Not done here

…l#1395, lanl#1371)

From run 41076909610 on main de59f5b, after lanl#1384: the twelve check
kinds it landed get their 14,066 rows, so a regression in any of them
alerts. No regressions against the old baseline; three klu_vs_dense
rows lanl#1295 fixed are recorded as PASS (lanl#1371). Each new failing row's
reason is in the pull request.
@wshlavacek
wshlavacek merged commit cec9b79 into lanl:main Oct 31, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants