Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,17 @@ if(BLASTGPU_BUILD_SOLVER)
endif()
if(BLASTGPU_BUILD_TESTS AND BLASTGPU_GPU_BACKEND STREQUAL "NONE")
find_package(Python3 REQUIRED COMPONENTS Interpreter)
add_executable(test_fluxregister tests/test_fluxregister.cpp)
target_link_libraries(test_fluxregister PRIVATE AMReX::amrex)
add_test(NAME fluxregister COMMAND test_fluxregister)
add_test(NAME m1b-p1-ml-u0-smoke COMMAND ${Python3_EXECUTABLE}
${CMAKE_SOURCE_DIR}/tests/m1b_p0_check.py --uniform
$<TARGET_FILE:blastgpu> ${CMAKE_SOURCE_DIR}/inputs/m1b_p1_ml_u0.inp
"amr.n_cell=8 8 8" amr.max_grid_size=8 max_step=2 stop_time=1)
add_test(NAME m1b-p1-ml-sx-smoke COMMAND ${Python3_EXECUTABLE}
${CMAKE_SOURCE_DIR}/tests/m1b_p0_check.py
$<TARGET_FILE:blastgpu> ${CMAKE_SOURCE_DIR}/inputs/m1b_p1_ml_sx.inp
"amr.n_cell=32 8 8" amr.max_grid_size=32 max_step=2 stop_time=1)
add_test(NAME m1b-u0-smoke COMMAND ${Python3_EXECUTABLE}
${CMAKE_SOURCE_DIR}/tests/m1b_p0_check.py --uniform
$<TARGET_FILE:blastgpu> ${CMAKE_SOURCE_DIR}/inputs/m1b_p0_u0.inp
Expand Down
23 changes: 23 additions & 0 deletions docs/perf/m1b-p1-amrex-api-audit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# M1b-P1 local AMReX FluxRegister API audit

Authority: AMReX submodule `2cf4fbcde02a700e01c62bebb4a730aeeb66c661`.

| Operation | Local source | Selected contract |
|---|---|---|
| constructor/define | `Src/AmrCore/AMReX_FluxRegister.cpp:11-80` | Fine BoxArray and DistributionMapping, ratio 2, fine level 1, six register components. |
| `CrseInit` | `AMReX_FluxRegister.cpp:160-281` | Source/destination ranges are independently checked. `mult` and supplied area are multiplied once. COPY overwrites the oriented boundary register; ADD accumulates. P1 uses negative coarse scale. |
| `FineAdd` | `AMReX_FluxRegister.cpp:363-496` | Fine boundary fluxes are coarsened/summed by the refinement ratio and added with the supplied multiplier. P1 uses positive fine scale. |
| `Reflux` | `AMReX_FluxRegister.cpp:527-647` | Register components are copied to oriented coarse faces and applied to destination cells divided by coarse volume. P1 uses `scale=1`, register source 0, state destination 1, count 6. |
| example signs | `Tests/Amr/Advection_AmrCore/Source/AdvancePhiAtLevel.cpp:372-378` | Coarse `CrseInit` uses -1 and fine `FineAdd` uses +1. |
| subcycling order | `Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp:680-706` | Fine recursive advances complete, then reflux, then average-down. |

The overload without an explicit area creates a unit-area MultiFab internally;
therefore P1 passes the complete `dt*area*stage_weight` through `mult`. This is
an AMReX-internal logical temporary, not solver-owned persistent storage, and
must be checked in the steady underlying-allocation audit. Periodic handling
is supplied to register boundary copies during reflux. GPU paths use AMReX
launch machinery and contain stream synchronization in several internal copy
and reflux paths; these are measured, not bypassed.

The focused manufactured CPU test is the executable authority for the exact
sign, scaling, component-offset, and corrected-cell behavior used by P1.
10 changes: 10 additions & 0 deletions docs/perf/m1b-p1-authorization.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# M1b-P1 authorization

Authority: `BLASTGPU-M1B-P1-OWNER-AUTHORIZATION-TWO-LEVEL-MULTILEVEL-HYDRO-SSP-RK2-SUBCYCLING-SIX-COMPONENT-CONSERVATIVE-REFLUX-STATIC-REFINEMENT-AND-AMR-EFFICIENCY-BASELINE-2026-08-22`.

P1 implements at most two hydro levels, ratio-two subcycling, time-interpolated
coarse/fine fills, six-component conservative reflux, separate alpha
coarse/fine treatment, static sphere/box tagging, composite validation, and a
matched uniform-fine/AMR baseline. C2, dynamic tagging, remap, gauges, EB,
MPI, reactions, manual streams, fusion, arena/grid tuning, blastCFD work, and
cross-solver claims remain excluded.
4 changes: 4 additions & 0 deletions docs/perf/m1b-p1-build-manifest.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
build source compiler configuration architecture
cpu 318f485 GCC 13.3 Release C++20 MPI=OFF EB=OFF OpenMP=ON; sha256=61b208d630280cdf2a23f420d57ceb5d8abe6c1aa2bdf487d32d6c3a4453506c CPU
cuda-opt 318f485 NVCC 12.6.85 Release C++20 MPI=OFF EB=OFF fast-math/maxreg unchanged; sha256=c4e2bd9695ae0da691c56ac6861147f1251e12f4a0c237bd702562d6ef5ffdb8 CMAKE=75 AMReX=7.5 sm_75
control-cuda 9817caa NVCC 12.6.85 Release C++20 MPI=OFF EB=OFF fast-math/maxreg unchanged CMAKE=75 AMReX=7.5 sm_75
8 changes: 8 additions & 0 deletions docs/perf/m1b-p1-case-manifest.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
case base_grid levels ref_ratio coarse_steps fine_substeps output
ML-U0 32x32x32 2 2 100 200 final validation only
ML-SX 128x8x8 2 2 269 538 validation only
ML-AC 64x8x8 2 2 100 200 validation only
A64X2-R0 64x64x64 2 2 45 90 disabled in timing
U128-R0 128x128x128 1 - 90 - disabled in timing
A96X2-R0 96x96x96 2 2 68 136 disabled in timing
U192-R0 192x192x192 1 - 135 - disabled in timing
4 changes: 4 additions & 0 deletions docs/perf/m1b-p1-conservation-results.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
case mode arho1_initial arho1_final rhoE_initial rhoE_final result
ML-SX reflux_on 0.002197265603 0.002197265603 0.005371093768 0.005371093768 PASS
ML-SX reflux_off 0.002197265603 0.002197920663 0.005371093768 0.005373901780 DIAGNOSTIC_MISMATCH
ML-R0 reflux_on 1.224999988 1.224999988 254368.6704 254368.6704 PASS
29 changes: 29 additions & 0 deletions docs/perf/m1b-p1-correctness-results.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# M1b-P1 correctness results

All seven CPU CTests pass: EOS, M0 hydro, manufactured FluxRegister,
two P1 smoke cases, and the two P0 smoke cases. Canonical M1a remains valid on
three levels and its seven CPU/CUDA plotfile files are byte-identical.

- ML-U0: 100 coarse steps and 200 fine substeps; alpha, momentum, and energy
spreads are exactly zero, the coarse partial-density spread is one ulp
(`2.22e-16`, below the frozen `5e-15` envelope), counters are zero, and
composite conservative totals are unchanged.
- ML-U0-REGRID: ten coarse steps with regrid interval two preserve the uniform
state and hierarchy with zero counters and no stale register state.
- ML-SX: 269 coarse steps to `t=0.2`; CPU and CUDA complete with zero counters.
Reflux-on preserves displayed phase masses and energy; reflux-off produces
clear drift.
- ML-AC: 100 coarse steps; alpha remains in `[0.2,0.8]`, pressure remains
constant within the frozen envelope, phase masses are unchanged, and no
ordinary alpha register call exists.
- ML-R0: base 64/refined-by-2 reaches `t=2e-4` in 45 coarse steps. Phase
masses and energy are unchanged at reported precision, momentum is at
roundoff, counters are zero, and axis symmetry is approximately `1.6e-15`.

Composite AMR versus uniform-fine normalized L1 errors are `1.51e-6` (density),
`2.13e-6` (velocity), `2.18e-6` (pressure), and `2.33e-6` (energy) at
128-equivalent. At 192-equivalent they are `2.70e-7`, `3.91e-7`, `4.89e-7`,
and `5.00e-7`. All are well inside the committed tolerance authority.

Final Compute Sanitizer results: memcheck 0 errors, synccheck 0 errors,
initcheck 0 errors, and racecheck 0 hazards/errors/warnings.
23 changes: 23 additions & 0 deletions docs/perf/m1b-p1-implementation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# M1b-P1 implementation

P1 extends the P0 face-flux architecture to two hydro levels with refinement
ratio two. Each level owns persistent border, x/y/z flux, counter, and reflux
correction storage. Fine advances use two substeps per coarse step. Stage
fills request the fine stage time from the existing coarse old/new bracket,
so `FillPatchTwoLevels` performs time interpolation without retaining a stale
coarse stage-one state.

Both SSP-RK2 face fields are accumulated with weight one half. Register
components 0--5 map to state components 1--6; alpha and contact speed are
excluded. Reflux precedes seven-component average-down. A persistent
correction MultiFab distinguishes interface corrections from roundoff before
updating the coarse state. The same roundoff distinction in the cell update
preserves uniform free streams across interpolation and summation boundaries.

Static sphere tagging remains the default. A deterministic static box option
supports planar interface tests. Hydro mode asserts at most two levels;
identity mode and the canonical M1a input remain unchanged.

Physical outflow ghosts use an explicit first-order clamped interior copy.
This avoids concurrent corner reads from uninitialized ghost cells and is
validated by Compute Sanitizer initcheck.
8 changes: 8 additions & 0 deletions docs/perf/m1b-p1-kernel-results.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
family instances_or_share interpretation
directional_flux 80.8_percent_kernel_time dominant hydro work
cell_update 2.8_percent_kernel_time combined seven-component update
coarse_fine_interp about_10_percent_kernel_time FillPatchTwoLevels interpolation
CFL_reduction 2.5_percent_kernel_time one host-visible dt selection per level advance
FineAdd 0.9_percent_kernel_time six-component fine register accumulation
Reflux 0.5_percent_kernel_time six-component coarse correction
steady_cudaMalloc_cudaFree 0 PASS
9 changes: 9 additions & 0 deletions docs/perf/m1b-p1-memory-results.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
case arena_reserved_mb arena_max_used_mb matched_memory_efficiency
U128 5823 933 1.0
A64X2 5823 333 2.8018
U192 5823 3131 1.0
A96X2 5823 960 3.2615
P0-control-128 5823 837 1.0
P1-single-level-128 5823 837 1.0
P0-control-192 5823 2807 1.0
P1-single-level-192 5823 2807 1.0
9 changes: 9 additions & 0 deletions docs/perf/m1b-p1-numerical-results.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
pair field normalized_L1 normalized_L2 normalized_Linf
A64X2_vs_U128 rho 1.506302e-6 3.300128e-6 1.549482e-5
A64X2_vs_U128 u 2.134036e-6 7.085275e-6 5.791795e-5
A64X2_vs_U128 p 2.177851e-6 5.263779e-6 3.006390e-5
A64X2_vs_U128 rhoE 2.326179e-6 5.544679e-6 3.048631e-5
A96X2_vs_U192 rho 2.701837e-7 6.206293e-7 3.989132e-6
A96X2_vs_U192 u 3.908286e-7 1.507684e-6 1.815454e-5
A96X2_vs_U192 p 4.894288e-7 1.219134e-6 7.635313e-6
A96X2_vs_U192 rhoE 5.000204e-7 1.258989e-6 8.323991e-6
21 changes: 21 additions & 0 deletions docs/perf/m1b-p1-performance-results.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# M1b-P1 AMR efficiency baseline

All authoritative measurements use the committed tolerance authority, the
same optimized SM 7.5 executable, no output, and seven fresh processes per
case. The refinement region was frozen before timing.

| Pair | Uniform median (s) | AMR median (s) | Runtime efficiency | Uniform used MB | AMR used MB | Memory efficiency | CV uniform / AMR |
|---|---:|---:|---:|---:|---:|---:|---:|
| U128-R0 / A64X2-R0 | 28.0674 | 7.84385 | 3.578x | 933 | 333 | 2.802x | 0.650% / 1.816% |
| U192-R0 / A96X2-R0 | 142.284 | 31.5994 | 4.503x | 3131 | 960 | 3.261x | 0.183% / 0.417% |

The default arena reservation is 5823 MB in every case and is not presented
as working storage. Maximum-used values establish the memory ratio. Both
pairs retain matched-profile accuracy and therefore meet the package's
transformative runtime and memory categories.

The final single-level guardrail uses five alternating P0-control/P1-candidate
pairs. At 128 cubed the medians are 31.0393/31.2944 s (candidate +0.822%);
at 192 cubed they are 104.923/104.995 s (+0.068%). Arena maximum used is
identical at 837 MB and 2807 MB. Thus the max-level-zero path passes the 1%
runtime and memory gates.
19 changes: 19 additions & 0 deletions docs/perf/m1b-p1-profile-summary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# M1b-P1 Nsight Systems summary

The authoritative steady-window report is external at
`profiles/a64x2-final-steady-window.nsys-rep`, SHA-256
`36012b4145a4630dbe9fe450de9da08b6251b9e6d829954f0d811e27818c098f`.
It captures a three-second output-free static-AMR interval.

Directional Hydro3D flux kernels account for approximately 80.8% of kernel
time; updates account for 2.8%, coarse/fine interpolation kernels about 10%,
CFL reductions 2.5%, FineAdd 0.9%, and Reflux 0.5%. The trace contains no
steady `cudaMalloc` or `cudaFree`. `cudaStreamSynchronize` has high inclusive
time because it waits for GPU work, but its median host call is 0.86 microsecond
and the GPU remains flux-kernel dominated. The earlier C1 experiment already
removed a proven iterator synchronization family without large-grid benefit.
There is no measured >=5% batching opportunity, so C2 is not recommended.

Atomics remain limited to diagnostics/reductions/framework internals; no
face-to-cell residual atomic exists. Reflux uses the original face arrays and
does not introduce a second flux path.
8 changes: 8 additions & 0 deletions docs/perf/m1b-p1-raw-artifact-manifest.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
artifact location sha256_or_identity
evidence_root /home/tim/Downloads/blastgpu-perf-runs/m1b-p1-20260822T192505Z external
steady_nsys profiles/a64x2-final-steady-window.nsys-rep 36012b4145a4630dbe9fe450de9da08b6251b9e6d829954f0d811e27818c098f
tolerances docs/perf/m1b-p1-tolerances.json f0ae174788e371c00c4d783721ce5c03f5227e97c62c69e3e0832f9c1a7365a9
U128_profile references/profiles/u128.tsv 00726a6f5964e6ed2e1088bf96c961c76fad78954cfad1cd8a11ecf8e27bad05
A64X2_profile references/profiles/a64x2.tsv 20a3f794a6640033ad2a3357c82f8fe9a332393ed8e9286e40f78f58d6109f0f
U192_profile references/profiles/u192.tsv d50b96ad4fa9c3fb1f363fa58203dab3267fb44c545a00775d33053322b912b4
A96X2_profile references/profiles/a96x2.tsv fdfbf3088975fdfa775293275b1ecbd7009029b0ce5ce84335204d516cc840b8
79 changes: 79 additions & 0 deletions docs/perf/m1b-p1-reflux-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# M1b-P1 reflux design

This document freezes the two-level reflux contract before solver integration.

## Component classification

The cell state is ordered as `alpha1, arho1, arho2, rhou, rhov, rhow, rhoE`.
Only components 1--6 are conservative variables handled by the standard
AMReX `FluxRegister`. The register therefore has six components; face source
component 1 maps to register component 0, and reflux writes register components
0--5 to cell-state components 1--6.

The P0 alpha discretization is

```
rhs_alpha = -sum_d ((Falpha_hi - alpha_cell*Sstar_hi)
-(Falpha_lo - alpha_cell*Sstar_lo)) / dx_d.
```

It is a paired nonconservative flux/source operator. A mismatch in `Falpha`
alone is not its complete coarse/fine update, so component 0 is deliberately
excluded from ordinary conservative reflux. Fine alpha is advanced with the
same paired operator, filled through normal coarse/fine FillPatch, and averaged
onto covered coarse cells after subcycling. Uncovered coarse alpha retains its
coarse update. P1 validates boundedness, uniform-alpha invariance, pressure
consistency, and interface error; it does not claim global alpha conservation.

Face components 1--6 are directly compatible with conservative AMReX reflux.
Face component 0 participates in the nonconservative alpha flux/source pair
and requires separate coarse-fine consistency treatment. Face component 7 is
the HLLC contact speed and is never accumulated in a standard FluxRegister.

## SSP-RK2 time integration and units

For `U1 = Un + dt L(Un)` and
`Un+1 = 0.5*(Un + U1 + dt L(U1))`, the integrated face flux is
`0.5*dt*F(Un) + 0.5*dt*F(U1)`. Each stage is accumulated with weight 0.5.
Hydro3D face arrays store unweighted physical fluxes. In direction `d`, the
register multiplier contains exactly once the level timestep and transverse
face area:

```
stage_scale = 0.5 * dt_level * product(dx[q], q != d)
```

Coarse contributions use `CrseInit(..., mult=-stage_scale)` (COPY for the
first stage and ADD thereafter). Fine contributions use
`FineAdd(..., mult=+stage_scale)` for both stages and every fine substep.
`Reflux(..., scale=1)` divides the accumulated extensive mismatch by the
coarse cell volume. Fine calls use fine area and fine timestep; neither cell
volume nor any area/time factor is applied twice.

## Lifecycle and ordering

`flux_reg[1]` represents the level-0/level-1 interface and contains six
components. It is defined with the fine BoxArray, fine DistributionMapping,
ratio two, and fine-level index one. Its first coarse-stage `CrseInit(COPY)`
resets every directional interface contribution for a parent coarse step;
later coarse stages use ADD. Both stages of both fine substeps then use
`FineAdd`. After fine subcycling, P1 performs:

1. reflux components 1--6 into coarse `phi_new`;
2. average all seven fine components onto covered coarse cells;
3. validate/floor the resulting coarse state;
4. invalidate/refill ghosts only when next needed.

Registers are recreated only with level creation/remake, cleared with the
level, and never cleared from a fine advance. Regridding discards stale face
and register contents. The same stage face arrays used by the update supply
the register; there is no second flux path.

## Validation

The manufactured FluxRegister test freezes sign, component offsets, timestep
and face-area scaling, and the corrected coarse-cell set. ML-U0 requires a
zero register correction. ML-SX-INTERFACE requires a measurable reflux-off
conservation mismatch and at least an order-of-magnitude improvement (or the
frozen numerical floor) with reflux enabled. ML-AC verifies bounded alpha,
pressure consistency, and that no standard alpha-register call occurs.
31 changes: 31 additions & 0 deletions docs/perf/m1b-p1-result.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# M1b-P1 result

Disposition: `BLASTGPU_M1B_P1_COMPLETE_MULTILEVEL_REFLUX_SUBCYCLING_AND_AMR_EFFICIENCY_BASELINE_READY_FOR_REVIEW`.

P1 delivers two-level Hydro3D, ratio-two time subcycling, coarse old/new time
interpolation for both fine RK stages, stage-weighted six-component
FluxRegister accumulation, reflux-before-average-down, separately validated
alpha treatment, static sphere/box tagging, persistent per-level storage, and
composite reductions/profiles.

The manufactured register test fixes the signs and scaling. Reflux-on removes
the deliberately exposed planar interface conservation error; reflux-off
retains a measurable mismatch. Alpha is never inserted into the conservative
register. ML-U0, regrid, shock, contact, and sphere cases pass the frozen
envelopes on CPU/CUDA. All four Compute Sanitizer modes pass. M0/P0 tests and
the canonical byte-identical M1a CPU/CUDA result remain valid.

Matched R0 efficiency is transformative: 3.578x runtime and 2.802x used-memory
efficiency at 128-equivalent, rising to 4.503x and 3.261x at 192-equivalent,
while composite profile errors remain far below the frozen limits. The final
single-level guardrail is +0.822% at 128 cubed and +0.068% at 192 cubed with
identical maximum-used memory, so P0 preservation passes.

The final steady trace contains no underlying allocation/free and remains
dominated by directional flux work. Inclusive synchronization time represents
GPU waits rather than measured host overhead; together with the negative C1
result, it does not satisfy the C2 reconsideration gate. The next package is
therefore M1C-P0, not launch batching.

No blastCFD work, merge, C2 experiment, dynamic tagging, remap, gauges, or
cross-solver claim occurred.
5 changes: 5 additions & 0 deletions docs/perf/m1b-p1-timing-results.tsv
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
case run1 run2 run3 run4 run5 run6 run7 median_s cv_percent
U128 27.816338 28.067420 27.971697 28.033955 28.239506 28.279862 28.314495 28.067420 0.650
A64X2 7.799938 7.800669 7.820214 7.843846 8.203249 7.890213 7.846024 7.843846 1.816
U192 142.286145 142.284202 142.359906 142.265092 142.283801 142.951549 142.177345 142.284202 0.183
A96X2 31.629693 31.601831 31.599399 31.547933 31.867318 31.428734 31.578346 31.599399 0.417
36 changes: 36 additions & 0 deletions docs/perf/m1b-p1-tolerances.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
{
"frozen_before_authoritative_performance": "2026-08-22T22:15:00Z",
"basis": [
"M1b-P0 uniform-grid convergence and CPU/CUDA repeatability",
"manufactured FluxRegister sign/scaling test",
"pre-authority multilevel correctness and composite-profile runs"
],
"uniform": {
"alpha_momentum_energy_spread": 0.0,
"partial_density_absolute_spread": 5e-15,
"composite_conservative_relative_drift": 1e-14,
"floor_nan_inf_events": 0
},
"planar_composite_vs_uniform_fine": {
"normalized_L1_max": {"rho": 0.005, "u": 0.005, "p": 0.005, "rhoE": 0.005},
"normalized_Linf_max": {"rho": 0.06, "u": 0.25, "p": 0.06, "rhoE": 0.06},
"transverse_momentum_absolute": 1e-12
},
"sphere_composite_vs_uniform_fine": {
"normalized_L1_max": {"rho": 0.005, "u": 0.02, "p": 0.005, "rhoE": 0.005},
"normalized_Linf_max": {"rho": 0.06, "u": 0.15, "p": 0.06, "rhoE": 0.06},
"axis_symmetry_relative_Linf": 1e-9,
"cpu_cuda_relative_Linf": 1e-12
},
"reflux_interface": {
"minimum_conservation_improvement": 10.0,
"conservative_relative_drift": 1e-10
},
"alpha_contact": {
"lower_bound_slack": 1e-12,
"upper_bound_slack": 1e-12,
"relative_pressure_oscillation": 1e-8,
"cpu_cuda_relative_Linf": 1e-12
},
"all_cases": {"floor_nan_inf_events": 0}
}
Loading
Loading