Native CPU transforms by shape and one forward pack (#584, #587, #595)

changelog.d/598.changed.md

@@ -0,0 +1 @@
+The native backend's CPU transforms are faster on many threads ([#598](https://github.com/LupoLab/solprop/pull/598); [ADR-075](docs/dev/decisions.md)): a batch of transforms now threads by its shape, about one thread per row and the whole team only for many rows, instead of by its element count, and a multi-mode stage transforms its forward rows (projection, absorber, plasma currents) as one batch, as on a GPU (`SOLPROP_NATIVE_FORWARD_PACK=0` turns that off). The tracking cases ran 0.98 of their time on geometric mean on 18–36 cores of an Intel Xeon Gold 6240 and 16 of an AMD EPYC 7763, the few-mode capillaries and the one-dimensional free-space beam 0.80–0.95. `solprop calibrate` now also tunes the transforms' threading, reaches a decision on FFT plan efforts (it no longer tries `patient` plans, and skips efforts whose plans are slower alone), and takes about 30 s, two minutes or seven minutes (from 20 s, 90 s and five minutes). Existing calibration profiles no longer apply: the next native run on each machine calibrates quickly again by itself, or run `solprop calibrate`.

changelog.d/sep-0046-machine-calibration.added.md

@@ -1 +1 @@
-`solprop calibrate` fits the native backend to the machine it runs on ([SEP 46](docs/dev/seps/archive/sep-0046-machine-calibration.md); [Calibrating solprop to your machine](docs/guide/performance/calibration.md)): in about 20 s (`--quick`), a minute and a half, or five minutes (`--thorough`), child processes pinned to the job's CPUs measure how idle threads should wait, how wide the team should be, where each kind of operation should start to thread and which FFTW plan effort runs fastest inside the step, and a setting replaces the built-in value only if it wins every round by more than the measured noise, slows none of the problems (production-sized ones among them) and validates with the other winners. The profile is kept per user in the platform's cache directory (`SOLPROP_CACHE_DIR` moves it; `SOLPROP_CALIBRATION_PATH` adds shared read-only directories, such as a cluster's per node type), keyed by machine, build and allocation, and every later run applies it and records the settings it applied in its metadata (`calibration`). The first native run with no profile for its machine, build and allocation calibrates quickly by itself after a `CalibrationNotice`, and a scan calibrates its workers' allocation first (unless they run one thread each, which has nothing to choose); `SOLPROP_CALIBRATION=use` never measures and `off` ignores profiles. `Simulation.calibrate()` and `solprop calibrate --recipe` refine the settings for one simulation, reused by every run and scan point of the same execution. `--show`, `--export`, `--import` and `--clear` manage profiles. Benchmark series calibrate each build first and record the profile per row. Without a profile, every run uses exactly the former settings, now a built-in table keyed by operating system; `BuildInfo`'s threshold fields report that table for the process.
+`solprop calibrate` fits the native backend to the machine it runs on ([SEP 46](docs/dev/seps/archive/sep-0046-machine-calibration.md); [Calibrating solprop to your machine](docs/guide/performance/calibration.md)): in about 30 s (`--quick`), two minutes, or seven minutes (`--thorough`), child processes pinned to the job's CPUs measure how idle threads should wait, how wide the team should be, where each kind of operation should start to thread and which FFTW plan effort runs fastest inside the step, and a setting replaces the built-in value only if it wins every round by more than the measured noise, slows none of the problems (production-sized ones among them) and validates with the other winners. The profile is kept per user in the platform's cache directory (`SOLPROP_CACHE_DIR` moves it; `SOLPROP_CALIBRATION_PATH` adds shared read-only directories, such as a cluster's per node type), keyed by machine, build and allocation, and every later run applies it and records the settings it applied in its metadata (`calibration`). The first native run with no profile for its machine, build and allocation calibrates quickly by itself after a `CalibrationNotice`, and a scan calibrates its workers' allocation first (unless they run one thread each, which has nothing to choose); `SOLPROP_CALIBRATION=use` never measures and `off` ignores profiles. `Simulation.calibrate()` and `solprop calibrate --recipe` refine the settings for one simulation, reused by every run and scan point of the same execution. `--show`, `--export`, `--import` and `--clear` manage profiles. Benchmark series calibrate each build first and record the profile per row. Without a profile, every run uses exactly the former settings, now a built-in table keyed by operating system; `BuildInfo`'s threshold fields report that table for the process.

docs/dev/backends.md

@@ -1586,11 +1586,14 @@ at most:
   lengths below, at and past one block, partial tiles, both `beta` and row subviews, so
   the kernel's blocking and order are free. Vendor-BLAS and device builds use
   `KokkosBlas::gemm`.
-- **Small transforms on one FFTW thread.** A modal or spatial plan whose real side holds
-  fewer than **16 384** samples per execution (32 768 on macOS; `default_threaded_fft_min_samples`;
-  `SOLPROP_NATIVE_THREADED_FFT_MIN_SAMPLES` overrides it) is made on a one-thread
-  execution instance, because below that a threaded FFTW plan only costs time: the
-  threshold is a speed threshold. FFTW's planner may choose a transform's factorisation by
+- **Small transforms on one FFTW thread; larger ones by shape.** A modal or spatial plan
+  whose real side holds fewer than **8 192** samples per execution (32 768 on macOS; the
+  execution record's `fft_threshold`; `SOLPROP_NATIVE_THREADED_FFT_MIN_SAMPLES`
+  overrides it) is made on a one-thread execution instance, because below that a
+  threaded FFTW plan only costs time: the threshold is a speed threshold. Above it a
+  batch takes a thread per `fft_thread_elements` samples, at most one a row unless its
+  rows are `fft_long_row` long, when it may take the medium width
+  ([transforms by shape](benchmarks.md#transforms-by-shape-and-one-forward-pack)). FFTW's planner may choose a transform's factorisation by
   the thread count (a two-column 256-point inverse took radix 4 with 64-point leaves at
   three and four threads and radix 8 with 32-point leaves at others), which moves ten
   realistic steps by at most 8 u: within the transform's rounding bound, so it constrains

docs/dev/benchmarks.md

@@ -2942,6 +2942,81 @@ time does not change with the thread count, so its two columns are one result. N
 machine has a case that reaches the 10 % at the default width that
 [#337](https://github.com/LupoLab/solprop/issues/337) asks of run-time dispatch.
 
+### Transforms by shape and one forward pack
+
+Issues [#587](https://github.com/LupoLab/solprop/issues/587) and
+[#584](https://github.com/LupoLab/solprop/issues/584), November 2026. The transforms'
+width was one element count (serial below 16384, four threads below 131072, the full
+pool above), and a modal stage ran each forward transform as its own threaded call over
+the modes' rows only.
+
+**Width by shape, timed alone.** A batched real-to-complex forward transform of each
+shape the tracking set runs, at FFTW's estimate and measure, on 1 to 36 threads, with
+native's OpenMP loop callback and spinning waits (an Intel Xeon Gold 6240 node, 16
+cores of an AMD EPYC 7763 on one NUMA domain, an Apple M5 Max). Microseconds a
+transform, measured plans; the table's width in brackets:
+
+| rows × length | Xeon 1 | 4 | 8 | 18 | EPYC 1 | 4 | 8 | 16 |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| 4 × 3072 | [20.5] | 15.5 | 8.5 | 16.0 | [16.7] | 5.8 | 7.8 | 9.3 |
+| 4 × 10240 | 73.5 | [21.8] | 24.5 | 29.1 | 66.5 | [17.7] | 19.9 | 20.9 |
+| 1 × 32768 | 70.2 | [47.6] | 50.5 | 62.7 | 71.1 | [34.2] | 29.0 | 38.8 |
+| 7 × 6400 | 77.7 | [38.5] | 16.9 | 18.8 | 67.0 | [20.6] | 13.6 | 14.2 |
+| 289 × 128 | 46.5 | [16.2] | 22.7 | 9.8 | 40.4 | [12.2] | 7.5 | 7.7 |
+| 128 × 576 | 119.5 | [30.4] | 16.4 | 28.4 | 89.7 | [23.5] | 13.0 | 9.5 |
+| 16 × 6144 | 198.9 | [46.9] | 29.5 | 45.1 | 149.8 | [38.9] | 19.8 | 13.3 |
+| 12 × 10240 | 279.1 | [61.7] | 44.4 | 31.5 | 205.0 | [52.5] | 37.6 | 23.9 |
+
+A few rows ran best at about a thread a row, more threads splitting rows at a loss;
+many rows of a few thousand samples wanted 8 to 18 threads, where four were two to three
+times slower. One row gained from being split only from about 16384 samples on these
+two Linux machines (from 4608 on the M5 Max's top tier, at four threads). Hence the rule
+(`operation_threads`): from `fft_threshold`, a thread per `fft_thread_elements`
+samples, at most one a row, or up to the medium width for rows of `fft_long_row`
+samples; Linux 8192, 4096 and 16384, macOS 32768, 32768 and 65536 (a sixteenth on a
+spinning top tier). Estimate plans of a prime length (4096 × 193) at more than four
+threads ran 15 to 47 times slower than at four; measured ones did not.
+
+**One forward pack.** A modal stage's forward rows -- the projection's fine and coarse,
+the in-stage absorber's, the two held plasma currents' fine and coarse -- now go
+through one plan, as on a device: `rdw` transforms 28 rows of 10240 at once where it
+transformed seven batches of four. The host keeps the coarse rows after the fine ones,
+so that a deferred quadrature transforms the fine block in its stages and the coarse
+block once at its finish. Pack on against off, one build (5 % windows, three rounds,
+fields within 1e-14):
+
+| case | Xeon Gold 6240, 18 | 36 | EPYC 7763, 16 | M5 Max, 6 (contended) |
+|---|---:|---:|---:|---:|
+| `argon_blueshift` | 0.88 | 0.90 | 0.88 | 0.92 |
+| `rdw` | 0.90 | 0.94 | 0.95 | 0.90 |
+| `ctc` | 0.92 | 0.94 | 0.94 | 0.97 |
+| `he_rdw_averaged` (one column) | 1.35 | 1.32 | 1.11 | 0.97 |
+
+On the one-column case the pack's four rows, threaded on the compact team, took 152 µs
+a stage against 90 µs for its three serial transforms (Kokkos section timer, Xeon, 18
+threads), though the same transform alone took 25 µs on eight threads; measured plans
+did not change it (1.34). One-column problems keep their separate transforms.
+
+**Against the previous build** (both together; one-column problems unpacked; 5 %
+windows, three rounds, fields within 1e-12):
+
+| case | Xeon 18 | Xeon 36 | EPYC 16 | M5 Max 6 (contended) |
+|---|---:|---:|---:|---:|
+| `argon_blueshift` | 0.87 | 0.80 | 0.86 | 0.92 |
+| `ctc` | 0.89 | 0.91 | 0.92 | 0.98 |
+| `rdw` | 0.91 | 0.95 | 0.95 | 0.95 |
+| `line` | 0.93 | 0.87 | 0.88 | 1.00 |
+| `radial_filament_256` | 0.97 | 1.01 | 0.99 | 1.00 |
+| `gnlse_raman_shock` | 0.94 | 0.99 | 0.98 | 1.00 |
+| `grin_nonlinear` | 1.01 | 0.93 | 1.01 | 1.00 |
+| `kerr_beam_192` | 1.02 | 1.04 | 1.01 | 0.99 |
+| geometric mean, with the one-thread rows | 0.977 | | 0.981 | 0.993 |
+
+The shape rule alone (pack off) gave `line` 0.91 and 0.90 (Xeon 18 and 36) and 0.89
+(EPYC), and `grin_nonlinear` 0.95 on the EPYC; the one-thread rows stay within
+0.98-1.02. `he_rdw_averaged` and `h2_raman_averaged`, whose code is unchanged, ranged
+0.92-1.12 on the Xeon between runs of either arm: the per-process modes of #81.
+
 ## Reference tables
 
 These tables back the indicative results above.
@@ -2967,7 +3042,7 @@ partial chunk.
 | General-modal synthesis/projection/maps | 65,536 | 65,536 | none |
 | Coupled matrix functions | 32,768 | 32,768 | none |
 | Exponential Adams trial passes | 8,192 | 65,536 | up to four threads |
-| Modal/spatial FFT | 32,768 | 131,072 | up to four threads |
+| Modal/spatial FFT | 8,192 (macOS 32,768) | 4,194,304 | by shape: a thread per 4,096 samples (macOS 32,768), at most one a row; rows of 16,384 samples (macOS 65,536) or more split up to four threads ([transforms by shape](#transforms-by-shape-and-one-forward-pack)) |
 | Raman FFT | 65,536 | 262,144 | up to four threads |
 
 A response column is one active node and one component. Every width is capped by the

docs/dev/cpu-performance.md

@@ -160,6 +160,22 @@ measured while it existed.
   pool's cores for the whole run: at 16 threads the four-mode capillary became 2.2 times faster on a Xeon Gold
   6240 (1.3 times on an AMD EPYC 7763), the radial filament 1.4 (1.6) times and averaged
   helium 1.4 (2.9) times; one thread is unchanged.
+- **Transforms thread by shape** ([measurements](benchmarks.md#transforms-by-shape-and-one-forward-pack),
+  issue [#587](https://github.com/LupoLab/solprop/issues/587)). A batch of transforms took
+  its width from its element count, though FFTW divides a batch's rows among its
+  threads: a few rows ran fastest at about a thread a row and full width was 1.1–1.6
+  times slower, while many short rows wanted 8–18 threads where four were two to three
+  times slower. Now a batch takes a thread per `fft_thread_elements` samples, at most
+  one a row, or up to the medium width when its rows are `fft_long_row` long, and the
+  calibration moves both. `line` takes 0.87–0.93 of its time at 16–36 threads on an
+  Intel Xeon Gold 6240 and an AMD EPYC 7763.
+- **A modal stage's forward transforms as one batch** (issue
+  [#584](https://github.com/LupoLab/solprop/issues/584)). The device's pack of a stage's
+  forward rows (projection, absorber, held currents) now runs on the host too, so four
+  modes no longer leave most threads idle: `argon_blueshift` 0.80–0.87, `ctc`
+  0.89–0.92 and `rdw` 0.91–0.95 at 16–36 threads on those two machines. One-column
+  problems keep their separate transforms: there the pack's few rows on the compact
+  team ran averaged helium 1.1–1.35 times slower.
 - **Measured FFTW wisdom only for serial transforms by default**
   ([measurements](benchmarks.md#threaded-fft-plan-selection)). FFTW's construction-time
   trials sometimes chose a slower *threaded* plan: on a 32-core AVX-512 server at 32
@@ -180,9 +196,43 @@ measured while it existed.
   (`fftw_set_timelimit` of 0.01–0.2 s) gave plans 10–30 % slower to execute, and
   estimate plans ran one-thread windows up to 1.6 times slower on an AMD EPYC 7763.
 
+- **Measured threaded plans through the calibration, not by default** (issue
+  [#595](https://github.com/LupoLab/solprop/issues/595)). In the step, measured plans for
+  the threaded rows (`SOLPROP_FFTW_PLAN=measure`) ran the tracking cases 0.94 on an
+  Intel Xeon Gold 6240 (`kerr_beam_192` 0.54–0.79, `ctc` 0.90–0.94), 0.96 on 16 cores
+  of an AMD EPYC 7763 (`grin_nonlinear` 0.83, `ctc` 0.90) and 0.97 on an Apple M5 Max,
+  but planning them took up to 12 s more construction (`ctc` on the EPYC 7763: 13.8 s
+  against 1.6 s), more than a whole run of it gains. So the default stays at estimate,
+  and a calibration that admits `threaded_rows measure` stores the wisdom once.
+  - Why the calibration's measured and patient plans "rarely won": they were rarely
+    timed. On all three machines the thorough level's patient capture never finished
+    within its share of the stage, and the timing in the step then ran out of rounds
+    before a verdict (no noise estimate), so every group fell back to the policy. The
+    previous build did the same on the EPYC 7763 (thorough: no verdict; standard: no
+    transform planned within the stage's share), and so still does this one without
+    patient plans: the stage needs more of the levels' time than it gets.
+    Timed alone, the measured threaded plans ran 0.53–1.05 of estimate's time
+    (medians 0.74 on the EPYC 7763, 0.91 on the Xeon, about 0.95 on the M5 Max), and
+    estimate's serial plans 1.06–2.8 of measure's. Those timings ran outside the native
+    wait scope, so LLVM's runtime woke its idle threads at each loop; the timing alone
+    now runs inside it, as a step's transforms do (on the M5 Max a 4 × 10240 plan took
+    11 µs there against 22 µs outside). Inside it, on the EPYC 7763: measured threaded
+    plans 0.72–0.92 of estimate's time (medians by level and allocation), estimate's
+    serial rows 2.1–3.0 of measure's and its planes 1.05–1.34, so both are pruned. In
+    the step `threaded_rows measure` won every round on an 8-core allocation (0.915)
+    and lost one on 16 cores (1.039).
+  - The thorough level no longer plans at patient; a candidate whose plans are no
+    faster alone (median ratio 0.98 or more) is not timed in the step; and the levels
+    take 30 s, 2 min and 7 min (from 20 s, 90 s and 5 min). On the EPYC 7763 the stage
+    then reached a verdict: estimate's serial rows (2.11 alone) and planes (1.26) were
+    pruned, and `threaded_rows measure` scored 0.99 on the calibration's problems but
+    tied a round, so the policy held. Each candidate's plans timed alone are kept beside
+    its verdict (`evidence.planning.alone`, `pruned`).
+
 **Tried and not adopted**
 - **`FFTW_PATIENT` planning.** It adds seconds of construction and gave no measured
-  throughput gain.
+  throughput gain. SEP 46's thorough calibration tried it for serial rows; its capture
+  never finished in time on three machines, and it was dropped (#595).
 
 ## Kernels