diff --git a/CHANGELOG.md b/CHANGELOG.md index 98393e72d..55c013eff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,149 @@ All notable changes to solprop are recorded here. The format follows ## Unreleased +### Added + +- Performance-history rows for 0.2.0 (`benchmarks/history/`) on the same nine hardware sets + as 0.1.0. +- The benchmark harness has a `sep47` set of short windows for SEP 47's performance matrix: statistics rows at every step under `Multistep()` and `RungeKutta()`, snapshot controllers idle, sparse and at every endpoint, repeated `with_input` runs, a four-point scan on `Local` and the same points on `Batch(4)`, saved and field-free radial and Cartesian planes in memory and in HDF5, and coupled flows of 4 and 17 channels. `track --counters` and `ab --counters` add one separately reported run per worker with structural counters from the native backend, counted the same way whether the native execution session or the transitional driver session serves the run: its returns to Python by cause, native march calls, attempts, planes, statistics rows, transverse-edge reads, the fields copied to the host and loaded into the native state, and their bytes. +- The desktop front end runs every simulation and scan as a process of its own, which outlives the window and is followed from its record, and runs them on other machines over SSH too: a + workstation, a Slurm cluster or a rented machine, picked on the Runs on line. Each run + keeps its record in its folder there, so it survives a closed window or a dropped network + and is found again on reconnecting; a Runs window lists every run on every machine, a + results file is analysed on its machine, and a machine can be set up with solprop from the + window. See [Running on other machines](docs/guide/front-end/machines.md). +- `solprop calibrate` fits the native backend to the machine it runs on ([SEP 46](docs/dev/seps/archive/sep-0046-machine-calibration.md); [Calibrating solprop to your machine](docs/guide/performance/calibration.md)): in about 30 s (`--quick`), two minutes, or seven minutes (`--thorough`), child processes pinned to the job's CPUs measure how idle threads should wait, how wide the team should be, where each kind of operation should start to thread and which FFTW plan effort runs fastest inside the step, and a setting replaces the built-in value only if it wins every round by more than the measured noise, slows none of the problems (production-sized ones among them) and validates with the other winners. The profile is kept per user in the platform's cache directory (`SOLPROP_CACHE_DIR` moves it; `SOLPROP_CALIBRATION_PATH` adds shared read-only directories, such as a cluster's per node type), keyed by machine, build and allocation, and every later run applies it and records the settings it applied in its metadata (`calibration`). The first native run with no profile for its machine, build and allocation calibrates quickly by itself after a `CalibrationNotice`, and a scan calibrates its workers' allocation first (unless they run one thread each, which has nothing to choose); `SOLPROP_CALIBRATION=use` never measures and `off` ignores profiles. `Simulation.calibrate()` and `solprop calibrate --recipe` refine the settings for one simulation, reused by every run and scan point of the same execution. `--show`, `--export`, `--import` and `--clear` manage profiles. Benchmark series calibrate each build first and record the profile per row. Without a profile, every run uses exactly the former settings, now a built-in table keyed by operating system; `BuildInfo`'s threshold fields report that table for the process. + +### Changed + +- The native backend's CPU transforms are faster on many threads ([#598](https://github.com/LupoLab/solprop/pull/598); [ADR-075](docs/dev/decisions.md)): a batch of transforms now threads by its shape, about one thread per row and the whole team only for many rows, instead of by its element count, and a multi-mode stage transforms its forward rows (projection, absorber, plasma currents) as one batch, as on a GPU (`SOLPROP_NATIVE_FORWARD_PACK=0` turns that off). The tracking cases ran 0.98 of their time on geometric mean on 18–36 cores of an Intel Xeon Gold 6240 and 16 of an AMD EPYC 7763, the few-mode capillaries and the one-dimensional free-space beam 0.80–0.95. `solprop calibrate` now also tunes the transforms' threading, reaches a decision on FFT plan efforts (it no longer tries `patient` plans, and skips efforts whose plans are slower alone), and takes about 30 s, two minutes or seven minutes (from 20 s, 90 s and five minutes). Existing calibration profiles no longer apply: the next native run on each machine calibrates quickly again by itself, or run `solprop calibrate`. +- On a Mac whose chip has cores of more than one kind, the thread budget is now the cores of the fastest kind (`hw.perflevel0.physicalcpu`, 6 on an Apple M5 Max rather than all 18), unless `SOLPROP_THREADS` is set: every thread of a parallel pass waits for the slowest, and five of six benchmark cases ran 1.17 to 1.49 times faster on the 6 top-tier cores (#333, ADR-070). Set `SOLPROP_THREADS` to use every core; the run's `thread_budget_source` records `"top-tier cores"`. +- On Arm and other non-x86-64 hosts, the native plasma column's band filter sums each sample's taps in one unrolled pass over blocks of samples, with the same values: on an Apple M5 Max the plasma cases' windows ran 1-6 % faster (#560). +- `python -m benchmarks ab` now calibrates each build at the `standard` level by default instead of `thorough`, so a development A/B no longer spends several minutes per arm calibrating; pass `--calibrate thorough` for release benchmarks. `track` keeps `thorough`. +- The benchmark harness (`python -m benchmarks track` and `ab`) now runs workers with the backend's own OpenMP waiting by default (`--threading backend`), so that a row measures a user's default run; `--threading passive` keeps the previous policy. History rows record the policy, so rows taken from now on start a new series. +- Python processes solprop starts (a run from the window, directly or as a Slurm job, the window's kernel, and `Pool` workers) take `OPENBLAS_THREAD_TIMEOUT=20` unless it is set, so NumPy's and SciPy's OpenBLAS pools no longer spin on every CPU for a moment as they load: importing the driver used 13.8 CPU seconds on a 72-CPU Linux machine, and now 0.75 ([#383](https://github.com/LupoLab/solprop/issues/383)). The performance guide says how to set it for your own scripts. +- The README, the package metadata, the desktop front end's Help and the native backend's + error messages now link the manual of the newest release (`/en/stable/`) rather than the + development version (`/en/latest/`), and the installation page's advice for a platform + without a native wheel now allows for the yanked 0.1.0. +- The transverse edge's read spacing has a growing backstop: `min(B, 4g/v_q)`, with `B` starting at `L/16` and doubling at each read whose reach has not grown by a tenth since it started, up to `L/4`. A free-space run whose spectrum holds far from its edge no longer takes extra steps for its reads. On the suite's `line` case the edge costs 2.9 % of the run where it cost 12.8 %, with 13 reads where it took 23 ([#396](https://github.com/LupoLab/solprop/issues/396)). A planar run also no longer evaluates its slab modes again at each run to set up the edge. On scaled versions of SEP 40's linear witnesses (414 packets that must stop) and on a self-focusing line beam in narrowed boxes (80 configurations), every run that stopped or warned under the former spacing still does. +- The native backend now keeps the FFTW plans its runs measure by default, so each transform shape is measured once per machine and build instead of on every run. Plans are kept per transform in `wisdom.json` in the calibration store's directory for the machine and build, and a calibration profile's own plans take precedence. `SOLPROP_FFTW_WISDOM` now only moves this file, and the file it names no longer turns off a profile's FFT planning. `SOLPROP_CALIBRATION=off` keeps no default file, and `solprop calibrate --clear` removes the plans. Plans kept by earlier releases are not read; their files can be deleted. A name that cannot be used now logs a warning and the run plans without stored plans, where it used to raise. The store is keyed by the machine fingerprint, which tells Apple silicon models apart (#136). +- The native backend forms free space's separable stage propagators a tile at a time through its vector maths (SLEEF on x86-64 and macOS arm64), rather than one libm `sincos` per element, which took 18 % of a one-thread `line` run. `line` takes 0.89 of its time at one thread and 0.93 at 16, and the Cartesian filament 0.93 at 16 ([#338](https://github.com/LupoLab/solprop/issues/338)). +- The front end's service layer (its kernel process, run workers, live reductions and their + messages) moved from `solprop_gui.service` into solprop as `solprop.service`, so a machine + that only runs simulations for the desktop front end needs solprop alone, without Qt. It is + front-end support, not part of the user interface, and may change without notice. +- A run's fixed host cost is smaller ([#347](https://github.com/LupoLab/solprop/issues/347)): the first statistics no longer import `scipy.signal` (0.7 s), opening a run's output no longer serialises and digests its configuration a second time, and package versions are read once per process. A large free-space launch builds and translates faster, its column energies summed and its admittance copied in threads. +- The native backend refuses the benchmark-only Adams comparison variants off the selected family, by name and before it allocates anything: `ComparisonMultistep` with PECE, explicit Adams–Bashforth (`corrector="none"`) or a matched pair (`corrector_order = order`). They ran on solprop's Python march with a `HostMarchWarning`; they are now the NumPy reference's alone (SEP 47 section 6.3), so run them with `backend="numpy_ref"`. `ComparisonMultistep(corrector="pec", corrector_order=order + 1)`, the selected method, still runs natively. The native bridge's copy of the general Adams path is removed with them. +- The native backend's idle OpenMP threads now spin briefly within native work instead of sleeping at once: `GOMP_SPINCOUNT=10000` on GNU's runtime and, on LLVM's (macOS and Windows), a 1 ms block time inside native calls only. Waking sleeping threads cost 16 to 72 µs for each of a step's many short parallel regions; benchmark windows ran in about 0.8 of the time on an Apple M5 Max, two Intel Xeon Gold 6240 and a shared AMD EPYC 7763 node, and in 0.87 of it on an AMD EPYC 9B45 under Windows (ADR-071). Teams wider than a chip's fastest cores keep sleeping at once. Set `OMP_WAIT_POLICY=passive` to restore the previous behaviour. +- The native backend evaluates and finishes its statistics natively from a plan lowered once (SEP 47 section 6.4): the half-maximum widths, the species reduction, the multi-channel electron density and peak ionisation rate, the plasma integration ratios, the profile statistics and a general modal basis's flux statistics (`omega0`, `physical_energy`, `spectral_flux`, `analytic_flux`, `fwhm_t`, `fwhm_t_outer`) no longer return to Python, and a general modal plane's statistics no longer copy the field to the host. Their z-dependent inputs are read from the problem's own tables. A profile statistic (`density`, `pressure`, `core_radius`) reads the propagation's table of its path where one reproduces the profile, or the profile tabulated once over the run; its measured error, at most 1e-13 of the profile's magnitude (about 1e-15 on a gas-filled capillary), is recorded in the run's execution record under `statistics_tables`. A modal power weight that varies in z is summed in a different order, a rounding-level change. The native execution session takes every periodic-capable statistic as a native periodic row. +- Single-mode problems on the native backend now run their synthesis and projection on the response team with Apple's Accelerate too, as they already did with OpenBLAS: on an Apple M5 Max at six threads the argon blueshift window ran 0.96 of its time. +- Host passes too small to thread no longer read the CPU topology first, which took up to a sixth of a short window's host statistics on Linux (#556). +- The native backend's transverse-edge reads on radial grids synthesise the rows they use in one matrix product, once for every table, rather than one sum per sample per table: on an Apple M5 Max the 256- and 512-node radial filaments ran 0.83 and 0.75 of their one-thread windows, and 0.92 and 0.87 at six threads, with the same steps. +- A saved plane makes fewer field-sized copies, with the same stored bits. A field the backend made for the observation alone (the native backend's host copy, the reference's widened single-precision state) is converted to true-power amplitudes where it is, so a modal plane on `native` makes two field-sized arrays where it made three, and a unit-weight one no longer copies the field at all before storing it. The reference's single-precision observation scales its widened state in place, one array fewer, and a plane observer and a failure checkpoint at a plane that kept no field now share one fetch of it. This partly addresses [#346](https://github.com/LupoLab/solprop/issues/346), as SEP 47's early copy elisions; a large spatial plane on `native` still makes its host copy and its stored copy. +- The native backend's Adams march takes the boundary monitor's checks inside its compiled loop: a run whose monitor is busy, as when light lingers near the time window's edge, no longer returns to Python at every step. The monitor's record, its warning and a stop are unchanged, at the same step ([SEP 45](docs/dev/seps/archive/sep-0045-native-rk-march.md)). The split collar's crossing limit sums its bucket occupancies in one sequential order, so that the compiled march reproduces it exactly (Q7); in a near-tie its bucket can move one place, which changes a split-scheme run's steps at rounding level. The native backend also steps the adaptive Runge–Kutta loop of `RungeKutta()` in compiled code, from the stage tables it already holds, taking each trial's stage rows and its error norm natively: a run's metadata says which loop stepped it (`rk_march`), including interpolated planes, replayed schedules, coupled flows, z tables, the transverse edge and a weighted error norm, on CPU and GPU builds; a controller with its own methods runs solprop's loop with a `HostMarchWarning` naming the reason. The norm, weighted or not, now sums in one sequential order in both loops, which moves native Runge–Kutta runs once at rounding level. +- A scan's workers ask for passive waiting instead of setting `OMP_WAIT_POLICY=passive` themselves, so that a user's own `OMP_WAIT_POLICY`, `GOMP_SPINCOUNT` or `KMP_BLOCKTIME` now reaches them, and so does a machine calibration's row for the workers' allocation ([SEP 46](docs/dev/seps/archive/sep-0046-machine-calibration.md)); without either they wait as before. Their recorded `openmp_wait_policy` reads `passive`. +- Snapshots now work with interpolated planes (`interpolate_planes=True`), which they refused before: a requested snapshot is taken at the end of the step that crossed the plane being saved, once every plane that step crossed has been saved, or at the next accepted endpoint, so its running checkpoint is a state the march passed through, which `resume=True` continues without `resume="interpolated"`, saving each later plane once. The native execution session (SEP 47's prototype, selected by tests and the benchmark harness) now interpolates planes itself, for Adams and the embedded Runge-Kutta tableau alike, where the state lives: no stage or field crosses to the host but the exported plane. Its Runge-Kutta march evaluates the next step's first stage afresh after an observation, as the native bridge does, so the two take the same steps to the bit and their planes differ only at the last bits. A step plane owed by a step that crossed an interpolated plane and ended on the next one is no longer saved twice at that position ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md)). +- A kept plane's field is exported once, straight into its stored form, with the same stored bits ([#346](https://github.com/LupoLab/solprop/issues/346), SEP 47). `MemoryOutput` hands the run the block row the plane will occupy, so the field is written once, where it is kept, with no host volume of its own; `HDF5Output` is handed one new array, which h5py writes into the plane's chunk without a copy. A plane observer and a failure checkpoint at the same plane share that export. On a 108 MiB Cartesian plane the native session's saved plane no longer allocates a field into memory and one fewer into a file, and the driver session's file output converts its observation where it is instead of allocating a second field. `PlaneStore` gains `takes_exports`, `field_destination` and `save_exported` for this, with `solprop.output.schema.StoredField` beside them; a store subclass that overrides `save` is still given every plane, with its field, through it. Export destinations are refused by what is wrong with them (shape, dtype, layout or a read-only array). +- A `Batch` scan's group of native points now runs, on the CPU build, in the native group session: each member in its own native execution session, which steps it natively from one save plane to the next and keeps its own events, statistics rows, boundary monitor and exports, with the same fields, statistics and step records as before. An Adams group no longer warns `HostMarchWarning` for its replay. A member's monitor that refuses its launch now ends it after its launch plane is committed, as a run of the point alone does, and in a group-wide failure the members that had already taken their turn report the steps they committed, eligible for a checkpoint. A group the native group session does not run (a GPU build, a noise model, a feature the native session refuses) runs on the lockstep batch as before ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md)). +- The native execution session (SEP 47's prototype, selected by tests and the benchmark harness) now runs the low-storage replay (`LowStorageRungeKutta`) itself: each step is one in-place native step whose stage scalars the native state forms from its tables, so the replay no longer returns to Python once a step, with the same steps, statistics rows and fields as before. A failure partway through a step ends the session poisoned, with no checkpoint of its own, and the run's service loop records it against the latest running checkpoint ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md)). +- The native execution session (SEP 47's prototype, selected by tests and the benchmark harness) now runs a run's temporal boundary monitor itself: its launch judgement, cadence, quiet collar, split-collar cap, warnings, stops, plane rows, finish and resumed record are native, with the same decisions and record as solprop's monitor. A boundary warning reaches the caller as a `ServiceRequired` warning barrier, which the run's service loop emits; under warning filters that raise it, the run fails at that endpoint with its field available for the checkpoint ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md)). A free-space run's transverse edge now runs on the native session too, no longer refused: its launch judgement, read spacing, reads, a requested collar's current and return, warnings, stops, record and resumed record are native, with the same decisions and record as solprop's edge on the same readings, and a point that raises several warnings serves each as its own barrier. The safety ramp's share in its band at the launch and its removal over each read interval are summed natively in another order, within a rounding bound of solprop's values that accumulates across reads, so their warnings and stops may differ from solprop's for a share within that bound of a threshold. +- Runs on a GPU build (CUDA or HIP) of the native backend now go through the native execution session when they use Adams or an embedded Runge-Kutta tableau on a modal route, with saved planes, periodic statistics rows, step planes, the temporal boundary monitor and resumed Adams history, with the same steps, statistics and fields as before ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md)). A saved plane's field crosses from the device once, into its destination, with no extra host copy. On a GPU build, spatial routes, the low-storage replay, dense output, a transverse edge, noise models and `Batch` groups still use the backend's own march. +- Runs on the native backend's CPU build now go through the native execution session whenever it admits the run ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md)), with the same steps, fields, statistics, step records and call ledgers as before. The session also runs resumed runs, carrying the Adams history and the call ledgers, the input and modified noise models, the boundary monitor (including a free-space run's transverse edge), dense output and the low-storage replay, and it exports the Adams history beside each saved plane and checkpoint. A signal during a run is still handled once the native call returns, within 256 attempts or 0.1 s; the run's checkpoint then includes every attempt made. Some runs still use the backend's own march, as before: comparison methods and methods other than Adams and the embedded Runge-Kutta tableaux; controllers that override `PIController`'s methods; TDSE responses; the input noise beside a transverse edge; a noise model on a propagator stored as a z table; marches the backend steps on the host (a coupled flow off its eigenbasis route or wider than 16 channels, or low-storage stage rows the host forms); and the plane statistics the host finishes, such as `beam_radius` and `fwhm_r`. A `Batch` scan's group refuses noise models in the native group session and runs them on the lockstep batch. +- A progress reporter now hears a run's attempted steps in batches, as the run's session delivers them: once per save plane, and at least every 256 attempts or 0.1 s in between. It no longer hears each step as it is taken ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md) section 5.3). Each attempt is still reported once and in order. As a result, a stop flag that a reporter sets lands up to one batch later, at most 256 attempts or 0.1 s, on every backend, the NumPy reference included. A flag set from outside the run is still polled between batches, as before. +- Each execution record now states what a restart keeps and restores, under `outcome.execution["restart"]`: whether a plane's or checkpoint's method history was `retained`, `omitted` or is `not_applicable`, with a reason (`explicit_policy`, `automatic_policy`, `automatic_size_limit`, `snapshot_required`, `unavailable_history`, `output_cannot_store`, `write_failed`, `incompatible_representation`, `unsupported_method`, `no_method_history`), and in a resumed execution what it `restored`, `reset` (a Runge–Kutta resume's proposal and norm memory, an Adams history set aside) or found `not_applicable` (low-storage), and that the boundary monitor was reconstructed from its persisted records. A resume now warns once (`SolpropWarning`) when an Adams history it expected was omitted by the automatic 256 MiB limit, was not exportable, or cannot be read back (including a write cut short); `resume_history=False` is recorded without a warning. Files written before the record read as unknown, and resume as before ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md)). +- Snapshots no longer require `step_planes=1`: a requested snapshot is taken at the next accepted endpoint, and where no plane is saved there the run writes a running checkpoint, in the existing terminal-checkpoint record with a matching `running` outcome, from which the snapshot and `resume=True` continue. The attempts and periodic rows since the last plane stay with that plane, once, and a later saved plane supersedes the checkpoint. A low-storage step abandoned in flight now falls back to the latest running checkpoint, or to the checkpoint the run resumed from, rather than to the last saved plane ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md)). +- Building a graded-index guide and run, and a large free-space launch, take less time: a mode solve evaluates its material once instead of once per finite element, a free-space launch forms each distinct column's admittance once, and neither repeats a table evaluation or a pass over the beam it already made. On the `grin_nonlinear` benchmark the build and translation go from 24 s to 16 s on one core, and on `kerr_beam_384` the launch's build from 12.6 s to 7.1 s. Every number of the translated problem is unchanged. +- On macOS arm64 the native backend's CPU kernels now take SLEEF's 1-ulp AdvSIMD vector `exp`, `log` and `sincos` by default (`SOLPROP_NATIVE_VECTOR_MATH=AUTO`), as x86-64 builds do: on an Apple M5 Max the comb-to-continuum case ran 0.84 of its time at one thread, and the tracking cases 0.93 on geometric mean, with none slower beyond its spread (ADR-072, #339). Linux aarch64 keeps the system libm, where SLEEF measured level (1.00 on a Neoverse V2). +- On Apple silicon, where the native backend's idle threads spin within native calls on the top-tier cores, it threads smaller passes: every size threshold is divided by 16, since a parallel region there costs about a microsecond and the cluster shares its L2 cache. On an Apple M5 Max at six threads the Raman GNLSE ran 0.70 of its time and the comb-to-continuum 0.91, none slower (ADR-073). Other platforms keep the previous thresholds, which measured faster there. + +### Changed (breaking) + +- The backend seam's problem descriptor is now one trajectory's typed description + ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md) sections 6.2 and 6.7). + `solprop.translate.problem.KernelProblem` is renamed `PropagationProblem`, with no + alias. The nominal leading batch axis is gone: `PropagationProblem.shape`, + `initial.field0` and `initial.noise0`, `Backend.set_initial`, `Observation.field` and + `noise_phase`, `Backend.stats()`, the periodic rows of `StepStats` and + `ResponseBudgetError.field` are all `(n_modes, n_pol, n_omega)`-shaped with no batch + axis, `Initial.batch_params`, `Flags.single_batch` and `assemble(batch_params=...)` are + removed, `assemble` refuses a stacked field, and `RunEstimate.shape` has three entries. A + group of trajectories is a sequence of member problems, as a batched scan already runs + them; `solprop.driver.batch.split_observation` and `plane_from_observation(member=...)` + are removed. A march snapshot's stored state keeps its leading axis of one on disk, and + existing files stay readable. The route-specific optional fields `transform`, `flux`, + `transverse`, `tdse` and `current_numerator` move into a closed union, `problem.route`, + from the new module `solprop.translate.routes`: a `ModalRoute`, a `GeneralRoute` or a + `SpatialRoute` of kind radial, Cartesian, planar or line, each with the fields its + executor needs; the TDSE response is the modal route's reference-only `TDSESource`. + `problem.transform` still reads the route's transform, and `tdse_of`, `flux_of` and + `transverse_of` read a route's own leaves. Both backends refuse a route they do not + execute by name before they allocate. The public `Simulation` and `SimulationSettings` + interfaces are unchanged, and every result is bitwise the same. +- The run's policy has left the backend seam's problem descriptor + ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md) sections 5.2 and 6.2). + `PropagationProblem` no longer has `stepper` or `stats_spec`: the tableau, controller, + step limits, schedule, save planes and statistics travel in the immutable + `solprop.translate.run_request.RunRequest`. `solprop.translate.problem.assemble` returns + a `Translation(problem, request)` pair (`problem, request = assemble(...)`), whose + request has the route's step limits resolved, and `RunRequest.stepper` reads the stepper + settings as one `StepperSettings` record. Backends receive the request where they are + built: `solprop.backends.registry.select(name, problem, request, ...)` and the backend + constructors take it after the problem, check the pair with the new + `solprop.translate.problem.validate_run(problem, request)` before they allocate, and + expose it as `Backend.request`. A run starts through the backend's preparation: + `Backend.prepare(problem, request)` returns a `PreparedExecution` whose `start(...)` + opens the backend's own session, the reference's `HostSession`, the native backend's + `NativeSession` where it admits the run and the transitional session otherwise, and + `solprop.backends.sessions.start_session` only starts it; a preparation refuses a + request that is not one run of the backend's own (`RunRequest.for_run` of it). + `PropagationProblem.validate()` now checks what the problem + promises alone; `validate_run` adds the checks that read the run's policy, with the same + refusals and messages, and `validate_launch(request)` takes the request. + `describe_run` now takes a `StepperSettings`; `RunRequest.for_run(...)` gives one run's + start, retention, monitor, history and snapshot policy, and `RunRequest.with_stepper(...)` + replaces the stepper section. `metadata_for(problem, request, grid, ...)`, + `field_retention(request, ...)`, `check_snapshot_request(request, ...)`, + `run_summary(problem, request, metadata, ...)` and + `check_derived_problem(constructed, derived, constructed_request, derived_request)` take + the request; `run_problem` reads it from the backend. The public `Simulation` and + `SimulationSettings` interfaces are unchanged, configuration hashes and resume identity + are unchanged (a file written before this change resumes), and every result is bitwise + the same. +- A run now starts from an explicit start rather than from a backend its caller loaded + ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md) section 5.2): + `solprop.driver.run.run_problem` takes a required `start=`, a + `solprop.backends.session.Start` (`Start.entrance(problem)` for a fresh run), and loads + it into the backend through the run's session; its `start_plane=` argument is removed, + the start's `plane_index` taking its place. A caller no longer calls + `backend.set_initial` before `run_problem`. `Simulation.run`, `SimulationRunner`, scans + and the benchmark suite start their runs this way, and a resumed run's call ledgers + and Adams history travel in the start, a history the backend cannot install being + warned about and recorded as reset as before. A batched scan's points run as one group + through the group protocol (`solprop.backends.session.GroupSession`, SEP 47 section + 6.7): each point keeps its own planes, statistics and outcome, a boundary stop still + fails its point alone, and a group-wide failure now reports each member's actual + position. +- The backend seam's `timed_step()` is replaced by an explicit timing probe + ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md) section 6.6): + `Backend.timing_probe(start)` loads a `solprop.backends.session.Start` and returns a + `solprop.backends.protocol.TimingProbe`, whose `trial(z, dz)` takes one trial step that + commits nothing and whose `field()` reads that trial's end state back on the host. + `Simulation.estimate` and the SEP 46 calibration time and compare trials through it, + instead of loading the backend themselves and reaching private driver members; the + calibration's plan comparison on the native backend now compares the trial fields, + where it read the unchanged launch field before. +- The node chunk is the backend's execution choice, planned when the backend is prepared, not part of the translated problem (SEP 47, #361): `Transform.node_chunk` is gone, and `assemble`, `multimode_transform`, `free_transform`, `planar_transform` and `radial_transform` no longer take `node_budget` (`free_transform` no longer takes the time grid, nor `radial_transform` `n_pol`). `SimulationSettings.node_budget` keeps its meaning and its chunks, is now an execution setting that may change after translation and stays outside the resume key, and reaches the backend through `ExecutionOptions(node_budget=...)` (`select(..., options=...)` or the backend's `options=`); an explicit budget is authoritative. `from_capillary`, `from_waveguide`, `prop_capillary` and `prop_waveguide` now default `node_budget` to `None`, the backend's choice (the same 64 MiB on these modal routes), so a later device default can apply to them; the general-guide refusal of `node_budget` now refuses any value other than `None`. A budget may be any integral value, a NumPy integer included, and one that is not a positive integer is refused when the settings are made (`DriverError`); a budget too small for one node is refused by the backend with a `BackendError`, no longer by the translation. The default budgets and the rule are in `solprop.backends.node_chunk` (`DEFAULT_NODE_BUDGET`, `DEFAULT_SPATIAL_NODE_BUDGET`, `plan_node_chunk`), and every run records its `node_chunk`, `node_budget` and `node_chunk_source` (`"default"`, `"explicit"` or `"forced"`) in its backend metadata. + +### Removed + +- The experimental `mlx` backend for Apple GPUs is retired ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md) §6.6): `backend="mlx"`, in a call or a recipe, raises `BackendError` and never runs on another backend, and output that records it still opens. The `solprop[mlx]` extra, the front end's MLX choice and the `metal` entry of the kernel's run options are gone; the front end shows a recipe's unavailable backend as it is named, and picking a binary64 backend for a single-precision recipe returns it to double precision. Single precision runs on `numpy_ref`, named explicitly; `precision="single"` with `backend="auto"` is refused. With MLX's transform costs gone, the `transforms` keyword of `FieldGrid.from_limits` and `EnvelopeGrid.from_limits` (a recipe grid that gives it is refused, naming SEP 47), the `library` argument of `solprop.grid.fft_cost`, `solprop.grid.TransformLibrary` and `TRANSFORM_LIBRARIES`, `solprop.backends.registry.metal_available` and `SINGLE_ONLY`, and the `mlx_*` and `single_precision_rows` sizes of `solprop.backends.memory` are removed. + +### Fixed + +- The boundary monitor's record counts each committed step once: a step left unchecked and then checked at its saved plane is no longer counted among both the checked (`steps`) and the skipped steps, so the two add up to the committed steps. Under interpolated planes, a plane that no check read since the previous one, as a later plane inside one step, records its collar and guard occupancies as unmeasured (`None`) rather than the previous collar reading and empty guards (#133). +- A native run on a Python thread other than the one that imported `solprop_native`, such as a front end's worker thread, now keeps to the thread budget: each native call sets that thread's OpenMP default width to the runtime's and restores it afterwards, so FFTW's threaded loops and an OpenMP OpenBLAS no longer take every CPU between narrower teams. Issue #130's reproduction ran 9 to 19 times slower on such a thread on two Intel Xeon Gold 6240, and 1.45 times slower on an Apple M5 Max, where it now matches the main thread (#130). + ## [0.2.0] - 2026-11-07 solprop 0.2.0 adds a desktop front end, describes every run as a recipe that can be diff --git a/changelog.d/598.changed.md b/changelog.d/598.changed.md deleted file mode 100644 index 3f4cd47c3..000000000 --- a/changelog.d/598.changed.md +++ /dev/null @@ -1 +0,0 @@ -The native backend's CPU transforms are faster on many threads ([#598](https://github.com/LupoLab/solprop/pull/598); [ADR-075](docs/dev/decisions.md)): a batch of transforms now threads by its shape, about one thread per row and the whole team only for many rows, instead of by its element count, and a multi-mode stage transforms its forward rows (projection, absorber, plasma currents) as one batch, as on a GPU (`SOLPROP_NATIVE_FORWARD_PACK=0` turns that off). The tracking cases ran 0.98 of their time on geometric mean on 18–36 cores of an Intel Xeon Gold 6240 and 16 of an AMD EPYC 7763, the few-mode capillaries and the one-dimensional free-space beam 0.80–0.95. `solprop calibrate` now also tunes the transforms' threading, reaches a decision on FFT plan efforts (it no longer tries `patient` plans, and skips efforts whose plans are slower alone), and takes about 30 s, two minutes or seven minutes (from 20 s, 90 s and five minutes). Existing calibration profiles no longer apply: the next native run on each machine calibrates quickly again by itself, or run `solprop calibrate`. diff --git a/changelog.d/apple-top-tier-budget.changed.md b/changelog.d/apple-top-tier-budget.changed.md deleted file mode 100644 index 54470f061..000000000 --- a/changelog.d/apple-top-tier-budget.changed.md +++ /dev/null @@ -1 +0,0 @@ -On a Mac whose chip has cores of more than one kind, the thread budget is now the cores of the fastest kind (`hw.perflevel0.physicalcpu`, 6 on an Apple M5 Max rather than all 18), unless `SOLPROP_THREADS` is set: every thread of a parallel pass waits for the slowest, and five of six benchmark cases ran 1.17 to 1.49 times faster on the 6 top-tier cores (#333, ADR-070). Set `SOLPROP_THREADS` to use every core; the run's `thread_budget_source` records `"top-tier cores"`. diff --git a/changelog.d/arm-band-filter.changed.md b/changelog.d/arm-band-filter.changed.md deleted file mode 100644 index c03053019..000000000 --- a/changelog.d/arm-band-filter.changed.md +++ /dev/null @@ -1 +0,0 @@ -On Arm and other non-x86-64 hosts, the native plasma column's band filter sums each sample's taps in one unrolled pass over blocks of samples, with the same values: on an Apple M5 Max the plasma cases' windows ran 1-6 % faster (#560). diff --git a/changelog.d/benchmark-ab-standard-calibration.changed.md b/changelog.d/benchmark-ab-standard-calibration.changed.md deleted file mode 100644 index a40303523..000000000 --- a/changelog.d/benchmark-ab-standard-calibration.changed.md +++ /dev/null @@ -1 +0,0 @@ -`python -m benchmarks ab` now calibrates each build at the `standard` level by default instead of `thorough`, so a development A/B no longer spends several minutes per arm calibrating; pass `--calibrate thorough` for release benchmarks. `track` keeps `thorough`. diff --git a/changelog.d/benchmark-backend-default.changed.md b/changelog.d/benchmark-backend-default.changed.md deleted file mode 100644 index 90ca26ed1..000000000 --- a/changelog.d/benchmark-backend-default.changed.md +++ /dev/null @@ -1 +0,0 @@ -The benchmark harness (`python -m benchmarks track` and `ab`) now runs workers with the backend's own OpenMP waiting by default (`--threading backend`), so that a row measures a user's default run; `--threading passive` keeps the previous policy. History rows record the policy, so rows taken from now on start a new series. diff --git a/changelog.d/benchmark-history-0-2-0.added.md b/changelog.d/benchmark-history-0-2-0.added.md deleted file mode 100644 index 80b66264d..000000000 --- a/changelog.d/benchmark-history-0-2-0.added.md +++ /dev/null @@ -1,2 +0,0 @@ -Performance-history rows for 0.2.0 (`benchmarks/history/`) on the same nine hardware sets -as 0.1.0. diff --git a/changelog.d/benchmark-sep47-matrix.added.md b/changelog.d/benchmark-sep47-matrix.added.md deleted file mode 100644 index 40be50e6c..000000000 --- a/changelog.d/benchmark-sep47-matrix.added.md +++ /dev/null @@ -1 +0,0 @@ -The benchmark harness has a `sep47` set of short windows for SEP 47's performance matrix: statistics rows at every step under `Multistep()` and `RungeKutta()`, snapshot controllers idle, sparse and at every endpoint, repeated `with_input` runs, a four-point scan on `Local` and the same points on `Batch(4)`, saved and field-free radial and Cartesian planes in memory and in HDF5, and coupled flows of 4 and 17 channels. `track --counters` and `ab --counters` add one separately reported run per worker with structural counters from the native backend, counted the same way whether the native execution session or the transitional driver session serves the run: its returns to Python by cause, native march calls, attempts, planes, statistics rows, transverse-edge reads, the fields copied to the host and loaded into the native state, and their bytes. diff --git a/changelog.d/blas-idle-children.changed.md b/changelog.d/blas-idle-children.changed.md deleted file mode 100644 index 7fb69c2cf..000000000 --- a/changelog.d/blas-idle-children.changed.md +++ /dev/null @@ -1 +0,0 @@ -Python processes solprop starts (a run from the window, directly or as a Slurm job, the window's kernel, and `Pool` workers) take `OPENBLAS_THREAD_TIMEOUT=20` unless it is set, so NumPy's and SciPy's OpenBLAS pools no longer spin on every CPU for a moment as they load: importing the driver used 13.8 CPU seconds on a 72-CPU Linux machine, and now 0.75 ([#383](https://github.com/LupoLab/solprop/issues/383)). The performance guide says how to set it for your own scripts. diff --git a/changelog.d/docs-stable-links.changed.md b/changelog.d/docs-stable-links.changed.md deleted file mode 100644 index 8e097075d..000000000 --- a/changelog.d/docs-stable-links.changed.md +++ /dev/null @@ -1,4 +0,0 @@ -The README, the package metadata, the desktop front end's Help and the native backend's -error messages now link the manual of the newest release (`/en/stable/`) rather than the -development version (`/en/latest/`), and the installation page's advice for a platform -without a native wheel now allows for the yanked 0.1.0. diff --git a/changelog.d/edge-read-backstop.changed.md b/changelog.d/edge-read-backstop.changed.md deleted file mode 100644 index 21d26def2..000000000 --- a/changelog.d/edge-read-backstop.changed.md +++ /dev/null @@ -1 +0,0 @@ -The transverse edge's read spacing has a growing backstop: `min(B, 4g/v_q)`, with `B` starting at `L/16` and doubling at each read whose reach has not grown by a tenth since it started, up to `L/4`. A free-space run whose spectrum holds far from its edge no longer takes extra steps for its reads. On the suite's `line` case the edge costs 2.9 % of the run where it cost 12.8 %, with 13 reads where it took 23 ([#396](https://github.com/LupoLab/solprop/issues/396)). A planar run also no longer evaluates its slab modes again at each run to set up the edge. On scaled versions of SEP 40's linear witnesses (414 packets that must stop) and on a self-focusing line beam in narrowed boxes (80 configurations), every run that stopped or warned under the former spacing still does. diff --git a/changelog.d/fftw-plans-kept.changed.md b/changelog.d/fftw-plans-kept.changed.md deleted file mode 100644 index a86261955..000000000 --- a/changelog.d/fftw-plans-kept.changed.md +++ /dev/null @@ -1 +0,0 @@ -The native backend now keeps the FFTW plans its runs measure by default, so each transform shape is measured once per machine and build instead of on every run. Plans are kept per transform in `wisdom.json` in the calibration store's directory for the machine and build, and a calibration profile's own plans take precedence. `SOLPROP_FFTW_WISDOM` now only moves this file, and the file it names no longer turns off a profile's FFT planning. `SOLPROP_CALIBRATION=off` keeps no default file, and `solprop calibrate --clear` removes the plans. Plans kept by earlier releases are not read; their files can be deleted. A name that cannot be used now logs a warning and the run plans without stored plans, where it used to raise. The store is keyed by the machine fingerprint, which tells Apple silicon models apart (#136). diff --git a/changelog.d/free-space-vector-propagators.changed.md b/changelog.d/free-space-vector-propagators.changed.md deleted file mode 100644 index 1a4fb1f30..000000000 --- a/changelog.d/free-space-vector-propagators.changed.md +++ /dev/null @@ -1 +0,0 @@ -The native backend forms free space's separable stage propagators a tile at a time through its vector maths (SLEEF on x86-64 and macOS arm64), rather than one libm `sincos` per element, which took 18 % of a one-thread `line` run. `line` takes 0.89 of its time at one thread and 0.93 at 16, and the Cartesian filament 0.93 at 16 ([#338](https://github.com/LupoLab/solprop/issues/338)). diff --git a/changelog.d/front-end-machines.added.md b/changelog.d/front-end-machines.added.md deleted file mode 100644 index e0a315255..000000000 --- a/changelog.d/front-end-machines.added.md +++ /dev/null @@ -1,6 +0,0 @@ -The desktop front end runs every simulation and scan as a process of its own, which outlives the window and is followed from its record, and runs them on other machines over SSH too: a -workstation, a Slurm cluster or a rented machine, picked on the Runs on line. Each run -keeps its record in its folder there, so it survives a closed window or a dropped network -and is found again on reconnecting; a Runs window lists every run on every machine, a -results file is analysed on its machine, and a machine can be set up with solprop from the -window. See [Running on other machines](docs/guide/front-end/machines.md). diff --git a/changelog.d/front-end-service.changed.md b/changelog.d/front-end-service.changed.md deleted file mode 100644 index 7ebf7424c..000000000 --- a/changelog.d/front-end-service.changed.md +++ /dev/null @@ -1,4 +0,0 @@ -The front end's service layer (its kernel process, run workers, live reductions and their -messages) moved from `solprop_gui.service` into solprop as `solprop.service`, so a machine -that only runs simulations for the desktop front end needs solprop alone, without Qt. It is -front-end support, not part of the user interface, and may change without notice. diff --git a/changelog.d/host-fixed-cost.changed.md b/changelog.d/host-fixed-cost.changed.md deleted file mode 100644 index f87740f3d..000000000 --- a/changelog.d/host-fixed-cost.changed.md +++ /dev/null @@ -1 +0,0 @@ -A run's fixed host cost is smaller ([#347](https://github.com/LupoLab/solprop/issues/347)): the first statistics no longer import `scipy.signal` (0.7 s), opening a run's output no longer serialises and digests its configuration a second time, and package versions are read once per process. A large free-space launch builds and translates faster, its column energies summed and its admittance copied in threads. diff --git a/changelog.d/monitor-step-count.fixed.md b/changelog.d/monitor-step-count.fixed.md deleted file mode 100644 index ba9c3e250..000000000 --- a/changelog.d/monitor-step-count.fixed.md +++ /dev/null @@ -1 +0,0 @@ -The boundary monitor's record counts each committed step once: a step left unchecked and then checked at its saved plane is no longer counted among both the checked (`steps`) and the skipped steps, so the two add up to the committed steps. Under interpolated planes, a plane that no check read since the previous one, as a later plane inside one step, records its collar and guard occupancies as unmeasured (`None`) rather than the previous collar reading and empty guards (#133). diff --git a/changelog.d/native-comparison-variants.changed.md b/changelog.d/native-comparison-variants.changed.md deleted file mode 100644 index 17d0459f2..000000000 --- a/changelog.d/native-comparison-variants.changed.md +++ /dev/null @@ -1 +0,0 @@ -The native backend refuses the benchmark-only Adams comparison variants off the selected family, by name and before it allocates anything: `ComparisonMultistep` with PECE, explicit Adams–Bashforth (`corrector="none"`) or a matched pair (`corrector_order = order`). They ran on solprop's Python march with a `HostMarchWarning`; they are now the NumPy reference's alone (SEP 47 section 6.3), so run them with `backend="numpy_ref"`. `ComparisonMultistep(corrector="pec", corrector_order=order + 1)`, the selected method, still runs natively. The native bridge's copy of the general Adams path is removed with them. diff --git a/changelog.d/native-short-spin.changed.md b/changelog.d/native-short-spin.changed.md deleted file mode 100644 index b5c423623..000000000 --- a/changelog.d/native-short-spin.changed.md +++ /dev/null @@ -1 +0,0 @@ -The native backend's idle OpenMP threads now spin briefly within native work instead of sleeping at once: `GOMP_SPINCOUNT=10000` on GNU's runtime and, on LLVM's (macOS and Windows), a 1 ms block time inside native calls only. Waking sleeping threads cost 16 to 72 µs for each of a step's many short parallel regions; benchmark windows ran in about 0.8 of the time on an Apple M5 Max, two Intel Xeon Gold 6240 and a shared AMD EPYC 7763 node, and in 0.87 of it on an AMD EPYC 9B45 under Windows (ADR-071). Teams wider than a chip's fastest cores keep sleeping at once. Set `OMP_WAIT_POLICY=passive` to restore the previous behaviour. diff --git a/changelog.d/native-statistics-plan.changed.md b/changelog.d/native-statistics-plan.changed.md deleted file mode 100644 index f7c3fee7a..000000000 --- a/changelog.d/native-statistics-plan.changed.md +++ /dev/null @@ -1 +0,0 @@ -The native backend evaluates and finishes its statistics natively from a plan lowered once (SEP 47 section 6.4): the half-maximum widths, the species reduction, the multi-channel electron density and peak ionisation rate, the plasma integration ratios, the profile statistics and a general modal basis's flux statistics (`omega0`, `physical_energy`, `spectral_flux`, `analytic_flux`, `fwhm_t`, `fwhm_t_outer`) no longer return to Python, and a general modal plane's statistics no longer copy the field to the host. Their z-dependent inputs are read from the problem's own tables. A profile statistic (`density`, `pressure`, `core_radius`) reads the propagation's table of its path where one reproduces the profile, or the profile tabulated once over the run; its measured error, at most 1e-13 of the profile's magnitude (about 1e-15 on a gas-filled capillary), is recorded in the run's execution record under `statistics_tables`. A modal power weight that varies in z is summed in a different order, a rounding-level change. The native execution session takes every periodic-capable statistic as a native periodic row. diff --git a/changelog.d/one-mode-accelerate.changed.md b/changelog.d/one-mode-accelerate.changed.md deleted file mode 100644 index ec11bc7f2..000000000 --- a/changelog.d/one-mode-accelerate.changed.md +++ /dev/null @@ -1 +0,0 @@ -Single-mode problems on the native backend now run their synthesis and projection on the response team with Apple's Accelerate too, as they already did with OpenBLAS: on an Apple M5 Max at six threads the argon blueshift window ran 0.96 of its time. diff --git a/changelog.d/parallel-rows-budget.changed.md b/changelog.d/parallel-rows-budget.changed.md deleted file mode 100644 index a2921f052..000000000 --- a/changelog.d/parallel-rows-budget.changed.md +++ /dev/null @@ -1 +0,0 @@ -Host passes too small to thread no longer read the CPU topology first, which took up to a sixth of a short window's host statistics on Linux (#556). diff --git a/changelog.d/python-thread-openmp-width.fixed.md b/changelog.d/python-thread-openmp-width.fixed.md deleted file mode 100644 index 4c4e7dcb8..000000000 --- a/changelog.d/python-thread-openmp-width.fixed.md +++ /dev/null @@ -1 +0,0 @@ -A native run on a Python thread other than the one that imported `solprop_native`, such as a front end's worker thread, now keeps to the thread budget: each native call sets that thread's OpenMP default width to the runtime's and restores it afterwards, so FFTW's threaded loops and an OpenMP OpenBLAS no longer take every CPU between narrower teams. Issue #130's reproduction ran 9 to 19 times slower on such a thread on two Intel Xeon Gold 6240, and 1.45 times slower on an Apple M5 Max, where it now matches the main thread (#130). diff --git a/changelog.d/radial-edge-gemm.changed.md b/changelog.d/radial-edge-gemm.changed.md deleted file mode 100644 index 1ebcef767..000000000 --- a/changelog.d/radial-edge-gemm.changed.md +++ /dev/null @@ -1 +0,0 @@ -The native backend's transverse-edge reads on radial grids synthesise the rows they use in one matrix product, once for every table, rather than one sum per sample per table: on an Apple M5 Max the 256- and 512-node radial filaments ran 0.83 and 0.75 of their one-thread windows, and 0.92 and 0.87 at six threads, with the same steps. diff --git a/changelog.d/saved-plane-copies.changed.md b/changelog.d/saved-plane-copies.changed.md deleted file mode 100644 index cd504e287..000000000 --- a/changelog.d/saved-plane-copies.changed.md +++ /dev/null @@ -1 +0,0 @@ -A saved plane makes fewer field-sized copies, with the same stored bits. A field the backend made for the observation alone (the native backend's host copy, the reference's widened single-precision state) is converted to true-power amplitudes where it is, so a modal plane on `native` makes two field-sized arrays where it made three, and a unit-weight one no longer copies the field at all before storing it. The reference's single-precision observation scales its widened state in place, one array fewer, and a plane observer and a failure checkpoint at a plane that kept no field now share one fetch of it. This partly addresses [#346](https://github.com/LupoLab/solprop/issues/346), as SEP 47's early copy elisions; a large spatial plane on `native` still makes its host copy and its stored copy. diff --git a/changelog.d/sep-0045-native-march.changed.md b/changelog.d/sep-0045-native-march.changed.md deleted file mode 100644 index 6bcb8f311..000000000 --- a/changelog.d/sep-0045-native-march.changed.md +++ /dev/null @@ -1 +0,0 @@ -The native backend's Adams march takes the boundary monitor's checks inside its compiled loop: a run whose monitor is busy, as when light lingers near the time window's edge, no longer returns to Python at every step. The monitor's record, its warning and a stop are unchanged, at the same step ([SEP 45](docs/dev/seps/archive/sep-0045-native-rk-march.md)). The split collar's crossing limit sums its bucket occupancies in one sequential order, so that the compiled march reproduces it exactly (Q7); in a near-tie its bucket can move one place, which changes a split-scheme run's steps at rounding level. The native backend also steps the adaptive Runge–Kutta loop of `RungeKutta()` in compiled code, from the stage tables it already holds, taking each trial's stage rows and its error norm natively: a run's metadata says which loop stepped it (`rk_march`), including interpolated planes, replayed schedules, coupled flows, z tables, the transverse edge and a weighted error norm, on CPU and GPU builds; a controller with its own methods runs solprop's loop with a `HostMarchWarning` naming the reason. The norm, weighted or not, now sums in one sequential order in both loops, which moves native Runge–Kutta runs once at rounding level. diff --git a/changelog.d/sep-0046-machine-calibration.added.md b/changelog.d/sep-0046-machine-calibration.added.md deleted file mode 100644 index 8ef828685..000000000 --- a/changelog.d/sep-0046-machine-calibration.added.md +++ /dev/null @@ -1 +0,0 @@ -`solprop calibrate` fits the native backend to the machine it runs on ([SEP 46](docs/dev/seps/archive/sep-0046-machine-calibration.md); [Calibrating solprop to your machine](docs/guide/performance/calibration.md)): in about 30 s (`--quick`), two minutes, or seven minutes (`--thorough`), child processes pinned to the job's CPUs measure how idle threads should wait, how wide the team should be, where each kind of operation should start to thread and which FFTW plan effort runs fastest inside the step, and a setting replaces the built-in value only if it wins every round by more than the measured noise, slows none of the problems (production-sized ones among them) and validates with the other winners. The profile is kept per user in the platform's cache directory (`SOLPROP_CACHE_DIR` moves it; `SOLPROP_CALIBRATION_PATH` adds shared read-only directories, such as a cluster's per node type), keyed by machine, build and allocation, and every later run applies it and records the settings it applied in its metadata (`calibration`). The first native run with no profile for its machine, build and allocation calibrates quickly by itself after a `CalibrationNotice`, and a scan calibrates its workers' allocation first (unless they run one thread each, which has nothing to choose); `SOLPROP_CALIBRATION=use` never measures and `off` ignores profiles. `Simulation.calibrate()` and `solprop calibrate --recipe` refine the settings for one simulation, reused by every run and scan point of the same execution. `--show`, `--export`, `--import` and `--clear` manage profiles. Benchmark series calibrate each build first and record the profile per row. Without a profile, every run uses exactly the former settings, now a built-in table keyed by operating system; `BuildInfo`'s threshold fields report that table for the process. diff --git a/changelog.d/sep-0046-scan-worker-wait.changed.md b/changelog.d/sep-0046-scan-worker-wait.changed.md deleted file mode 100644 index 099791373..000000000 --- a/changelog.d/sep-0046-scan-worker-wait.changed.md +++ /dev/null @@ -1 +0,0 @@ -A scan's workers ask for passive waiting instead of setting `OMP_WAIT_POLICY=passive` themselves, so that a user's own `OMP_WAIT_POLICY`, `GOMP_SPINCOUNT` or `KMP_BLOCKTIME` now reaches them, and so does a machine calibration's row for the workers' allocation ([SEP 46](docs/dev/seps/archive/sep-0046-machine-calibration.md)); without either they wait as before. Their recorded `openmp_wait_policy` reads `passive`. diff --git a/changelog.d/sep-0047-dense-snapshots.changed.md b/changelog.d/sep-0047-dense-snapshots.changed.md deleted file mode 100644 index 458c2342a..000000000 --- a/changelog.d/sep-0047-dense-snapshots.changed.md +++ /dev/null @@ -1 +0,0 @@ -Snapshots now work with interpolated planes (`interpolate_planes=True`), which they refused before: a requested snapshot is taken at the end of the step that crossed the plane being saved, once every plane that step crossed has been saved, or at the next accepted endpoint, so its running checkpoint is a state the march passed through, which `resume=True` continues without `resume="interpolated"`, saving each later plane once. The native execution session (SEP 47's prototype, selected by tests and the benchmark harness) now interpolates planes itself, for Adams and the embedded Runge-Kutta tableau alike, where the state lives: no stage or field crosses to the host but the exported plane. Its Runge-Kutta march evaluates the next step's first stage afresh after an observation, as the native bridge does, so the two take the same steps to the bit and their planes differ only at the last bits. A step plane owed by a step that crossed an interpolated plane and ended on the next one is no longer saved twice at that position ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md)). diff --git a/changelog.d/sep-0047-descriptor-contract.breaking.md b/changelog.d/sep-0047-descriptor-contract.breaking.md deleted file mode 100644 index 8e0bc1928..000000000 --- a/changelog.d/sep-0047-descriptor-contract.breaking.md +++ /dev/null @@ -1,21 +0,0 @@ -The backend seam's problem descriptor is now one trajectory's typed description -([SEP 47](docs/dev/seps/sep-0047-run-ownership.md) sections 6.2 and 6.7). -`solprop.translate.problem.KernelProblem` is renamed `PropagationProblem`, with no -alias. The nominal leading batch axis is gone: `PropagationProblem.shape`, -`initial.field0` and `initial.noise0`, `Backend.set_initial`, `Observation.field` and -`noise_phase`, `Backend.stats()`, the periodic rows of `StepStats` and -`ResponseBudgetError.field` are all `(n_modes, n_pol, n_omega)`-shaped with no batch -axis, `Initial.batch_params`, `Flags.single_batch` and `assemble(batch_params=...)` are -removed, `assemble` refuses a stacked field, and `RunEstimate.shape` has three entries. A -group of trajectories is a sequence of member problems, as a batched scan already runs -them; `solprop.driver.batch.split_observation` and `plane_from_observation(member=...)` -are removed. A march snapshot's stored state keeps its leading axis of one on disk, and -existing files stay readable. The route-specific optional fields `transform`, `flux`, -`transverse`, `tdse` and `current_numerator` move into a closed union, `problem.route`, -from the new module `solprop.translate.routes`: a `ModalRoute`, a `GeneralRoute` or a -`SpatialRoute` of kind radial, Cartesian, planar or line, each with the fields its -executor needs; the TDSE response is the modal route's reference-only `TDSESource`. -`problem.transform` still reads the route's transform, and `tdse_of`, `flux_of` and -`transverse_of` read a route's own leaves. Both backends refuse a route they do not -execute by name before they allocate. The public `Simulation` and `SimulationSettings` -interfaces are unchanged, and every result is bitwise the same. diff --git a/changelog.d/sep-0047-destination-export.changed.md b/changelog.d/sep-0047-destination-export.changed.md deleted file mode 100644 index b121962be..000000000 --- a/changelog.d/sep-0047-destination-export.changed.md +++ /dev/null @@ -1 +0,0 @@ -A kept plane's field is exported once, straight into its stored form, with the same stored bits ([#346](https://github.com/LupoLab/solprop/issues/346), SEP 47). `MemoryOutput` hands the run the block row the plane will occupy, so the field is written once, where it is kept, with no host volume of its own; `HDF5Output` is handed one new array, which h5py writes into the plane's chunk without a copy. A plane observer and a failure checkpoint at the same plane share that export. On a 108 MiB Cartesian plane the native session's saved plane no longer allocates a field into memory and one fewer into a file, and the driver session's file output converts its observation where it is instead of allocating a second field. `PlaneStore` gains `takes_exports`, `field_destination` and `save_exported` for this, with `solprop.output.schema.StoredField` beside them; a store subclass that overrides `save` is still given every plane, with its field, through it. Export destinations are refused by what is wrong with them (shape, dtype, layout or a read-only array). diff --git a/changelog.d/sep-0047-native-groups.changed.md b/changelog.d/sep-0047-native-groups.changed.md deleted file mode 100644 index 9e5278120..000000000 --- a/changelog.d/sep-0047-native-groups.changed.md +++ /dev/null @@ -1 +0,0 @@ -A `Batch` scan's group of native points now runs, on the CPU build, in the native group session: each member in its own native execution session, which steps it natively from one save plane to the next and keeps its own events, statistics rows, boundary monitor and exports, with the same fields, statistics and step records as before. An Adams group no longer warns `HostMarchWarning` for its replay. A member's monitor that refuses its launch now ends it after its launch plane is committed, as a run of the point alone does, and in a group-wide failure the members that had already taken their turn report the steps they committed, eligible for a checkpoint. A group the native group session does not run (a GPU build, a noise model, a feature the native session refuses) runs on the lockstep batch as before ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md)). diff --git a/changelog.d/sep-0047-native-low-storage.changed.md b/changelog.d/sep-0047-native-low-storage.changed.md deleted file mode 100644 index 7a0e40a98..000000000 --- a/changelog.d/sep-0047-native-low-storage.changed.md +++ /dev/null @@ -1 +0,0 @@ -The native execution session (SEP 47's prototype, selected by tests and the benchmark harness) now runs the low-storage replay (`LowStorageRungeKutta`) itself: each step is one in-place native step whose stage scalars the native state forms from its tables, so the replay no longer returns to Python once a step, with the same steps, statistics rows and fields as before. A failure partway through a step ends the session poisoned, with no checkpoint of its own, and the run's service loop records it against the latest running checkpoint ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md)). diff --git a/changelog.d/sep-0047-native-monitor.changed.md b/changelog.d/sep-0047-native-monitor.changed.md deleted file mode 100644 index 7d4c6fc93..000000000 --- a/changelog.d/sep-0047-native-monitor.changed.md +++ /dev/null @@ -1 +0,0 @@ -The native execution session (SEP 47's prototype, selected by tests and the benchmark harness) now runs a run's temporal boundary monitor itself: its launch judgement, cadence, quiet collar, split-collar cap, warnings, stops, plane rows, finish and resumed record are native, with the same decisions and record as solprop's monitor. A boundary warning reaches the caller as a `ServiceRequired` warning barrier, which the run's service loop emits; under warning filters that raise it, the run fails at that endpoint with its field available for the checkpoint ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md)). A free-space run's transverse edge now runs on the native session too, no longer refused: its launch judgement, read spacing, reads, a requested collar's current and return, warnings, stops, record and resumed record are native, with the same decisions and record as solprop's edge on the same readings, and a point that raises several warnings serves each as its own barrier. The safety ramp's share in its band at the launch and its removal over each read interval are summed natively in another order, within a rounding bound of solprop's values that accumulates across reads, so their warnings and stops may differ from solprop's for a share within that bound of a threshold. diff --git a/changelog.d/sep-0047-native-session-device.changed.md b/changelog.d/sep-0047-native-session-device.changed.md deleted file mode 100644 index 00bd45828..000000000 --- a/changelog.d/sep-0047-native-session-device.changed.md +++ /dev/null @@ -1 +0,0 @@ -Runs on a GPU build (CUDA or HIP) of the native backend now go through the native execution session when they use Adams or an embedded Runge-Kutta tableau on a modal route, with saved planes, periodic statistics rows, step planes, the temporal boundary monitor and resumed Adams history, with the same steps, statistics and fields as before ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md)). A saved plane's field crosses from the device once, into its destination, with no extra host copy. On a GPU build, spatial routes, the low-storage replay, dense output, a transverse edge, noise models and `Batch` groups still use the backend's own march. diff --git a/changelog.d/sep-0047-native-session-production.changed.md b/changelog.d/sep-0047-native-session-production.changed.md deleted file mode 100644 index 0436df953..000000000 --- a/changelog.d/sep-0047-native-session-production.changed.md +++ /dev/null @@ -1 +0,0 @@ -Runs on the native backend's CPU build now go through the native execution session whenever it admits the run ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md)), with the same steps, fields, statistics, step records and call ledgers as before. The session also runs resumed runs, carrying the Adams history and the call ledgers, the input and modified noise models, the boundary monitor (including a free-space run's transverse edge), dense output and the low-storage replay, and it exports the Adams history beside each saved plane and checkpoint. A signal during a run is still handled once the native call returns, within 256 attempts or 0.1 s; the run's checkpoint then includes every attempt made. Some runs still use the backend's own march, as before: comparison methods and methods other than Adams and the embedded Runge-Kutta tableaux; controllers that override `PIController`'s methods; TDSE responses; the input noise beside a transverse edge; a noise model on a propagator stored as a z table; marches the backend steps on the host (a coupled flow off its eigenbasis route or wider than 16 channels, or low-storage stage rows the host forms); and the plane statistics the host finishes, such as `beam_radius` and `fwhm_r`. A `Batch` scan's group refuses noise models in the native group session and runs them on the lockstep batch. diff --git a/changelog.d/sep-0047-reporter-cadence.changed.md b/changelog.d/sep-0047-reporter-cadence.changed.md deleted file mode 100644 index ee70cb62e..000000000 --- a/changelog.d/sep-0047-reporter-cadence.changed.md +++ /dev/null @@ -1 +0,0 @@ -A progress reporter now hears a run's attempted steps in batches, as the run's session delivers them: once per save plane, and at least every 256 attempts or 0.1 s in between. It no longer hears each step as it is taken ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md) section 5.3). Each attempt is still reported once and in order. As a result, a stop flag that a reporter sets lands up to one batch later, at most 256 attempts or 0.1 s, on every backend, the NumPy reference included. A flag set from outside the run is still polled between batches, as before. diff --git a/changelog.d/sep-0047-restart-record.changed.md b/changelog.d/sep-0047-restart-record.changed.md deleted file mode 100644 index a77df5456..000000000 --- a/changelog.d/sep-0047-restart-record.changed.md +++ /dev/null @@ -1 +0,0 @@ -Each execution record now states what a restart keeps and restores, under `outcome.execution["restart"]`: whether a plane's or checkpoint's method history was `retained`, `omitted` or is `not_applicable`, with a reason (`explicit_policy`, `automatic_policy`, `automatic_size_limit`, `snapshot_required`, `unavailable_history`, `output_cannot_store`, `write_failed`, `incompatible_representation`, `unsupported_method`, `no_method_history`), and in a resumed execution what it `restored`, `reset` (a Runge–Kutta resume's proposal and norm memory, an Adams history set aside) or found `not_applicable` (low-storage), and that the boundary monitor was reconstructed from its persisted records. A resume now warns once (`SolpropWarning`) when an Adams history it expected was omitted by the automatic 256 MiB limit, was not exportable, or cannot be read back (including a write cut short); `resume_history=False` is recorded without a warning. Files written before the record read as unknown, and resume as before ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md)). diff --git a/changelog.d/sep-0047-retire-mlx.removed.md b/changelog.d/sep-0047-retire-mlx.removed.md deleted file mode 100644 index 0d55d5d57..000000000 --- a/changelog.d/sep-0047-retire-mlx.removed.md +++ /dev/null @@ -1 +0,0 @@ -The experimental `mlx` backend for Apple GPUs is retired ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md) §6.6): `backend="mlx"`, in a call or a recipe, raises `BackendError` and never runs on another backend, and output that records it still opens. The `solprop[mlx]` extra, the front end's MLX choice and the `metal` entry of the kernel's run options are gone; the front end shows a recipe's unavailable backend as it is named, and picking a binary64 backend for a single-precision recipe returns it to double precision. Single precision runs on `numpy_ref`, named explicitly; `precision="single"` with `backend="auto"` is refused. With MLX's transform costs gone, the `transforms` keyword of `FieldGrid.from_limits` and `EnvelopeGrid.from_limits` (a recipe grid that gives it is refused, naming SEP 47), the `library` argument of `solprop.grid.fft_cost`, `solprop.grid.TransformLibrary` and `TRANSFORM_LIBRARIES`, `solprop.backends.registry.metal_available` and `SINGLE_ONLY`, and the `mlx_*` and `single_precision_rows` sizes of `solprop.backends.memory` are removed. diff --git a/changelog.d/sep-0047-run-request.breaking.md b/changelog.d/sep-0047-run-request.breaking.md deleted file mode 100644 index c87f5f2aa..000000000 --- a/changelog.d/sep-0047-run-request.breaking.md +++ /dev/null @@ -1,30 +0,0 @@ -The run's policy has left the backend seam's problem descriptor -([SEP 47](docs/dev/seps/sep-0047-run-ownership.md) sections 5.2 and 6.2). -`PropagationProblem` no longer has `stepper` or `stats_spec`: the tableau, controller, -step limits, schedule, save planes and statistics travel in the immutable -`solprop.translate.run_request.RunRequest`. `solprop.translate.problem.assemble` returns -a `Translation(problem, request)` pair (`problem, request = assemble(...)`), whose -request has the route's step limits resolved, and `RunRequest.stepper` reads the stepper -settings as one `StepperSettings` record. Backends receive the request where they are -built: `solprop.backends.registry.select(name, problem, request, ...)` and the backend -constructors take it after the problem, check the pair with the new -`solprop.translate.problem.validate_run(problem, request)` before they allocate, and -expose it as `Backend.request`. A run starts through the backend's preparation: -`Backend.prepare(problem, request)` returns a `PreparedExecution` whose `start(...)` -opens the backend's own session, the reference's `HostSession`, the native backend's -`NativeSession` where it admits the run and the transitional session otherwise, and -`solprop.backends.sessions.start_session` only starts it; a preparation refuses a -request that is not one run of the backend's own (`RunRequest.for_run` of it). -`PropagationProblem.validate()` now checks what the problem -promises alone; `validate_run` adds the checks that read the run's policy, with the same -refusals and messages, and `validate_launch(request)` takes the request. -`describe_run` now takes a `StepperSettings`; `RunRequest.for_run(...)` gives one run's -start, retention, monitor, history and snapshot policy, and `RunRequest.with_stepper(...)` -replaces the stepper section. `metadata_for(problem, request, grid, ...)`, -`field_retention(request, ...)`, `check_snapshot_request(request, ...)`, -`run_summary(problem, request, metadata, ...)` and -`check_derived_problem(constructed, derived, constructed_request, derived_request)` take -the request; `run_problem` reads it from the backend. The public `Simulation` and -`SimulationSettings` interfaces are unchanged, configuration hashes and resume identity -are unchanged (a file written before this change resumes), and every result is bitwise -the same. diff --git a/changelog.d/sep-0047-running-checkpoints.changed.md b/changelog.d/sep-0047-running-checkpoints.changed.md deleted file mode 100644 index 5c100c3cf..000000000 --- a/changelog.d/sep-0047-running-checkpoints.changed.md +++ /dev/null @@ -1 +0,0 @@ -Snapshots no longer require `step_planes=1`: a requested snapshot is taken at the next accepted endpoint, and where no plane is saved there the run writes a running checkpoint, in the existing terminal-checkpoint record with a matching `running` outcome, from which the snapshot and `resume=True` continue. The attempts and periodic rows since the last plane stay with that plane, once, and a later saved plane supersedes the checkpoint. A low-storage step abandoned in flight now falls back to the latest running checkpoint, or to the checkpoint the run resumed from, rather than to the last saved plane ([SEP 47](docs/dev/seps/sep-0047-run-ownership.md)). diff --git a/changelog.d/sep-0047-start-and-group.breaking.md b/changelog.d/sep-0047-start-and-group.breaking.md deleted file mode 100644 index f709235e5..000000000 --- a/changelog.d/sep-0047-start-and-group.breaking.md +++ /dev/null @@ -1,14 +0,0 @@ -A run now starts from an explicit start rather than from a backend its caller loaded -([SEP 47](docs/dev/seps/sep-0047-run-ownership.md) section 5.2): -`solprop.driver.run.run_problem` takes a required `start=`, a -`solprop.backends.session.Start` (`Start.entrance(problem)` for a fresh run), and loads -it into the backend through the run's session; its `start_plane=` argument is removed, -the start's `plane_index` taking its place. A caller no longer calls -`backend.set_initial` before `run_problem`. `Simulation.run`, `SimulationRunner`, scans -and the benchmark suite start their runs this way, and a resumed run's call ledgers -and Adams history travel in the start, a history the backend cannot install being -warned about and recorded as reset as before. A batched scan's points run as one group -through the group protocol (`solprop.backends.session.GroupSession`, SEP 47 section -6.7): each point keeps its own planes, statistics and outcome, a boundary stop still -fails its point alone, and a group-wide failure now reports each member's actual -position. diff --git a/changelog.d/sep-0047-timing-probe.breaking.md b/changelog.d/sep-0047-timing-probe.breaking.md deleted file mode 100644 index f844c3177..000000000 --- a/changelog.d/sep-0047-timing-probe.breaking.md +++ /dev/null @@ -1,9 +0,0 @@ -The backend seam's `timed_step()` is replaced by an explicit timing probe -([SEP 47](docs/dev/seps/sep-0047-run-ownership.md) section 6.6): -`Backend.timing_probe(start)` loads a `solprop.backends.session.Start` and returns a -`solprop.backends.protocol.TimingProbe`, whose `trial(z, dz)` takes one trial step that -commits nothing and whose `field()` reads that trial's end state back on the host. -`Simulation.estimate` and the SEP 46 calibration time and compare trials through it, -instead of loading the backend themselves and reaching private driver members; the -calibration's plan comparison on the native backend now compares the trial fields, -where it read the unchanged launch field before. diff --git a/changelog.d/sep47-backend-node-chunk.breaking.md b/changelog.d/sep47-backend-node-chunk.breaking.md deleted file mode 100644 index cdf61976f..000000000 --- a/changelog.d/sep47-backend-node-chunk.breaking.md +++ /dev/null @@ -1 +0,0 @@ -The node chunk is the backend's execution choice, planned when the backend is prepared, not part of the translated problem (SEP 47, #361): `Transform.node_chunk` is gone, and `assemble`, `multimode_transform`, `free_transform`, `planar_transform` and `radial_transform` no longer take `node_budget` (`free_transform` no longer takes the time grid, nor `radial_transform` `n_pol`). `SimulationSettings.node_budget` keeps its meaning and its chunks, is now an execution setting that may change after translation and stays outside the resume key, and reaches the backend through `ExecutionOptions(node_budget=...)` (`select(..., options=...)` or the backend's `options=`); an explicit budget is authoritative. `from_capillary`, `from_waveguide`, `prop_capillary` and `prop_waveguide` now default `node_budget` to `None`, the backend's choice (the same 64 MiB on these modal routes), so a later device default can apply to them; the general-guide refusal of `node_budget` now refuses any value other than `None`. A budget may be any integral value, a NumPy integer included, and one that is not a positive integer is refused when the settings are made (`DriverError`); a budget too small for one node is refused by the backend with a `BackendError`, no longer by the translation. The default budgets and the rule are in `solprop.backends.node_chunk` (`DEFAULT_NODE_BUDGET`, `DEFAULT_SPATIAL_NODE_BUDGET`, `plan_node_chunk`), and every run records its `node_chunk`, `node_budget` and `node_chunk_source` (`"default"`, `"explicit"` or `"forced"`) in its backend metadata. diff --git a/changelog.d/setup-repeated-work.changed.md b/changelog.d/setup-repeated-work.changed.md deleted file mode 100644 index e5741311d..000000000 --- a/changelog.d/setup-repeated-work.changed.md +++ /dev/null @@ -1 +0,0 @@ -Building a graded-index guide and run, and a large free-space launch, take less time: a mode solve evaluates its material once instead of once per finite element, a free-space launch forms each distinct column's admittance once, and neither repeats a table evaluation or a pass over the beam it already made. On the `grin_nonlinear` benchmark the build and translation go from 24 s to 16 s on one core, and on `kerr_beam_384` the launch's build from 12.6 s to 7.1 s. Every number of the translated problem is unchanged. diff --git a/changelog.d/sleef-macos.changed.md b/changelog.d/sleef-macos.changed.md deleted file mode 100644 index 3efb87f91..000000000 --- a/changelog.d/sleef-macos.changed.md +++ /dev/null @@ -1 +0,0 @@ -On macOS arm64 the native backend's CPU kernels now take SLEEF's 1-ulp AdvSIMD vector `exp`, `log` and `sincos` by default (`SOLPROP_NATIVE_VECTOR_MATH=AUTO`), as x86-64 builds do: on an Apple M5 Max the comb-to-continuum case ran 0.84 of its time at one thread, and the tracking cases 0.93 on geometric mean, with none slower beyond its spread (ADR-072, #339). Linux aarch64 keeps the system libm, where SLEEF measured level (1.00 on a Neoverse V2). diff --git a/changelog.d/spin-thresholds.changed.md b/changelog.d/spin-thresholds.changed.md deleted file mode 100644 index 5d93e4e20..000000000 --- a/changelog.d/spin-thresholds.changed.md +++ /dev/null @@ -1 +0,0 @@ -On Apple silicon, where the native backend's idle threads spin within native calls on the top-tier cores, it threads smaller passes: every size threshold is divided by 16, since a parallel region there costs about a microsecond and the cluster shares its L2 cache. On an Apple M5 Max at six threads the Raman GNLSE ran 0.70 of its time and the comb-to-continuum 0.91, none slower (ADR-073). Other platforms keep the previous thresholds, which measured faster there.