diff --git a/AGENTS.md b/AGENTS.md index 4c18260..61ad816 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ Test dependencies live in `test/Project.toml` (Julia workspace: `[workspace] pro | `src/distributions.jl` | `DNormal`, the diagonal normal prior on initial positions | | `src/gaussianpsf.jl` | `CircularGaussian` (2D), `CircularGaussianLorentzian` (3D), pixel integration through `erf` | | `src/pixeldetector.jl` | Code shared by pixel detectors: construction helpers, expected counts (dark counts + PSF × brightness), likelihood sums over unfiltered pixels | -| `src/spad.jl` | `SPAD`: binomial likelihood over `batchsize` binary frames, readout simulation; `SPADLogLikelihood`, the sparse likelihood workspace (evaluated at nonzero readouts plus a separable sum over pixels) | +| `src/spad.jl` | `SPAD`: binomial likelihood over `batchsize` binary frames, readout simulation; `SPADLogLikelihood`, the sparse likelihood workspace (stable hit/miss scores at nonzero readouts plus nonnegative zero-readout intensity sums) | | `src/emccd.jl` | `EMCCD` (Gaussian readout) and `EMCCDGamma` (gamma readout, not exported) | | `src/msd.jl` | `MeanSquaredDisplacement`, direct sampling from its inverse-gamma posterior (the optional rescaling move, which stretches the on tracks with the MSD, is in `src/tracks.jl`) | | `src/emittercount.jl` | `EmitterCount`, direct sampling of the number of emitters up to the weak limit | @@ -44,13 +44,13 @@ Test dependencies live in `test/Project.toml` (Julia workspace: `[workspace] pro - Array shapes: tracks are `(nframes, ndims, nparticles)`, with `ndims` 2 for `CircularGaussian` and 3 (z in column 3) for `CircularGaussianLorentzian`. `readouts` are `(nx, ny, nframes)` `UInt16`; `darkcounts` and `filter` are `(nx, ny)`. `active` is `(nframes, 1, nparticles)` (one entry shared by all coordinates, as in `examples/parametric`) or `(nframes, ndims, nparticles)`. Pixel bounds start at 0, so pixel `(i, j)` covers `[(i-1)·pxsize, i·pxsize] × [(j-1)·pxsize, j·pxsize]`. - Units are the caller's (the examples use μm and s). `msd` is the per-coordinate variance of a one-frame displacement, 2·D·period; the `Sample` field holding it is named `diffusivity`. `brightness` and `darkcounts` are expected photons per frame (rate × period); `brightness` is divided by `psf.A`, so it is the expected count in a pixel when the emitter sits at its center. - Everything is parametrized on `T<:AbstractFloat`. Public constructors are `Type{T}(; kwargs...)` and convert their inputs to `T` (`convert`, `elconvert`); methods expect one `T` across all arguments. -- One code path serves `Array` and `CuArray`: allocate with `similar(x, ...)`, avoid scalar indexing, prefer broadcasting and `batched_mul!`. `sum(vec(x))` instead of `sum(x)` is deliberate (faster with CUDA.jl). GPU-specific methods go in `ext/` as overloads of internal functions; `src/` never loads CUDA or Metal. Metal has no Float64, so code that runs on the GPU must not create Float64 values: write constants as `T(...)` and convert keyword inputs (e.g. `cutoffs`, `pixel_size`) to `T` (see `erfdiff`, `_init_pixel_detector_params`). Metal also cannot launch kernels over empty arrays (guard them, as `simulate!` does for an empty off chunk), and its `mul!` keeps a NaN or Inf already in the output, so zero an output buffer before `mul!` (as `framesum!` does). +- One code path serves `Array` and `CuArray`: allocate with `similar(x, ...)`, avoid scalar indexing, prefer broadcasting and `batched_mul!`. `sum(vec(x))` instead of `sum(x)` is deliberate (faster with CUDA.jl). GPU-specific methods go in `ext/` as overloads of internal functions; `src/` never loads CUDA or Metal. Metal has no Float64, so code that runs on the GPU must not create Float64 values: write constants as `T(...)` and convert keyword inputs (e.g. `cutoffs`, `pixel_size`) to `T` (see `erfdiff`, `_init_pixel_detector_params`). Metal also cannot launch kernels over empty arrays (guard them, as `simulate!` does for an empty off chunk), and its `mul!` keeps a NaN or Inf already in the output, so zero an output buffer before `mul!`. `framesum!` instead masks zero-weight inputs and uses a backend-native reduction. - Random draws take `rng::AbstractRNG` as their first argument, threaded down from `runMCMC`'s `rng` keyword. Array draws use `devicerng(rng, x)`, which is `rng` for CPU arrays and the GPU package's default generator for CUDA and Metal arrays (defined in `ext/`). The exported `simulate!` and `bridge!`, and `simulate_readouts!`, also have methods without `rng` (using `Random.default_rng()`), which the example scripts call. - Mutating functions end in `!` and return the mutated object. Buffers are preallocated: in the likelihood workspaces, `means[1]` is current and `means[2]` proposed; in `Tracks`, index 1 is current and 2 the proposal. - `runMCMC!` builds its likelihood workspace with `loglikelihoodaux(detector)`: `SPADLogLikelihood` for `SPAD` (expected counts only at nonzero readouts), the dense `LogLikelihoodArray` otherwise. The sweep reaches it only through `reset!`, `add_poisson_mean!`, `set_poisson_mean(s)!`, `get_loglikelihood!`, `set_frame_Δloglikelihood!` (and `.frame`) and `get_Δloglikelihood!`; a workspace implements these. The dense path also serves SPAD and is the reference the sparse one is tested against. -- `onchunk`, `offchunk` and `proposals` are views into the `Tracks` arrays; call `reassign!` after `nemitters.value` changes. `effvalue = value ./ active` is `Inf` for inactive frames, so they contribute no photons. `active` is a fixed mask; proposals share it (`active[2]` is a separate buffer kept equal to `active[1]`, reserved for inferring the mask later; see F1 in `MEMORY.md`). +- `onchunk`, `offchunk` and `proposals` are views into the `Tracks` arrays; call `reassign!` after `nemitters.value` changes. Inactive effective positions use infinite lateral coordinates and zero axial coordinates in 3D, so they contribute no photons without `0/0` or infinite PSF width. `active` is a fixed mask; proposals share it (`active[2]` is a separate buffer kept equal to `active[1]`, reserved for inferring the mask later; see F1 in `MEMORY.md`). - Names: `ᵒ` current, `ᵖ` proposed, `ᵥ` a value (`msdᵥ = msd.value`), `𝑇` temperature, `Δ` proposed minus current. Work in log space. Likelihood terms are divided by `𝑇`, priors are not. -- `RandomVariable` subtypes have `value`, `prior` and `fixed`; `update!` returns at once when `isfixed`. `parametric = true` or a fixed `tracks.nemitters` selects the parametric sweep, which never updates the emitter count. `logprior` of `MeanSquaredDisplacement` and `Brightness` is unnormalized unless `normalization = true`. The track and `DNormal` log-priors are always normalized, since their constants depend on the MSD. `log𝒫` is the log posterior of the whole weak-limit state, all `max_ntracks` candidates, so every sample's density has the same units and samples with different emitter counts compare: the off candidates enter through `offlogprior`, their expected log prior given the MSD (or an exact χ² draw of it with `logposterior_rng`), without simulating their tracks. Its density is in the coordinates (first positions, standardized increments `ε = displacement/√msd`, MSD), through `logjacobian`, so the tracks' prior volume does not depend on the MSD (`docs/logposterior.qmd` derives both constructions and their equivalence). The MSD is drawn from the on tracks only (the off tracks integrated out), then, with `msdrescale = true`, moved once more by the rescaling move, which stretches the on tracks with it; the nonparametric sweep redraws the off tracks before the next count update. +- `RandomVariable` subtypes have `value`, `prior` and `fixed`; `update!` returns at once when `isfixed`. `parametric = true` or a fixed `tracks.nemitters` selects the parametric sweep, which never updates the emitter count. `logprior` of `MeanSquaredDisplacement` and `Brightness` is unnormalized unless `normalization = true`. The track and `DNormal` log-priors are always normalized, since their constants depend on the MSD. `log𝒫` is a standardized-coordinate ranking score including the brightness prior. With off candidates it uses a log-prior proxy, not a marginalized density or the density of the stored full state: the off candidates enter through `offlogprior`, their expected log prior given the MSD (or an exact χ² draw of it with `logposterior_rng`), without simulating their tracks. Its density is in the coordinates (first positions, standardized increments `ε = displacement/√msd`, MSD), through `logjacobian`, so the tracks' prior volume does not depend on the MSD (`docs/logposterior.qmd` derives both constructions and their equivalence). The MSD is drawn from the on tracks only (the off tracks integrated out), then, with `msdrescale = true`, moved once more by the rescaling move, which stretches the on tracks with it; the nonparametric sweep redraws the off tracks before the next count update. - Derived properties come from `Base.getproperty` overloads (`chain.msds`, `detector.framecenter`); list new ones in `Base.propertynames` too, as `Chain` does. - A new detector subtypes `PixelDetector{T}` with fields `period`, `pxsize`, `pxbounds`, `darkcounts`, `filter`, `readouts`, has a keyword constructor built on `check_pixel_dimsmatch` and `_init_pixel_detector_params`, and implements `set_pixel_loglikelihood!`, `set_pixel_Δloglikelihood!` and `simulate_readouts!`. - The export list in `src/SP2T.jl` is the public API; tests reach internals as `SP2T.name`. @@ -75,7 +75,7 @@ Test dependencies live in `test/Project.toml` (Julia workspace: `[workspace] pro - Tracks and moves: `Tracks` with an `active` mask (both layouts); the identity-swap move (frame-by-frame relabeling; crossing particles with swapped tails are untangled, with their masks); a fixed emitter count is never updated and selects the parametric sweep; a parametric step leaves `effvalue` up to date; the rescaling and joint count–brightness moves run only with `msdrescale` and `countbrightness` (their proposal counters) - Chain: `EmitterCount.limit`; `Chain`'s `propertynames`; thinning (at most `sizelimit` samples, equal power-of-two spacing); `findmap`, `findmle` with `burn_in`; `jumpdistances` - `PolynomialAnnealing`: 𝑇₀ at the start, 1 from `I` on, decreasing in between - - Samplers (`test/samplers.jl`), seeded with `StableRNG`, against their target distributions from theory, with tolerances of 4 standard errors (batch means for Markov chains): `randc`; the MSD's tempered inverse-gamma posterior; the brightness and position Metropolis–Hastings chains, which sample their priors when the likelihood is switched off (𝑇 = Inf); the joint count–brightness move with the brightness update, which then samples the count and brightness priors; one rescaling MSD move applied to exact prior draws keeps the MSD's inverse gamma and the tracks' χ²; the identity-swap acceptance rate; `simulate!` and `bridge!` variances; each detector's `simulate_readouts!`; `offlogprior`'s expectation and exact draw against simulated off tracks; and that `runMCMC` is reproducible for a given seed, also with `logposterior_rng`, which leaves the chain unchanged + - Samplers (`test/samplers.jl`), seeded with `StableRNG`, against their target distributions from theory, with tolerances of 4 standard errors (batch means for Markov chains): `randc`; the MSD's inverse-gamma conditional, unchanged under likelihood-only tempering; the brightness and position Metropolis–Hastings chains, which sample their priors when the likelihood is switched off (𝑇 = Inf); the joint count–brightness move with the brightness update, which then samples the count and brightness priors; one rescaling MSD move applied to exact prior draws keeps the MSD's inverse gamma and the tracks' χ²; the identity-swap acceptance rate; `simulate!` and `bridge!` variances; each detector's `simulate_readouts!`; `offlogprior`'s expectation and exact draw against simulated off tracks; and that `runMCMC` is reproducible for a given seed, also with `logposterior_rng`, which leaves the chain unchanged - Test dependencies: Test, Distributions, StableRNGs, Metal and Random. No data files. Tests that draw random numbers pass a `StableRNG`. -- `test/metal.jl` runs on a Mac with a functional Metal GPU (skipped elsewhere): Metal arrays in private storage with scalar indexing disallowed, `erfdiff`, `framesum!` into a buffer holding NaN and Inf, the log-likelihood (dense and sparse workspaces) against the CPU, and parametric and nonparametric runs that are reproducible when both `rng` and `Metal.default_rng()` are seeded. CI has no GPU, so there `ext/` is untested, and CUDA is untested everywhere. +- `test/metal.jl` runs on a Mac with a functional Metal GPU (skipped elsewhere): Metal arrays in private storage with scalar indexing disallowed, `erfdiff`, `framesum!` into a buffer holding NaN and Inf, the log-likelihood (dense and sparse workspaces) against the CPU, and parametric and nonparametric runs that are reproducible when both `rng` and `Metal.default_rng()` are seeded. The same Metal entry point runs the Float32 detector and sparse SPAD regression suites. A hosted macOS job exercises them when Metal.functional() succeeds. Optional CUDA regressions live in `test/cuda.jl` and require a separate environment with the declared CUDA weak dependency and test dependencies; see `docs/verification.qmd`. - New sampler or fit tests: use a `StableRNG` seed or fixed data, and assert against a known ground truth with a tolerance derived from the noise, not tuned to current output (as `test/samplers.jl` does). diff --git a/docs/logposterior.qmd b/docs/logposterior.qmd index 37c524d..be7b582 100644 --- a/docs/logposterior.qmd +++ b/docs/logposterior.qmd @@ -8,7 +8,7 @@ format: html-math-method: mathjax --- -SP2T stores a log posterior, `log𝒫`, with every sample, and `findmap` picks the sample where it is largest. This note shows that the posterior can be built in two ways, from track positions or from standardized increments; that both give the same posterior distribution; and that their densities still differ at a point by a factor that depends on the MSD. `log𝒫` uses the second construction, which is why `get_logposterior` adds `logjacobian`. +SP2T stores a posterior ranking score, `log𝒫`, with every sample, and `findmap` picks its largest finite value after burn-in. This note shows that the posterior can be built in two ways, from track positions or from standardized increments; that both give the same posterior distribution; and that their densities still differ at a point by a factor that depends on the MSD. `log𝒫` uses the second coordinate convention, which is why `get_logposterior` adds `logjacobian`. With off candidates, its expected-prior substitution makes it a ranking score rather than the exact joint log density of a stored full state or a marginalized log density. # Setup @@ -139,6 +139,8 @@ $$ They differ by $\tfrac{(K-1)D}{2}\log v$, that candidate's share of the Jacobian, as the change of variables requires. With `logposterior_rng`, the expectation is replaced by an exact draw: the sum of squared standardized increments and first positions of the off tracks is $\chi^2$-distributed with $(M-n)\,K\,D$ degrees of freedom. +An expected log prior is not the log of an integrated prior, and an independent draw is not the density of the particular stored off-track coordinates. These substitutions do not change the sampler, but limit interpretation of `findmap` when candidates are off. With every candidate on, the score is the standardized-coordinate joint log posterior up to sample-independent constants. + # What SP2T computes $$ @@ -148,15 +150,16 @@ $$ + \underbrace{\tfrac{N}{2} \log v}_{\texttt{logjacobian}} + \underbrace{\log p(v)}_{\texttt{logprior(msd)}} + \underbrace{\lambda\, n}_{\texttt{logprior(nemitters)}} ++ \underbrace{\log p(b)}_{\texttt{logprior(brightness)}} $$ -That is $\log \pi_B$, up to a constant shared by all samples. The track and first-position terms are normalized, because their constants depend on $v$; $\log p(v)$ and $\log p(n)$ are not, since their constants are the same for every sample. Standardizing the first positions by $\sigma_d$ as well would only add a constant, so it is left out. The brightness prior is not included. +This is a score in construction B, with the off-candidate qualification above. The track and first-position terms are normalized, because their constants depend on $v$; omitted MSD/count prior normalizers are shared by all samples. Standardizing the first positions by $\sigma_d$ as well would only add a constant, so it is left out. Saved samples include the normalized brightness prior. The three-argument internal `get_logposterior(loglikelihood, tracks, msd)` helper remains a reduced score without brightness; the brightness-aware overload is used when saving samples. `test/runtests.jl` checks this against a direct evaluation in construction B with Distributions.jl, and that with the increments $\varepsilon$ and the log-likelihood fixed, changing $v$ changes `log𝒫` only by $\log p(v)$. # Example: the 2D example -Two states of the 2D example (`examples/2D`, 50 × 50 pixels, $K = 2550$, $D = 2$, $M = 10$, so $N = 50{,}980$): the ground truth, and the state the sampler stays in when started from the example's own values (tracks at zero, brightness 0.1, MSD 2 × 10⁻⁵). +The following historical calculation omits the brightness prior and predates the repaired sampler. It illustrates the coordinate Jacobian, not the current complete saved-score calculation or current sampler behavior. It compares two states of the 2D example (`examples/2D`, 50 × 50 pixels, $K = 2550$, $D = 2$, $M = 10$, so $N = 50{,}980$): the ground truth and an earlier stuck state. | | Ground truth | Stuck state | Truth − stuck | |---|---:|---:|---:| diff --git a/docs/verification.qmd b/docs/verification.qmd new file mode 100644 index 0000000..43cf1fe --- /dev/null +++ b/docs/verification.qmd @@ -0,0 +1,75 @@ +--- +title: "Regression verification" +format: html +--- + +Run the repository's CPU CI command with Julia 1.13: + +```julia +using Pkg +Pkg.test() +``` + +The suite includes detector, score, activity-mask, continuation, and identity-swap +regressions. The MSD reference uses likelihood-only tempering: its conditional is +`InverseGamma(alpha + n/2, beta + sum(displacement_squared)/2)` at every temperature. +The identity test enumerates a finite conditional target and checks both stationarity +and detailed balance, including the actual sampler's event mapping. Distributional +sampler tests use fixed seeds and theory-based tolerances. + +For SPAD readout `r` out of `B` binary trials, the pixel score (apart from its +binomial coefficient) is `r*log(1-exp(-lambda)) - (B-r)*lambda`. Evaluate the hit +term with stable `expm1`/`log1p` branches, and skip zero hit or miss counts before +multiplying. The dense and sparse paths are checked against high-precision binomial +probabilities at zero, near the smallest representable means, and at saturation. +The sparse workspace scores nonzero readouts directly and accumulates nonnegative +zero-readout intensity separately. The generic implementation reduces bounded +tiles. On CUDA, one block handles each frame, updates the clicked photon means, +and sums the intensity at the included zero readouts. Both computations preserve +small positive tails at saturation. Filtered pixels and padding are excluded +before evaluating a likelihood at an invalid excluded mean. + +On a Mac with a functional Metal GPU, the package tests also run the Float32 +detector and SPAD reference suites on private device arrays with scalar indexing +disabled. Metal allows a mathematically subnormal Float32 result to flush to +zero. The full-hit tests check against a high-precision count-weighted reference +and require a nonzero tail whenever that result is normal, even if the +individual hit contribution is subnormal. + +The identity move draws two indices independently and uniformly. Distinct indices +define a transposition shared across frame boundaries; equal indices give a no-op. +Only tracks with identical fixed activity histories may exchange identities. This +keeps the observation likelihood and fixed masks unchanged. For a shared involution, +accepted boundary flips reverse their energy differences and rejected-boundary +probabilities agree under reversal. Arbitrary shared permutations do not satisfy +this condition for multiple cuts. Whole-track shuffling also preserves fixed masks. + +For optional CUDA verification, use a separate Julia 1.13 environment containing this +checkout, its declared CUDA weak dependency, and Test, Distributions, StableRNGs, and +Random. Resolve the environment under Julia 1.13; do not reuse a manifest from an older +Julia version. Then run: + +```julia +include("test/cuda.jl") +``` + +This entry point requires a functional CUDA device, disables scalar indexing, and +runs the detector reference controls for Float32 and Float64. A CUDA-free CI runner +does not establish CUDA compatibility. Record the exact source, Julia, CUDA package, +toolkit, compiler, driver, device, and dependency manifest with validation results. +Retain compile/runtime errors separately from failed numerical assertions. + +`Chain.iteration` counts completed transitions independently of retained samples. +When reconstructing a chain from a separately saved live sampler state, supply its +actual iteration explicitly; retained samples cannot recover an unsaved final state. +Same-process continuation tests do not establish disk recovery or checkpoint durability. + +The saved score includes the brightness prior and uses standardized increments. +Off candidates retain the documented expected or independently drawn log-prior proxy; +this is not their marginalized density or the density of stored off coordinates. +See `logposterior.qmd` for the resulting interpretation of `findmap`. + +Passing deterministic regressions and finite-kernel checks does not by itself establish +posterior convergence, physical calibration, unknown-count recovery, or portability +to untested hardware and operating systems. Bind any additional posterior study to its +actual model, data, source snapshot, seeds, diagnostic gates, and environment. diff --git a/ext/SP2TCUDAExt.jl b/ext/SP2TCUDAExt.jl index 1c33119..4959f0a 100644 --- a/ext/SP2TCUDAExt.jl +++ b/ext/SP2TCUDAExt.jl @@ -7,5 +7,6 @@ SP2T.devicerng(::AbstractRNG, ::CuArray) = CUDA.default_rng() include("permutation.jl") include("track.jl") +include("spad.jl") end diff --git a/ext/SP2TMetalExt.jl b/ext/SP2TMetalExt.jl index 4c58c47..392a6fe 100644 --- a/ext/SP2TMetalExt.jl +++ b/ext/SP2TMetalExt.jl @@ -7,6 +7,30 @@ using Random: AbstractRNG SP2T.devicerng(::AbstractRNG, ::MtlArray) = Metal.default_rng() +function SP2T.addincident!( + intensity::MtlArray{T,3}, + psfcomponents::NTuple{2,<:AbstractArray{T,3}}, + relativepos::NTuple{2,<:AbstractArray{T,3}}, + tracksᵥ::AbstractArray{T,3}, + brightnessᵥ::T, + bounds::NTuple{2,<:AbstractVector{T}}, + psf::SP2T.GaussianPSF{T}, +) where {T<:AbstractFloat} + if iszero(brightnessᵥ) || iszero(size(tracksᵥ, 3)) || isempty(intensity) + return intensity + end + px, py = SP2T.integratepsf!(psfcomponents, relativepos, tracksᵥ, bounds, psf) + scale = brightnessᵥ / psf.A + # Update each calibrated pixel independently, including when a neighboring + # masked pixel holds a nonfinite background. + for m in axes(tracksᵥ, 3) + x = view(px, :, m:m, :) + y = PermutedDimsArray(view(py, :, m:m, :), (2, 1, 3)) + intensity .+= scale .* x .* y + end + return intensity +end + function SP2T.addΔlogprior₁!( ln𝓇::MtlVector{T}, x::MtlArray{T,3}, diff --git a/ext/spad.jl b/ext/spad.jl new file mode 100644 index 0000000..d48bd4c --- /dev/null +++ b/ext/spad.jl @@ -0,0 +1,84 @@ +# One block owns one frame. Updating clicked means and summing the intensity +# at zero readouts together avoids temporary gathers and per-tile launches. +const SPAD_WARPS_PER_BLOCK = 4 + +function SP2T.add_spad_signal!( + llarray::SP2T.SPADLogLikelihood{T}, detector::SPAD{T}, + px::CuArray{T,3}, py::CuArray{T,3}, scale::T, i::Integer, +) where {T<:Union{Float32,Float64}} + no_zeros = length(llarray.zeroruns) == 1 && only(llarray.zeroruns)[2] === :none + threads = 32 * SPAD_WARPS_PER_BLOCK + CUDA.@cuda threads=threads blocks=size(px, 3) add_spad_signal_kernel!( + llarray.means[i], llarray.linear[i], px, py, scale, + llarray.xindex, llarray.yindex, llarray.weights, detector.readouts, + llarray.filter, no_zeros, + ) + return llarray +end + +@inline function warp_sum(value) + for offset in (16, 8, 4, 2, 1) + value += CUDA.shfl_down_sync(0xffffffff, value, offset) + end + return value +end + +function add_spad_signal_kernel!( + means, linear, px, py, scale::T, xindex, yindex, weights, readouts, filter, no_zeros, +) where {T} + k = CUDA.blockIdx().x + thread = CUDA.threadIdx().x + width = CUDA.warpsize() + lane = (thread - Int32(1)) % width + Int32(1) + warp = (thread - Int32(1)) ÷ width + Int32(1) + @inbounds for h in thread:CUDA.blockDim().x:size(means, 1) + means[h, k] += scale * px[xindex[h, k]] * py[yindex[h, k]] + end + no_zeros && return + + nx, ny, _ = size(readouts) + if isnothing(filter) && iszero(@inbounds weights[1, k]) + # A frame with no clicks and no filter has separable zero-readout mass. + if warp == 1 + xsum, ysum = zero(T), zero(T) + @inbounds for x in lane:width:nx + xsum += px[x, 1, k] + end + @inbounds for y in lane:width:ny + ysum += py[y, 1, k] + end + mass = warp_sum(xsum) * warp_sum(ysum) + if lane == 1 + @inbounds linear[k] += SP2T._spad_scale_mass(scale, mass) + end + end + return + end + + # Sum only included zero readouts. In particular, subtracting the signal + # at clicks from the total could erase a small positive saturated tail. + mass = zero(T) + nwarps = CUDA.blockDim().x ÷ width + @inbounds for y in warp:nwarps:ny, x in lane:width:nx + if iszero(readouts[x, y, k]) + weight = isnothing(filter) ? one(T) : filter[x, y] + if !iszero(weight) + mass += weight * px[x, 1, k] * py[y, 1, k] + end + end + end + mass = warp_sum(mass) + partial = CUDA.CuStaticSharedArray(T, SPAD_WARPS_PER_BLOCK) + if lane == 1 + @inbounds partial[warp] = mass + end + CUDA.sync_threads() + if thread == 1 + total = zero(T) + @inbounds for w in 1:SPAD_WARPS_PER_BLOCK + total += partial[w] + end + @inbounds linear[k] += SP2T._spad_scale_mass(scale, total) + end + return +end diff --git a/src/base.jl b/src/base.jl index 975defb..2cf77f9 100644 --- a/src/base.jl +++ b/src/base.jl @@ -62,11 +62,16 @@ LogLikelihoodArray{T}(frames::AbstractArray) where {T<:AbstractFloat} = LogLikel """ framesum!(r::AbstractVector{T}, A::AbstractArray{T,3}, b::AbstractMatrix{T}) where {T} -Store in `r` the `b`-weighted sum over pixels of each frame of `A`; a more efficient way to implement `r .= vec(sum(A .* b, dims = (1, 2)))`. -""" -framesum!(r::AbstractVector{T}, A::AbstractArray{T,3}, b::AbstractMatrix{T}) where {T} = - # zeroed first: Metal's mul! computes 0 * r, which keeps a NaN or Inf already in `r` - mul!(fill!(r, zero(T)), transpose(reshape(A, length(b), :)), vec(b)) +Store in `r` the `b`-weighted sum over pixels of each frame of `A`. +Zero weights exclude their inputs, including `NaN` and `Inf`, before reduction. +Nonzero weights preserve nonfinite included values. Neither input is modified. +""" +function framesum!(r::AbstractVector{T}, A::AbstractArray{T,3}, b::AbstractMatrix{T}) where {T} + selected = ifelse.(iszero.(b), zero(T), A .* b) + # Reduction works on the array's backend without routing device pointers to BLAS. + sum!(reshape(r, 1, 1, :), selected) + return r +end """ devicerng(rng::AbstractRNG, x::AbstractArray) @@ -137,7 +142,8 @@ dimsmatch(A::AbstractArray, B::AbstractArray; dims::Union{Integer,AbstractUnitRa size(A)[dims] == size(B)[dims] Δlogpdf(P::Gamma{T}, xᵖ::T, xᵒ::T) where {T<:AbstractFloat} = - (shape(P) - 1) * log(xᵖ / xᵒ) - (xᵖ - xᵒ) / scale(P) + (isone(shape(P)) ? zero(T) : (shape(P) - 1) * log(xᵖ / xᵒ)) - + (xᵖ - xᵒ) / scale(P) Δlogpdf(P::InverseGamma{T}, xᵖ::T, xᵒ::T) where {T<:AbstractFloat} = -(shape(P) + 1) * log(xᵖ / xᵒ) - scale(P) * (1 / xᵖ - 1 / xᵒ) @@ -190,6 +196,7 @@ function boolcopyto!( src::AbstractArray{T,3}, i::AbstractVector{T}, ) where {T} - @. dest .+= i .* (src .- dest) + # Arithmetic masking contaminates rejected values and can overflow src - dest. + @. dest = ifelse(iszero(i), dest, src) return dest end diff --git a/src/brightness.jl b/src/brightness.jl index 36eb44b..f6a4c5a 100644 --- a/src/brightness.jl +++ b/src/brightness.jl @@ -36,7 +36,8 @@ Calculate the log density for `b`'s prior given its current value. `normalizatio """ logprior(b::Brightness{T,P}; normalization::Bool = false) where {T,P<:Gamma{T}} = normalization ? logpdf(b.prior, b.value) : - (shape(b.prior) - 1) * log(b.value) - b.value / scale(b.prior) + (isone(shape(b.prior)) ? zero(T) : (shape(b.prior) - 1) * log(b.value)) - + b.value / scale(b.prior) """ Δlogprior(b::Brightness{T}, xᵖ::T) diff --git a/src/chain.jl b/src/chain.jl index 002bbdf..3e7ea9f 100644 --- a/src/chain.jl +++ b/src/chain.jl @@ -4,7 +4,7 @@ struct Sample{T<:AbstractFloat,A<:AbstractArray{T,3}} brightness::T iteration::Int # iteration 𝑇::T # temperature - log𝒫::T # log posterior + log𝒫::T # standardized-coordinate score; off candidates use offlogprior logℒ::T # log likelihood end @@ -38,6 +38,17 @@ mutable struct Chain{T<:AbstractFloat,VofS<:Vector{<:Sample{T}},A<:Annealing{T}} samples::VofS sizelimit::Int annealing::A + iteration::Int # completed transitions, including states not retained after thinning +end + +# Saved samples do not identify an unsaved live state. Supply iteration explicitly +# when reconstructing a chain together with a separately preserved sampler state. +function Chain(samples::Vector{<:Sample}, sizelimit::Integer, annealing::Annealing; + iteration::Integer = isempty(samples) ? 0 : samples[end].iteration, +) + sizelimit >= 2 || throw(ArgumentError("sizelimit must be at least two")) + iteration >= 0 || throw(ArgumentError("iteration must be nonnegative")) + return Chain(samples, Int(sizelimit), annealing, Int(iteration)) end Base.propertynames(::Chain) = ( @@ -87,16 +98,34 @@ tosave(chain::Chain, iter::Real) = iter % savestride(chain) == 0 """ findmap(chain::Chain; burn_in::Integer = 0) -Return the sample with the largest log posterior (`chain.logposteriors`) after the first `burn_in` samples, and its index. The log posterior is a density, in the coordinates of `docs/logposterior.qmd` (first positions, standardized increments, MSD). Where a density is largest depends on the coordinates, and with thousands of track coordinates that point is not a typical posterior sample: use the result as a convenient point estimate, not as a summary of the posterior. +Return the sample with the largest finite saved score after the first `burn_in` +stored samples, and its index. The score uses first positions and standardized +increments. With off candidates it includes an expected or independently drawn +off-prior score, not their marginal density or the density of their stored +coordinates. Consequently this is a score-maximizing point estimate, not general +full-state MAP or a summary of the posterior distribution. """ function findmap(chain::Chain; burn_in::Integer = 0) - ~, i = findmax(chain.logposteriors[burn_in+1:end]) - return chain.samples[i+burn_in], i + burn_in + return _findbest(chain, :log𝒫, burn_in) end function findmle(chain::Chain; burn_in::Integer = 0) - ~, i = findmax(chain.loglikelihoods[burn_in+1:end]) - return chain.samples[i+burn_in], i + burn_in + return _findbest(chain, :logℒ, burn_in) +end + +function _findbest(chain::Chain, field::Symbol, burn_in::Integer) + 0 <= burn_in < length(chain) || throw(ArgumentError( + "burn_in must be nonnegative and smaller than the number of stored samples", + )) + best_index, best_score = 0, -Inf + for i in burn_in+1:length(chain) + score = getproperty(chain.samples[i], field) + if isfinite(score) && (best_index == 0 || score > best_score) + best_index, best_score = i, score + end + end + best_index > 0 || throw(ArgumentError("no finite $field scores remain after burn-in")) + return chain.samples[best_index], best_index end jumpdistances(tracks::AbstractArray{<:Real}) = sqrt.(sum(diff(tracks, dims=1) .^ 2, dims=2)) @@ -128,7 +157,7 @@ function extend!( logposterior_rng::Union{Nothing,AbstractRNG} = nothing, ) where {T} loglikelihood = get_loglikelihood!(llarray, tracks, brightness, detector, psf) - logposterior = get_logposterior(loglikelihood, tracks, msd, logposterior_rng) + logposterior = get_logposterior(loglikelihood, tracks, msd, brightness, logposterior_rng) push!( chain.samples, Sample(tracks.onchunk, msd, brightness, iter, 𝑇, logposterior, loglikelihood), @@ -148,10 +177,9 @@ function get_loglikelihood!( return get_loglikelihood!(llarray, detector) end -# The log posterior of the whole weak-limit state, all `limit` candidates, so every sample's -# density has the same units; the off candidates enter through `offlogprior`. Its density is -# in the coordinates (first positions, standardized increments, MSD) (`logjacobian`), where -# the tracks' prior does not depend on the MSD; see docs/logposterior.qmd. +# Reduced standardized-coordinate score, retained for callers that do not supply +# brightness. Off candidates contribute an expected or independently drawn +# log-prior proxy, not the marginalized density or their stored-state log density. get_logposterior( loglikelihood::T, tracks::Tracks{T}, @@ -165,6 +193,33 @@ get_logposterior( logprior(msd) + logprior(tracks.nemitters) +# Saved scores include the brightness prior as well. Keep the standardized- +# increment Jacobian and the declared off-candidate proxy from the current model. +get_logposterior( + loglikelihood::T, + tracks::Tracks{T}, + msd::MeanSquaredDisplacement{T}, + brightness::Brightness{T}, + logposterior_rng::Union{Nothing,AbstractRNG} = nothing, +) where {T} = get_logposterior(loglikelihood, tracks, msd, logposterior_rng) + + logprior(brightness; normalization = true) + +# GPU kernel launches can consume the host task RNG during deterministic scratch +# initialization. Rebuilding scratch at a continuation boundary must not consume +# proposal draws. This guard does not restore device RNGs or cover sampling/scoring. +function _continuation_likelihood_array(tracks::Tracks{T}, detector::Detector{T}) where {T} + host_rng = Random.default_rng() + saved_rng = copy(host_rng) + try + llarray = loglikelihoodaux(detector) + reset!(llarray, detector, 1) + tracks.nemitters.loglikelihood[1] = get_loglikelihood!(llarray, detector) + return llarray + finally + copy!(host_rng, saved_rng) + end +end + function parametricMCMC!( rng::AbstractRNG, tracks::Tracks{T}, @@ -253,11 +308,14 @@ Continue running a MCMC sampling with the specified keyword arguments. Requires - `niters`: The number of additional iterations for the MCMC simulation. - `parametric`: A boolean indicating whether to use parametric or non-parametric MCMC. A fixed emitter count (`tracks.nemitters.fixed`) also selects parametric MCMC. - `rng`: The random number generator (default: `Random.default_rng()`). On GPU, array draws use `CUDA.default_rng()` instead; seed it separately for reproducible GPU runs. -- `logposterior_rng`: The log posterior of each sample covers all `max_ntracks` candidates, so that samples with different emitter counts compare. The off candidates' tracks enter through their expected log prior (default: `nothing`), or, given a random number generator, through an exact draw of it, as if their tracks had been simulated. That generator is kept apart from `rng`, so the chain itself does not change. +- `logposterior_rng`: Saved scores include the brightness prior and use standardized increments. Off candidates contribute their expected log prior (default: `nothing`) or an independent draw of that log prior using this generator. This is an off-prior score proxy, not a marginalized density or the density of the stored off tracks. Use a generator distinct from `rng` to leave sampling unchanged. - `msdrescale`: Also run the rescaling MSD move after the MSD's Gibbs draw, which stretches the on tracks with the MSD (default: `false`; `docs/msdrescale.qmd`). It costs one likelihood evaluation per iteration. - `countbrightness`: Also run the joint count–brightness move after the emitter-count update, which switches one emitter on or off and rescales the brightness by n/n′ (default: `false`; `docs/countbrightness.qmd`). Nonparametric sweep only. Both moves leave the posterior unchanged; they change only how the chain moves. The chain's size limit and annealing schedule are those stored in `chain`. +`chain.iteration` counts completed transitions independently of retained samples. +Free brightness must be finite and strictly positive: its multiplicative proposal +cannot leave zero. Fixed zero brightness remains permitted. """ function runMCMC!(; chain::Chain{T}, @@ -273,10 +331,17 @@ function runMCMC!(; msdrescale::Bool = false, countbrightness::Bool = false, ) where {T} - prev_niters = chain.samples[end].iteration - llarray = loglikelihoodaux(detector) - reset!(llarray, detector, 1) - tracks.nemitters.loglikelihood[1] = get_loglikelihood!(llarray, detector) + niters >= 0 || throw(ArgumentError("niters must be nonnegative")) + isfixed(brightness) || (isfinite(brightness.value) && brightness.value > zero(T)) || + throw(ArgumentError( + "free brightness must be finite and positive; its multiplicative proposal cannot leave zero", + )) + prev_niters = chain.iteration + llarray = _continuation_likelihood_array(tracks, detector) + if isempty(chain.samples) + extend!(chain, tracks, msd, brightness, llarray, detector, psf, + prev_niters, temperature(chain, prev_niters), logposterior_rng) + end parametric = parametric || isfixed(tracks.nemitters) nextsample! = parametric ? parametricMCMC! : nonparametricMCMC! @showprogress dt = 1 desc = "Computing..." for iter in prev_niters .+ (1:niters) @@ -293,6 +358,7 @@ function runMCMC!(; msdrescale, countbrightness, ) + chain.iteration = iter if tosave(chain, iter) isfull(chain) && shrink!(chain) extend!( @@ -309,6 +375,7 @@ function runMCMC!(; ) end end + return chain end """ @@ -358,7 +425,7 @@ Run a MCMC sampling with the specified keyword arguments. Requires the following - `annealing`: An `Annealing` object representing the annealing schedule (default: `ConstantAnnealing{T}(1)`). - `parametric`: A boolean indicating whether to use parametric or non-parametric MCMC (default: `false`). A fixed emitter count (`tracks.nemitters.fixed`) also selects parametric MCMC. - `rng`: The random number generator (default: `Random.default_rng()`). On GPU, array draws use `CUDA.default_rng()` instead; seed it separately for reproducible GPU runs. -- `logposterior_rng`: The log posterior of each sample covers all `max_ntracks` candidates, so that samples with different emitter counts compare. The off candidates' tracks enter through their expected log prior (default: `nothing`), or, given a random number generator, through an exact draw of it, as if their tracks had been simulated. That generator is kept apart from `rng`, so the chain itself does not change. +- `logposterior_rng`: Saved scores include the brightness prior and use standardized increments. Off candidates contribute their expected log prior (default: `nothing`) or an independent draw of that log prior using this generator. This is an off-prior score proxy, not a marginalized density or the density of the stored off tracks. Use a generator distinct from `rng` to leave sampling unchanged. - `msdrescale`: Also run the rescaling MSD move after the MSD's Gibbs draw, which stretches the on tracks with the MSD (default: `false`; `docs/msdrescale.qmd`). It costs one likelihood evaluation per iteration. - `countbrightness`: Also run the joint count–brightness move after the emitter-count update, which switches one emitter on or off and rescales the brightness by n/n′ (default: `false`; `docs/countbrightness.qmd`). Nonparametric sweep only. Both moves leave the posterior unchanged; they change only how the chain moves. @@ -379,7 +446,8 @@ function runMCMC(; countbrightness::Bool = false, ) where {T} isnothing(annealing) && (annealing = ConstantAnnealing{T}(1)) - chain = Chain([Sample(tracks.onchunk, msd, brightness)], sizelimit, annealing) + sample_type = typeof(Sample(tracks.onchunk, msd, brightness)) + chain = Chain(sample_type[], sizelimit, annealing) runMCMC!( chain = chain, tracks = tracks, diff --git a/src/emccd.jl b/src/emccd.jl index 1971511..8502b67 100644 --- a/src/emccd.jl +++ b/src/emccd.jl @@ -60,15 +60,36 @@ function set_pixel_loglikelihood!( return llarray end +# Factor the residual-square difference before device arithmetic. This preserves +# exact identity increments despite device fused multiply-add contraction. +function _emccd_Δloglikelihood( + readout, + oldmean::T, + newmean::T, + gain::T, + offset::T, + variance::T, +) where {T} + if oldmean == newmean && isfinite(oldmean) && isfinite(gain) && + isfinite(offset) && isfinite(variance) && variance > zero(T) + return zero(T) + end + return gain * (newmean - oldmean) * + (2 * (T(readout) - offset) - gain * (newmean + oldmean)) / (2 * variance) +end + function set_pixel_Δloglikelihood!( llarray::LogLikelihoodArray{T}, detector::EMCCD{T}, ) where {T} - @. llarray.pixel = - ( - (detector.readouts - detector.gain * llarray.means[1] - detector.offset)^2 - - (detector.readouts - detector.gain * llarray.means[2] - detector.offset)^2 - ) / (2 * detector.variance) + @. llarray.pixel = _emccd_Δloglikelihood( + detector.readouts, + llarray.means[1], + llarray.means[2], + detector.gain, + detector.offset, + detector.variance, + ) return llarray end @@ -127,14 +148,24 @@ function EMCCDGamma{T}(; ) end -function set_pixel_loglikelihood!( - llarray::LogLikelihoodArray{T}, - detector::EMCCDGamma{T}, -) where {T} - llarray.means[2] .= llarray.means[1] ./ detector.noise_excess_factor - llarray.pixel .= - (llarray.means[2] .- 1) .* log.(detector.readouts) .- loggamma.(llarray.means[2]) .- - llarray.means[2] .* log(detector.noise_excess_factor * detector.gain) +function _gamma_pixel(weight::T, readout, mean::T, excess::T, logscale::T) where {T} + # Exclude before calling loggamma: a filtered negative calibration can have + # a shape outside the real log-gamma domain. + iszero(weight) && return zero(T) + k = mean / excess + return (k - one(T)) * log(T(readout)) - loggamma(k) - k * logscale +end + +function _gamma_pixel_delta(weight::T, readout, old::T, new::T, excess::T, logscale::T) where {T} + iszero(weight) && return zero(T) + ko, kn = old / excess, new / excess + return (kn - ko) * (log(T(readout)) - logscale) - (loggamma(kn) - loggamma(ko)) +end + +function set_pixel_loglikelihood!(llarray::LogLikelihoodArray{T}, detector::EMCCDGamma{T}) where {T} + logscale = log(detector.noise_excess_factor * detector.gain) + llarray.pixel .= _gamma_pixel.(detector.filter, detector.readouts, llarray.means[1], + detector.noise_excess_factor, logscale) return llarray end @@ -142,12 +173,9 @@ function set_pixel_Δloglikelihood!( Δllarray::LogLikelihoodArray{T}, detector::EMCCDGamma{T}, ) where {T} - Δllarray.means[1] ./= detector.noise_excess_factor - Δllarray.means[2] ./= detector.noise_excess_factor - @. Δllarray.pixel = - (Δllarray.means[2] .- Δllarray.means[1]) .* - (log.(detector.readouts) .- log(detector.noise_excess_factor * detector.gain)) .- - loggamma.(Δllarray.means[2]) .+ loggamma.(Δllarray.means[1]) + logscale = log(detector.noise_excess_factor * detector.gain) + Δllarray.pixel .= _gamma_pixel_delta.(detector.filter, detector.readouts, + Δllarray.means[1], Δllarray.means[2], detector.noise_excess_factor, logscale) return Δllarray end @@ -166,4 +194,4 @@ function simulate_readouts!( ) detector.readouts .= round.(UInt16, clamp.(floatreadouts, 0, typemax(UInt16))) return detector -end \ No newline at end of file +end diff --git a/src/msd.jl b/src/msd.jl index abc805b..a76da09 100644 --- a/src/msd.jl +++ b/src/msd.jl @@ -42,7 +42,9 @@ logprior(msd::MeanSquaredDisplacement; normalization::Bool = false) = """ update!(rng::AbstractRNG, msd::MeanSquaredDisplacement{T}, displacement²::AbstractArray{T,3}, 𝑇::T) -Directly sample `msd`'s new value from its posterior given the `displacement²` array and the temperature `𝑇`. +Directly sample `msd`'s new value from its posterior given `displacement²`. +Only the detector likelihood is tempered, so the motion-prior sufficient +statistics and this conditional do not depend on `𝑇`. """ function update!( rng::AbstractRNG, @@ -51,7 +53,7 @@ function update!( 𝑇::T, ) where {T} isfixed(msd) && return msd - Δparams = (length(displacement²), sum(vec(displacement²))) ./ (2 * 𝑇) #* sum(vec(displacement²))) as summing vec is currently faster using CUDA.jl + Δparams = (T(length(displacement²)) / 2, sum(vec(displacement²)) / 2) msd.value = _rand(rng, msd.prior, Δparams) return msd -end \ No newline at end of file +end diff --git a/src/permutation.jl b/src/permutation.jl index 21e3268..b0f1e44 100644 --- a/src/permutation.jl +++ b/src/permutation.jl @@ -33,11 +33,15 @@ end """ onshuffle!(rng::AbstractRNG, t::Tracks) -Shuffles the tracks of the emitting particles in `t`. +Shuffle the emitting tracks only when the proposed relabeling preserves their fixed +activity masks. Different fixed activity histories are not exchangeable. """ function onshuffle!(rng::AbstractRNG, t::Tracks) perm = randperm(rng, t.nemitters.value) - isequal(perm, 1:t.nemitters.value) || _permute!(t, perm) + if !isequal(perm, 1:t.nemitters.value) + @views all(t.onchunk.active .== t.onchunk.active[:, :, perm]) || return t + _permute!(t, perm) + end return t end @@ -81,7 +85,17 @@ end """ update!(rng::AbstractRNG, tracksᵒ::TrackChunk{T}, tracksᵖ::MHTrackChunk{T}, msdᵥ::T) -Swap particle identities partway through the tracks. Propose a random relabeling `perm` of the particles and, independently for each boundary between frames `k - 1` and `k`, accept relabeling frames `k` onward by `perm` according to the change in the motion log-prior across that boundary. Positions in each frame are unchanged, so the likelihood and the first-frame prior are too. Accepted relabelings compose: frames after `c` accepted boundaries are relabeled by `perm` applied `c` times, which leaves the steps across every other boundary unchanged. `active` is relabeled with the positions. +Swap two particle identities partway through their tracks. Draw two indices uniformly +with replacement; equal indices give a no-op. Otherwise use their transposition at +every boundary, with independent Metropolis decisions from the motion log-prior. +Only particles with identical fixed activity histories may be exchanged, so neither +the mask nor the image likelihood changes. The first frame is never relabeled. + +The shared permutation must be an involution. For a transposition, accepted boundary +flips commute and are their own inverses; rejected-boundary probabilities also match +in the reverse move. Arbitrary shared permutations do not have this property when +several boundaries are tested. With two particles the proposal law remains one-half +identity and one-half swap. """ function update!( rng::AbstractRNG, @@ -90,8 +104,12 @@ function update!( msdᵥ::T, ) where {T} nparticles = size(tracksᵒ.value, 3) - perm = randperm(rng, nparticles) - isequal(perm, 1:nparticles) && return tracksᵒ + (nparticles < 2 || size(tracksᵒ.value, 1) < 2) && return tracksᵒ + a, b = rand(rng, 1:nparticles), rand(rng, 1:nparticles) + a == b && return tracksᵒ + @views all(tracksᵒ.active[:, :, a:a] .== tracksᵒ.active[:, :, b:b]) || return tracksᵒ + perm = collect(1:nparticles) + perm[a], perm[b] = b, a initmh!(rng, tracksᵖ) permuteto!(tracksᵖ.value, tracksᵒ.value, perm, 2) setdisplacement²!(tracksᵒ) @@ -103,7 +121,6 @@ function update!( @views tracksᵖ.counter[:, 1] .+= count(accepted), length(accepted) - 1 any(accepted) || return tracksᵒ relabel!(tracksᵒ.value, relabelindices(tracksᵒ.value, perm, accepted), tracksᵖ.value) - relabel!(tracksᵒ.active, relabelindices(tracksᵒ.active, perm, accepted), tracksᵖ.active) - copyto!(tracksᵖ.active, tracksᵒ.active) # proposals share the current mask + copyto!(tracksᵖ.active, tracksᵒ.active) # the exchanged masks are identical return tracksᵒ end diff --git a/src/pixeldetector.jl b/src/pixeldetector.jl index 0832b93..333ce88 100644 --- a/src/pixeldetector.jl +++ b/src/pixeldetector.jl @@ -201,6 +201,9 @@ function set_poisson_means!( psf::PointSpreadFunction{T}, ) where {T} set_poisson_mean!(llarray, detector, tracksᵥ, brightnessᵥ₁, psf, 1) + if iszero(brightnessᵥ₁) + return set_poisson_mean!(llarray, detector, tracksᵥ, brightnessᵥ₂, psf, 2) + end llarray.means[2] .= detector.darkcounts .+ (llarray.means[1] .- detector.darkcounts) .* (brightnessᵥ₂ / brightnessᵥ₁) @@ -236,4 +239,4 @@ function getincident( bounds, psf, ) -end \ No newline at end of file +end diff --git a/src/spad.jl b/src/spad.jl index edd9b88..472df81 100644 --- a/src/spad.jl +++ b/src/spad.jl @@ -35,19 +35,63 @@ function SPAD{T}(; ) end -# Most SPAD readouts are 0, so `logexpm1` is evaluated only where it contributes: a real -# branch (not `ifelse`) is 5× faster on CPU and up to 3.5× on Metal. -clickterm(r::Integer, λ::T) where {T} = iszero(r) ? zero(T) : r * logexpm1(λ) -Δclickterm(r::Integer, λ₁::T, λ₂::T) where {T} = - iszero(r) ? zero(T) : r * (logexpm1(λ₂) - logexpm1(λ₁)) +# log(1-exp(-λ)): use expm1 at small λ and preserve the exponentially small +# negative tail at saturation. Never obtain that tail by subtracting λ from λ. +_spad_log_hit(λ::T) where {T} = + λ <= log(T(2)) ? log(-expm1(-λ)) : log1p(-exp(-λ)) + +function _spad_Δlog_hit(old::T, new::T) where {T} + old == new && old > zero(T) && return zero(T) + lo, hi = min(old, new), max(old, new) + lo <= zero(T) && return _spad_log_hit(new) - _spad_log_hit(old) + # p(hi)-p(lo) = exp(-lo) * (1-exp(-(hi-lo))). This also resolves + # close proposals for which subtraction of two log-hit scores loses bits. + ratio = exp(-lo) * (-expm1(-(hi - lo))) / (-expm1(-lo)) + d = isinf(ratio) ? _spad_log_hit(hi) - _spad_log_hit(lo) : log1p(ratio) + return new > old ? d : -d +end + +_spad_included_full(f::T, r, B, λ::T) where {T} = + iszero(f) ? zero(T) : _spad_loglikelihood(r, B, λ) +_spad_included_delta(f::T, r, B, old::T, new::T) where {T} = + iszero(f) ? zero(T) : _spad_Δloglikelihood(r, B, old, new) + +# At exp(-λ) < eps(T), -log(1-exp(-λ)) equals exp(-λ) to working +# precision. Scale by the count before forming the final product: the +# per-hit tail can be subnormal even when the count-weighted tail is normal. +function _spad_counted_tail(r::Integer, λ::T) where {T} + half = exp(-λ / T(2)) + return (T(r) * half) * half +end + +function _spad_loglikelihood(r::Integer, B::Integer, λ::T) where {T} + if r == B && !iszero(r) && λ > -log(eps(T)) + return -_spad_counted_tail(r, λ) + end + hits = iszero(r) ? zero(T) : T(r) * _spad_log_hit(λ) + misses = r == B ? zero(T) : (T(B) - T(r)) * λ + return hits - misses +end + +function _spad_Δloglikelihood(r::Integer, B::Integer, old::T, new::T) where {T} + lo, hi = min(old, new), max(old, new) + if r == B && !iszero(r) && lo > -log(eps(T)) + old == new && return zero(T) + delta = _spad_counted_tail(r, lo) * (-expm1(-(hi - lo))) + return new > old ? delta : -delta + end + hits = iszero(r) ? zero(T) : T(r) * _spad_Δlog_hit(old, new) + misses = r == B ? zero(T) : (T(B) - T(r)) * (new - old) + return hits - misses +end function set_pixel_loglikelihood!( llarray::LogLikelihoodArray{T}, detector::SPAD{T}, ) where {T} - @. llarray.pixel = - clickterm(detector.readouts, llarray.means[1]) - - detector.batchsize * llarray.means[1] + @. llarray.pixel = _spad_included_full( + detector.filter, detector.readouts, detector.batchsize, llarray.means[1], + ) return llarray end @@ -55,16 +99,24 @@ function set_pixel_Δloglikelihood!( Δllarray::LogLikelihoodArray{T}, detector::SPAD{T}, ) where {T} - @. Δllarray.pixel = - Δclickterm(detector.readouts, Δllarray.means[1], Δllarray.means[2]) - - detector.batchsize * (Δllarray.means[2] - Δllarray.means[1]) + @. Δllarray.pixel = _spad_included_delta( + detector.filter, detector.readouts, detector.batchsize, + Δllarray.means[1], Δllarray.means[2], + ) return Δllarray end """ SPADLogLikelihood{T} <: LogLikelihoodAux{T} -The SPAD log-likelihood, evaluated only where it is nonlinear. Per pixel it is `r·logexpm1(λ) − B·λ`. The first term vanishes where the readout `r` is 0, and the second, summed over a frame's pixels, separates: `Σ f·λ = Σ f·darkcounts + (b/A) Σₘ pxₘᵀ F pyₘ`, with `pxₘ`, `pyₘ` emitter `m`'s pixel-integrated PSF along x and y and `F` the filter. So `means[i]` holds the expected counts only at the clicks (nonzero, unfiltered readouts), and `linear[i]` each frame's `Σ f·(λ − darkcounts)`. The results equal those of `LogLikelihoodArray` up to rounding. +The SPAD log-likelihood, storing photon means only at nonzero, unfiltered readouts. +At those pixels use the stable hit/miss form `r·log(1-exp(-λ)) − (B-r)·λ`. +Zero-readout pixels contribute only `−B·λ`: `darksum[k]` and `linear[i][k]` +are the nonnegative filtered background and emitter signal sums **only over those +zero-readout pixels**. This avoids cancellation between two large terms at saturated +full hits. `zeroruns` identifies frames with no such pixels, all-zero readouts, or +a mixture. The generic reduction uses bounded spatial/time tiles. CUDA sums each +frame directly in one block while updating the clicked photon means. The clicks are padded to one column per frame, with as many rows as the busiest frame has clicks, so per-frame sums are column sums on CPUs and GPUs alike. Padding entries have readout 0, so they contribute nothing, and point at pixel (1, 1). `xindex` and `yindex` index a PSF component of size `(nx, 1, nframes)` and `(ny, 1, nframes)`. `filter` is `nothing` when no pixel is filtered out. """ @@ -82,37 +134,85 @@ struct SPADLogLikelihood{ xindex::I yindex::I filter::F - darksum::T # Σ f·darkcounts over a frame + darksum::V # filtered background mass at zero-readout pixels, per frame + zeroruns::Vector{Tuple{UnitRange{Int},Symbol}} means::NTuple{2,A} - linear::NTuple{2,V} + linear::NTuple{2,V} # emitter signal mass at zero-readout pixels, per frame pixel::A frame::V end +# Walk the included pixels once per frame, in column-major order. Only the +# spatial index of each click is needed to pack the padded workspace below. +function _spad_click_metadata(readouts, included, filter, darkcounts::Matrix{T}) where {T} + npixels, nframes = length(filter), size(readouts, 3) + background = zeros(Float64, npixels) + for p in included + background[p] = Float64(filter[p]) * Float64(darkcounts[p]) + end + clickpixels = Int[] + counts = Vector{Int}(undef, nframes) + darkzero = Vector{T}(undef, nframes) + zeroruns = Tuple{UnitRange{Int},Symbol}[] + for k in 1:nframes + offset = (k - 1) * npixels + start = length(clickpixels) + total = 0.0 + @inbounds for p in included + if iszero(readouts[offset + p]) + total += background[p] + else + push!(clickpixels, p) + end + end + n = length(clickpixels) - start + counts[k], darkzero[k] = n, T(total) + kind = n == length(included) ? :none : iszero(n) ? :all : :mixed + if !isempty(zeroruns) && last(zeroruns)[2] == kind + zeroruns[end] = (first(last(zeroruns)[1]):k, kind) + else + push!(zeroruns, (k:k, kind)) + end + end + return (; clickpixels, counts, darkzero, zeroruns) +end + function SPADLogLikelihood(detector::SPAD{T}) where {T} - # indexed on the CPU, once per run readouts, filter = Array(detector.readouts), Array(detector.filter) darkcounts = Array(detector.darkcounts) nx, ny, nframes = size(readouts) - clicks = [findall((readouts[:, :, k] .> 0) .& (filter .!= 0)) for k = 1:nframes] - nrows = max(1, maximum(length, clicks)) # Metal cannot launch kernels over empty arrays - x, y = ones(Int, nrows, nframes), ones(Int, nrows, nframes) + included = all(!iszero, filter) ? eachindex(filter) : findall(!iszero, vec(filter)) + metadata = _spad_click_metadata(readouts, included, filter, darkcounts) + nrows = max(1, maximum(metadata.counts)) # GPU kernels need a nonempty row dimension + x, y = Matrix{Int}(undef, nrows, nframes), Matrix{Int}(undef, nrows, nframes) r, w = zeros(UInt16, nrows, nframes), zeros(T, nrows, nframes) - for k = 1:nframes, (j, c) in enumerate(clicks[k]) - x[j, k], y[j, k] = Tuple(c) - r[j, k], w[j, k] = readouts[c, k], filter[c] + d = fill(darkcounts[1], nrows, nframes) + coords = CartesianIndices(filter) + firstclick = 1 + @inbounds for k in 1:nframes + xoffset, yoffset = (k - 1) * nx, (k - 1) * ny + for j in 1:metadata.counts[k] + p = metadata.clickpixels[firstclick + j - 1] + ix, iy = Tuple(coords[p]) + x[j, k], y[j, k] = xoffset + ix, yoffset + iy + r[j, k] = readouts[(k - 1) * length(filter) + p] + w[j, k], d[j, k] = filter[p], darkcounts[p] + end + for j in (metadata.counts[k] + 1):nrows + x[j, k], y[j, k] = xoffset + 1, yoffset + 1 + end + firstclick += metadata.counts[k] end - d = darkcounts[CartesianIndex.(x, y)] - frameoffsets = (0:nframes-1)' todevice(a) = copyto!(similar(detector.darkcounts, eltype(a), size(a)), a) return SPADLogLikelihood( todevice(r), todevice(w), todevice(d), - todevice(x .+ nx .* frameoffsets), - todevice(y .+ ny .* frameoffsets), + todevice(x), + todevice(y), all(isone, filter) ? nothing : detector.filter, - convert(T, sum(filter .* darkcounts)), + todevice(metadata.darkzero), + metadata.zeroruns, (todevice(d), todevice(d)), (similar(detector.darkcounts, nframes), similar(detector.darkcounts, nframes)), todevice(w), @@ -135,27 +235,42 @@ function reset!(llarray::SPADLogLikelihood{T}, ::SPAD{T}, i::Integer) where {T} return llarray end -# Add `scale · pxᵀ F py` of each frame to `linear`; with no pixel filtered out, F is all -# ones and this is `scale · Σpx · Σpy`. -addframesums!( - linear::AbstractVector{T}, - px::AbstractArray{T,3}, - py::AbstractArray{T,3}, - ::Nothing, - scale::T, -) where {T} = linear .+= scale .* vec(sum(px, dims = 1)) .* vec(sum(py, dims = 1)) - -function addframesums!( - linear::AbstractVector{T}, - px::AbstractArray{T,3}, - py::AbstractArray{T,3}, - filter::AbstractMatrix{T}, - scale::T, -) where {T} - Px, Py = reshape(px, size(px, 1), :), reshape(py, size(py, 1), :) - # zeroed first: Metal's mul! keeps a NaN or Inf already in the output - FPy = mul!(fill!(similar(Px), zero(T)), filter, Py) - return linear .+= scale .* vec(sum(Px .* FPy, dims = 1)) +_spad_zero_mass(r, f::T, x::T, y::T) where {T} = + iszero(f) || !iszero(r) ? zero(T) : f * x * y +_spad_scale_mass(scale::T, mass::T) where {T} = + iszero(mass) ? zero(T) : scale * mass + +function add_zero_framesums!(linear, px, py, detector, llarray, scale) + nx, ny = size(detector) + for (frames, kind) in llarray.zeroruns + kind == :none && continue + if kind == :all && isnothing(llarray.filter) + @views mass = vec(sum(px[:, :, frames], dims = 1)) .* + vec(sum(py[:, :, frames], dims = 1)) + @views linear[frames] .+= _spad_scale_mass.(scale, mass) + continue + end + # <= 128*8*16 scalar entries per product tile, independent of movie size. + # Unlike subtracting click signal from total signal, all terms are + # nonnegative and absent pixels are branched away before multiplication. + for k in first(frames):16:last(frames), j in 1:8:ny, i in 1:128:nx + K = k:min(k + 15, last(frames)); J = j:min(j + 7, ny); I = i:min(i + 127, nx) + @views reads = detector.readouts[I, J, K] + @views filter = detector.filter[I, J] + @views x = px[I, :, K] + @views y = reshape(py[J, :, K], 1, length(J), length(K)) + tile = _spad_zero_mass.(reads, filter, x, y) + mass = vec(sum(tile, dims = (1, 2))) + @views linear[K] .+= _spad_scale_mass.(scale, mass) + end + end + return linear +end + +function add_spad_signal!(llarray, detector, px, py, scale, i) + llarray.means[i] .+= scale .* px[llarray.xindex] .* py[llarray.yindex] + add_zero_framesums!(llarray.linear[i], px, py, detector, llarray, scale) + return llarray end function add_poisson_mean!( @@ -166,6 +281,7 @@ function add_poisson_mean!( psf::PointSpreadFunction{T}, i::Integer, ) where {T} + iszero(brightnessᵥ) && return llarray nframes = size(tracksᵥ, 1) psfcomponents = ( similar(tracksᵥ, size(detector, 1), 1, nframes), @@ -184,8 +300,7 @@ function add_poisson_mean!( detector.pxbounds, psf, ) - llarray.means[i] .+= scale .* px[llarray.xindex] .* py[llarray.yindex] - addframesums!(llarray.linear[i], px, py, llarray.filter, scale) + add_spad_signal!(llarray, detector, px, py, scale, i) end return llarray end @@ -199,17 +314,28 @@ function set_poisson_means!( psf::PointSpreadFunction{T}, ) where {T} set_poisson_mean!(llarray, detector, tracksᵥ, brightnessᵥ₁, psf, 1) + if iszero(brightnessᵥ₁) + return set_poisson_mean!(llarray, detector, tracksᵥ, brightnessᵥ₂, psf, 2) + end + iszero(brightnessᵥ₂) && return reset!(llarray, detector, 2) ratio = brightnessᵥ₂ / brightnessᵥ₁ llarray.means[2] .= llarray.darkcounts .+ (llarray.means[1] .- llarray.darkcounts) .* ratio - llarray.linear[2] .= llarray.linear[1] .* ratio + llarray.linear[2] .= _spad_scale_mass.(ratio, llarray.linear[1]) return llarray end +_spad_weighted_full(w::T, r, B, λ::T) where {T} = + iszero(w) ? zero(T) : w * _spad_loglikelihood(r, B, λ) +_spad_weighted_delta(w::T, r, B, old::T, new::T) where {T} = + iszero(w) ? zero(T) : w * _spad_Δloglikelihood(r, B, old, new) + function get_loglikelihood!(llarray::SPADLogLikelihood{T}, detector::SPAD{T}) where {T} - @. llarray.pixel = llarray.weights * clickterm(llarray.readouts, llarray.means[1]) + @. llarray.pixel = _spad_weighted_full(llarray.weights, llarray.readouts, detector.batchsize, llarray.means[1]) sum!(reshape(llarray.frame, 1, :), llarray.pixel) - llarray.frame .-= detector.batchsize .* (llarray.darksum .+ llarray.linear[1]) + if !iszero(detector.batchsize) + llarray.frame .-= detector.batchsize .* (llarray.darksum .+ llarray.linear[1]) + end return sum(llarray.frame) end @@ -217,11 +343,13 @@ function set_frame_Δloglikelihood!( llarray::SPADLogLikelihood{T}, detector::SPAD{T}, ) where {T} - @. llarray.pixel = - llarray.weights * - Δclickterm(llarray.readouts, llarray.means[1], llarray.means[2]) + @. llarray.pixel = _spad_weighted_delta( + llarray.weights, llarray.readouts, detector.batchsize, llarray.means[1], llarray.means[2], + ) sum!(reshape(llarray.frame, 1, :), llarray.pixel) - llarray.frame .-= detector.batchsize .* (llarray.linear[2] .- llarray.linear[1]) + if !iszero(detector.batchsize) + llarray.frame .-= detector.batchsize .* (llarray.linear[2] .- llarray.linear[1]) + end return llarray end @@ -239,4 +367,4 @@ function simulate_readouts!( detector.readouts .= rand.(Ref(rng), Binomial.(detector.batchsize, -expm1.(-means))) end return detector -end \ No newline at end of file +end diff --git a/src/tracks.jl b/src/tracks.jl index f369296..e30e4a3 100644 --- a/src/tracks.jl +++ b/src/tracks.jl @@ -74,13 +74,28 @@ setacceptance!(t::MHTrackChunk; start::Integer, step::Integer) = """ seteffvalue!(tracks::AbstractTrackChunk{T}) -Set the effective value of the track chunk. The effective value is calculated as `value / active`. Note that `effvalue == value` when `active` is 1, and `effvalue == Inf` (a particle is infinitely far away so cannot contribute any photons) when `active` is 0. Currently, `active` should ONLY be used in parametric runs. +Set the effective positions, preserving active coordinates. Inactive particles +have infinite lateral coordinates and zero axial coordinate in 3D, so they +contribute no photons without forming `0/0` or an infinite Lorentzian width. +Currently, `active` should ONLY be used in parametric runs. """ function seteffvalue!(tracks::AbstractTrackChunk{T}) where {T} - @. tracks.effvalue = tracks.value / tracks.active + _seteffvalue!(tracks.effvalue, tracks.value, tracks.active) return tracks end +function _seteffvalue!(effective::AbstractArray{T,3}, value, active) where {T} + isempty(effective) && return effective + inactive_position = T(Inf) + @. effective = ifelse(iszero(active), inactive_position, value) + if size(value, 2) == 3 + mask_column = size(active, 2) == 1 ? 1 : 3 + axial_active = @view active[:, mask_column:mask_column, :] + @views effective[:, 3:3, :] .= ifelse.(iszero.(axial_active), zero(T), value[:, 3:3, :]) + end + return effective +end + """ Tracks{T<:AbstractFloat, A<:AbstractArray{T,3}, NT<:EmitterCount{T}, TR<:TrackChunk{T}, MH<:MHTrackChunk{T}} @@ -140,7 +155,7 @@ end """ offlogprior([rng::AbstractRNG,] tracks::Tracks{T}, msdᵥ::T) -The off candidates' `logprior` without simulating their tracks: its expectation over tracks drawn from the prior given `msdᵥ` or, with `rng`, an exact draw of it. For such tracks, every displacement divided by `√msdᵥ` and every first position standardized by the `DNormal` prior is standard normal, so the sum of their squares is χ²-distributed with `nframes · ndims` degrees of freedom per track; the expectation replaces it by its mean. +The off candidates' `logprior` without simulating their tracks: its expectation over tracks drawn from the prior given `msdᵥ` or, with `rng`, an exact draw of it. For such tracks, every displacement divided by `√msdᵥ` and every first position standardized by the `DNormal` prior is standard normal, so the sum of their squares is χ²-distributed with `nframes · ndims` degrees of freedom per track; the expectation replaces it by its mean. This is a score proxy, not a marginalized log density or the log density of the particular off-track values stored in `tracks`. """ offlogprior(tracks::Tracks{T}, msdᵥ::T) where {T} = offlogprior(identity, tracks, msdᵥ) # E[χ²(k)] = k @@ -180,17 +195,24 @@ function Tracks{T}(; ) where {T,P} guess = elconvert(T, guess) nframes, ndims, nguesses = size(guess) + max_ntracks >= nguesses || throw(ArgumentError("max_ntracks must include every guessed track")) + if !isnothing(active) + size(active, 1) == nframes && size(active, 2) in (1, ndims) && + size(active, 3) == nguesses || throw(DimensionMismatch( + "active must match guessed frames and particles, with 1 or $ndims columns", + )) + all(x -> iszero(x) || isone(x), active) || throw(ArgumentError( + "active must contain only zero or one", + )) + size(active, 2) == 1 || all(active .== view(active, :, 1:1, :)) || + throw(ArgumentError("active must agree across coordinates of each particle")) + end value = copyto!(similar(guess, nframes, ndims, max_ntracks), guess) values = (value, similar(value)) scaling isa Real && (scaling = fill!(similar(value), convert(T, scaling))) active = if !isnothing(active) - dimsmatch(guess, active, dims = 1) && size(active, 2) in (1, ndims) || throw( - DimensionMismatch( - "active must have as many frames as guess and 1 or $ndims columns", - ), - ) _resize3(elconvert(T, active), max_ntracks, fill = one(T)) else fill!(similar(value, nframes, ndims, max_ntracks), true) @@ -227,7 +249,7 @@ function setdisplacement²!(tracks::Tracks, i::Integer = 1) end function seteffvalue!(tracks::Tracks, i::Integer = 1) - @. tracks.effvalue[i] = tracks.value[i] / tracks.active[i] + _seteffvalue!(tracks.effvalue[i], tracks.value[i], tracks.active[i]) return tracks end @@ -312,6 +334,8 @@ bridge!(x::AbstractArray{T,3}, msd::T, xend::AbstractArray{T,3}) where {T} = bridge!(Random.default_rng(), x, msd, xend) function propose!(rng::AbstractRNG, proposals::MHTrackChunk{T}, tracks::TrackChunk{T}) where {T} + # Activity is fixed conditioning data, not part of the position proposal. + copyto!(proposals.active, tracks.active) arw_propose!( rng, proposals.value, diff --git a/test/core_regressions.jl b/test/core_regressions.jl new file mode 100644 index 0000000..55d5368 --- /dev/null +++ b/test/core_regressions.jl @@ -0,0 +1,218 @@ +using Test, Random, StableRNGs, Distributions + +@testset "Likelihood-only tempering leaves the MSD conditional unchanged" begin + # Four independent Gaussian increments with sum of squares 30 give + # IG(alpha + 4/2, beta + 30/2) = IG(4,18), regardless of image temperature. + for F in (Float32, Float64), temperature in (F(1), F(2), F(Inf)) + displacement2 = reshape(F[1, 4, 9, 16], 2, 2, 1) + msd = MeanSquaredDisplacement{F}(guess = 1, priorparams = (2, 3)) + rng, reference_rng = StableRNG(901), StableRNG(901) + expected = InverseGamma(F(4), F(18)) + for _ = 1:8 + @test SP2T.update!(rng, msd, displacement2, temperature).value == + rand(reference_rng, expected) + end + end +end + +@testset "Gamma brightness boundary and variable brightness score" begin + for F in (Float32, Float64) + b = Brightness{F}(guess = 0, priorparams = (1, 3)) + @test iszero(SP2T.logprior(b)) + @test SP2T.logprior(b; normalization = true) == logpdf(Gamma(F(1), F(3)), F(0)) + @test SP2T.Δlogprior(b, F(2)) ≈ -F(2) / 3 + @test SP2T.Δlogprior(b, F(0)) == zero(F) + @test SP2T.logprior(Brightness{F}(guess = 0, priorparams = (2, 3))) == -Inf + @test SP2T.logprior(Brightness{F}(guess = 0, priorparams = (0.5, 3))) == Inf + end + + # Independent normal-coordinate reference: one actual on track and one off + # candidate represented by E[log prior], with standardized increments. + epsilon = [0.3 -0.2; -0.4 0.5] + first = [0.5, 0.8] + mu, sigma = [0.4, 0.6], [0.7, 0.9] + msd = MeanSquaredDisplacement{Float64}(guess = 0.2, priorparams = (2, 0.1)) + value = reshape(vcat(first', first' .+ sqrt(msd.value) .* cumsum(epsilon; dims = 1)), 3, 2, 1) + tracks = Tracks{Float64}(guess = value, prior = DNormal{Float64}(mu, sigma), + max_ntracks = 2, scaling = 0.01, logonprob = -2) + b = Brightness{Float64}(guess = 1.7, priorparams = (2, 3)) + likelihood = -7.5 + expected_off = -6 * (1 + log(2pi)) / 2 - sum(log, sigma) + expected = likelihood + sum(logpdf.(Normal(), epsilon)) + + sum(logpdf.(Normal.(mu, sigma), first)) + expected_off - + 3 * log(msd.value) - 0.1 / msd.value - 2 + logpdf(Gamma(2, 3), b.value) + actual = SP2T.get_logposterior(likelihood, tracks, msd, b) + @test actual ≈ expected + @test actual - SP2T.get_logposterior(likelihood, tracks, msd) ≈ + logpdf(Gamma(2, 3), b.value) + b.value = 2.4 + @test SP2T.get_logposterior(likelihood, tracks, msd, b) - actual ≈ + logpdf(Gamma(2, 3), 2.4) - logpdf(Gamma(2, 3), 1.7) +end + +@testset "Finite retained-score selection" begin + samples = [Sample(zeros(1, 2, 1), 1.0, 1.0, i - 1, 1.0, p, l) + for (i, (p, l)) in enumerate(zip( + [NaN, Inf, -Inf, -3.0, 2.0, 2.0], + [Inf, NaN, -Inf, 4.0, 1.0, 3.0]))] + chain = Chain(samples, 10, ConstantAnnealing{Float64}(1)) + @test findmap(chain)[2] == 5 + @test findmle(chain)[2] == 4 + @test findmap(chain; burn_in = 5)[2] == 6 + @test findmle(chain; burn_in = 4)[2] == 6 + @test_throws ArgumentError findmap(chain; burn_in = -1) + @test_throws ArgumentError findmle(chain; burn_in = length(chain)) + invalid = Chain(samples[1:3], 10, ConstantAnnealing{Float64}(1)) + @test_throws ArgumentError findmap(invalid) + @test_throws ArgumentError findmle(invalid) + @test chain.iteration == 5 + @test :iteration in propertynames(chain) + @test Chain(samples, 10, chain.annealing; iteration = 13).iteration == 13 +end + +@testset "Fixed activity contract and inactive PSF safety" begin + for F in (Float32, Float64), dims in (2, 3) + guess = zeros(F, 3, dims, 1) + prior = DNormal{F}(zeros(F, dims), ones(F, dims)) + make(active) = Tracks{F}(; guess, prior, max_ntracks = 2, + scaling = 0, logonprob = -1, active) + for columns in (1, dims) + active = ones(F, 3, columns, 1) + active[1, :, :] .= 0 + preserved = copy(active) + tracks = make(active) + SP2T.seteffvalue!(tracks.onchunk) + @test active == preserved + @test tracks.onchunk.active == preserved + @test tracks.proposals.active == tracks.onchunk.active + @test all(isone, tracks.active[1][:, :, 2:2]) + # A reused proposal buffer must not change the fixed activity history. + fill!(tracks.proposals.active, one(F)) + SP2T.propose!(StableRNG(503), tracks.proposals, tracks.onchunk) + @test tracks.proposals.active == preserved + SP2T.seteffvalue!(tracks.proposals) + @test tracks.proposals.effvalue == tracks.onchunk.effvalue # scaling is zero + @test all(isinf, tracks.onchunk.effvalue[1:1, 1:2, :]) + dims == 3 && @test all(iszero, tracks.onchunk.effvalue[1:1, 3:3, :]) + @test tracks.onchunk.effvalue[2:3, :, :] == guess[2:3, :, :] + end + for bad in (F(-1), F(0.5), F(2), F(NaN)) + @test_throws ArgumentError make(fill(bad, 3, 1, 1)) + end + @test_throws DimensionMismatch make(ones(F, 2, 1, 1)) + @test_throws DimensionMismatch make(ones(F, 3, 1, 2)) + @test_throws DimensionMismatch make(ones(F, 3, dims + 1, 1)) + inconsistent = ones(F, 3, dims, 1) + inconsistent[1, 1, 1] = 0 + @test_throws ArgumentError make(inconsistent) + + # All-off tracks must give background-only means, including positive + # axial coordinates that formerly produced an infinite PSF width. + for coordinate in (zero(F), F(0.7)) + tracks = Tracks{F}(guess = fill(coordinate, 3, dims, 1), prior = prior, + max_ntracks = 1, scaling = 0, logonprob = -1, active = zeros(F, 3, 1, 1)) + SP2T.seteffvalue!(tracks.onchunk) + detector = SPAD{F}(period = 0.1, pixel_size = 1, darkcounts = fill(F(0.1), 2, 2), + cutoffs = (0, Inf), readouts = ones(UInt16, 2, 2, 3), batchsize = 3) + psf = dims == 2 ? CircularGaussian{F}(0.4, 1) : + CircularGaussianLorentzian{F}(0.4, 0.8, 1) + dense = SP2T.LogLikelihoodArray{F}(detector.readouts) + sparse = SP2T.loglikelihoodaux(detector) + SP2T.set_poisson_mean!(dense, detector, tracks.onchunk.effvalue, F(2), psf) + SP2T.set_poisson_mean!(sparse, detector, tracks.onchunk.effvalue, F(2), psf) + @test dense.means[1] == repeat(reshape(detector.darkcounts, 2, 2, 1), 1, 1, 3) + @test sparse.means[1] == sparse.darkcounts + @test all(iszero, sparse.linear[1]) + end + end +end + +function core_regression_state(::Type{F}) where {F} + tracks = Tracks{F}(guess = reshape(F.(1:6) ./ 20, 3, 2, 1), + prior = DNormal{F}(fill(F(0.5), 2), fill(F(2), 2)), + max_ntracks = 1, scaling = F(0.02), logonprob = -1) + return (; tracks, + msd = MeanSquaredDisplacement{F}(guess = 0.15, priorparams = (2, 0.1)), + brightness = Brightness{F}(guess = 1.4, priorparams = (2, 3)), + detector = SPAD{F}(period = 0.1, pixel_size = 1, darkcounts = fill(F(0.1), 2, 2), + cutoffs = (0, Inf), readouts = ones(UInt16, 2, 2, 3), batchsize = 3), + psf = CircularGaussian{F}(0.4, 1)) +end + +@testset "Free brightness starts and fixed zero boundary" begin + for F in (Float32, Float64) + for invalid in (F(0), F(-1), F(Inf), F(NaN)) + state = core_regression_state(F) + state.brightness.value = invalid + rng, reference_rng = StableRNG(611), StableRNG(611) + @test_throws ArgumentError runMCMC(; state..., niters = 0, + sizelimit = 4, parametric = true, rng) + @test rand(rng, UInt64) == rand(reference_rng, UInt64) + + state = core_regression_state(F) + chain = runMCMC(; state..., niters = 0, sizelimit = 4, + parametric = true, rng = StableRNG(611)) + state.brightness.value = invalid + @test_throws ArgumentError runMCMC!(; chain, state..., niters = 1, + parametric = true, rng = StableRNG(612)) + @test chain.iteration == 0 + @test chain.iterations == [0] + end + fixed = merge(core_regression_state(F), ( + brightness = Brightness{F}(guess = 0, priorparams = (1, 3), fixed = true), + )) + chain = runMCMC(; fixed..., niters = 0, sizelimit = 4, + parametric = true, rng = StableRNG(613)) + @test only(chain.brightnesses) == zero(F) + @test isfinite(only(chain.loglikelihoods)) + @test isfinite(only(chain.logposteriors)) + end +end + +@testset "Initial scores and thinned same-process continuation" begin + for F in (Float32, Float64) + initial = core_regression_state(F) + zerochain = runMCMC(; initial..., niters = 0, sizelimit = 4, parametric = true, + rng = StableRNG(421), annealing = PolynomialAnnealing{F}(2, 26, 2)) + @test zerochain.iteration == 0 + @test zerochain.iterations == [0] + @test isfinite(only(zerochain.loglikelihoods)) + @test isfinite(only(zerochain.logposteriors)) + @test only(zerochain.samples).𝑇 == F(2) + @test findmap(zerochain)[2] == findmle(zerochain)[2] == 1 + + for limit in (4, 64), optional in (false, true) + function fit(split) + state = core_regression_state(F) + rng = StableRNG(421) + score_rng = StableRNG(422) + chain = runMCMC(; state..., niters = split ? 13 : 26, sizelimit = limit, + annealing = PolynomialAnnealing{F}(2, 26, 2), parametric = true, + rng, logposterior_rng = score_rng, msdrescale = optional) + if split + @test chain.iteration == 13 + limit == 4 && @test last(chain.iterations) < chain.iteration + runMCMC!(; chain, state..., niters = 13, parametric = true, + rng, logposterior_rng = score_rng, msdrescale = optional) + end + return chain, state, rng, score_rng + end + whole, a, arng, ascore = fit(false) + split, b, brng, bscore = fit(true) + @test whole.iteration == split.iteration == 26 + @test whole.iterations == split.iterations + @test whole.msds == split.msds + @test whole.brightnesses == split.brightnesses + @test whole.loglikelihoods == split.loglikelihoods + @test whole.logposteriors == split.logposteriors + @test [s.tracks for s in whole.samples] == [s.tracks for s in split.samples] + @test a.tracks.onchunk.value == b.tracks.onchunk.value + @test a.msd.value == b.msd.value + @test a.brightness.value == b.brightness.value + @test a.msd.counter == b.msd.counter + @test a.brightness.counter == b.brightness.counter + @test rand(arng, UInt64) == rand(brng, UInt64) + @test rand(ascore, UInt64) == rand(bscore, UInt64) + end + end +end diff --git a/test/cuda.jl b/test/cuda.jl new file mode 100644 index 0000000..d50bf7e --- /dev/null +++ b/test/cuda.jl @@ -0,0 +1,10 @@ +# Optional CUDA regression entry point. Run in an environment with the declared +# CUDA weak dependency and test dependencies installed; ordinary CPU CI is unchanged. +using SP2T, Test, Distributions, StableRNGs, Random, CUDA +CUDA.functional() || error("CUDA regression tests require a functional CUDA device") +CUDA.allowscalar(false) +include("detector_regressions.jl") +detector_regression_suite(CuArray) +include("spad_saturation_regressions.jl") +spad_saturation_regression_suite(CuArray) +CUDA.synchronize() diff --git a/test/detector_regressions.jl b/test/detector_regressions.jl new file mode 100644 index 0000000..11175d4 --- /dev/null +++ b/test/detector_regressions.jl @@ -0,0 +1,132 @@ +# References use scalar distribution densities or explicitly enumerated weighted sums. +# This same suite runs on CPU and, from cuda.jl, CUDA with scalar indexing disabled. +function detector_regression_suite(adapt = identity; precisions = (Float32, Float64)) + @testset "Detector regressions $(adapt) $(T)" for T in precisions + atol = T === Float32 ? 3e-5 : 2e-10 + rtol = T === Float32 ? 5e-5 : 5e-12 + weights = reshape(T[1, 0, 0.5, 0], 2, 2) + values = reshape(T[2, Inf, 6, NaN, -4, NaN, 8, -Inf], 2, 2, 2) + output = adapt(T[Inf, NaN]) + devicevalues = adapt(values) + SP2T.framesum!(output, devicevalues, adapt(weights)) + @test Array(output) == T[5, 0] + @test isequal(Array(devicevalues), values) + SP2T.framesum!(output, devicevalues, adapt(zeros(T, 2, 2))) + @test Array(output) == zeros(T, 2) + included = copy(values) + included[1, 1, 1] = T(Inf) + SP2T.framesum!(output, adapt(included), adapt(weights)) + @test isinf(Array(output)[1]) + + dest = adapt(reshape(T[1, 2, -floatmax(T)], 3, 1, 1)) + proposed = adapt(reshape(T[NaN, Inf, floatmax(T)], 3, 1, 1)) + SP2T.boolcopyto!(dest, proposed, adapt(T[0, 0, 1])) + @test vec(Array(dest)) == T[1, 2, floatmax(T)] + + counts = reshape(UInt16[1, 2, 3, 4], 2, 2, 1) + dark = reshape(T[0.1, 0.2, 0.3, 0.4], 2, 2) + common = ( + period = T(0.1), pixel_size = T(1), darkcounts = adapt(dark), + cutoffs = (0, Inf), readouts = adapt(counts), + ) + detector = SP2T.EMCCDGamma{T}(; common..., gain = T(2), noise_excess_factor = T(2)) + ll = SP2T.LogLikelihoodArray{T}(detector.readouts) + old = reshape(T[0.5, 1, 2, 4], 2, 2, 1) + new = reshape(T[0.7, 1.2, 1.8, 3.4], 2, 2, 1) + ll.means[1] .= adapt(old) + ll.means[2] .= adapt(new) + reference = sum(logpdf(Gamma(Float64(new[i]) / 2, 4), Float64(counts[i])) - + logpdf(Gamma(Float64(old[i]) / 2, 4), Float64(counts[i])) + for i in eachindex(old)) + delta = SP2T.get_Δloglikelihood!(ll, detector) + @test isapprox(delta, reference; atol, rtol) + @test isequal(SP2T.get_Δloglikelihood!(ll, detector), delta) + @test Array(ll.means[1]) == old + @test Array(ll.means[2]) == new + full = SP2T.get_loglikelihood!(ll, detector) + # The implementation drops the readout-only -y/scale term. + referencefull = sum(logpdf(Gamma(Float64(old[i]) / 2, 4), Float64(counts[i])) + + Float64(counts[i]) / 4 for i in eachindex(old)) + @test isapprox(full, referencefull; atol, rtol) + @test isequal(SP2T.get_loglikelihood!(ll, detector), full) + @test Array(ll.means[1]) == old + @test Array(ll.means[2]) == new + + # Excluded pixels must not enter loggamma, even when their shapes are invalid. + maskedgamma = SP2T.EMCCDGamma{T}(period = 1, pixel_size = 1, + darkcounts = adapt(reshape(T[-1, NaN, 0.3, 0.4], 2, 2)), cutoffs = (0, 10), + readouts = adapt(counts), gain = 2, noise_excess_factor = 2) + gold, gnew = copy(old), copy(new) + gold[1:2] .= T[-1, NaN] + gnew[1:2] .= T[NaN, -1] + ll.means[1] .= adapt(gold) + ll.means[2] .= adapt(gnew) + expectedmasked = sum(logpdf(Gamma(Float64(old[i]) / 2, 4), Float64(counts[i])) + + Float64(counts[i]) / 4 for i in 3:4) + expecteddelta = sum(logpdf(Gamma(Float64(new[i]) / 2, 4), Float64(counts[i])) - + logpdf(Gamma(Float64(old[i]) / 2, 4), Float64(counts[i])) for i in 3:4) + @test isapprox(SP2T.get_loglikelihood!(ll, maskedgamma), expectedmasked; atol, rtol) + @test isapprox(SP2T.get_Δloglikelihood!(ll, maskedgamma), expecteddelta; atol, rtol) + @test isequal(Array(ll.means[1]), gold) && isequal(Array(ll.means[2]), gnew) + + gaussian = EMCCD{T}(; common..., gain = T(2), offset = T(0.3), variance = T(1.2)) + gll = SP2T.LogLikelihoodArray{T}(gaussian.readouts) + gll.means[1] .= adapt(old) + gll.means[2] .= adapt(new) + SP2T.set_pixel_Δloglikelihood!(gll, gaussian) + gaussreference = [ + logpdf(Normal(2 * Float64(new[i]) + Float64(T(0.3)), sqrt(Float64(T(1.2)))), counts[i]) - + logpdf(Normal(2 * Float64(old[i]) + Float64(T(0.3)), sqrt(Float64(T(1.2)))), counts[i]) + for i in eachindex(old) + ] + @test isapprox(vec(Array(gll.pixel)), gaussreference; atol, rtol) + gll.means[2] .= gll.means[1] + SP2T.set_pixel_Δloglikelihood!(gll, gaussian) + @test all(iszero, Array(gll.pixel)) + @test isnan(SP2T._emccd_Δloglikelihood(1, T(NaN), T(NaN), T(2), T(0), T(1))) + + spad = SPAD{T}(; common..., batchsize = 4) + psf = CircularGaussian{T}(T(0.4), T(1)) + coords = adapt(reshape(T[0.3, 0.7], 1, 2, 1)) + # Direct Normal CDF integration is independent of the PSF buffer implementation. + massx = [cdf(Normal(0.3, 0.4), i) - cdf(Normal(0.3, 0.4), i - 1) for i in 1:2] + massy = [cdf(Normal(0.7, 0.4), i) - cdf(Normal(0.7, 0.4), i - 1) for i in 1:2] + for oldbrightness in (T(0), T(1.5)), newbrightness in (T(0), T(2)) + dense = SP2T.LogLikelihoodArray{T}(spad.readouts) + sparse = SP2T.loglikelihoodaux(spad) + expected = Float64.(dark) .+ Float64(newbrightness / psf.A) .* (massx * massy') + for aux in (dense, sparse) + SP2T.set_poisson_means!(aux, spad, coords, oldbrightness, newbrightness, psf) + @test all(isfinite, Array(aux.means[2])) + end + @test isapprox(Array(dense.means[2])[:, :, 1], expected; atol, rtol) + @test isapprox(SP2T.get_Δloglikelihood!(dense, spad), + SP2T.get_Δloglikelihood!(sparse, spad); atol, rtol) + end + + calibration = adapt(reshape(T[0, Inf, -1, 2], 2, 2)) + original = Array(calibration) + SPAD{T}(period = 0.1, pixel_size = 1, darkcounts = calibration, + cutoffs = (0, Inf), readouts = adapt(counts)) + @test isequal(Array(calibration), original) + + # Excluded calibration values must not enter the new sparse background sum. + filtered = SPAD{T}( + period = T(0.1), pixel_size = T(1), + darkcounts = adapt(reshape(T[NaN, Inf, 0.1, 0.2], 2, 2)), + cutoffs = (0, 10), readouts = adapt(ones(UInt16, 2, 2, 1)), batchsize = 4, + ) + referencefiltered = sum(logpdf(Binomial(4, -expm1(-Float64(d))), 1) - log(4) + for d in T[0.1, 0.2]) + for aux in (SP2T.LogLikelihoodArray{T}(filtered.readouts), SP2T.loglikelihoodaux(filtered)) + SP2T.reset!(aux, filtered, 1) + @test isapprox(SP2T.get_loglikelihood!(aux, filtered), referencefiltered; atol, rtol) + SP2T.set_poisson_mean!(aux, filtered, coords, T(2), psf) + expected = [Float64(T(0.1)) + 2 / Float64(psf.A) * massx[1] * massy[2], + Float64(T(0.2)) + 2 / Float64(psf.A) * massx[2] * massy[2]] + referencepositive = sum(logpdf(Binomial(4, -expm1(-d)), 1) - log(4) + for d in expected) + @test isapprox(SP2T.get_loglikelihood!(aux, filtered), referencepositive; atol, rtol) + end + end +end diff --git a/test/identity_regressions.jl b/test/identity_regressions.jl new file mode 100644 index 0000000..cde573c --- /dev/null +++ b/test/identity_regressions.jl @@ -0,0 +1,168 @@ +using Test, Random, StableRNGs + +# Deterministic draws select a proposed pair and a point inside each acceptance or +# rejection interval. This exercises the actual update!, not a replica of its map. +mutable struct IdentityFixtureRNG <: AbstractRNG + pair::NTuple{2,Int} + draw::Int + uniforms::Vector{Float64} +end + +function Random.rand(rng::IdentityFixtureRNG, range::UnitRange{Int}) + rng.draw += 1 + @assert rng.draw <= 2 + value = rng.pair[rng.draw] + @assert value in range + return value +end + +function Random.rand!(rng::IdentityFixtureRNG, x::AbstractArray{T}) where {T} + @assert length(x) == length(rng.uniforms) + copyto!(x, T.(rng.uniforms)) + return x +end + +struct IdentityShuffleRNG <: AbstractRNG + permutation::Vector{Int} +end +Random.randperm(rng::IdentityShuffleRNG, n::Integer) = + (@assert length(rng.permutation) == n; copy(rng.permutation)) + +function identity_permutations(n) + result = Vector{Vector{Int}}() + function visit!(prefix, unused) + if isempty(unused) + push!(result, copy(prefix)) + else + for value in unused + visit!([prefix; value], filter(!=(value), unused)) + end + end + end + visit!(Int[], collect(1:n)) + return result +end + +function identity_fixture(x; active = ones(size(x, 1), 1, size(x, 3))) + T = eltype(x) + return Tracks{T}( + guess = x, + prior = SP2T.DNormal{T}(zeros(T, size(x, 2)), ones(T, size(x, 2))), + max_ntracks = size(x, 3), + scaling = zero(T), + logonprob = -one(T), + active = T.(active), + ) +end + +# Independent reference: edit each accepted suffix in chronological order. This +# deliberately does not use relabelindices, cumulative powers, or package energies. +function identity_tail_swap(x, a, b, boundary) + result = copy(x) + result[boundary:end, :, a] = x[boundary:end, :, b] + result[boundary:end, :, b] = x[boundary:end, :, a] + return result +end +identity_motion_logweight(x) = -sum(abs2, diff(x, dims = 1)) / 2 + +@testset "Identity swap exact finite detailed balance" begin + # Dyadic coordinates make independent energy differences exactly representable. + # Fix frame1 labels; frame2 and frame3 each admit all 3! labelings: 36 states. + points = [0.0 1.0 3.0; 0.25 2.0 3.5; -0.5 1.25 4.0] + permutations = identity_permutations(3) + states = Array{Float64,3}[] + for p in permutations, q in permutations + x = zeros(3, 2, 3) + x[1, 1, :] = points[1, :] + x[2, 1, :] = points[2, p] + x[3, 1, :] = points[3, q] + push!(states, x) + end + lookup = Dict(Tuple(vec(x)) => i for (i, x) in enumerate(states)) + logweights = identity_motion_logweight.(states) + weights = exp.(logweights .- maximum(logweights)) + weights ./= sum(weights) + transition = zeros(36, 36) + checked_events = 0 + for (i, x) in enumerate(states), a = 1:3, b = 1:3 + deltas = [ + identity_motion_logweight(identity_tail_swap(x, a, b, k)) - + identity_motion_logweight(x) for k = 2:3 + ] + probabilities = min.(1.0, exp.(deltas)) + for mask = 0:3 + accepted = [!iszero(mask & (1 << (k - 1))) for k = 1:2] + probability = prod(accepted[k] ? probabilities[k] : 1 - probabilities[k] + for k = 1:2) / 9 + iszero(probability) && continue + expected = copy(x) + uniforms = [0.5; [accepted[k] ? probabilities[k] / 2 : + (1 + probabilities[k]) / 2 for k = 1:2]] + for k = 1:2 + accepted[k] && (expected = identity_tail_swap(expected, a, b, k + 1)) + end + tracks = identity_fixture(copy(x)) + rng = IdentityFixtureRNG((a, b), 0, uniforms) + SP2T.update!(rng, tracks.onchunk, tracks.proposals, 1.0) + @test tracks.onchunk.value == expected + @test tracks.onchunk.active == tracks.proposals.active == ones(3, 1, 3) + @test tracks.onchunk.value[1, :, :] == x[1, :, :] + transition[i, lookup[Tuple(vec(tracks.onchunk.value))]] += probability + checked_events += 1 + end + end + # These are roundoff bounds for a finite sum, not Monte Carlo tolerances. + @test checked_events > 324 + @test maximum(abs.(sum(transition, dims = 2) .- 1)) < 2e-14 + @test maximum(abs.(vec(weights' * transition) .- weights)) < 2e-14 + flux = weights .* transition + @test maximum(abs.(flux .- transpose(flux))) < 2e-14 +end + +@testset "Identity swap fixed activity and degenerate cases" begin + x = reshape(collect(1.0:18.0), 3, 2, 3) + # A fixed history cannot follow a proposed physical identity across a cut. + active = reshape([1., 0, 1, 0, 1, 1, 1, 1, 0], 3, 1, 3) + for a = 1:3, b = 1:3 + tracks = identity_fixture(copy(x); active) + SP2T.update!(IdentityFixtureRNG((a, b), 0, fill(0.5, 3)), + tracks.onchunk, tracks.proposals, 1.0) + @test tracks.onchunk.value == x + @test tracks.onchunk.active == tracks.proposals.active == active + @test all(iszero, tracks.proposals.counter) + end + for p in identity_permutations(3) + tracks = identity_fixture(copy(x); active) + SP2T.onshuffle!(IdentityShuffleRNG(p), tracks) + @test tracks.onchunk.value == x + @test tracks.onchunk.active == tracks.proposals.active == active + end + # A whole-track permutation within an equal-mask class remains permissible. + active[:, :, 2] = active[:, :, 1] + tracks = identity_fixture(copy(x); active) + SP2T.onshuffle!(IdentityShuffleRNG([2, 1, 3]), tracks) + @test tracks.onchunk.value == x[:, :, [2, 1, 3]] + @test tracks.onchunk.active == tracks.proposals.active == active + for T in (Float32, Float64), nframes in (1, 3), nparticles in (0, 1) + tracks = identity_fixture(zeros(T, nframes, 2, nparticles)) + SP2T.update!(IdentityFixtureRNG((1, 1), 0, fill(0.5, nframes)), + tracks.onchunk, tracks.proposals, one(T)) + @test size(tracks.onchunk.value) == (nframes, 2, nparticles) + @test all(iszero, tracks.proposals.counter) + end +end + +@testset "Two-emitter multi-cut swap with common blinking" begin + nframes = 10 + x = [d == 1 ? 0.1k : Float64(k <= 5 ? m - 1 : 2 - m) + for k = 1:nframes, d = 1:2, m = 1:2] + active = [Float64(!(k in (3, 8))) for k = 1:nframes, d = 1:1, m = 1:2] + tracks = identity_fixture(copy(x); active) + rng = StableRNG(2) + for _ = 1:50 + SP2T.update!(rng, tracks.onchunk, tracks.proposals, 1e-4) + end + @test all(m -> allequal(tracks.onchunk.value[:, 2, m]), 1:2) + @test tracks.onchunk.value[:, 1, :] == x[:, 1, :] + @test tracks.onchunk.active == tracks.proposals.active == active +end diff --git a/test/metal.jl b/test/metal.jl index b220faa..56029ec 100644 --- a/test/metal.jl +++ b/test/metal.jl @@ -14,6 +14,29 @@ else private(x::AbstractArray) = MtlArray{eltype(x),ndims(x),Metal.PrivateStorage}(x) + @testset "Metal Float32 detector and SPAD regressions" begin + detector_regression_suite(private; precisions = (Float32,)) + spad_saturation_regression_suite(private; precisions = (Float32,), allow_subnormal_zero = true) + Metal.synchronize() + end + + @testset "Inactive Metal track coordinates" begin + for dims in (2, 3), mask_width in (1, dims) + positions = reshape(Float32.(1:2*dims), 2, dims, 1) + mask = ones(Float32, 2, mask_width, 1) + mask[1, :, :] .= 0 + expected = copy(positions) + expected[1, 1:2, :] .= Inf32 + if dims == 3 + expected[1, 3, :] .= 0 + end + device_positions = private(positions) + effective = similar(device_positions) + SP2T._seteffvalue!(effective, device_positions, private(mask)) + @test Array(effective) == expected + end + end + @testset let x = Float32.(range(-6, 6, length = 101)) @test Array(SP2T.erfdiff.(private(x[1:end-1]), private(x[2:end]))) ≈ SP2T.erfdiff.(x[1:end-1], x[2:end]) rtol = 1e-5 diff --git a/test/runtests.jl b/test/runtests.jl index 6c3ed2c..6127471 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -255,12 +255,12 @@ end # With a small MSD, swapping back at boundary 5 → 6 is overwhelmingly favored and any # other swap overwhelmingly disfavored, so once a non-identity relabeling is proposed # (probability 1/2 per call; 1 - 2^-50 within 50 calls), the tracks are untangled and -# stay so. The particle on y = 0 is dark in frames 3 and 8; its mask must follow it. +# stay so. Both particles share the same fixed dark frames, so they are exchangeable. @testset let nframes = 10, line = [0.0, 1.0], swapped = [k <= 5 ? m : 3 - m for k = 1:nframes, m = 1:2], guess = [d == 1 ? 0.1k : line[swapped[k, m]] for k = 1:nframes, d = 1:2, m = 1:2], - dark = [swapped[k, m] == 1 && k in (3, 8) for k = 1:nframes, d = 1:1, m = 1:2], + dark = [k in (3, 8) for k = 1:nframes, d = 1:1, m = 1:2], tracks = Tracks{Float64}( guess = guess, prior = SP2T.DNormal{Float64}([0.5, 0.5], [1.0, 1.0]), @@ -279,7 +279,7 @@ end @test all(m -> allequal(value[:, 2, m]), 1:2) @test sort(value[1, 2, :]) == line for m = 1:2 - @test active[:, :, m] == (value[1, 2, m] == 0 ? [!(k in (3, 8)) for k = 1:nframes, d = 1:1] : trues(nframes, 1)) + @test active[:, :, m] == [!(k in (3, 8)) for k = 1:nframes, d = 1:1] end @test tracks.proposals.active == active @test tracks.proposals.counter[2, 1] > 0 @@ -646,4 +646,11 @@ end include("samplers.jl") +include("detector_regressions.jl") +detector_regression_suite() +include("spad_saturation_regressions.jl") +spad_saturation_regression_suite() +include("core_regressions.jl") +include("identity_regressions.jl") + Sys.isapple() && include("metal.jl") diff --git a/test/samplers.jl b/test/samplers.jl index bd9e207..1908ca3 100644 --- a/test/samplers.jl +++ b/test/samplers.jl @@ -19,13 +19,13 @@ end end end -# The MSD is drawn from its conjugate inverse-gamma posterior, with the motion likelihood -# tempered by 𝑇: shape α + n/(2𝑇), scale β + Σd²/(2𝑇). -@testset for 𝑇 in (1.0, 2.0) +# Only the observation likelihood is tempered. The conditional motion prior gives +# shape α + n/2 and scale β + Σd²/2, independently of the temperature. +@testset for 𝑇 in (1.0, 2.0, Inf) rng = StableRNG(12) displacement² = reshape([0.01 * i for i = 1:40], 10, 2, 2) msd = MeanSquaredDisplacement{Float64}(guess = 1.0, priorparams = (2, 0.1)) - posterior = InverseGamma(2 + 40 / (2𝑇), 0.1 + sum(displacement²) / (2𝑇)) + posterior = InverseGamma(2 + 40 / 2, 0.1 + sum(displacement²) / 2) ndraws = 20_000 draws = [SP2T.update!(rng, msd, displacement², 𝑇).value for _ = 1:ndraws] @test abs(mean(draws) - mean(posterior)) < 4 * std(posterior) / √ndraws diff --git a/test/spad_saturation_regressions.jl b/test/spad_saturation_regressions.jl new file mode 100644 index 0000000..0a5ddb4 --- /dev/null +++ b/test/spad_saturation_regressions.jl @@ -0,0 +1,206 @@ +using Test, Distributions + +# Independent binomial probability, not the implementation's log-space formula. +function spad_probability_reference(r, B, mean) + setprecision(BigFloat, 2048) do + λ = BigFloat(mean) + hit, miss = -expm1(-λ), exp(-λ) + log(hit^Int(r) * miss^(Int(B) - Int(r))) + end +end + +function spad_reference_match(actual, expected, ::Type{T}, mult; allow_subnormal_zero = false) where {T} + # Metal Shading Language §8.1 permits Float32 arithmetic to flush a + # subnormal result to zero. Classify the host-rounded BigFloat reference. + if allow_subnormal_zero && T === Float32 && issubnormal(expected) && iszero(actual) + return true + end + return actual == expected || (isfinite(actual) && isfinite(expected) && + isapprox(actual, expected; rtol = mult * eps(T), atol = 16 * nextfloat(zero(T)))) +end + +function spad_saturation_regression_suite(adapt = identity; precisions = (Float32, Float64), + allow_subnormal_zero = false) + @testset "SPAD finite, saturated, and close-mean references $(adapt) $(T)" for T in precisions + B = UInt16(8) + for r in UInt16.((0, 1, 4, 8)) + detector = SPAD{T}(period = 1, pixel_size = 1, darkcounts = adapt(ones(T, 1, 1)), + cutoffs = (0, Inf), readouts = adapt(fill(r, 1, 1, 1)), batchsize = B) + dense = SP2T.LogLikelihoodArray{T}(detector.readouts) + sparse = SP2T.loglikelihoodaux(detector) + # Supply the same one-pixel photon means to each workspace. In sparse + # storage the sole pixel is either a click or a zero-readout linear term. + fill!(sparse.darksum, 0) + function setmeans!(λ, i) + fill!(dense.means[i], λ) + fill!(sparse.means[i], iszero(r) ? zero(T) : λ) + fill!(sparse.linear[i], iszero(r) ? λ : zero(T)) + end + metal_full_hit = allow_subnormal_zero && T === Float32 && r == B + means = T[0, floatmin(T), 1e-8, 0.1, 1, 10, 100, 1e5, Inf] + # The second per-hit tail is subnormal, but both count-weighted + # results are normal. A flush-to-zero exception cannot apply. + tail_means = T === Float32 ? (T(80), T(88)) : (T(700), T(709)) + r == B && append!(means, tail_means) + for λ in means + setmeans!(λ, 1) + expected = T(spad_probability_reference(r, B, λ)) + for aux in (dense, sparse) + actual = SP2T.get_loglikelihood!(aux, detector) + @test spad_reference_match(actual, expected, T, 32; + allow_subnormal_zero = metal_full_hit) + end + end + pairs = [(T(1e-8), T(2e-8)), (T(0.1), T(0.2)), (T(10), T(11)), + (T(100), T(101)), (T(1e5), T(1e5 + 1)), + (T(0.7), nextfloat(T(0.7))), (nextfloat(T(0.7)), T(0.7))] + if r == B + append!(pairs, [(T(Inf), T(Inf)), (zero(T), one(T)), (one(T), zero(T))]) + append!(pairs, [(λ, λ + one(T)) for λ in tail_means]) + elseif iszero(r) + append!(pairs, [(zero(T), zero(T)), (zero(T), T(0.2))]) + end + for (old, new) in pairs + setmeans!(old, 1); setmeans!(new, 2) + expected = setprecision(BigFloat, 2048) do + T(spad_probability_reference(r, B, new) - spad_probability_reference(r, B, old)) + end + for aux in (dense, sparse) + actual = SP2T.get_Δloglikelihood!(aux, detector) + @test spad_reference_match(actual, expected, T, 64; + allow_subnormal_zero = metal_full_hit) + end + end + end + + # No Bernoulli trials: the unique zero count has probability one even + # at infinite intensity. In particular, do not evaluate 0*(Inf-Inf). + no_trials = SPAD{T}(period = 1, pixel_size = 1, darkcounts = adapt(ones(T, 1, 1)), + cutoffs = (0, Inf), readouts = adapt(zeros(UInt16, 1, 1, 1)), batchsize = 0) + dense0 = SP2T.LogLikelihoodArray{T}(no_trials.readouts) + sparse0 = SP2T.loglikelihoodaux(no_trials) + for i = 1:2 + fill!(dense0.means[i], T(Inf)); fill!(sparse0.linear[i], T(Inf)) + end + for aux in (dense0, sparse0) + @test iszero(SP2T.get_loglikelihood!(aux, no_trials)) + @test iszero(SP2T.get_Δloglikelihood!(aux, no_trials)) + end + + # All-zero, all-click and mixed frames; filtered negative/NaN entries + # include positive readouts. Padding points at the negative first pixel. + dark = T[-1 0.2; NaN 0.3; 0.4 0.6] + counts = cat(UInt16[2 0; 1 0; 0 0], UInt16[3 8; 4 2; 5 1], + UInt16[1 8; 2 0; 0 3], UInt16[2 0; 2 7; 0 0]; dims = 3) + d = SPAD{T}(period = 1, pixel_size = 1, darkcounts = adapt(dark), cutoffs = (0, Inf), + readouts = adapt(counts), batchsize = B) + coords = [T(dim == 1 ? 0.3 + 0.1k : 0.7 - 0.04k) for k = 1:4, dim = 1:2, m = 1:1] + psf = CircularGaussian{T}(T(0.45), one(T)) + σ = Float64(T(0.45)); A = (cdf(Normal(0, σ), 0.5) - cdf(Normal(0, σ), -0.5))^2 + signal = zeros(Float64, 3, 2, 4) + for k = 1:4, j = 1:2, i = 1:3 + px = Normal(Float64(coords[k, 1, 1]), σ); py = Normal(Float64(coords[k, 2, 1]), σ) + signal[i, j, k] = (cdf(px, i) - cdf(px, i - 1)) * + (cdf(py, j) - cdf(py, j - 1)) / A + end + valid = isfinite.(dark) .& (dark .> 0) + function framescores(b) + [sum(Float64(spad_probability_reference(counts[i, j, k], B, + Float64(dark[i, j]) + Float64(b) * signal[i, j, k])) + for j = 1:2, i = 1:3 if valid[i, j]) for k = 1:4] + end + dense = SP2T.LogLikelihoodArray{T}(d.readouts); sparse = SP2T.loglikelihoodaux(d) + # Column-major click order, padding and PSF offsets for this 3 × 2 movie. + packed = UInt16[0 5 8 7; 0 8 3 0; 0 2 0 0; 0 1 0 0] + @test Array(sparse.readouts) == packed + @test Array(sparse.weights) == T.(packed .> 0) + @test Array(sparse.xindex) == [1 6 7 11; 1 4 9 10; 1 5 7 10; 1 6 7 10] + @test Array(sparse.yindex) == [1 3 6 8; 1 4 6 7; 1 4 5 7; 1 4 5 7] + @test Array(sparse.darkcounts) == T[-1 0.4 0.2 0.3; -1 0.2 0.6 -1; + -1 0.3 -1 -1; -1 0.6 -1 -1] + @test sparse.zeroruns == [(1:1, :all), (2:2, :none), (3:4, :mixed)] + excluded = SPAD{T}(period = 1, pixel_size = 1, darkcounts = adapt(fill(T(-1), 3, 2)), + cutoffs = (0, Inf), readouts = adapt(counts), batchsize = B) + empty_clicks = SP2T.loglikelihoodaux(excluded) + @test Array(empty_clicks.readouts) == zeros(UInt16, 1, 4) + @test Array(empty_clicks.darksum) == zeros(T, 4) + @test empty_clicks.zeroruns == [(1:4, :none)] + expected_dark = [sum((Float64(dark[i, j]) for j = 1:2, i = 1:3 + if valid[i, j] && iszero(counts[i, j, k])); init = 0.0) for k = 1:4] + atol, rtol = T === Float32 ? (3e-5, 5e-5) : (2e-10, 5e-12) + @test isapprox(Array(sparse.darksum), expected_dark; atol, rtol) + for (oldb, newb) in ((T(0), T(2)), (T(0.4), T(1.2)), (T(2), T(0))) + oldscore, newscore = framescores(oldb), framescores(newb) + @testset "$(nameof(typeof(aux))) brightness $oldb to $newb" for aux in (dense, sparse) + SP2T.set_poisson_means!(aux, d, adapt(coords), oldb, newb, psf) + score = SP2T.get_loglikelihood!(aux, d) + @test isapprox(score, sum(oldscore); atol, rtol) + @test isapprox(Array(aux.frame), oldscore; atol, rtol) + SP2T.set_frame_Δloglikelihood!(aux, d) + @test isapprox(Array(aux.frame), newscore - oldscore; atol, rtol) + end + for (slot, b) in ((1, oldb), (2, newb)) + expected_signal = [sum((Float64(b) * signal[i, j, k] for j = 1:2, i = 1:3 + if valid[i, j] && iszero(counts[i, j, k])); init = 0.0) for k = 1:4] + @test isapprox(Array(sparse.linear[slot]), expected_signal; atol, rtol) + end + end + + # Cross every tile boundary (x>128, y>8, frames>16). Binary fractions + # make the direct positive zero-readout mass exactly representable. + nx, ny, nf = 129, 9, 17 + reads = [UInt16((i + 2j + k) % 5 == 0) for i = 1:nx, j = 1:ny, k = 1:nf] + calibration = [T((i + j) % 7 == 0 ? -1 : 1) for i = 1:nx, j = 1:ny] + td = SPAD{T}(period = 1, pixel_size = 1, darkcounts = adapt(calibration), + cutoffs = (0, Inf), readouts = adapt(reads), batchsize = 1) + s = SP2T.loglikelihoodaux(td) + px = [T(mod(i + k, 4) + 1) / 4 for i = 1:nx, m = 1:1, k = 1:nf] + py = [T(mod(j + 2k, 4) + 1) / 4 for j = 1:ny, m = 1:1, k = 1:nf] + result = adapt(zeros(T, nf)); scale = T(0.5) + SP2T.add_zero_framesums!(result, adapt(px), adapt(py), td, s, scale) + expected = [sum(scale * px[i, 1, k] * py[j, 1, k] for j = 1:ny, i = 1:nx + if calibration[i, j] > 0 && iszero(reads[i, j, k])) for k = 1:nf] + @test Array(result) == expected + + # The public sampler also accumulates more than one emitter. This + # fixture has more than 128 clicks per frame and a partial final warp. + X, Y = adapt(px), adapt(py) + SP2T.reset!(s, td, 1) + fill!(s.linear[1], T(0.25)) + fill!(s.means[2], T(23)) + fill!(s.linear[2], T(13)) + for _ in 1:2 + SP2T.add_spad_signal!(s, td, X, Y, scale, 1) + end + @test Array(s.linear[1]) == T(0.25) .+ 2 .* expected + click_increment = scale .* px[Array(s.xindex)] .* py[Array(s.yindex)] + @test Array(s.means[1]) == Array(s.darkcounts) .+ 2 .* click_increment + @test all(==(T(23)), Array(s.means[2])) + @test all(==(T(13)), Array(s.linear[2])) + + # A clicked pixel contains unit signal; the sole included miss contains + # less than half an ulp at 1. An excluded column carries a nonfinite PSF. + tail = eps(T) / T(8) + td = SPAD{T}(period = 1, pixel_size = 1, darkcounts = adapt(reshape(T[1, 1, -1], 3, 1)), + cutoffs = (0, Inf), readouts = adapt(reshape(UInt16[1, 0, 0], 3, 1, 1)), batchsize = 1) + s = SP2T.loglikelihoodaux(td) + SP2T.reset!(s, td, 1) + SP2T.add_spad_signal!(s, td, adapt(reshape(T[1, tail, NaN], 3, 1, 1)), + adapt(ones(T, 1, 1, 1)), one(T), 1) + @test only(Array(s.linear[1])) == tail + @test only(Array(s.means[1])) == T(2) + + # A click in frame 2 leaves frame 1 fully separable. Both spatial axes + # exceed a warp and the number of pixels is not a multiple of 32. + counts = zeros(UInt16, 37, 39, 3) + counts[37, 39, 2] = 1 + counts[:, :, 3] .= 1 + td = SPAD{T}(period = 1, pixel_size = 1, darkcounts = adapt(ones(T, 37, 39)), + cutoffs = (0, Inf), readouts = adapt(counts), batchsize = 1) + s = SP2T.loglikelihoodaux(td) + SP2T.reset!(s, td, 1) + SP2T.add_spad_signal!(s, td, adapt(ones(T, 37, 1, 3)), + adapt(ones(T, 39, 1, 3)), T(0.5), 1) + @test Array(s.linear[1]) == T[37 * 39 / 2, (37 * 39 - 1) / 2, 0] + end +end