Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Test dependencies live in `test/Project.toml` (Julia workspace: `[workspace] pro
| `src/distributions.jl` | `DNormal`, the diagonal normal prior on initial positions |
| `src/gaussianpsf.jl` | `CircularGaussian` (2D), `CircularGaussianLorentzian` (3D), pixel integration through `erf` |
| `src/pixeldetector.jl` | Code shared by pixel detectors: construction helpers, expected counts (dark counts + PSF × brightness), likelihood sums over unfiltered pixels |
| `src/spad.jl` | `SPAD`: binomial likelihood over `batchsize` binary frames, readout simulation; `SPADLogLikelihood`, the sparse likelihood workspace (evaluated at nonzero readouts plus a separable sum over pixels) |
| `src/spad.jl` | `SPAD`: binomial likelihood over `batchsize` binary frames, readout simulation; `SPADLogLikelihood`, the sparse likelihood workspace (stable hit/miss scores at nonzero readouts plus nonnegative zero-readout intensity sums) |
| `src/emccd.jl` | `EMCCD` (Gaussian readout) and `EMCCDGamma` (gamma readout, not exported) |
| `src/msd.jl` | `MeanSquaredDisplacement`, direct sampling from its inverse-gamma posterior (the optional rescaling move, which stretches the on tracks with the MSD, is in `src/tracks.jl`) |
| `src/emittercount.jl` | `EmitterCount`, direct sampling of the number of emitters up to the weak limit |
Expand All @@ -44,13 +44,13 @@ Test dependencies live in `test/Project.toml` (Julia workspace: `[workspace] pro
- Array shapes: tracks are `(nframes, ndims, nparticles)`, with `ndims` 2 for `CircularGaussian` and 3 (z in column 3) for `CircularGaussianLorentzian`. `readouts` are `(nx, ny, nframes)` `UInt16`; `darkcounts` and `filter` are `(nx, ny)`. `active` is `(nframes, 1, nparticles)` (one entry shared by all coordinates, as in `examples/parametric`) or `(nframes, ndims, nparticles)`. Pixel bounds start at 0, so pixel `(i, j)` covers `[(i-1)·pxsize, i·pxsize] × [(j-1)·pxsize, j·pxsize]`.
- Units are the caller's (the examples use μm and s). `msd` is the per-coordinate variance of a one-frame displacement, 2·D·period; the `Sample` field holding it is named `diffusivity`. `brightness` and `darkcounts` are expected photons per frame (rate × period); `brightness` is divided by `psf.A`, so it is the expected count in a pixel when the emitter sits at its center.
- Everything is parametrized on `T<:AbstractFloat`. Public constructors are `Type{T}(; kwargs...)` and convert their inputs to `T` (`convert`, `elconvert`); methods expect one `T` across all arguments.
- One code path serves `Array` and `CuArray`: allocate with `similar(x, ...)`, avoid scalar indexing, prefer broadcasting and `batched_mul!`. `sum(vec(x))` instead of `sum(x)` is deliberate (faster with CUDA.jl). GPU-specific methods go in `ext/` as overloads of internal functions; `src/` never loads CUDA or Metal. Metal has no Float64, so code that runs on the GPU must not create Float64 values: write constants as `T(...)` and convert keyword inputs (e.g. `cutoffs`, `pixel_size`) to `T` (see `erfdiff`, `_init_pixel_detector_params`). Metal also cannot launch kernels over empty arrays (guard them, as `simulate!` does for an empty off chunk), and its `mul!` keeps a NaN or Inf already in the output, so zero an output buffer before `mul!` (as `framesum!` does).
- One code path serves `Array` and `CuArray`: allocate with `similar(x, ...)`, avoid scalar indexing, prefer broadcasting and `batched_mul!`. `sum(vec(x))` instead of `sum(x)` is deliberate (faster with CUDA.jl). GPU-specific methods go in `ext/` as overloads of internal functions; `src/` never loads CUDA or Metal. Metal has no Float64, so code that runs on the GPU must not create Float64 values: write constants as `T(...)` and convert keyword inputs (e.g. `cutoffs`, `pixel_size`) to `T` (see `erfdiff`, `_init_pixel_detector_params`). Metal also cannot launch kernels over empty arrays (guard them, as `simulate!` does for an empty off chunk), and its `mul!` keeps a NaN or Inf already in the output, so zero an output buffer before `mul!`. `framesum!` instead masks zero-weight inputs and uses a backend-native reduction.
- Random draws take `rng::AbstractRNG` as their first argument, threaded down from `runMCMC`'s `rng` keyword. Array draws use `devicerng(rng, x)`, which is `rng` for CPU arrays and the GPU package's default generator for CUDA and Metal arrays (defined in `ext/`). The exported `simulate!` and `bridge!`, and `simulate_readouts!`, also have methods without `rng` (using `Random.default_rng()`), which the example scripts call.
- Mutating functions end in `!` and return the mutated object. Buffers are preallocated: in the likelihood workspaces, `means[1]` is current and `means[2]` proposed; in `Tracks`, index 1 is current and 2 the proposal.
- `runMCMC!` builds its likelihood workspace with `loglikelihoodaux(detector)`: `SPADLogLikelihood` for `SPAD` (expected counts only at nonzero readouts), the dense `LogLikelihoodArray` otherwise. The sweep reaches it only through `reset!`, `add_poisson_mean!`, `set_poisson_mean(s)!`, `get_loglikelihood!`, `set_frame_Δloglikelihood!` (and `.frame`) and `get_Δloglikelihood!`; a workspace implements these. The dense path also serves SPAD and is the reference the sparse one is tested against.
- `onchunk`, `offchunk` and `proposals` are views into the `Tracks` arrays; call `reassign!` after `nemitters.value` changes. `effvalue = value ./ active` is `Inf` for inactive frames, so they contribute no photons. `active` is a fixed mask; proposals share it (`active[2]` is a separate buffer kept equal to `active[1]`, reserved for inferring the mask later; see F1 in `MEMORY.md`).
- `onchunk`, `offchunk` and `proposals` are views into the `Tracks` arrays; call `reassign!` after `nemitters.value` changes. Inactive effective positions use infinite lateral coordinates and zero axial coordinates in 3D, so they contribute no photons without `0/0` or infinite PSF width. `active` is a fixed mask; proposals share it (`active[2]` is a separate buffer kept equal to `active[1]`, reserved for inferring the mask later; see F1 in `MEMORY.md`).
- Names: `ᵒ` current, `ᵖ` proposed, `ᵥ` a value (`msdᵥ = msd.value`), `𝑇` temperature, `Δ` proposed minus current. Work in log space. Likelihood terms are divided by `𝑇`, priors are not.
- `RandomVariable` subtypes have `value`, `prior` and `fixed`; `update!` returns at once when `isfixed`. `parametric = true` or a fixed `tracks.nemitters` selects the parametric sweep, which never updates the emitter count. `logprior` of `MeanSquaredDisplacement` and `Brightness` is unnormalized unless `normalization = true`. The track and `DNormal` log-priors are always normalized, since their constants depend on the MSD. `log𝒫` is the log posterior of the whole weak-limit state, all `max_ntracks` candidates, so every sample's density has the same units and samples with different emitter counts compare: the off candidates enter through `offlogprior`, their expected log prior given the MSD (or an exact χ² draw of it with `logposterior_rng`), without simulating their tracks. Its density is in the coordinates (first positions, standardized increments `ε = displacement/√msd`, MSD), through `logjacobian`, so the tracks' prior volume does not depend on the MSD (`docs/logposterior.qmd` derives both constructions and their equivalence). The MSD is drawn from the on tracks only (the off tracks integrated out), then, with `msdrescale = true`, moved once more by the rescaling move, which stretches the on tracks with it; the nonparametric sweep redraws the off tracks before the next count update.
- `RandomVariable` subtypes have `value`, `prior` and `fixed`; `update!` returns at once when `isfixed`. `parametric = true` or a fixed `tracks.nemitters` selects the parametric sweep, which never updates the emitter count. `logprior` of `MeanSquaredDisplacement` and `Brightness` is unnormalized unless `normalization = true`. The track and `DNormal` log-priors are always normalized, since their constants depend on the MSD. `log𝒫` is a standardized-coordinate ranking score including the brightness prior. With off candidates it uses a log-prior proxy, not a marginalized density or the density of the stored full state: the off candidates enter through `offlogprior`, their expected log prior given the MSD (or an exact χ² draw of it with `logposterior_rng`), without simulating their tracks. Its density is in the coordinates (first positions, standardized increments `ε = displacement/√msd`, MSD), through `logjacobian`, so the tracks' prior volume does not depend on the MSD (`docs/logposterior.qmd` derives both constructions and their equivalence). The MSD is drawn from the on tracks only (the off tracks integrated out), then, with `msdrescale = true`, moved once more by the rescaling move, which stretches the on tracks with it; the nonparametric sweep redraws the off tracks before the next count update.
- Derived properties come from `Base.getproperty` overloads (`chain.msds`, `detector.framecenter`); list new ones in `Base.propertynames` too, as `Chain` does.
- A new detector subtypes `PixelDetector{T}` with fields `period`, `pxsize`, `pxbounds`, `darkcounts`, `filter`, `readouts`, has a keyword constructor built on `check_pixel_dimsmatch` and `_init_pixel_detector_params`, and implements `set_pixel_loglikelihood!`, `set_pixel_Δloglikelihood!` and `simulate_readouts!`.
- The export list in `src/SP2T.jl` is the public API; tests reach internals as `SP2T.name`.
Expand All @@ -75,7 +75,7 @@ Test dependencies live in `test/Project.toml` (Julia workspace: `[workspace] pro
- Tracks and moves: `Tracks` with an `active` mask (both layouts); the identity-swap move (frame-by-frame relabeling; crossing particles with swapped tails are untangled, with their masks); a fixed emitter count is never updated and selects the parametric sweep; a parametric step leaves `effvalue` up to date; the rescaling and joint count–brightness moves run only with `msdrescale` and `countbrightness` (their proposal counters)
- Chain: `EmitterCount.limit`; `Chain`'s `propertynames`; thinning (at most `sizelimit` samples, equal power-of-two spacing); `findmap`, `findmle` with `burn_in`; `jumpdistances`
- `PolynomialAnnealing`: 𝑇₀ at the start, 1 from `I` on, decreasing in between
- Samplers (`test/samplers.jl`), seeded with `StableRNG`, against their target distributions from theory, with tolerances of 4 standard errors (batch means for Markov chains): `randc`; the MSD's tempered inverse-gamma posterior; the brightness and position Metropolis–Hastings chains, which sample their priors when the likelihood is switched off (𝑇 = Inf); the joint count–brightness move with the brightness update, which then samples the count and brightness priors; one rescaling MSD move applied to exact prior draws keeps the MSD's inverse gamma and the tracks' χ²; the identity-swap acceptance rate; `simulate!` and `bridge!` variances; each detector's `simulate_readouts!`; `offlogprior`'s expectation and exact draw against simulated off tracks; and that `runMCMC` is reproducible for a given seed, also with `logposterior_rng`, which leaves the chain unchanged
- Samplers (`test/samplers.jl`), seeded with `StableRNG`, against their target distributions from theory, with tolerances of 4 standard errors (batch means for Markov chains): `randc`; the MSD's inverse-gamma conditional, unchanged under likelihood-only tempering; the brightness and position Metropolis–Hastings chains, which sample their priors when the likelihood is switched off (𝑇 = Inf); the joint count–brightness move with the brightness update, which then samples the count and brightness priors; one rescaling MSD move applied to exact prior draws keeps the MSD's inverse gamma and the tracks' χ²; the identity-swap acceptance rate; `simulate!` and `bridge!` variances; each detector's `simulate_readouts!`; `offlogprior`'s expectation and exact draw against simulated off tracks; and that `runMCMC` is reproducible for a given seed, also with `logposterior_rng`, which leaves the chain unchanged
- Test dependencies: Test, Distributions, StableRNGs, Metal and Random. No data files. Tests that draw random numbers pass a `StableRNG`.
- `test/metal.jl` runs on a Mac with a functional Metal GPU (skipped elsewhere): Metal arrays in private storage with scalar indexing disallowed, `erfdiff`, `framesum!` into a buffer holding NaN and Inf, the log-likelihood (dense and sparse workspaces) against the CPU, and parametric and nonparametric runs that are reproducible when both `rng` and `Metal.default_rng()` are seeded. CI has no GPU, so there `ext/` is untested, and CUDA is untested everywhere.
- `test/metal.jl` runs on a Mac with a functional Metal GPU (skipped elsewhere): Metal arrays in private storage with scalar indexing disallowed, `erfdiff`, `framesum!` into a buffer holding NaN and Inf, the log-likelihood (dense and sparse workspaces) against the CPU, and parametric and nonparametric runs that are reproducible when both `rng` and `Metal.default_rng()` are seeded. The same Metal entry point runs the Float32 detector and sparse SPAD regression suites. A hosted macOS job exercises them when Metal.functional() succeeds. Optional CUDA regressions live in `test/cuda.jl` and require a separate environment with the declared CUDA weak dependency and test dependencies; see `docs/verification.qmd`.
- New sampler or fit tests: use a `StableRNG` seed or fixed data, and assert against a known ground truth with a tolerance derived from the noise, not tuned to current output (as `test/samplers.jl` does).
9 changes: 6 additions & 3 deletions docs/logposterior.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ format:
html-math-method: mathjax
---

SP2T stores a log posterior, `log𝒫`, with every sample, and `findmap` picks the sample where it is largest. This note shows that the posterior can be built in two ways, from track positions or from standardized increments; that both give the same posterior distribution; and that their densities still differ at a point by a factor that depends on the MSD. `log𝒫` uses the second construction, which is why `get_logposterior` adds `logjacobian`.
SP2T stores a posterior ranking score, `log𝒫`, with every sample, and `findmap` picks its largest finite value after burn-in. This note shows that the posterior can be built in two ways, from track positions or from standardized increments; that both give the same posterior distribution; and that their densities still differ at a point by a factor that depends on the MSD. `log𝒫` uses the second coordinate convention, which is why `get_logposterior` adds `logjacobian`. With off candidates, its expected-prior substitution makes it a ranking score rather than the exact joint log density of a stored full state or a marginalized log density.

# Setup

Expand Down Expand Up @@ -139,6 +139,8 @@ $$

They differ by $\tfrac{(K-1)D}{2}\log v$, that candidate's share of the Jacobian, as the change of variables requires. With `logposterior_rng`, the expectation is replaced by an exact draw: the sum of squared standardized increments and first positions of the off tracks is $\chi^2$-distributed with $(M-n)\,K\,D$ degrees of freedom.

An expected log prior is not the log of an integrated prior, and an independent draw is not the density of the particular stored off-track coordinates. These substitutions do not change the sampler, but limit interpretation of `findmap` when candidates are off. With every candidate on, the score is the standardized-coordinate joint log posterior up to sample-independent constants.

# What SP2T computes

$$
Expand All @@ -148,15 +150,16 @@ $$
+ \underbrace{\tfrac{N}{2} \log v}_{\texttt{logjacobian}}
+ \underbrace{\log p(v)}_{\texttt{logprior(msd)}}
+ \underbrace{\lambda\, n}_{\texttt{logprior(nemitters)}}
+ \underbrace{\log p(b)}_{\texttt{logprior(brightness)}}
$$

That is $\log \pi_B$, up to a constant shared by all samples. The track and first-position terms are normalized, because their constants depend on $v$; $\log p(v)$ and $\log p(n)$ are not, since their constants are the same for every sample. Standardizing the first positions by $\sigma_d$ as well would only add a constant, so it is left out. The brightness prior is not included.
This is a score in construction B, with the off-candidate qualification above. The track and first-position terms are normalized, because their constants depend on $v$; omitted MSD/count prior normalizers are shared by all samples. Standardizing the first positions by $\sigma_d$ as well would only add a constant, so it is left out. Saved samples include the normalized brightness prior. The three-argument internal `get_logposterior(loglikelihood, tracks, msd)` helper remains a reduced score without brightness; the brightness-aware overload is used when saving samples.

`test/runtests.jl` checks this against a direct evaluation in construction B with Distributions.jl, and that with the increments $\varepsilon$ and the log-likelihood fixed, changing $v$ changes `log𝒫` only by $\log p(v)$.

# Example: the 2D example

Two states of the 2D example (`examples/2D`, 50 × 50 pixels, $K = 2550$, $D = 2$, $M = 10$, so $N = 50{,}980$): the ground truth, and the state the sampler stays in when started from the example's own values (tracks at zero, brightness 0.1, MSD 2 × 10⁻⁵).
The following historical calculation omits the brightness prior and predates the repaired sampler. It illustrates the coordinate Jacobian, not the current complete saved-score calculation or current sampler behavior. It compares two states of the 2D example (`examples/2D`, 50 × 50 pixels, $K = 2550$, $D = 2$, $M = 10$, so $N = 50{,}980$): the ground truth and an earlier stuck state.

| | Ground truth | Stuck state | Truth − stuck |
|---|---:|---:|---:|
Expand Down
Loading
Loading