Skip to content

feat: Add pois_lnorm distribution functions - #173

Closed
joethorley wants to merge 1 commit into
mainfrom
pois-lnorm
Closed

joethorley wants to merge 1 commit into
mainfrom
pois-lnorm

Conversation

@joethorley

Copy link
Copy Markdown
Member

Closes #172

Adds the Poisson-lognormal family, mirroring gamma_pois: log_lik_pois_lnorm(), ran_pois_lnorm(), dev_pois_lnorm(), res_pois_lnorm(), prob_pois_lnorm(), quant_pois_lnorm() and sens_pois_lnorm().

X | z ~ Poisson(lambda * exp(sdlog * z - sdlog^2 / 2)), z ~ Normal(0, 1), so E[X] = lambda and Var[X] = lambda + lambda^2 * (exp(sdlog^2) - 1).
sdlog = 0 gives exactly the Poisson results (identical() for log_lik, dev, res, prob and quant, and the same random stream for ran_).

Implementation

  • log_lik_: adaptive Gauss-Hermite quadrature with a 10-node rule computed once at build time (Golub-Welsch), centred on the mode of the integrand and scaled by its curvature.
  • The Newton search for the mode is safeguarded.
    The proposed start overshoots and overflows when x is much larger than lambda (for example x = 100, lambda = 0.1, sdlog = 0.5 returned Inf).
    The log integrand's score is concave and decreasing, so a start left of the root takes one Newton step and is then capped at an upper bound where lam = max(x + 0.5, lambda).
    From there Newton converges monotonically, iterating to a tolerance instead of a fixed 10 steps (at most 16 iterations over a grid with x up to 1e6, lambda from 1e-6 to 1e8, and sdlog from 1e-6 to 10).
  • dev_: no exact closed form exists for the saturated lambda.
    It tends to x * exp(sdlog^2 / 2) as x grows but is smaller for small x (for example 1.29x vs 1.65x at x = 1, sdlog = 1).
    The optimum always lies in [x, x * exp(sdlog^2 / 2)], which was checked numerically across a grid along with monotonicity of the score.
    It is found by safeguarded Newton steps on log(lambda), using the posterior mean and variance of the Poisson rate from the same quadrature nodes, with bisection fallback.
  • prob_: sums the probability masses from 0 to floor(x).
    quant_ accumulates the same sums up to an analytic upper bound on the quantile, so both cost more as x grows.
  • log_lik_ supports tlower/tupper truncation like the other count families.
  • sens_ expands the sd through sdlog and errors for sd_mult < 1, as sens_gamma_pois() does.

Accuracy (absolute error in log-likelihood vs stats::integrate())

sdlog 10 nodes (used) 20 nodes
<= 1 1.5e-5 2.4e-8
1.5 1.6e-4 3.5e-6
2 5.8e-4 4.5e-5
3 6.5e-3 6.7e-4
5 1.6e-2 6.1e-3

The grid covered x from 0 to 1e6 and lambda from 1e-6 to 1e8.
The error comes from small counts with large sdlog, where the integrand is skewed.
The validated range (x from 0 to 84, lambda from 0.6 to 36, sdlog from 0.9 to 1.2) is within 4.4e-5.
The tests check < 1e-4 for sdlog <= 1 and < 1e-3 for sdlog <= 2 against stats::integrate() across a grid, plus finiteness at extreme inputs.

Speed

For 1e5 values with lambda = 20 and sdlog = 1 (Apple Silicon):

function time (s)
log_lik_pois_lnorm() 0.10
dev_pois_lnorm() 0.56
log_lik_gamma_pois() 0.006
dev_gamma_pois() 0.002

That is about 1 µs per value for log_lik_ and 5.6 µs per value for dev_/res_ (type "dev"), roughly 15 and 250 times slower than gamma_pois.
For an embr PPC with 4000 draws of 500 observations, the deviance residuals take about 11 s.
The tests include skip_on_cran() timing checks (1e5 values: log_lik_ < 2 s, dev_ < 10 s).

For review

  • Deviance residual sign. dev_pois_lnorm(res = TRUE) uses the package convention sign(x - lambda) (dev_res(), as in dev_beta_binom()).
    Because the saturated lambda is above x, residuals jump near the mean: at lambda = 5, sdlog = 0.8, x = 4, 5, 6 give -0.007, 0, 0.48.
    At x == lambda the residual is 0 although the deviance is positive.
    The alternative, sign(lambda_sat(x) - lambda), would be continuous and monotone in x with res^2 == dev everywhere, but would give small positive residuals for some x just below lambda.
  • Zero-inflated (_zi) variants are not included.

🤖 Generated with Claude Code

Add log_lik_, ran_, dev_, res_, prob_, quant_ and sens_pois_lnorm() for
the Poisson-lognormal distribution parameterised by its mean lambda and
sdlog. The marginal log-likelihood uses adaptive 10-node Gauss-Hermite
quadrature with a safeguarded Newton search for the mode, and the
saturated lambda for the deviance uses safeguarded Newton steps on the
score bracketed by x and x * exp(sdlog^2 / 2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joethorley

Copy link
Copy Markdown
Member Author

I don't need after all

@joethorley joethorley closed this Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Poisson-lognormal (pois_lnorm) distribution functions

1 participant