Repository navigation
feat: Add pois_lnorm distribution functions - #173
Closed
joethorley wants to merge 1 commit into
Closed
joethorley wants to merge 1 commit into
joethorley wants to merge 1 commit into
Conversation
Add log_lik_, ran_, dev_, res_, prob_, quant_ and sens_pois_lnorm() for the Poisson-lognormal distribution parameterised by its mean lambda and sdlog. The marginal log-likelihood uses adaptive 10-node Gauss-Hermite quadrature with a safeguarded Newton search for the mode, and the saturated lambda for the deviance uses safeguarded Newton steps on the score bracketed by x and x * exp(sdlog^2 / 2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
I don't need after all |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #172
Adds the Poisson-lognormal family, mirroring
gamma_pois:log_lik_pois_lnorm(),ran_pois_lnorm(),dev_pois_lnorm(),res_pois_lnorm(),prob_pois_lnorm(),quant_pois_lnorm()andsens_pois_lnorm().X | z ~ Poisson(lambda * exp(sdlog * z - sdlog^2 / 2)), z ~ Normal(0, 1), so E[X] = lambda and Var[X] = lambda + lambda^2 * (exp(sdlog^2) - 1).
sdlog = 0gives exactly the Poisson results (identical()for log_lik, dev, res, prob and quant, and the same random stream forran_).Implementation
log_lik_: adaptive Gauss-Hermite quadrature with a 10-node rule computed once at build time (Golub-Welsch), centred on the mode of the integrand and scaled by its curvature.The proposed start overshoots and overflows when x is much larger than lambda (for example x = 100, lambda = 0.1, sdlog = 0.5 returned
Inf).The log integrand's score is concave and decreasing, so a start left of the root takes one Newton step and is then capped at an upper bound where
lam = max(x + 0.5, lambda).From there Newton converges monotonically, iterating to a tolerance instead of a fixed 10 steps (at most 16 iterations over a grid with x up to 1e6, lambda from 1e-6 to 1e8, and sdlog from 1e-6 to 10).
dev_: no exact closed form exists for the saturated lambda.It tends to
x * exp(sdlog^2 / 2)as x grows but is smaller for small x (for example 1.29x vs 1.65x at x = 1, sdlog = 1).The optimum always lies in
[x, x * exp(sdlog^2 / 2)], which was checked numerically across a grid along with monotonicity of the score.It is found by safeguarded Newton steps on log(lambda), using the posterior mean and variance of the Poisson rate from the same quadrature nodes, with bisection fallback.
prob_: sums the probability masses from 0 tofloor(x).quant_accumulates the same sums up to an analytic upper bound on the quantile, so both cost more as x grows.log_lik_supportstlower/tuppertruncation like the other count families.sens_expands the sd throughsdlogand errors forsd_mult < 1, assens_gamma_pois()does.Accuracy (absolute error in log-likelihood vs
stats::integrate())The grid covered x from 0 to 1e6 and lambda from 1e-6 to 1e8.
The error comes from small counts with large
sdlog, where the integrand is skewed.The validated range (x from 0 to 84, lambda from 0.6 to 36, sdlog from 0.9 to 1.2) is within 4.4e-5.
The tests check < 1e-4 for sdlog <= 1 and < 1e-3 for sdlog <= 2 against
stats::integrate()across a grid, plus finiteness at extreme inputs.Speed
For 1e5 values with lambda = 20 and sdlog = 1 (Apple Silicon):
log_lik_pois_lnorm()dev_pois_lnorm()log_lik_gamma_pois()dev_gamma_pois()That is about 1 µs per value for
log_lik_and 5.6 µs per value fordev_/res_(type"dev"), roughly 15 and 250 times slower thangamma_pois.For an embr PPC with 4000 draws of 500 observations, the deviance residuals take about 11 s.
The tests include
skip_on_cran()timing checks (1e5 values:log_lik_< 2 s,dev_< 10 s).For review
dev_pois_lnorm(res = TRUE)uses the package conventionsign(x - lambda)(dev_res(), as indev_beta_binom()).Because the saturated lambda is above x, residuals jump near the mean: at lambda = 5, sdlog = 0.8, x = 4, 5, 6 give -0.007, 0, 0.48.
At
x == lambdathe residual is 0 although the deviance is positive.The alternative,
sign(lambda_sat(x) - lambda), would be continuous and monotone in x withres^2 == deveverywhere, but would give small positive residuals for some x just below lambda._zi) variants are not included.🤖 Generated with Claude Code