mcsamplerNFlow: make it usable as a portfolio member, and fix its evidence

[oshaughnessy-junior]

mcsamplerNFlow could not run as a mcsamplerPortfolio member. --sampler-portfolio NFlow is offered by util_ConstructIntrinsicPosterior_GenericCoordinates and util_ConstructEOSPosterior, so the configuration is reachable. Four defects, each hidden behind the previous one.

1. Return order. draw_simplified returned (rv, p_s, p_prior). MCSamplerGeneric and every other implementation return (p_s, p_prior, rv), which mcsamplerPortfolio.draw() unpacks, so an NFlow member had its samples assigned to joint_p_s. Not always loud. On base 40ceccb, [AV, NFlow] over a unit Gaussian in [-5,5]^d, with portfolio_allow_stratified_density=True since defect 2 otherwise refuses the portfolio at setup():

d	base 40ceccb	this branch
2	ValueError: could not broadcast input array from shape (2,200) into shape (200,)	runs
1	completes, d = -3.69, -3.94, -4.06, -4.21 nats (seeds 1-4)	-0.003, +0.046, -0.023, +0.005

At d=1 the (1,n) sample array broadcasts into the (n,) density slot, so the run returns a number several nats low with no error.

2. No sampling_density. The portfolio builds q_mix = sum_m frac_m q_m from it. Since the member p_s contract merged (80bd766), [AV, NFlow] is refused at setup() rather than falling back.

3. Truncation. enforce_bounds discarded the out-of-box flow samples, with two consequences. A trained flow returned fewer samples than asked, and the portfolio aborted copying into its fixed-width slice. And the p_s reported for the survivors was the raw q(x) rather than q(x)/A, so NFlow's own evidence was biased high by ln(1/A). draw_simplified now refills to the requested count and divides by the batch acceptance. A is per-batch on purpose: the flow is retrained between chunks, and a cumulative A left ln Z biased -0.179 nats over 6 seeds.

4. d=1 could not train. num_layers = int(d/2) is 0, so the transform had no trainable weights, and np.diag(np.cov(...)) raises on a (1,n) array.

NFlow's own evidence

test_NF_reuse.py (rostered OPTDEP), bias in nats, same seeds:

arm	base 40ceccb	this branch
TRAIN+SAVE	+0.465	-0.004
COLD	+0.561	-0.005
WARM(reuse)	+0.351	-0.030
WARM(polish)	+0.321	+0.010

That test passes on both. Its tolerance is 4*|cold bias| + 0.1, derived from the arm it compares against, so a bias shared by all four arms inflates the bound along with it. It checks reuse against cold, and cannot see an absolute offset.

Portfolio evidence against an exact answer

8 seeds per arm, unit Gaussian in [-5,5]^d, prior uniform on the box:

arm	d	neff	mean d	sem
AV+NFlow	2	300	-0.0072	0.0097
NFlow	2	300	-0.0001	0.0085
GMM+NFlow	2	300	+0.0008	0.0059
AV+AV	2	300	+0.0032	0.0081
AV+GMM	2	300	-0.0040	0.0066
AV+NFlow	1	300	+0.0069	0.0112

Every arm is within its sem. GMM+NFlow holds at acceptance 0.46 to 0.66, where the 1/A factor reaches about 0.7 nats, which is what exercises the normalization.

An earlier revision normalized by the LATEST batch rather than the current flow generation. That is unbiased, but at the ~46 draws a floored portfolio member actually gets it injects about 0.13 nats of per-chunk noise, and it left AV+NFlow at -0.0201 +- 0.0042, resolved. Accumulating within one generation is the same estimator over a larger sample and removes it.

Normalization, swept over dimension

Every ln Z measurement above is at d=1 or d=2. E[prior/p_s] over fresh draws is 1 exactly when the reported p_s is a normalized density on the box, and 1/A when it is not, so it separates the two without relying on the 1/A algebra. One trained flow per dimension, 4000 fresh draws:

d	base n_ret/n_req	base E[prior/p_s]	branch n_ret/n_req	branch E[prior/p_s]	branch A
2	242/1500	6.29	1500/1500	1.0014	0.43
4	383/1500	4.21	1500/1500	0.9925	0.30
6	534/1500	2.88	1500/1500	1.0620	0.51
8	737/1500	2.18	1500/1500	0.9699	0.10

The base misses the requested count at every dimension and its reported density is off by 1/A. The branch fills every chunk, including at acceptance 0.10, and sampling_density agrees with the reported p_s to 1e-6 (the flow is float32). Acceptance is not monotonic in d here; it is set by where training happened to leave each flow.

Verification

nflows and torch are absent from the CIT IGWN conda python. Real-dependency runs used ~/.conda/envs/myigwn-py310-testing (python 3.10.14, torch 2.1.2) on ldas-grid, no cupy.

check	result
new gate, real torch + nflows	21 passed
new gate, IGWN python, no nflows	21 passed
test_NF_reuse.py --as-test	PASS, biases above
.travis/test-ci-roster.py	PASS
gate block executed standalone	exit 0 clean, exit 1 under xfail(run=False)
19 mutations, isolated worktree	all caught but the headroom knob, control green either side

The gate needs neither package: it drives the untrained uniform branch plus strict in-file flow stand-ins, and stubs both at import when absent. Registered in .travis/test-integrate.sh with a collection count and a passed floor. The passed floor was wrong first time, scoring 0 on a green run because the count starts the line, so the block is executed in the checks above rather than read.

Two mutations survived the first battery. Capping the refill at one pass changes nothing, because one pass already over-asks by 1/A, so the faithful mutation removes the headroom too. And int(n_to_get) alone no longer decides anything, because the refill loop sizes each pass with int(np.ceil(...)); only removing both breaks the contract. The code comment says this.

Adversarial review

An adversarial subagent review of cbb551d found eight items. Seven are acted on here.

finding	action
the flow stand-ins had a CONSTANT density, so no test could see a row misalignment between rv, log_ps and log_p; it demonstrated three silent-wrong-answer mutations passing the gate	stand-ins are position-dependent now (_GradedFlow), and the alignment is asserted elementwise. All three are caught
sampling_density is documented READ-ONLY and advances the torch RNG	documented precisely, including why the portfolio never hits it at a q_mix evaluation
flow_acceptance's cache was never invalidated when the flow was replaced; measured 1.05x wrong on a retrain-then-query path	_reset_flow_acceptance() at all three install sites, pinned by an AST test that the SITES call it
draw_simplified(0) raised need at least one array to concatenate -- new in this branch, in the configuration the branch is for	returns empty arrays, matching mcsamplerAdaptiveVolume
enforce_bounds=False drew 8x what it was asked for and took a bounds-enforcing probe on behalf of a caller that asked for no bounds	single pass of exactly n, no probe, no 1/A
the per-pass memory cap did not bound what its comment claimed at the default n_chunk	absolute cap lowered to the figure that binds, comment corrected
per-batch A is unbiased but noisy at a floored member's allocation	per-generation accumulation, numbers above

The eighth was that E[1/Ahat] != 1/A costs under 0.003 nats, measured against a closed-form A over 400 repeats. That is a confirmation, not a defect.

Second battery, 19 mutations: all caught except changing the refill headroom from 1.15 to 1.00, which is a performance knob (the loop runs until the chunk is full, so it changes only the pass count). The code says so where it could be mistaken for a guard.

Not changed

nf_method='iterative' builds a TanhTransformFrozen layer, supported on the box by construction, so its acceptance is 1 and defect 3 never applied to it. The default nf_method uses an unbounded PointwiseAffineTransform, hence acceptance below 1. I did not change which architecture is the default.

🤖 Generated with Claude Code