Importance sampling for the stochastic Frank-Wolfe variants

[GeoffNN]

Importance sampling for the stochastic Frank-Wolfe variants

minimize_sfw's SAG-style estimator keeps one stale scalar per datapoint and refreshes a
uniformly sampled one each step. Its error constant is

Psi(q) = sum_{m>=1} max_j d_j (1 - q_j)^m,    d_j = max_{u,v in C} |a_j^T (u - v)|

where q_j is the probability of sampling datapoint j and d_j is how far that
datapoint's prediction can move across the constraint set. Uniform sampling gives exactly
Psi = (n-1) max_j d_j. Taking q_j proportional to d_j reduces it to about sum_j d_j
(up to a log factor) — a gain of max_j d_j / mean_j d_j, which is 1 on a homogeneous
design and large on a heavy-tailed one. For the l1 ball of radius alpha,
d_j = 2 * alpha * ||a_j||_inf.

What this adds

• sfw_importance_probs(A, alpha, ord=1) — computes the weights for an l1 ball.
• sampling_probs=None on minimize_sfw — pass the weights to use them.

Scope, and why

Restricted to the SAG and SAGA variants at batch_size=1. Both restrictions are real
rather than conservative: the batch sampler draws without replacement, which non-uniform
probabilities do not describe, and the analysis assumes unit batches. Probabilities must be
strictly positive — a datapoint that is never resampled keeps a stale gradient forever.

The improvement is asserted for both SAG and SAGA, but only after fixing a second thing —
see below. Psi(q) is derived for the SAG-style estimator, whose per-datapoint error decays at
rate q_j, so these weights are principled for SAG and merely effective for SAGA; they
are not claimed to be variance-optimal for the latter.

SAGA's correction had to be made unbiased first

SAGA rescaled the change in the sampled dual variable by the constant n_samples - 1. That
constant is 1/q_j - 1 evaluated at the uniform q_j = 1/n: the estimator is
grad_agg_prev + (1/(n q_j)) a_j (f'_j(fresh) - f'_j(stale)), and since dual_var already
carries the 1/n, the factor is 1/q_j — of which grad_agg has supplied 1. Left constant,
SAGA's gradient estimate is biased the moment sampling stops being uniform.

Computing the factor from the sampled probability restores unbiasedness and is exactly
equivalent under uniform sampling, so nothing changes when sampling_probs is not passed.
SAG is untouched either way — it reads grad_agg directly and never forms grad_est.

The effect is large. Mean suboptimality against a projected-gradient reference over 12 seeds,
importance sampling vs uniform:

	epoch 30	epoch 100	epoch 300
before	1.4× (8/12 seeds)	1.1× (5/12)	1.6× (12/12)
after	13.7× (12/12)	34.0× (12/12)	85.1× (12/12)

Also documented, not fixed

step_size='DR' builds its certificate from the stochastic gap
<-grad_agg, update_direction>, which is not a lower bound on the true directional
derivative of the objective. It is therefore a heuristic step size rather than one backed by
a sufficient-decrease guarantee. This matters if anyone tries to build a backtracking line
search on it: when the stochastic gap overestimates the true gap, no Lipschitz estimate
satisfies the sufficient-decrease test and the loop diverges — observed reaching ~1e17
within a few thousand iterations. Only a docstring note here; no behaviour change.

Follow-up deliberately left out

A backtracking line search using the exact Frank-Wolfe gap. For this problem class the
exact gap costs O(n) rather than O(nd) — maintaining theta = A @ x makes both the
objective and the gap cheap, since the vertex is sparse. But evaluating the objective from
the linear predictions needs a value counterpart to partial_deriv, which the loss classes
do not have. Adding one to every loss class is an API decision that belongs to the
maintainer, so it is not in this PR. Happy to follow up if you want it.

Tests

pytest tests/test_stochastic_fw.py — 67 passed (60 pre-existing, 7 new).

A second, separable commit: the CI workflow

CI on this repo was red before this PR — master included. Every job died after
~3s at Set up Python with Version 3.9 with arch x64 not found, because
ubuntu-latest is now ubuntu-24.04 (no Python below 3.10 in the image) and
setup-python@v1 only consumes pre-installed interpreters instead of downloading
one. So there was no way to show this PR green without touching the workflow.

The second commit does the minimum for that: checkout@v1→v4,
setup-python@v1→v5, matrix 3.8/3.9/3.10→3.10/3.11/3.12, and
pipconflictchecker→pip check (pip-conflict-checker was last released in 2016
and imports pkg_resources, which setuptools 84 removed, so it now dies on
import). flake8 and pytest become explicit installs since they were only
arriving as transitive dependencies of the packages being dropped.

Result: 208 passed on each of 3.10, 3.11 and 3.12. It is a separate commit on
purpose — drop or rework it freely if you would rather fix CI on its own terms.

Attribution

The estimator and its analysis are from Négiar, Dresdner, Tsai, El Ghaoui, Locatello,
Freund and Pedregosa, Stochastic Frank-Wolfe for Constrained Finite-Sum Minimization,
ICML 2020 (arXiv:2002.11860). The importance-sampling
refinement is work in preparation by the same first author.

---

🤖 Generated with Claude Code

[fabianp]

hmm something's weird. its trying to duplicate everything into a build/ directory and comitting that it seems