Skip to content

Add PWKAC: the PWK quantifier with leave-one-out rates - #17

Open
luizfernandolj wants to merge 1 commit into
masterfrom
pwk-loo
Open

luizfernandolj wants to merge 1 commit into
masterfrom
pwk-loo

Conversation

@luizfernandolj

Copy link
Copy Markdown
Collaborator

What

Adds mlquantify.neighbors.PWKAC, the quantifier Barranquero, González, Díez and del Coz propose in On the study of nearest neighbor algorithms for prevalence estimation in binary problems (Pattern Recognition 46(2), 2013): the weighted k-NN rule inside the adjusted count, with TPR and FPR estimated by leave-one-out.

Why

mlquantify.neighbors had only PWK, which is class PWK(CC) — the weighted rule under plain classify-and-count. That is the classifier of the paper inside a different quantifier. The method the paper proposes is an adjusted count, and its whole argument for nearest neighbours is the way the rates are estimated: the correction is only as good as the rates it divides by, and k-NN is the one learner that can afford leave-one-out, because the neighbour query is made once and every fold reads from it.

So this is a missing method, not a change to an existing one. PWK is untouched and stays in the API: classify-and-count over the weighted rule is a legitimate baseline, and several comparative studies catalogue PWK that way.

How

PWKAC subclasses ACC and overrides _fit_estimator_predictions, replacing the cross-validation of BaseAdjustCount with one neighbour query: ask for n_neighbors + 1 neighbours of each training point and drop the first, which is the point itself.

The two alternatives are worse for this method:

  • Cross-validation is unbiased but estimates the rates of a classifier trained on a fraction of the data, and costs k neighbour structures.
  • Resubstitution — labelling the training set with the fitted rule — lets each point vote for itself at distance zero, and under the PWK weights that self-vote is the heaviest of all when the point belongs to the minority class.

Evidence

Measured on 12 binary datasets, comparing each estimate against the truth (the rates of the fitted rule on a held-out half):

estimate closest on mean deviation
leave-one-out 6 of 12 0.0769
resubstitution 5 of 12 0.0903
5-fold cross-validation 1 of 12 0.0877

The spread matters more than the average: resubstitution is accurate where the problem is easy and optimistic where it is not. On haberman the true TPR is 0.415 and resubstitution reports 0.675; on sonar, 0.729 against 0.857 — precisely the datasets where the adjustment is doing work.

The correction itself is what PWK lacks. On an imbalanced problem (400 examples, 25% positives) PWK returns 0.385 against a true 0.250, while PWKAC returns 0.254.

Also

PWK's reference pointed at the wrong paper: it cited Barranquero, Díez & del Coz (2013), Quantification-Oriented Learning Based on Reliable Classifiers, Pattern Recognition 48(2), 591–604 — which is the Q-measure paper, from 2015. Corrected to the 2013 nearest-neighbour one.

Tests

Seven new tests in tests/test_neighbors.py: binary and multiclass prevalence, that the stored training labels are not the resubstitution ones and are no more accurate than them, that the single-query trick agrees with an explicit leave-one-out loop that refits without each point, that the adjustment beats the plain count on an imbalanced problem, parameters, and n_neighbors above the training size. The whole suite passes (281 passed, 1 skipped), doctests included.

🤖 Generated with Claude Code

mlquantify.neighbors had only PWK, which is class PWK(CC): the weighted
k-NN rule under plain classify-and-count. That is the classifier of
Barranquero et al. (2013) inside a different quantifier. The method the
paper proposes is an adjusted count, and its whole argument for nearest
neighbours is how the rates are estimated -- the correction is only as
good as the TPR and FPR it divides by, and k-NN is the one learner that
can afford leave-one-out, because the neighbour query is made once and
every fold reads from it.

PWKAC subclasses ACC and replaces the cross-validation of
BaseAdjustCount with one neighbour query: ask for n_neighbors + 1
neighbours of each training point and drop the first, which is the point
itself. Cross-validation would estimate the rates of a classifier
trained on a fraction of the data and cost k neighbour structures;
resubstitution would let each point vote for itself at distance zero,
and under the PWK weights that self-vote is the heaviest of all when the
point belongs to the minority class.

Measured on 12 binary datasets against the truth (the rates of the
fitted rule on a held-out half), leave-one-out is closest on 6 with a
mean deviation of 0.0769, resubstitution on 5 with 0.0903 and 5-fold
cross-validation on 1 with 0.0877. Resubstitution is accurate where the
problem is easy and optimistic where it is not: on haberman the true TPR
is 0.415 and it reports 0.675; on sonar, 0.729 against 0.857 -- the
datasets where the adjustment is doing work.

PWK is unchanged and stays in the API. Its reference, which pointed at
the 2015 Q-measure paper instead of the 2013 nearest-neighbour one, is
corrected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant