Repository navigation
Add PWKAC: the PWK quantifier with leave-one-out rates - #17
Open
luizfernandolj wants to merge 1 commit into
Open
luizfernandolj wants to merge 1 commit into
luizfernandolj wants to merge 1 commit into
Conversation
mlquantify.neighbors had only PWK, which is class PWK(CC): the weighted k-NN rule under plain classify-and-count. That is the classifier of Barranquero et al. (2013) inside a different quantifier. The method the paper proposes is an adjusted count, and its whole argument for nearest neighbours is how the rates are estimated -- the correction is only as good as the TPR and FPR it divides by, and k-NN is the one learner that can afford leave-one-out, because the neighbour query is made once and every fold reads from it. PWKAC subclasses ACC and replaces the cross-validation of BaseAdjustCount with one neighbour query: ask for n_neighbors + 1 neighbours of each training point and drop the first, which is the point itself. Cross-validation would estimate the rates of a classifier trained on a fraction of the data and cost k neighbour structures; resubstitution would let each point vote for itself at distance zero, and under the PWK weights that self-vote is the heaviest of all when the point belongs to the minority class. Measured on 12 binary datasets against the truth (the rates of the fitted rule on a held-out half), leave-one-out is closest on 6 with a mean deviation of 0.0769, resubstitution on 5 with 0.0903 and 5-fold cross-validation on 1 with 0.0877. Resubstitution is accurate where the problem is easy and optimistic where it is not: on haberman the true TPR is 0.415 and it reports 0.675; on sonar, 0.729 against 0.857 -- the datasets where the adjustment is doing work. PWK is unchanged and stays in the API. Its reference, which pointed at the 2015 Q-measure paper instead of the 2013 nearest-neighbour one, is corrected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
mlquantify.neighbors.PWKAC, the quantifier Barranquero, González, Díez and del Coz propose in On the study of nearest neighbor algorithms for prevalence estimation in binary problems (Pattern Recognition 46(2), 2013): the weighted k-NN rule inside the adjusted count, withTPRandFPRestimated by leave-one-out.Why
mlquantify.neighborshad onlyPWK, which isclass PWK(CC)— the weighted rule under plain classify-and-count. That is the classifier of the paper inside a different quantifier. The method the paper proposes is an adjusted count, and its whole argument for nearest neighbours is the way the rates are estimated: the correction is only as good as the rates it divides by, and k-NN is the one learner that can afford leave-one-out, because the neighbour query is made once and every fold reads from it.So this is a missing method, not a change to an existing one.
PWKis untouched and stays in the API: classify-and-count over the weighted rule is a legitimate baseline, and several comparative studies catalogue PWK that way.How
PWKACsubclassesACCand overrides_fit_estimator_predictions, replacing the cross-validation ofBaseAdjustCountwith one neighbour query: ask forn_neighbors + 1neighbours of each training point and drop the first, which is the point itself.The two alternatives are worse for this method:
Evidence
Measured on 12 binary datasets, comparing each estimate against the truth (the rates of the fitted rule on a held-out half):
The spread matters more than the average: resubstitution is accurate where the problem is easy and optimistic where it is not. On
habermanthe trueTPRis 0.415 and resubstitution reports 0.675; onsonar, 0.729 against 0.857 — precisely the datasets where the adjustment is doing work.The correction itself is what
PWKlacks. On an imbalanced problem (400 examples, 25% positives)PWKreturns 0.385 against a true 0.250, whilePWKACreturns 0.254.Also
PWK's reference pointed at the wrong paper: it cited Barranquero, Díez & del Coz (2013), Quantification-Oriented Learning Based on Reliable Classifiers, Pattern Recognition 48(2), 591–604 — which is the Q-measure paper, from 2015. Corrected to the 2013 nearest-neighbour one.Tests
Seven new tests in
tests/test_neighbors.py: binary and multiclass prevalence, that the stored training labels are not the resubstitution ones and are no more accurate than them, that the single-query trick agrees with an explicit leave-one-out loop that refits without each point, that the adjustment beats the plain count on an imbalanced problem, parameters, andn_neighborsabove the training size. The whole suite passes (281 passed, 1 skipped), doctests included.🤖 Generated with Claude Code