Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,44 @@ All notable changes to mlquantify will be documented in this file.

## [Unreleased]

### New Features

- **`PWKAC`: the PWK quantifier as Barranquero et al. (2013) propose it,
with leave-one-out rates.** `mlquantify.neighbors` had only `PWK`, which is
`class PWK(CC)` -- the weighted k-NN rule under plain classify-and-count.
That is the classifier of the paper inside a different quantifier. The
method the paper proposes is an adjusted count, and its whole argument for
nearest neighbours is the way the rates are estimated: the correction is
only as good as the `TPR` and `FPR` it divides by, and k-NN is the one
learner that can afford leave-one-out, because the neighbour query is made
once and every fold reads from it. `PWKAC` is that method.

`PWKAC` replaces the cross-validation of `ACC` with one neighbour query:
it asks for `n_neighbors + 1` neighbours of each training point and drops
the first, which is the point itself. Cross-validation would estimate the
rates of a classifier trained on a fraction of the data and cost k
neighbour structures; resubstitution -- labelling the training set with the
fitted rule -- would let each point vote for itself at distance zero, and
under the PWK weights that self-vote is the heaviest of all when the point
belongs to the minority class.

Measured on 12 binary datasets, against the truth (the rates of the fitted
rule on a held-out half), leave-one-out is closest on 6 of them with a mean
deviation of 0.0769, resubstitution on 5 with 0.0903, and 5-fold
cross-validation on 1 with 0.0877. The spread matters more than the
average: resubstitution is accurate where the problem is easy and
optimistic where it is not. On `haberman` the true `TPR` is 0.415 and
resubstitution reports 0.675; on `sonar`, 0.729 against 0.857 -- precisely
the datasets where the adjustment is doing work.

The correction itself is what `PWK` lacks. On an imbalanced problem
(400 examples, 25% positives) `PWK` returns 0.385 against a true 0.250,
while `PWKAC` returns 0.254.

`PWK` is unchanged and stays in the API: classify-and-count over the
weighted rule is a legitimate baseline, and several comparative studies
catalogue PWK that way.

### Breaking Changes

- **`MLPE` no longer takes an estimator, and is no longer aggregative.**
Expand Down Expand Up @@ -68,6 +106,13 @@ All notable changes to mlquantify will be documented in this file.

### Documentation

- **`PWK`'s reference pointed at the wrong paper.** It cited Barranquero,
Diez & del Coz (2013), *Quantification-Oriented Learning Based on Reliable
Classifiers*, *Pattern Recognition* 48(2), 591-604 -- which is the Q-measure
paper, from 2015. PWK is from Barranquero, Gonzalez, Diez & del Coz (2013),
*On the study of nearest neighbor algorithms for prevalence estimation in
binary problems*, *Pattern Recognition* 46(2), 472-482.

- `CDE` documents when the iteration helps and when it turns against you. The
cost update has an amplifying fixed point, which is what corrects `CC`'s
shrinkage towards the training prevalence and also what turns a weak scorer
Expand Down
19 changes: 19 additions & 0 deletions docs/source/api/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2163,6 +2163,25 @@ reference on core concepts, see the :ref:`Foundations <quantification_foundation



* - :obj:`~mlquantify.neighbors.PWKAC`

- .. div:: sk-apisearch-desc

.. currentmodule:: mlquantify.neighbors

.. autoshortsummary:: mlquantify.neighbors.PWKAC

.. div:: caption

:mod:`mlquantify.neighbors`









* - :obj:`~mlquantify.neural.QuaNet`

Expand Down
1 change: 1 addition & 0 deletions docs/source/api/mlquantify.neighbors.rst
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,4 @@ Neighbor-based Methods


PWK
PWKAC
1 change: 1 addition & 0 deletions docs/source/api_reference.py
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,7 @@ def _get_submodule(module_name, submodule_name):
"title": "Neighbor-based Methods",
"autosummary": [
"PWK",
"PWKAC",
],
}
],
Expand Down
3 changes: 2 additions & 1 deletion mlquantify/neighbors/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from ._classes import (
PWK,
PWKAC,
)

__all__ = ["PWK"]
__all__ = ["PWK", "PWKAC"]
204 changes: 200 additions & 4 deletions mlquantify/neighbors/_classes.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
from mlquantify.utils._constraints import Interval, Options
from mlquantify.neighbors._classification import PWKCLF
from mlquantify.utils import validate_data
from mlquantify.counting import CC
from mlquantify.counting import CC, ACC


class PWK(CC):
Expand Down Expand Up @@ -78,9 +78,10 @@ class PWK(CC):
----------
.. dropdown:: References

.. [1] Barranquero, J., Díez, J., & del Coz, J. J. (2013).
Quantification-Oriented Learning Based on Reliable Classifiers.
*Pattern Recognition*, 48(2), 591–604.
.. [1] Barranquero, J., González, P., Díez, J., & del Coz, J. J. (2013).
On the study of nearest neighbor algorithms for prevalence
estimation in binary problems.
*Pattern Recognition*, 46(2), 472–482.
"""

_parameter_constraints = {
Expand Down Expand Up @@ -150,3 +151,198 @@ def classify(self, X):
"""
X = validate_data(self, X, ensure_2d=True)
return self.estimator_.predict(X)


class PWKAC(ACC):
r"""Adjusted Probabilistic Weighted k-Nearest Neighbour (PWKAC) quantifier.

Targets prior probability shift. This is the quantifier Barranquero et al.
propose: the weighted k-NN rule of :class:`PWKCLF` inside the adjusted
count, with ``TPR`` and ``FPR`` estimated by **leave-one-out** on the
training set. Their argument for nearest neighbours in quantification is
exactly this estimate: the adjustment is only as good as the rates it
corrects by, and k-NN is the one learner that can afford leave-one-out,
because the neighbour query is made once and every fold reads from it.
:class:`PWK`, in the same module, is the same classifier under plain
classify-and-count, without the correction.

Like :class:`~mlquantify.counting.ACC`, this is a **binary-only** method;
multiclass problems are decomposed one-vs-rest, and each sub-problem gets
its own class weights and its own leave-one-out estimate.

Parameters
----------
alpha : float, default=1
Imbalance-correction exponent of the weighted k-NN rule. ``1`` is the
value of the original study.
n_neighbors : int, default=10
Number of nearest neighbours considered for each instance.
algorithm : {'auto', 'ball_tree', 'kd_tree', 'brute'}, default='auto'
Neighbour-search algorithm.
metric : str, default='euclidean'
Distance metric for the neighbour search.
leaf_size : int, default=30
Leaf size for the tree-based algorithms (speed/memory trade-off).
p : int, default=2
Power parameter for the Minkowski metric (``1`` = Manhattan,
``2`` = Euclidean).
metric_params : dict or None, default=None
Additional keyword arguments for the metric function.
n_jobs : int or None, default=None
Number of parallel jobs for the neighbour search and for the
one-vs-rest decomposition.
strategy : {'ovr', 'ovo'}, default='ovr'
Multiclass decomposition strategy.

Attributes
----------
estimator : PWKCLF
The underlying weighted k-NN classifier (built from the parameters
above; not an argument).
estimator_ : PWKCLF
The fitted classifier.
classes_ : ndarray of shape (n_classes,)
Class labels seen during ``fit``.
train_predictions : ndarray of shape (n_samples,)
Leave-one-out labels for the training set, from which ``TPR`` and
``FPR`` are read.
y_train : ndarray of shape (n_samples,)
True labels corresponding to ``train_predictions``.

Notes
-----
Why leave-one-out, and not cross-validation or the training predictions
themselves:

* Predicting the training set with the fitted rule is resubstitution: a
training point is its own nearest neighbour at distance zero, and under
the PWK weights that self-vote carries the largest weight of all when
the point belongs to the minority class. The resulting rates are
optimistic and the adjustment they produce is too small. The bias grows
with the difficulty of the problem, which is where the correction
matters most.
* k-fold cross-validation is unbiased but estimates the rates of a
classifier trained on a fraction of the data, and costs k neighbour
structures.
* Leave-one-out estimates the rates of the deployed classifier and costs
one neighbour query: ask for ``n_neighbors + 1`` neighbours of each
training point and drop the first, which is the point itself.

The usual caveat of that trick applies: with exact duplicates in the
training set the dropped neighbour may be a copy rather than the point
itself. The vote then still excludes one of the identical points, so the
effect is limited to that duplicate's label.

See Also
--------
PWK : The same classifier under plain classify-and-count.
ACC : Adjusted count over a user-supplied classifier, with the rates
estimated by cross-validation.

Examples
--------
>>> from mlquantify.neighbors import PWKAC
>>> from sklearn.datasets import make_classification
>>> X, y = make_classification(n_samples=200, random_state=42)
>>> q = PWKAC(n_neighbors=10).fit(X, y)
>>> q.predict(X)
{0: ..., 1: ...}

References
----------
.. dropdown:: References

.. [1] Barranquero, J., Gonzalez, P., Diez, J., & del Coz, J. J. (2013).
On the study of nearest neighbor algorithms for prevalence
estimation in binary problems.
*Pattern Recognition*, 46(2), 472-482.
.. [2] Forman, G. (2008). Quantifying Counts and Costs via
Classification. *Data Mining and Knowledge Discovery*,
17(2), 164-206.
"""

_parameter_constraints = {
"alpha": [Interval(1, None, inclusive_right=False)],
"n_neighbors": [Interval(1, None, inclusive_right=False)],
"algorithm": [Options(["auto", "ball_tree", "kd_tree", "brute"])],
"metric": [str],
"leaf_size": [Interval(1, None, inclusive_right=False)],
"p": [Interval(1, None, inclusive_right=False)],
"metric_params": [dict, type(None)],
"n_jobs": [Interval(-1, None, inclusive_right=False), type(None)],
"strategy": [Options(["ovr", "ovo"])],
}

def __init__(self,
alpha=1,
n_neighbors=10,
algorithm="auto",
metric="euclidean",
leaf_size=30,
p=2,
metric_params=None,
n_jobs=None,
strategy="ovr"):
self.alpha = alpha
self.n_neighbors = n_neighbors
self.algorithm = algorithm
self.metric = metric
self.leaf_size = leaf_size
self.p = p
self.metric_params = metric_params
self.n_jobs = n_jobs
# Like PWK, the estimator is fixed to the quantification k-NN instead
# of being supplied by the user. The cross-validation parameters of
# ACC stay at their defaults and are never used: the rates come from
# leave-one-out.
super().__init__(
estimator=PWKCLF(alpha=alpha,
n_neighbors=n_neighbors,
algorithm=algorithm,
metric=metric,
leaf_size=leaf_size,
p=p,
metric_params=metric_params,
n_jobs=n_jobs),
strategy=strategy,
n_jobs=n_jobs,
)

def _fit_estimator_predictions(self, X, y, estimator_fitted=False,
cv=None, stratified=None, shuffle=None,
random_state=None, cv_prediction="refit"):
"""Fit the weighted k-NN and label the training set leave-one-out.

Replaces the cross-validation of
:class:`~mlquantify.counting._base.BaseAdjustCount`; the
cross-validation arguments are accepted and ignored.
"""
X = np.asarray(X)
y = np.asarray(y)
if estimator_fitted:
self.estimator_ = self.estimator
return self.estimator_.predict(X), y
self.estimator_ = self.estimator.fit(X, y)
return self._leave_one_out(X), y

def _leave_one_out(self, X):
"""Label each training point with the neighbours that are not itself."""
estimator = self.estimator_
n_samples = X.shape[0]
classes = estimator.classes_
# One query for the whole training set: ask for one neighbour more
# than the rule uses and drop the first column, which is the point
# itself at distance zero.
n_query = min(estimator.nbrs.n_neighbors + 1, n_samples)
indices = estimator.nbrs.kneighbors(
X, n_neighbors=n_query, return_distance=False)[:, 1:]
if indices.shape[1] == 0:
# A single training point has no other neighbour to vote.
return np.repeat(classes[0], n_samples)

labels = np.asarray(estimator.y_train)[indices]
codes = np.searchsorted(classes, labels)
counts = np.zeros((n_samples, len(classes)), dtype=float)
np.add.at(counts, (np.arange(n_samples)[:, None], codes), 1.0)
counts *= estimator.class_weights
return classes[np.argmax(counts, axis=1)]
Loading
Loading