Skip to content

Categorical has poorly defiend behavior when some events are 0 probability #459

Description

@YeungOnion

Was looking to have more function implement try_inverse_cdf instead of wrapping the panicking version with Ok, found this.

Categorical::locate (used by both inverse_cdf and try_inverse_cdf) does:

    self.norm_cdf.binary_search_by(|v| v.partial_cmp(&x))

But this won't match the standard quantile definition inf{x | F(x) >= p} if this cdf isn't monotone because there's not an ordering if some categories have 0 weight - a weird but not outright unreasonable case.

Reproducer:

let norm_cdf = vec![1.0, 1.0, 1.0]; // categories [1.0, 0.0, 0.0]
norm_cdf.binary_search_by(|v| v.partial_cmp(&1.0).unwrap()); // Ok(1), want 0

let norm_cdf = vec![0.0, 0.0, 1.0]; // categories [0.0, 0.0, 1.0]
norm_cdf.binary_search_by(|v| v.partial_cmp(&0.0).unwrap()); // Ok(1), want 0,

let norm_cdf = vec![0.25, 0.5, 0.5, 0.5, 1.0]; // interior tie
norm_cdf.binary_search_by(|v| v.partial_cmp(&0.5).unwrap()); // Ok(2), want 1

inverse_cdf/try_inverse_cdf currently reject x == 0.0 and x == 1.0 outright, which happens to route around the two boundary cases above, but does nothing for interior ties (the third example).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions