Add off-policy evaluation for contextual bandits (decisionrl.ope) - #24
Merged
Merged
Conversation
New decisionrl.ope module estimates what a target policy would earn from a log of past decisions, without deploying it, which is how you decide whether a new pricing or recommendation rule is worth shipping. Four estimators over a (context, action, propensity, reward) log and the target's action probabilities: inverse propensity scoring (unbiased, high variance), self-normalized IPS (steadier), the direct method (a fitted reward model), and doubly robust (unbiased if either the propensities or the model is right). Includes collect_bandit_log and behaviour-policy helpers. Pure NumPy, no torch. On a synthetic bandit with a known value, all four recover it from a uniform-behaviour log; a test also covers the doubly-robust property, that DR stays accurate when the reward model is deliberately broken. Docs page and cross-links added.
Only CHANGELOG.md conflicted, and only because both sides appended to the same `### Unreleased` heading: main gained the `set_training(bool)` entry and a `### Fixed` section about the rollout buffer's global NumPy RNG, this branch gained the `decisionrl.ope` entry. Nothing overlapped in meaning, so all three survive unchanged — ope first under `### Added` because it is the larger addition, then `set_training`, then `### Fixed`. No source file conflicted: `decisionrl.ope` is a new module and touches nothing main changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
decisionrl.ope, off-policy evaluation for contextual bandits: estimate what a targetpolicy would earn from a log of past decisions, without deploying it. This is the question
you actually face before shipping a new pricing or recommendation rule, and it fits the
library's honest-evaluation theme.
Four estimators over a
(context, action, propensity, reward)log plus the target policy'saction probabilities:
inverse_propensity_score(IPS): unbiased, high variance.self_normalized_ips(SNIPS): consistent and much steadier.direct_method(DM): a fitted per-arm reward model averaged under the target.doubly_robust(DR): DM plus an IPS correction; unbiased if either the propensities orthe reward model is right.
Plus
collect_bandit_logand behaviour-policy helpers (uniform_behavior,epsilon_greedy_behavior,greedy_target_probs). Everything is pure NumPy with no torchdependency, so it works in the minimal install.
On a synthetic bandit with a known value (0.443), all four recover it from a
uniform-behaviour log: IPS 0.447, SNIPS 0.447, DM 0.444, DR 0.447. A test also pins the
doubly-robust property: with a deliberately broken reward model the direct method collapses
toward zero, but doubly robust still returns the right value.
Docs page (
docs/ope.md), nav entry, README bullet, a cross-link from the bandits page, anda CHANGELOG entry are included. ruff, mypy, the OPE tests, and the strict docs build pass.