[sarthakguptaquant]

xref #904. Follow-up to #965: adds the sixth function from the proposal,
max_undetectable_understatement, which was left out pending a decision. Kevin said he
was happy to have a follow-up PR.

A passing backtest says the reported VaR was not rejected. It does not say how close it
was. This adds a function that answers a concrete question: by what fraction could the
whole VaR series have been scaled down, with the same returns, and still have passed
the selected tests.

Adds to arch/univariate/backtest.py, exported from arch.univariate:

• max_undetectable_understatement(returns, var, p=0.05, size=0.05, tests=..., ignore_too_few=True, window=250):
  the largest fraction by which var could have been understated and still pass
  kupiec_pof, christoffersen_independence, christoffersen_conditional_coverage and
  traffic_light (all four by default). Returns a frozen MaxUndetectableUnderstatementResult
  with the limit (understatement), the test that binds just beyond it (binding), the
  exceptions the limit would add (extra_exceptions), whether the reported VaR passes to
  begin with (passes, failed), nobs and p.

The search is exact. The exception set of a proportional scaling (1-s)*var only
changes at the breakpoints s = 1 - (-returns/var), so one pass over the breakpoints
finds the limit with no optimizer. It evaluates the tests once at each distinct
breakpoint, which is exact for this family of series.

It is a lower bound. A series that understates VaR on some days and overstates it on
others can produce the same, or fewer, exceptions than a proportional scaling, and so can
be harder to detect than this result suggests. The docstring states this.

ignore_too_few. kupiec_pof and christoffersen_conditional_coverage are
two-sided, so a VaR with too few exceptions is otherwise rejected for being too
conservative. That is not evidence of an understated VaR, so by default
(ignore_too_few=True) it is not counted as a failure.

Implementation. Reuses _aligned, _exceptions, _lr_pof and _lr_independence for
the breakpoint search, and the four public test functions for the baseline (s=0) check,
so the baseline validates p, the length of the series and, for traffic_light, the
window, with the same errors as calling them directly.

Usage:

from arch.univariate import max_undetectable_understatement

result = max_undetectable_understatement(returns, var, p=0.01)
result.understatement      # the largest undetectable fraction, e.g. 0.065
result.binding             # ('traffic_light',)
result.extra_exceptions    # exceptions the limit would add, e.g. 5

Tests, added to arch/tests/univariate/test_backtest.py:

• A hand-computed 4-observation case with an exact breakpoint.
• A brute-force grid check, using the public test functions directly, that the
  computed limit is the largest understatement that still passes, and that any larger one fails.
• Edge cases: no exceptions (fully understatable), all exceptions (fails the baseline),
  and both settings of ignore_too_few on the same series.
• A traffic_light-only case with a closed-form limit (the 5th largest ratio).
• Input validation: non-positive var, an unknown or empty tests, a bare string passed
  where a sequence is expected, out-of-range p/size/window, length mismatches, and
  too few observations for the selected tests.
• Agreement with the public test functions at the computed boundary.
• Immutability of the result and pandas index alignment.

Docs: a new paragraph and autosummary/autoclass entries in
doc/source/univariate/utility.rst, and a changelog entry in doc/source/changes/9.0.rst.

Checked: pytest arch/tests/univariate/test_backtest.py (231 passed), the full
arch/tests/univariate suite (13678 passed, 336 skipped), numpydoc lint, ruff,
black and isort, all clean.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.51%. Comparing base (8a47786) to head (c99ac1e).
⚠️ Report is 41 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1015      +/-   ##
==========================================
+ Coverage   99.44%   99.51%   +0.07%     
==========================================
  Files         153      152       -1     
  Lines       34229    34461     +232     
  Branches     2591     2603      +12     
==========================================
+ Hits        34039    34295     +256     
+ Misses        114       94      -20     
+ Partials       76       72       -4     

Flag	Coverage Δ
adder	99.46% <100.00%> (+0.07%)	⬆️
subtractor	99.46% <100.00%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bashtage]

Thanks for this. The code is clean and the breakpoint search is exact. There are two problems with what it reports.

1. The limit can lie beyond a scaling that is rejected. The search returns the smallest passing threshold, scanning up from 0. The independence test, and so the conditional coverage test, is not monotone in the threshold. So a smaller understatement can be rejected while the reported one passes:

import numpy as np
from arch.univariate import christoffersen_independence, max_undetectable_understatement

rng = np.random.default_rng(1234)
for _ in range(32):
    returns = rng.standard_t(5, 250) * 0.01
    var = np.full(250, 1.645 * 0.01 * rng.uniform(1.0, 1.4))
res = max_undetectable_understatement(returns, var, p=0.05)
res.understatement  # 0.226, binding ('traffic_light',)
christoffersen_independence(returns, var * (1 - 0.187)).pval  # 0.0357, rejected
christoffersen_independence(returns, var * (1 - 0.226)).pval  # 0.1153

Understating by 18.7% would have been detected, but the result says 22.6% is undetectable. This happened in 1 of the 50 passing series of this kind that I tried. The natural definition is the first rejection met when scaling down from the reported VaR. Scanning the breakpoints downward from 1 and stopping at the first failure gives that, and it usually needs only a few evaluations. Please also add a test with a non-monotone case.

2. ignore_too_few also ignores rejections that have nothing to do with too few exceptions. For christoffersen_conditional_coverage the code ignores any rejection when the exception rate is below p, even when the rejection comes from clustering:

from arch.univariate import christoffersen_conditional_coverage

returns = np.full(500, 0.001)
returns[200:210] = -0.05  # 10 consecutive exceptions, rate 0.02
var = np.full(500, 0.02)
christoffersen_conditional_coverage(returns, var, 0.05).pval  # 4.1e-20
res = max_undetectable_understatement(
    returns, var, p=0.05, tests=("christoffersen_conditional_coverage",)
)
res.passes, res.understatement  # (True, 1.0)

When the rate is below p, only the coverage part of the statistic should be set aside. For example, use LR_pof = 0 in the joint statistic and keep the chi2(2) critical value.

Smaller points:

• The boundary: the baseline counts a rejection at pval <= size, but the search counts one only at LR > crit, so the two disagree at equality. Please use one rule.
• Coverage: codecov reports 1 missed line and 3 partial lines in backtest.py.
• Reference: please add one for the diagnostic itself, if there is one. The current references are for the tests.

[sarthakguptaquant]

Thanks, both were real. Fixed in c99ac1e.

1. The scan now starts at the reported VaR, moves down the breakpoints and stops at the first rejected scaling, so the result is the largest s for which every scaling from 0 to s passes. Your seed-1234 example now returns 0.161 with christoffersen_independence binding, below the rejected 0.187. test_non_monotone_independence_stops_at_first_rejection covers it.
2. When the exception rate is below p, ignore_too_few now sets only the coverage term to 0 and keeps the chi2(2) critical value, so clustering still rejects. Your clustered example now fails at the reported VaR. test_ignore_too_few_keeps_clustering_rejection covers it.

Smaller points:

• The reported VaR and every scaling now go through the same gate, with one rule: reject when the p-value is at most size.
• Coverage of backtest.py is 100% of lines and branches locally.
• Reference: I am not aware of a published one for the diagnostic itself, so I have not added one. The Notes now relate it to Table 1 of the 1996 Basel framework, already cited as [1], which tabulates how often the traffic light accepts a VaR whose true coverage is lower than stated.