Repository navigation
Second session + LOB-Bench, and a calibrated queue-reactive model - #15
Merged
Merged
Conversation
The repo's headline correctness claim rested on one symbol-day. A second MSFT day would have re-tested the same regime, so the second session is chosen to be a different kind of book: INTC 2024-08-02 sits at a one-tick spread 81.1% of the session with 2,400-3,000 shares at the touch, against MSFT's 1.0% and 63-68 -- large-tick and queue-dominated against small-tick and spread-dominated, and 2.2x the displayed fills on 53% of the message count. INTC reaches 100.0000% agreement with Databento's own MBP-10 across 60,128,762 cells, on the first run, with no code changes. That is the useful part: the three fixes found on MSFT were corrections to a misread of the vendor's event model, not adjustments fitted to one session. Also here: - scripts/fetch_databento_session.py prices a request with metadata.get_cost and downloads nothing without --confirm, so the spend against a metered entitlement is always visible before it is incurred. Both sessions: $0.00. - validate_mbp10_vs_vendor.py now prints explicit slot accounting. An agreement percentage is computed over mutually-present cells, so a disagreement about whether a level exists at all would be invisible in it; the reconciliation (compared + both-absent + one-sided = total) makes the "zero one-sided" claim reproducible rather than a number quoted once in prose. - scripts/run_lob_bench.py runs LOB-Bench (Nagy et al., ICML 2025) over both sessions. All 12 statistics score 0.000000 on L1 and Wasserstein-1. Where cell agreement is already exact those zeros are exact by construction, and the README says so rather than presenting them as independent evidence of realism. What they do establish is that the engine's LOBSTER export is consumable by the standard academic toolchain through a third-party parser, including the message-derived statistics the MBP-10 diff never touches. - Multi-level OFI now runs on both sessions, and the second one earns its place: in the large-tick book best-level OFI explains almost nothing at short horizons (0.0163 against MSFT's 0.1121) while the integrated variable recovers 0.1029 -- a 6.3x lift where MSFT sees 2.0x. With the spread pinned and the queue deep, the information is behind the touch, not at it. The latency and benchmark tables are deliberately left single-session: those are host-specific timings, and an INTC number from different hardware would corrupt the comparison rather than broaden it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
A few new documentation details are internally inconsistent or misleading (notably schema mapping comments / timezone wording and an MBO record-count mismatch) and should be reconciled to keep the new two-session correctness claim precise.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR strengthens the repository’s “vendor cross-check” correctness claim by validating exact MBP-10 agreement on a second Databento session (a different book regime), adds explicit slot accounting to make “zero one-sided” disagreements reproducible, and adds tooling to run LOB-Bench as an external compatibility/metric check. It also introduces safer Databento-fetch tooling (dry-run by default) plus new Python tests that lock in the DST session window and rounding-vs-truncation regression.
Changes:
- Extend
validate_mbp10_vs_vendor.pyto return richer per-level accounting (Counts) and print explicit slot reconciliation. - Add
fetch_databento_session.py(dry-run pricing unless--confirm) andrun_lob_bench.pyto score both sessions via LOB-Bench. - Add Python tests for rounding, sentinel encoding, DST-correct session windows, and slot-accounting identity; update README/DATA/handoff docs to reflect the two-session validation.
File summaries
| File | Description |
|---|---|
| tests/test_scripts.py | Adds regression/unit tests for rounding, sentinels, DST session window, and slot accounting. |
| scripts/validate_mbp10_vs_vendor.py | Adds Counts accounting and prints explicit compared/absent/one-sided reconciliation; updates result narrative. |
| scripts/run_lob_bench.py | New script to export aligned LOBSTER sequences and run LOB-Bench scoring against vendor MBP-10. |
| scripts/fetch_databento_session.py | New script to price requests and optionally download Databento schemas for a full extended session window. |
| README.md | Updates correctness and methodology narrative to cover two sessions and LOB-Bench results. |
| handoff.md | Adds a handoff snapshot describing the two-session validation and new tooling. |
| DATA.md | Documents the two sessions used and reproducing the extracts. |
Review details
Suppressed comments (1)
scripts/fetch_databento_session.py:36
session_window()returns datetimes in the exchange timezone (America/New_York), not UTC. Calling them “UTC-aware bounds” reads like the values are in UTC and can mislead callers/tests.
def session_window(date_str: str) -> tuple[datetime, datetime]:
"""UTC-aware bounds of the extended session on `date_str` in exchange time."""
- Files reviewed: 7/7 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+23
to
+26
| | session | why | MBO records | billable | cost | | ||
| |---|---|---:|---:|---:| | ||
| | MSFT 2024-06-03 | small-tick, spread-dominated (one tick = 0.24 bp of mid) | 4,003,833 | 224.2 MB | $0.00 | | ||
| | INTC 2024-08-02 | large-tick, queue-dominated (4.73 bp); post-earnings gap day | 2,356,988 | 132.0 MB | $0.00 | |
Comment on lines
+31
to
+32
| # filename stem -> databento schema id | ||
| _SCHEMA_SUFFIX = {"mbo": "mbo", "mbp-10": "mbp10"} |
This repository has only ever reconstructed books. This is its first fitted
model: Huang, Lehalle and Rosenbaum's queue-reactive process (JASA 2015),
in which 2K queues sit at fixed offsets from a reference price and the
intensity of limit, cancel and market-order flow at each queue is a function
of that queue's own size.
The geometry carries the model and is easy to get wrong. Q_i sits (|i| - 0.5)
ticks from p_ref, NOT at a fixed offset from the best quote -- that is what
lets Q_1 be genuinely empty when the spread widens, and what makes "market
orders reach Q_2 only once Q_1 has emptied" expressible at all. Indexing from
the best quote would make Q_1 non-empty by construction and delete the
mechanism. Since limit prices are whole ticks and the queues sit half a tick
off p_ref, the whole geometry is exact in integers.
Having two sessions is what makes this worth doing. The queue-reactive model
is a large-tick model, and the two sessions land on opposite sides of that:
INTC has the best quote inside the +/-3 window 100.0% of the session, MSFT
70.8% with only 12.7% of messages landing on a modelled queue at all. The
sharpest diagnostic is the implied theta -- p_ref moves over best-queue
depletions -- which is 0.51 on INTC and 7.34 on MSFT. In the paper theta is a
probability. An estimator returning 7.34 is the model failing loudly rather
than quietly, and says "wrong regime" better than any fit statistic.
Two of three stylized facts reproduce on INTC and one does not:
- execution concentrated at the touch, sharply: peak lambda^M 28.50/s at
Q+1, 1.96 at Q+2, 0.09 at Q+3
- limit arrival falling in queue size, 76.3/s empty to ~10.6/s past five AES
- cancellation is NOT proportional to queue size and is not close.
corr(n, lambda^C) = -0.425 at Q+1, and the per-order cancellation rate
falls thirty-fold from 9.73 to 0.321 between n=1 and n=25. A long queue is
one traders want to stay in.
Where it fails is the substance, and it is measured rather than asserted.
Fitting the paper's three intensities leaves Q+1 with 283,771 adds against
261,399 cancels and executions, a surplus of +16.1M shares. A closed
birth-death chain cannot run that, and the real queue does not grow, so the
missing outflow is real: content leaving by RE-INDEXING when p_ref moves,
which is not an order event and so appears in none of the three rates.
Simulated unbounded, Q1 runs to 3,125 AES against a real 17.8.
Closing the generator needed a fitted state-dependent price-move intensity in
place of the paper's scalar theta -- it turns out to be a step function,
16.1/s at an empty touch and ~0 once anything rests there. That exposes a
mutual deadlock: Q+1 can only drain on a down move, which needs Q-1 to empty,
and vice versa. The real book escapes through cross-queue dependence Model I
forbids: P(both touch queues > 25 AES) is 5.06% against 1.88% under
independence, and the real one-second drift of Q+1 at fixed own size swings
from -3.3 to +1.2 AES/s with the state of a queue Model I never looks at.
Adding the paper's own Model IIb coupling plus a reflecting cap produces a
usable comparison; the cap is an admission, not a fix, and the queue-size row
stays wrong on purpose.
Order sizes were checked as the obvious suspect and are not the issue here:
the imbalance is worse in shares (+1.86 AES/s) than in events (+0.96).
Burstiness fails as expected: real median inter-arrival on a modelled queue is
57 microseconds against a simulated 6.2 ms, while the means differ by under
50%. That gap is what a state-dependent Poisson model has no machinery for,
and it is the measurement that would justify the Hawkes extension of Wu,
Rambaldi, Muzy and Bacry (arXiv 1901.08938), who add self-excitation directly
to these arrival rates. Not implemented.
Tests use a synthetic Poisson book whose intensities are known in closed form,
written in the engine's own format and run through the real calibration path.
Coverage is tested as coverage: checking every bin against its own 95% band
would fail by construction, since one bin in twenty is supposed to exclude the
truth, so the test asserts the nominal rate over ~280 bins instead. That
catches both a biased estimate and a mis-scaled interval, and it caught the
real bugs here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Aroesler1
changed the title
Three gaps against the specification, found by re-reading it rather than by a test failing. 1. The per-event record was specified as carrying event type, side, level relative to best, queue size before the event, and inter-arrival time. It carried a signed queue index and no inter-arrival at all -- side was recoverable from the sign and inter-arrival was recomputed downstream, so nothing was wrong with the analysis, but neither was on the record as specified. Both are now fields, and the inter-arrival is per-queue, which is the waiting time the estimator is actually defined on. 2. That makes a real check possible, so it is now a test: Huang, Lehalle and Rosenbaum define the total intensity as the reciprocal of the mean waiting time at a queue, while this module uses events over exposure. They are the same estimator written two ways, and they now have to agree to within 10% on every well-populated bin of the synthetic book. If the two accountings ever diverged, one of them would be wrong. 3. The brief said to use the LOB-Bench battery. It was substituted with the same two metrics computed directly, justified on the grounds that LOB-Bench wants ten-level books while this model produces three levels a side. That justification does not hold: LOB-Bench ships cut_data_to_lvl and its volume statistics take an n_levels argument, so three levels is a shape it supports on purpose. The battery now runs for real, with the simulated book as the "generated" side. The battery result is worth having for a reason beyond compliance. The same suite scored the engine's reconstruction against the vendor at 0.000000 on every statistic; the fitted model scores 0.09 to 0.70. The earlier zeros were not the metric failing to notice a difference. Best row is limit-order depth at 0.094 on INTC -- the model does capture where traders post relative to the mid. Worst is inter-arrival at 0.62-0.64 on both sessions, which is the burstiness failure reached independently by someone else's code. MSFT's two zeros at the touch are recorded as a trap rather than a success: the model matches by being vacuously right about a queue that is essentially never there. time_to_cancel is excluded on purpose, since it needs order identity and a queue-size process has none; synthetic ids would produce a number rather than a measurement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI installs requirements-dev.txt when it exists and otherwise only pytest, so the Python tests failed at import with "No module named numpy" while the C++ tests passed. That reads like a broken suite rather than a missing environment, which is the worst way for it to fail. scipy is in the list and is not optional. It only appears inside function bodies, so an import-graph skim misses it, but validate_mbp10_vs_vendor and queue_reactive both use scipy.stats for the exact Garwood Poisson intervals and the tests exercise those paths. Verified by installing exactly this file into an empty virtualenv: 32 passed. numpy/pandas/pytest alone gives 1 failed, 15 errors. databento is deliberately excluded. It is imported only inside the functions that read a vendor .dbn file and no test touches one, so adding it would pull ~40MB into every CI run to install a package nothing under tests/ imports. matplotlib likewise: plotting entry points only. Also stops publishing handoff.md. It is cross-agent working notes -- scratch context for whoever picks the work up next -- not part of the repository. The file stays on disk and is now gitignored. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two tranches. The second is stacked on the first because it reuses the LOB-Bench tooling.
1. Validate on a second session, and score both with LOB-Bench
The headline correctness claim rested on one symbol-day. This adds INTC 2024-08-02, chosen to be a different kind of book: a one-tick spread 81.1% of the session with 2,400–3,000 shares at the touch, against MSFT's 1.0% and 63–68.
INTC reached 100.0000% on the first run with no code changes — the three MSFT fixes were corrections to a misread of Databento's event model, not tuning to one session.
Also:
fetch_databento_session.py(prices withmetadata.get_cost, downloads nothing without--confirm; both sessions $0.00); explicit slot accounting in the validator so "zero one-sided" is reproducible rather than prose;run_lob_bench.pyscoring both sessions at 0.000000 on all 12 statistics; multi-level OFI on both, where the large-tick book shows a 6.3× integration lift against MSFT's 2.0×.2. Calibrate the queue-reactive model, and measure where it breaks
The repo's first fitted model: Huang, Lehalle & Rosenbaum (JASA 2015).
The geometry carries it.
Q_isits(|i| − 0.5)ticks fromp_ref, not at a fixed offset from the best quote — that is what letsQ_1be genuinely empty when the spread widens. Indexing from the best quote would makeQ_1non-empty by construction and delete the mechanism.Having two sessions is what makes this worth doing. The queue-reactive model is a large-tick model, and the sessions land on opposite sides of that:
θ is a probability in the paper. An estimator returning 7.34 is the model failing loudly rather than quietly.
Stylized facts: two reproduce, one does not
λ^M28.50 / 1.96 / 0.09 per second at Q₊₁/₊₂/₊₃.corr(n, λ^C) = −0.425, and the per-order cancellation rate falls thirty-fold, 9.73 → 0.321, between n=1 and n=25.Where it fails
Fitting the paper's three intensities leaves Q₊₁ with 283,771 adds against 261,399 cancels and executions — +16.1M shares. A closed birth-death chain cannot run that surplus and the real queue does not grow, so the missing outflow is real: content leaving by re-indexing when
p_refmoves, which is not an order event. Simulated unbounded, Q₁ runs to 3,125 AES against a real 17.8.Closing the generator needs a fitted state-dependent price-move intensity replacing the scalar θ — it turns out to be a step function, 16.1/s at an empty touch, ~0 otherwise. That exposes a mutual deadlock: Q₊₁ drains only on a down move, which needs Q₋₁ empty, and vice versa. The real book escapes via cross-queue dependence Model I forbids: P(both touch queues > 25 AES) is 5.06% vs 1.88% under independence, and real 1s drift of Q₊₁ at fixed own size swings −3.3 → +1.2 AES/s with a queue Model I never looks at. Model IIb coupling plus a reflecting cap gives a usable comparison; the cap is an admission, not a fix, and the queue-size row stays wrong on purpose.
Order sizes were checked as the obvious suspect and are not the issue: the imbalance is worse in shares (+1.86 AES/s) than events (+0.96).
Burstiness fails as expected — real median inter-arrival 57 µs vs simulated 6.2 ms, while means differ by under 50%. That is the measurement that would justify the Hawkes extension of Wu, Rambaldi, Muzy & Bacry (arXiv 1901.08938), who add self-excitation directly to these arrival rates. Not implemented.
Tests
Synthetic Poisson book with intensities known in closed form, written in the engine's own format and run through the real calibration path. Coverage is tested as coverage — checking every bin against its own 95% band would fail by construction, since one bin in twenty is supposed to exclude the truth. Asserting the nominal rate over ~280 bins catches both a biased estimate and a mis-scaled interval, and it caught the real bugs here.
pytest tests -q30 pass (2 pre-existing + 28 new).ctest4/4.🤖 Generated with Claude Code