LOB Engine and Queue Research

A deterministic C++ market-by-order limit order book engine that rebuilds Nasdaq order books from Databento order-level data, plus Python studies of queue position, latency and order-flow simulation that run on those books.

Questions a market maker cares about, such as what a place at the front of the queue is worth or what a millisecond of delay costs, need the full order-by-order book: you have to know how many shares stood ahead of a specific order at a specific moment. Snapshot feeds cannot answer that. I built the engine first, checked it cell by cell against the vendor's own 10-level book, and then used the exact book to price queue position and latency and to test how realistic a fitted order-flow simulator is.

Results

Sample: 15 symbol-days in 2024, five each for MSFT, INTC and AAPL, from Databento XNAS.ITCH (Nasdaq TotalView-ITCH) MBO data. The vendor cross-check uses two of them, MSFT 2024-06-03 and INTC 2024-08-02. Session list in DATA.md.

Result	Value	Source
Engine book vs Databento's own MBP-10, all 10 levels	100.0000% agreement on 53,551,466 (MSFT) and 60,128,762 (INTC) compared cells; 0 slots where only one side shows a level	vendor_check/
Replay speed, MSFT 2024-06-03 (3,862,854 messages), map backend, one core of an Apple M4, Release build	69.4 ns/msg for the book replay (14.4M msgs/sec); 574 ns/msg including CSV parsing	replay_latency.csv
Front vs back of the queue, all fills marked to the mid 10 s later	6,323,682 new orders at the best quote; the front decile beats the back in 6 of 15 sessions, and its own markout is negative in 14 of 15	summary.csv
Cost of latency for a one-share quote at the touch	loses money at every delay on 13 of 15 sessions; on MSFT, net P&L per fill ranges from -0.511 to +0.260 ticks at zero delay and from -3.536 to -1.612 ticks at 100 ms	claims.csv, latency_curve.csv
Hawkes self-excitation added to the queue-reactive simulator	the distance between simulated and real event-timing distributions shrinks in 15 of 15 sessions (mean W₁ 1.556 to 0.460); the median time between events lands within 2x of real in 6 of 15 sessions, against 37x to 189x too long without it	claims.csv, compare_*.csv
How realistic the improved simulator is	still easy to tell apart from the market: lowest classifier AUC 0.972; mean touch queue size moves further from real in 10 of 15 sessions	claims.csv

Every row was regenerated on 2026-10-23 from the local session files after the fixes listed in the audit history. Rerunning the first two needs the licensed feed; the others are recomputed offline from the published aggregate CSVs by scripts/verify_evidence.py and scripts/verify_queue_payoff.py.

How it works

1. Convert: scripts/databento_to_lobster.py turns Databento MBO records into the six-column LOBSTER message format. Databento reports one displayed execution as three records (a trade print, a fill and a mirror cancel), so the converter drops the duplicate print and the mirror cancel, keyed on sequence number and order id. A sidecar file keeps each row's vendor sequence number.
2. Replay: The C++ engine applies every add, cancel and execution. Orders live in a hash map keyed by order id; price levels live in either a std::map or a sorted std::vector, behind one interface and tested for identical output. It can write the top-N book and per-message analytics (spread, depth, imbalance, order-flow imbalance, VWAP, realized volatility) after every message.
3. Validate: scripts/validate_mbp10_vs_vendor.py diffs the engine's 10-level book against Databento's MBP-10, aligned on sequence number, and accounts for every slot, including levels that only one side reports. Getting to exact agreement took three fixes, two in the converter and one in the comparison itself.
4. Price queue position: For each new order at the best bid or ask, scripts/queue_position_value.py and scripts/queue_payoff.py record the shares ahead of it, every fill, and the mid 10 seconds after each fill, then compare the front decile of the queue with the back.
5. Price latency: scripts/latency_cost_curve.py replays a one-share quote at the best bid and ask through the real message stream with delays from 0 to 100 ms, and marks each fill to the mid 10 seconds later.
6. Simulate and test: scripts/queue_reactive.py fits the Huang-Lehalle-Rosenbaum queue-reactive model, scripts/hawkes.py adds a Hawkes self-excitation term at the touch, and both simulators are scored against the real sessions with distribution distances and a classifier that tries to tell simulated windows from real ones (scripts/simulator_turing_test.py).

Limitations

• Small, hand-picked sample: three names on 15 days, including an earnings reaction day (INTC 2024-08-02). The results describe these sessions only.
• The vendor cross-check covers two sessions. Timings come from one unpinned laptop core with other jobs running, so they are host-specific.
• Queue deciles are retrospective groups that differ in order size and lifetime, so they do not give a causal value of priority. Markouts are gross, before fees and rebates.
• The September LOB-Bench scores of the two simulators were not re-run after the fixes, because that needs a clone of LOB-Bench. They stay in report/lob_bench_*.csv for the record and are not cited. The Hawkes excitation is fitted against the uncoupled queue-reactive baseline but simulated on the coupled one.

Repository layout

Path	Contents
include/lob/, src/	C++ engine: parser, order book (two backends), replay, analytics, lob_engine CLI
benchmark/	replay throughput harness (lob_benchmark)
tests/	C++ unit tests (run by ctest) and Python tests for the scripts and published evidence
scripts/	data conversion, vendor cross-check, research studies, offline verifiers
data/	five tiny synthetic message files for tests; no vendor data
report/	published aggregate results (CSV) and figures
docs/	full research notes, audit history, evidence audit, literature notes
DATA.md	data source, sessions, licensing

Running it

Everything below runs on a fresh clone with no market data. It needs CMake, a C++17 compiler and Python 3.11+.

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build
ctest --test-dir build --output-on-failure

python -m pip install -r requirements-dev.txt
python -m pytest -q
python scripts/verify_evidence.py --check
python scripts/verify_queue_payoff.py --check

build/lob_engine data/AAPL_sample_messages.csv --backend both --depth 5
build/lob_engine data/AAPL_sample_messages.csv --analytics-out build/analytics.csv

The research scripts need licensed Databento XNAS.ITCH data, which is not in this repository. With the extracts in $DATABENTO_RAW_DIR (layout in DATA.md):

python scripts/build_sessions.py          # convert and replay all 15 sessions
python scripts/queue_position_value.py
python scripts/queue_payoff.py
python scripts/latency_cost_curve.py
python scripts/session_profile.py
python scripts/hawkes.py                  # also writes the simulated paths
python scripts/simulator_turing_test.py
python scripts/multi_level_ofi.py --sweep-out report/ofi_sweep.csv
python scripts/backend_sweep.py           # timings: run on a quiet machine

The per-session vendor cross-check and replay-timing commands are in the research notes.

More detail

• Full research notes: every table, the three fixes behind exact vendor agreement, where the queue-reactive model breaks, the Hawkes and classifier results, multi-level order-flow imbalance, and performance by backend.
• Audit history: every number corrected by the October 2026 re-run, old and new, and why it changed.
• Data provenance and licensing
• Evidence audit, verified claims and file manifest with hashes
• Literature notes
• Benchmark note for the small test fixtures
• Archived tables from the earlier README