Repository navigation
14.0.0: same candles live and in replay, deadlines, adaptive pacing, trades and quotes - #24
Merged
Merged
Conversation
…nts) The plan lists the gaps of 13.1 for aggregating the same events live and in replay, and the breaking changes that close them, in the order of the following revisions.
No built-in rule reads the candle that should_trigger receives, but the parameter couples every rule to one candle type. AggregationRule<E>, ModularCandle<E> and CandleComponentUpdate<E> now take any event type, so that later revisions can feed quotes and other events. The timestamp moves to a new Timestamped trait with a required TIMESTAMP_RESOLUTION constant: rules on a clock need the resolution before an event arrives, and the old default of milliseconds silently scaled every period for sources with nanosecond timestamps. Breaking, so the version is 14.0.0.
A consumer that removes duplicates after a replay, or joins candles by the time they became known, needs a sequence number and the open and close time of each candle. Bar<C> carries them with the number of events and the kind (closed by the rule or provisional). update() now pushes into an Extend sink instead of returning an Option, because the next revision closes candles on deadlines, and one event after a gap can close several. provisional(now) snapshots the unfinished candle for windows that end now. The aggregator asserts that time does not run backwards.
Time candles closed only when the next trade arrived, advanced by one period per trigger (so after a gap of k periods each of the next k trades closed a one-trade candle), produced nothing for empty periods and were stamped with the last trade instead of the period end. AggregationRule gets deadline(), on_deadline() and skip_to(). The aggregator closes the candle at the deadline once an event or a watermark (advance_to) reaches it, so a live consumer closes time candles on a heartbeat. update() calls advance_to() first, so an event at or after the end of a period never lands in it. Gaps::Flat emits empty candles for periods without events, at most max_candles per gap; the counter is part of the state, so heartbeats do not change the output. A test checks that heartbeats at any frequency give the same bars as a replay. TimeRule and AlignedTimeRule share a Clock: half-open periods, an Option instead of the 0 sentinel, rem_euclid for negative timestamps, an optional epoch offset, and the resolution from the event type instead of the constructor. Open, High and Low carry the last price over a reset, so an empty candle is flat at the previous close. The revision is about 900 lines, half of it tests; the deadline mechanism cannot be tested without a rule that uses it.
AnyRule closes at the earlier deadline of its rules and resets the other one, so AnyRule<R, AlignedTimeRule> closes at every period end without a trade, which gives candles a fixed boundary (e.g. UTC midnight) where the state is reproducible. MaxDuration closes at open + max_duration as a deadline, so quiet markets produce candles without waiting for a trade; its boundary is now inclusive like the time rules ([open, open + max)). MinDuration closes a pending trigger at open + min_duration, AllRule takes the deadline of the rule that has not triggered yet, and Adaptive forwards to its inner rule.
Adaptive averaged the activity per millisecond, threshold / duration, which aims at the harmonic mean of the candle durations. With skewed durations (bursts close candles in milliseconds, calm periods hold one for hours) that gives far fewer candles than the target. It now averages duration / threshold^p and sets threshold = (target / average)^(1/p). Scaling::Diffusive (p = 2) is for rules whose activity grows with the square root of time, such as the net flow of SignedVolumeRule: adaptive delta bars. A test checks the thresholds against a reference implementation of that pacing, and that the mean duration of the bars is near the target. For constant activity the linear case is unchanged.
A live process that restarts must otherwise replay from an earlier point to rebuild the state of the rules, e.g. the moving averages of Adaptive and of the imbalance and run bars. With the serde feature, every rule, its parameters and the GenericAggregator with its candle and counters now implement Serialize and Deserialize. A test saves and restores the aggregator at random points and checks that the bars equal those of an uninterrupted run. MillisecondPeriod deserializes through TryFrom<u64>, so a zero period is rejected. CI runs the tests with the serde feature too, as the checkpoint tests need it.
By::Quote summed the raw size and By::Base divided it by the price, which is right only when sizes are in the quote currency (inverse contracts). For spot and linear markets the size is in the base currency, and a dollar volume needs size * price, which By could not express. TakerTrade gets a required SIZE_UNIT constant (no default, as a wrong unit silently swaps volumes and notionals) and provided base_size() and quote_size(). By::Base and By::Quote now mean the currency of the sum. Trade keeps SizeUnit::Quote, so its results do not change.
The Candle derive implemented ModularCandle for one concrete input type (Trade, or the type of a PhantomData field named input), needed its traits in scope at the call site, guessed the value type of a getter from the first generic argument (wrong for Trades<T>) and panicked on bad input. - CandleComponent gets an associated Value type instead of a type parameter, so the getters return <Component as CandleComponent>::Value. - The derive implements ModularCandle<E> for every E that all components accept, supports generic candle structs, ignores PhantomData fields, uses absolute ::trade_aggregation paths and reports errors at compile time. - It generates FIELDS with the component names, and with #[candle(record = Name, record_derive(...))] a plain record struct of the values, e.g. to store closed candles. - CandleComponentUpdate::close_at and ModularCandle::close_at run before a candle is emitted (also provisional ones), for statistics that are weighted by time up to the end of the candle. trade_aggregation_derive is 0.5.0.
Candles lose the path of the trades inside them, so statistics of that path have to be components: - BaseVolume and QuoteVolume sum the sizes in a given currency, whatever the size unit of the trades; Volume keeps summing the raw size. - LargestTrade is the largest size in the base currency. - RealizedVariance, BipowerVariation and RealizedQuarticity of the log returns between trades. Each keeps the last price over a reset, so the first return of a candle starts at the previous close, no return is lost at a boundary, and the sum over candles equals the whole series. Fixes: Vpin was public with todo!() in every method and is removed. VolumeBuys and VolumeSells no longer count a size of +0.0 or -0.0 as a side. TimeVelocity no longer rounds the duration down to whole seconds.
A candle that also holds quote statistics over the same interval as its trade statistics needs one aggregator that sees both streams of a symbol in time order. MarketEvent<T, Q> is a trade or a BestQuote (best bid and ask, with mid and weighted average price); both must have the same timestamp resolution, which is checked at compile time. Trade rules and trade components now take any E: AsTrade and ignore events that are not trades. A blanket impl makes every TakerTrade an AsTrade of itself, so plain trade streams keep working unchanged. Time rules and the timestamp components act on every event. Quote is a simple millisecond BestQuote, the counterpart of Trade.
With MarketEvent a candle can hold quote statistics next to its trade statistics, which a later stage cannot rebuild from OHLC values: - MidOpen, MidHigh, MidLow and MidClose of the mid price, starting at the mid in effect when the candle opened; - QuoteUpdates and WapRealizedVariance (of the weighted average price); - TimeWeightedSpread and TimeWeightedImbalance, averaged over time up to the close of the candle through close_at; - OrderFlowImbalance (Cont, Kukanov and Stoikov 2014); - EffectiveSpread of the trades against the last mid price. The components keep the quote in effect over a reset, so a candle without quote updates still has the market's spread and mid. Values are None while no quote has arrived, so a record can store them as NULL instead of 0. Order book depth stays out of scope: a full book is the wrong state for a candle component.
The chrono feature did not compile: OpenDateTime matched a TimestampResolution::Second that never existed, used deprecated chrono constructors, and its test expected 1985 for a timestamp of 2023. The value is now an Option, None before the first event or for a timestamp outside of the range of DateTime, instead of a panic or a silent default. CI runs the tests with all features, so a feature cannot break unnoticed again.
The plan now says which steps landed and where the implementation differs from it (associated Value type, MaxDuration and AllRule deadlines, open_ts after gaps, Quote, Option values of the quote components, the chrono fix), so a reader of the plan does not assume the original design.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
13.1 does not let one aggregator produce the same candles from a live stream and from a historical replay. It also cannot say when a candle became known. The full list of gaps and the plan are in docs/2026-11-12_v14_plan.md. The main gaps were:
• Time candles closed only when the next trade arrived. After a gap of k periods, each of the next k trades closed a candle with one trade in it.
• Empty periods produced no candle, and a time candle was stamped with its last trade instead of the end of its period.
• Adaptive aimed at the harmonic mean of the bar duration. When bar durations are skewed, this produced far fewer candles than the target.
• Closed candles carried no metadata, and no state could be saved for a restart.
• By::Base and By::Quote were correct only for inverse contracts.
• The Candle derive worked for only one input type.
What changes
Events and rules
• New Timestamped trait with a required TIMESTAMP_RESOLUTION constant.
• TakerTrade: Timestamped has a required SIZE_UNIT constant (Base or Quote) and provided base_size() and quote_size() methods. By::Base and By::Quote now mean the currency of the sum, for spot, linear and inverse markets.
• AggregationRule no longer receives the candle. No built-in rule read it.
• Rules get three provided methods: deadline(), on_deadline() and skip_to().
Aggregator
• update(&event, &mut out) sends each closed candle to out as a Bar with seq, open_ts, close_ts, duration(), num_events and kind (Rule, Deadline or Provisional).
• advance_to(watermark, &mut out) closes candles at their deadline without waiting for an event, for example on a heartbeat.
• with_gaps(Gaps::Flat { max_candles }) emits flat candles for empty periods. The default Gaps::Skip emits nothing.
• provisional(now) returns a snapshot of the unfinished candle with its metadata.
• The aggregator asserts that events and watermarks arrive in time order.
Time rules
• TimeRule and AlignedTimeRule use half-open periods [start, end) and close at the period end.
• They skip a gap in one step instead of producing one candle per trade.
• They handle negative timestamps.
• AlignedTimeRule::with_offset_ms shifts the periods, for example to start weekly candles on a Monday.
• The resolution comes from the event type, not from the constructor.
Combinators
• They pass the deadlines of their inner rules through.
• AnyRule<R, AlignedTimeRule> closes candles at fixed boundaries, for example UTC midnight.
• MaxDuration closes at exactly open + max without waiting for a trade.
Adaptive
• It now averages duration / threshold^p, which aims at the arithmetic mean duration.
• New with_scaling(Scaling::Diffusive) for rules whose activity grows with the square root of time, such as the net flow of a SignedVolumeRule (adaptive delta bars).
Saving state
• With the serde feature, all rules, their parameters, GenericAggregator and Bar implement Serialize and Deserialize.
• A zero MillisecondPeriod is rejected when deserialized.
Candle derive (trade_aggregation_derive 0.5.0)
• CandleComponent has an associated Value type instead of a type parameter.
• The derive implements ModularCandle for every event type that all components accept, and supports generic candle structs.
• It uses absolute paths, so nothing needs to be imported where it is used.
• It generates a FIELDS constant, and an optional record struct of the values with #[candle(record = Name, record_derive(...))].
• Invalid input is a compile error instead of a panic.
• New hook close_at(timestamp) lets time-weighted components integrate up to the end of the candle.
Trades and quotes in one candle
• MarketEvent<T, Q> holds either a trade or a quote, with a compile-time check that both use the same timestamp resolution.
• AsTrade and AsQuote let trade rules and trade components ignore quotes, and quote components ignore trades. Plain trade types work unchanged.
• New BestQuote trait and Quote type.
New components
• Trades: BaseVolume, QuoteVolume, LargestTrade, RealizedVariance, BipowerVariation, RealizedQuarticity.
• Quotes: MidOpen, MidHigh, MidLow, MidClose, QuoteUpdates, WapRealizedVariance, TimeWeightedSpread, TimeWeightedImbalance, OrderFlowImbalance, EffectiveSpread.
• The return-based components keep the last price across candles, so no return is lost at a candle boundary. Quote values are None until the first quote.
Fixes
• Removed Vpin, which was public but todo!() in every method.
• VolumeBuys and VolumeSells no longer count trades of size ±0.0.
• TimeVelocity no longer rounds down to whole seconds.
• Open, High and Low of an empty candle show the previous close instead of stale values.
• The chrono feature compiles again (OpenDateTime).
CI
• Tests also run with --all-features.
• The unused .buildkite directory is removed.