Skip to content

DynamicCollection: range filters and sorts on an indexed double field return wrong results (keys are little-endian BitConverter bytes) #254

Open

Description

@CaffeinatedCoder

Found while fixing #249 (see #252), and checked on its own. On a DynamicCollection with a B-tree index on a double field, BLQL range filters and sorts on that field return wrong results: documents are missing, or come back in the wrong order. The unindexed collection returns the right answer.

Repro

using var engine = new BLiteEngine(path);
var col = engine.GetOrCreateCollection("zz");
await col.CreateIndexAsync("ratio");
foreach (var (n, v) in new[] { ("a", 0.1), ("b", 2.2), ("c", 1.3), ("d", 5.7), ("e", -3.3), ("f", 1.0000001), ("nz", -0.0), ("pz", 0.0) })
    await col.InsertAsync(col.CreateDocument(["name", "ratio"], b => b.AddString("name", n).AddDouble("ratio", v)));

col.Query("{\"ratio\": {\"$lt\": 2.0}}").ToList();
col.Query("{\"ratio\": {\"$gte\": 1.0, \"$lt\": 3.0}}").ToList();
col.Query().OrderBy("ratio").ToList();
Query Without the index (correct) With the index
{ratio: {$lt: 2.0}} a, c, e, f, nz, pz pz
{ratio: {$gte: 1.0, $lt: 3.0}} b, c, f (empty)
{ratio: 0.0} nz, pz pz
OrderBy("ratio") e, pz, nz, a, f, c, b, d pz, nz, e, b, a, f, d, c
OrderByDescending("ratio").Take(3) d, b, c c, d, f
QueryIndexAsync(idx, 1.0, 3.0) — (empty)

{ratio: {$gte: 1.0}} happens to come out right with these values: its upper bound is MaxKey, and every document it should return has a key above the lower bound.

Mechanism (verified on 0450b97)

  • DynamicCollection keys a double as new IndexKey(BitConverter.GetBytes(d)). This happens for stored values in BsonValueToIndexKey (src/BLite.Core/DynamicCollection.cs:2725) and for query bounds in CreateIndexKeyFromObject (:2812). BitConverter.GetBytes is little-endian IEEE 754: the first byte compared is the lowest mantissa byte and the sign bit is compared last, so key order has no relation to numeric order, and -0.0 and +0.0 get different keys. The typed path uses the order-preserving IndexKey(double) (IndexKey.cs:94-105).
  • Equality on a non-zero value still works, because both sides use the same bytes. Ranges and ordering do not.
  • BLQL takes the index range as its only candidate source. BlqlQuery.GetScanSource (src/BLite.Core/Query/Blql/BlqlQuery.cs:302) returns QueryIndexAsync(index, min, max) whenever the filter field is indexed, and the residual filter can only drop documents, never add the ones the scan missed.
  • The single-key sort fast path (BlqlQuery.cs:233-260) iterates the same index in key order, so the result comes out in byte order.
  • Dynamic B-tree indexes do not persist a KeyFormat: PersistIndexMetadata sets it for full-text indexes only (DynamicCollection.cs:2209). Every existing dynamic B-tree index therefore reads back as IndexMetadata.LegacyKeyFormat.

Affected versions: the encoding dates from 39e6c89 (2026-02-21, BLiteEngine, first in v1.4.0). BLQL has used the index for range scans and sorts since 6364585 (2026-03-25, first in v3.8.1-preview.2), which is when this became visible through BLQL. Before that, only QueryIndexAsync with bounds was affected. Reproduced on v5.2.3 / 0450b97.

Fix considerations

  • Key doubles with new IndexKey(d) in both places. That is order-preserving, and with fix(index): -0.0 and +0.0 share one index key (#249) #252 it gives -0.0 and +0.0 one key.
  • Stored keys change, so existing dynamic indexes holding doubles must be rebuilt. An old key cannot be told apart by looking at it: it is a 9-byte 0x02… key, like an Int64. Two clean triggers:
    • Since dynamic B-tree indexes read back as LegacyKeyFormat, write CurrentKeyFormat for them from now on and rebuild every dynamic B-tree index that reads back as legacy on open. This is the same pattern as the typed decimal migration (MigrateLegacyDecimalIndexes) and the dynamic full-text rebuild. It is a one-time rebuild of every dynamic B-tree index, including those without doubles.
    • Alternatively, fold it into the v6 KeyFormat bump planned for Typed secondary index keys are not prefix-free: values containing bytes 0x00–0x03 are mis-bounded #212.
  • DateTime is unaffected: it is keyed through IndexKey(long).

Happy to send the fix if you want it, with the first option unless you prefer to wait for the v6 bump.

Activity

  1. CaffeinatedCoder commented on Oct 17, 2026

    @CaffeinatedCoder
    CollaboratorAuthor

    The DynamicCollection differential test (tests/BLite.Tests/Oracles/DynamicDifferentialTests.cs, #272) reproduces this in 48 rows (every range, sort and == 0.0 on the indexed double field through QueryIndexAsync and BLQL) and carries them as a known candidate linked to this issue; #269 is the sibling in the same key switch (cross-type numeric bounds, Decimal128 and Boolean).

  2. added
    severity/highWrong results, crash or failed writes in a realistic scenario
    and removed on Oct 18, 2026
  3. mrdevrobot commented on Oct 18, 2026

    @mrdevrobot
    Contributor

    Reproduced on 0450b97: with a B-tree index on a double field of a DynamicCollection, BLQL ranges return nothing and sorts follow byte order. High. A fix changes stored keys, so existing dynamic indexes would need a rebuild.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/indexingB-Tree and secondary indexes, index lifecyclebugSomething isn't workingpackage/coreAffects BLite.Coreseverity/highWrong results, crash or failed writes in a realistic scenario

    Type

    No type

    Projects

    • Status
      Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions