Repository navigation
DynamicCollection: range filters and sorts on an indexed double field return wrong results (keys are little-endian BitConverter bytes) #254
Description
Found while fixing #249 (see #252), and checked on its own. On a DynamicCollection with a B-tree index on a double field, BLQL range filters and sorts on that field return wrong results: documents are missing, or come back in the wrong order. The unindexed collection returns the right answer.
Repro
using var engine = new BLiteEngine(path);
var col = engine.GetOrCreateCollection("zz");
await col.CreateIndexAsync("ratio");
foreach (var (n, v) in new[] { ("a", 0.1), ("b", 2.2), ("c", 1.3), ("d", 5.7), ("e", -3.3), ("f", 1.0000001), ("nz", -0.0), ("pz", 0.0) })
await col.InsertAsync(col.CreateDocument(["name", "ratio"], b => b.AddString("name", n).AddDouble("ratio", v)));
col.Query("{\"ratio\": {\"$lt\": 2.0}}").ToList();
col.Query("{\"ratio\": {\"$gte\": 1.0, \"$lt\": 3.0}}").ToList();
col.Query().OrderBy("ratio").ToList();| Query | Without the index (correct) | With the index |
|---|---|---|
{ratio: {$lt: 2.0}} |
a, c, e, f, nz, pz | pz |
{ratio: {$gte: 1.0, $lt: 3.0}} |
b, c, f | (empty) |
{ratio: 0.0} |
nz, pz | pz |
OrderBy("ratio") |
e, pz, nz, a, f, c, b, d | pz, nz, e, b, a, f, d, c |
OrderByDescending("ratio").Take(3) |
d, b, c | c, d, f |
QueryIndexAsync(idx, 1.0, 3.0) |
— | (empty) |
{ratio: {$gte: 1.0}} happens to come out right with these values: its upper bound is MaxKey, and every document it should return has a key above the lower bound.
Mechanism (verified on 0450b97)
DynamicCollectionkeys a double asnew IndexKey(BitConverter.GetBytes(d)). This happens for stored values inBsonValueToIndexKey(src/BLite.Core/DynamicCollection.cs:2725) and for query bounds inCreateIndexKeyFromObject(:2812).BitConverter.GetBytesis little-endian IEEE 754: the first byte compared is the lowest mantissa byte and the sign bit is compared last, so key order has no relation to numeric order, and -0.0 and +0.0 get different keys. The typed path uses the order-preservingIndexKey(double)(IndexKey.cs:94-105).- Equality on a non-zero value still works, because both sides use the same bytes. Ranges and ordering do not.
- BLQL takes the index range as its only candidate source.
BlqlQuery.GetScanSource(src/BLite.Core/Query/Blql/BlqlQuery.cs:302) returnsQueryIndexAsync(index, min, max)whenever the filter field is indexed, and the residual filter can only drop documents, never add the ones the scan missed. - The single-key sort fast path (
BlqlQuery.cs:233-260) iterates the same index in key order, so the result comes out in byte order. - Dynamic B-tree indexes do not persist a
KeyFormat:PersistIndexMetadatasets it for full-text indexes only (DynamicCollection.cs:2209). Every existing dynamic B-tree index therefore reads back asIndexMetadata.LegacyKeyFormat.
Affected versions: the encoding dates from 39e6c89 (2026-02-21, BLiteEngine, first in v1.4.0). BLQL has used the index for range scans and sorts since 6364585 (2026-03-25, first in v3.8.1-preview.2), which is when this became visible through BLQL. Before that, only QueryIndexAsync with bounds was affected. Reproduced on v5.2.3 / 0450b97.
Fix considerations
- Key doubles with
new IndexKey(d)in both places. That is order-preserving, and with fix(index): -0.0 and +0.0 share one index key (#249) #252 it gives -0.0 and +0.0 one key. - Stored keys change, so existing dynamic indexes holding doubles must be rebuilt. An old key cannot be told apart by looking at it: it is a 9-byte
0x02…key, like anInt64. Two clean triggers:- Since dynamic B-tree indexes read back as
LegacyKeyFormat, writeCurrentKeyFormatfor them from now on and rebuild every dynamic B-tree index that reads back as legacy on open. This is the same pattern as the typed decimal migration (MigrateLegacyDecimalIndexes) and the dynamic full-text rebuild. It is a one-time rebuild of every dynamic B-tree index, including those without doubles. - Alternatively, fold it into the v6
KeyFormatbump planned for Typed secondary index keys are not prefix-free: values containing bytes 0x00–0x03 are mis-bounded #212.
- Since dynamic B-tree indexes read back as
DateTimeis unaffected: it is keyed throughIndexKey(long).
Happy to send the fix if you want it, with the first option unless you prefer to wait for the v6 bump.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsTodo
Activity
CaffeinatedCoder commentedon Oct 17, 2026
The DynamicCollection differential test (
tests/BLite.Tests/Oracles/DynamicDifferentialTests.cs, #272) reproduces this in 48 rows (every range, sort and== 0.0on the indexed double field throughQueryIndexAsyncand BLQL) and carries them as a known candidate linked to this issue; #269 is the sibling in the same key switch (cross-type numeric bounds, Decimal128 and Boolean).test(oracles): differential and model-based tests of DynamicCollection
docs(oracles): describe the DynamicCollection tests, issue drafts D1 …
mrdevrobot commentedon Oct 18, 2026
Reproduced on 0450b97: with a B-tree index on a double field of a DynamicCollection, BLQL ranges return nothing and sorts follow byte order. High. A fix changes stored keys, so existing dynamic indexes would need a rebuild.