Secondary index on a `decimal` property keys by `(double)value`: `==` and `>=` return wrong documents

[CaffeinatedCoder]

Version: BLite 5.1.1 (NuGet), .NET 10.0.12, macOS 27 arm64 (the code path is platform-independent)

What happens

decimal values round-trip exactly (thanks for GetBits), and typed LINQ without an index compares them exactly. But once the property has a secondary index, the index key is built from (double)value:

src/BLite.Core/Indexing/CollectionSecondaryIndex.cs:472 (current main)

decimal decimalVal => new IndexKey((double)decimalVal),

All decimals that map to the same double collapse onto one key, so index-backed queries return wrong results. BLQL has the same problem with or without an index: BsonValueComparer converts Decimal128 to double (src/BLite.Core/Query/Blql/BsonValueComparer.cs, ToDouble).

Repro

public sealed class Figure        { public ObjectId Id { get; set; } public decimal Amount { get; set; } }
public sealed class IndexedFigure { public ObjectId Id { get; set; } public decimal Amount { get; set; } }

public partial class ReproDbContext : DocumentDbContext
{
    public DocumentCollection<ObjectId, Figure> Plain { get; set; } = null!;
    public DocumentCollection<ObjectId, IndexedFigure> Indexed { get; set; } = null!;
    public ReproDbContext(string path) : base(path) => InitializeCollections();
    protected override void OnModelCreating(ModelBuilder mb)
    {
        mb.Entity<Figure>();
        mb.Entity<IndexedFigure>().HasIndex(x => x.Amount);
    }
}

decimal[] values = [1234567890123456.77m, 1234567890123456.78m, 1234567890123456.79m, 1234567890123456.80m];
var probe = 1234567890123456.78m;   // all four are 1234567890123456.8 as double

using var db = new ReproDbContext(path);
foreach (var v in values)
{
    await db.Plain.InsertAsync(new Figure { Amount = v });
    await db.Indexed.InsertAsync(new IndexedFigure { Amount = v });
}
var plain   = await db.Plain.AsQueryable().Where(f => f.Amount == probe).ToListAsync();
var indexed = await db.Indexed.AsQueryable().Where(f => f.Amount == probe).ToListAsync();

// BLQL
using var engine = new BLiteEngine(path2);
var col = engine.GetOrCreateCollection("figures");
foreach (var v in values)
    await col.InsertAsync(col.CreateDocument(["amount"], b => b.AddDecimal("amount", v)));
var blql = col.Query().Filter(BlqlFilter.Eq("amount", BsonValue.FromDecimal(probe))).ToList();

Output

Where(Amount == probe)
  no index:   1234567890123456.78
  with index: 1234567890123456.79, 1234567890123456.78, 1234567890123456.77, 1234567890123456.80
Where(Amount >= probe)
  no index:   1234567890123456.79, 1234567890123456.80, 1234567890123456.78
  with index: 1234567890123456.79, 1234567890123456.78, 1234567890123456.77, 1234567890123456.80
BLQL $eq probe (no index)
  1234567890123456.80, 1234567890123456.77, 1234567890123456.79, 1234567890123456.78

Expected: exactly …56.78 for == and …56.78, …56.79, …56.80 for >=, in all three cases.

The values in the repro are large to make the collision obvious, but any decimal with more than ~15–17 significant digits is affected. That is common for money amounts kept at 4 decimal places, and it is the reason to use decimal in the first place.

Possible fix (for discussion)

An order-preserving byte encoding for decimal in IndexKey, e.g. a sign/exponent-normalised 96-bit mantissa (strip trailing zeros so 5 and 5.00 get the same key), plus a Decimal128 branch in BsonValueComparer that compares as decimal when both sides are Decimal128 (and as decimal against Int32/Int64 too). Existing indexes built with double keys would need a rebuild, so this probably needs a new key prefix or an index-format version.