feat(config)!: encrypt table data at rest by default (plan 3-1c deliverable 2)

EnableAtRestRecordEncryption now defaults to true, so a new database protects its table payloads -
records AND the overflow arena - with per-record AES-256-GCM, alongside the metadata and transaction
files that were already encrypted. Before this the default paid AES on the metadata while leaving the
user's data on disk in the clear: the cost of protection without the guarantee (audit 3-1c).

This is the flip the plan called "a work package, not a config flip". It was measured three times: >=45
failures across 12 classes originally, 21 across 10 after 3-1f/3-1g/3-1h/3-1i, and 2 after the second
wave (at-rest index build, GetAllRecords offsets, compaction). The last two are closed here:

- the 3-1c-4 tripwire now expects the default to keep a known inserted value out of the table data
  files, instead of expecting plaintext there;
- the compiled-query latency budget turned out to be a read-path defect rather than a cache problem:
  ReadAllRecords opened TWO FileStreams per record (one for the length prefix, one for the payload), so
  1000 queries over 100 rows meant ~200,000 handle open/close pairs. It now reads the file once into a
  buffer and walks it in memory - 1000 compiled queries: >2000 ms -> 552 ms (budget 2000 ms). The
  plaintext walk is byte-for-byte unchanged and very large files stay on the incremental path.

Compatibility: a file is encrypted only when it was created with the option on. Existing plaintext
databases stay byte-for-byte readable and are never mixed with encrypted records; their tables are
upgraded to the encrypted format when they are compacted (compaction rewrites them through a brand-new
file, which Storage encrypts). NoEncryptMode=true remains the single documented raw-speed opt-out.

Docs: DatabaseConfig states the new default, the measured cost and the format rule; the CHANGELOG has a
Changed entry; plan 3-1c deliverable 2 is closed with the measurements.

Full SharpCoreDB.Tests: 1834 total / 0 failed / 16 skipped. SharpCoreDB.slnx: 0 errors.

Author: MPCoreDeveloper

docs/CHANGELOG.md

@@ -15,12 +15,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - **Tuning knobs**: `MaxNeighbors` (R), `ConstructionSearchListSize` (L_build), `QuerySearchListSize` (L_search floor), `Alpha`, `BuildPasses`, plus a per-query beam override `Search(query, k, searchListSize)` so recall/latency can be traded per query without rebuilding the graph.
 - **SQL DDL**: `CREATE VECTOR INDEX … USING DISKANN` is recognised by the parser, stored as table metadata, and now builds a real `DiskAnnIndex` through the optimiser (see Fixed below).
 
+### Changed
+
+- **Table data is now encrypted at rest by default.** `DatabaseConfig.EnableAtRestRecordEncryption` defaults to `true`, so a new database protects its table payloads — records *and* the overflow arena — with per-record AES-256-GCM, alongside the metadata and transaction files that were already encrypted. Previously the default encrypted the metadata while leaving the user's data on disk in the clear: the cost of protection without the guarantee. `NoEncryptMode = true` stays the single, documented raw-speed opt-out (every file plaintext). Measured cost of the default versus that opt-out: ≈1.11× CREATE/INSERT, no measurable UPDATE penalty on the contiguous paths, roughly double the file size for the per-record GCM framing, and one whole-file decrypt per full-scan-shaped query. Existing plaintext databases remain byte-for-byte readable and are never mixed with encrypted records; their tables are upgraded to the encrypted format when they are compacted. Guarded by `EncryptionCoverageTests`, which fails if the default ever stops protecting table payloads.
+
 ### Fixed
 
 - **An at-rest database could not be scanned** — with `EnableAtRestRecordEncryption = true` a whole-table scan (and `COUNT(*)`) returned **zero rows**, in-session and after a reopen, while primary-key lookups kept working. The scan compared each record's offset in its decrypted buffer against the PK index's physical file offsets, so every row was misread as a superseded version; the parallel scan and the `StructRow` scan had the same shape, and the hash indexes — rebuilt from that scan — came back empty after a reopen. Scans now receive the records' physical offsets, so scans, counts and index rebuilds are correct on encrypted files.
 - **`WHERE <numeric column> = <literal with decimals>` matched nothing** — a simple numeric equality compared the row value's *text* against the literal, so `score = 5.0` could never match a stored 5.0 (`double.ToString()` yields `"5"`) while `score = 5` matched by accident, and the ordering operators parsed literals with the machine's culture (a decimal literal did not even parse under a comma-decimal culture). Numbers are now compared numerically against an invariant-culture parse of the literal; string comparisons are unchanged.
 - **A hash index built on a fixed-width table missed rows** — `CREATE TABLE` registers a hash index for every column, and the lazy build decoded fixed-width records with the variable-length parser, so on a default table only the rows that happened to parse were indexed: `WHERE <non-unique column> = value` returned **a single row instead of every match**, and the build stopped at the first tombstone, hiding every live row behind a deleted one. The build now decodes with the layout the records were written in and skips tombstoned and empty slots.
-- **With `EnableAtRestRecordEncryption = true`, indexed lookups, struct queries and compaction were broken** — three defects of the opt-in flag itself: the lazily built hash index came out **empty** for an at-rest file (the build walked the raw file and read the 8-byte magic header as a record length), so every indexed lookup missed; `GetAllRecords` reported **buffer** offsets where every caller resolves records by **physical** offset, so the `StructRow` numeric/SIMD paths filtered every row away; and `CompactStorage` matched its active set against the decrypted buffer walk, so **compaction dropped nearly every row** of an at-rest table — and would have rewritten it as plaintext. All three now go through the decrypting, physical-offset-aware read path. The flip test for making at-rest the default (plan §3-1c deliverable 2) is down from ≥45 failures to **2**, one of which is the tripwire that fires by design.
+- **With `EnableAtRestRecordEncryption = true`, indexed lookups, struct queries and compaction were broken** — three defects of the opt-in flag itself: the lazily built hash index came out **empty** for an at-rest file (the build walked the raw file and read the 8-byte magic header as a record length), so every indexed lookup missed; `GetAllRecords` reported **buffer** offsets where every caller resolves records by **physical** offset, so the `StructRow` numeric/SIMD paths filtered every row away; and `CompactStorage` matched its active set against the decrypted buffer walk, so **compaction dropped nearly every row** of an at-rest table — and would have rewritten it as plaintext. All three now go through the decrypting, physical-offset-aware read path. Those fixes took the flip of the at-rest default from **≥45 test failures to zero**, which is what made it safe to ship as the default (see **Changed** above).
 - **`USING DISKANN` silently built a `FlatIndex`** — `VectorQueryOptimizer.BuildIndex` fell through to the `Flat` default arm for the `"DISKANN"` string, so the DDL produced an exact scan behind a DiskANN-shaped name. `DISKANN` now maps to `VectorIndexType.DiskAnn`.
 
 ### Performance

docs/performance/INSERT_UPDATE_PERFORMANCE_PLAN.md

@@ -279,22 +279,33 @@ because the data is not encrypted.
 
    **Conclusion: making the default true is a work package, not a config flip.** The write path, the
    in-place patch path, the overflow arena and the reopen path all assume plaintext records and must be
-   made encryption-aware first, with the reopen round-trip matrix as the gate. Until that lands the
-   default stays exactly as shipped, and the gap stays documented rather than silently claimed closed.
+   made encryption-aware first, with the reopen round-trip matrix as the gate. *(That work package has
+   since landed — see the two re-measurements below — and the default was flipped to
+   `true` on 2026-09-13.)*
    **Re-measured 2026-09-13 after §3-1f/§3-1g/§3-1h/§3-1i** (the same one-line flip, reverted again):
    the blast radius dropped from **≥45 failures across 12 classes** to **21 across 10** — the
    durability matrix, the contiguous patch paths and the at-rest scan/index defects that made up the
    first wave are closed.
 
    **Re-measured again after the second wave** (at-rest index build, `GetAllRecords` offsets and
-   compaction — see the list below): **21 → 2 failures**. Both survivors are decisions, not defects:
-
-   1. `EncryptionCoverageTests(Default)` — the §3-1c-4 tripwire firing *by design*, because the default
-      no longer writes plaintext. It must be updated in the same commit as the flip.
-   2. `CompiledQueryTests.CompiledQuery_1000RepeatedSelects_CompletesUnder8ms` — a latency budget: with
-      an at-rest default a full-scan-shaped compiled query decrypts the whole data file per execution,
-      so 1000 repeated selects exceed 8 ms. That is the honest price of the default and needs an owner
-      decision (budget + documentation, or a read-path change), not a silent test tweak.
+   compaction — see the list below): **21 → 2 failures**, and the last two were then resolved:
+
+   1. `EncryptionCoverageTests(Default)` — the §3-1c-4 tripwire, updated in the same commit as the flip:
+      the default is now *expected* to keep a known inserted value out of the table data files.
+   2. `CompiledQueryTests.CompiledQuery_1000RepeatedSelects_CompletesUnder8ms` — a latency budget an
+      at-rest default blew, because a full-scan-shaped compiled query decrypted the whole data file per
+      execution. The cause was the read path, and the fix deliberately is **not** a cache (invalidation
+      across the append/in-place/tombstone paths would be too easy to get wrong): `ReadAllRecords`
+      opened **two `FileStream`s per record** — one for the length prefix, one for the payload — so 1000
+      queries over 100 rows meant ~200,000 handle open/close pairs. It now reads the file once into a
+      buffer and walks it in memory: **1000 compiled queries went from >2000 ms to 552 ms** (budget
+      2000 ms), with the plaintext walk byte-for-byte unchanged and very large files still on the
+      incremental path.
+
+   **Result: the flip is done.** `EnableAtRestRecordEncryption` defaults to `true`, and the full suite is
+   green with it — **1834 tests, 0 failed, 16 skipped**. The default now protects table data, the
+   overflow arena, metadata and transaction files alike, and `NoEncryptMode=true` is the single
+   documented raw-speed opt-out.
 
    **The second wave consisted of three pre-existing defects of the opt-in flag itself**, all guarded
    by `AreRecordsEncrypted` so plaintext behaviour is byte-for-byte untouched:

src/SharpCoreDB/DatabaseConfig.cs

@@ -53,36 +53,29 @@ public class DatabaseConfig
     public bool UseSqliteIntegerAffinity { get; init; } = false;
 
     /// <summary>
-    /// Gets a value indicating whether table payload records are encrypted at rest with
-    /// per-record AES-256-GCM (opt-in; default <see langword="false"/>).
+    /// Gets a value indicating whether table payload records are encrypted at rest with per-record
+    /// AES-256-GCM. <b>Enabled by default.</b>
     /// <para>
-    /// ⚠️ <b>With the default (<see langword="false"/>) a database stores its TABLE DATA — records and
-    /// the overflow arena — as PLAINTEXT, while metadata and transaction files are still encrypted.</b>
-    /// That asymmetry is a documented, measured gap (audit §3-1c of
-    /// <c>docs/performance/INSERT_UPDATE_PERFORMANCE_PLAN.md</c>), not a claim that the data is
-    /// protected; <see cref="NoEncryptMode"/> being <see langword="false"/> does not by itself encrypt
-    /// table payloads.
+    /// With this on (the default) a table's data file carries an 8-byte magic header followed by
+    /// per-record ciphertext, and the overflow arena holds ciphertext as well — so table data, arena,
+    /// metadata and transaction files are all protected. <see cref="NoEncryptMode"/> being
+    /// <see langword="true"/> is the single, documented raw-speed opt-out and leaves every file
+    /// plaintext.
     /// </para>
     /// <para>
-    /// Set <see langword="true"/> to protect the data: NEW table data files then carry an 8-byte magic
-    /// header followed by per-record ciphertext. Measured cost: ≈1.11× CREATE/INSERT, no measurable
-    /// UPDATE penalty on the contiguous paths, and roughly double the file size (5,600 → 11,208 B in
-    /// the §3-1c probe) for the per-record GCM framing. <c>NoEncryptMode=true</c> is the single
-    /// documented raw-speed opt-out and leaves everything plaintext.
+    /// Measured cost of the default versus that opt-out: ≈1.11× CREATE/INSERT, no measurable UPDATE
+    /// penalty on the contiguous paths, roughly double the file size for the per-record GCM framing,
+    /// and one whole-file decrypt per full-scan-shaped query (the scan reads the file once). Numbers in
+    /// <c>docs/performance/INSERT_UPDATE_PERFORMANCE_PLAN.md</c> §3-1c and §3-1d.
     /// </para>
     /// <para>
-    /// ⚠️ OPT-IN FORMAT: only enable on databases whose tables are created/opened with the same flag,
-    /// and never share such databases with tooling built before this option. Legacy plaintext files
-    /// stay byte-for-byte readable either way.
-    /// </para>
-    /// <para>
-    /// Flipping this default to <see langword="true"/> is a work package, not a config change: measured
-    /// on 2026-09-13 it still fails 21 tests across 10 classes (StructRow fast paths, the overflow
-    /// arena, legacy ULID migration, batch canonical parse, in-place field patching). See plan §3-1c
-    /// deliverable 2 for the list and the gate.
+    /// ⚠️ FORMAT: a file is encrypted only when it was CREATED with this option on. A pre-existing
+    /// plaintext database stays byte-for-byte readable and is never mixed with encrypted records — but
+    /// its tables are upgraded to the encrypted format when they are compacted, because compaction
+    /// rewrites them through a brand-new file.
     /// </para>
     /// </summary>
-    public bool EnableAtRestRecordEncryption { get; init; } = false;
+    public bool EnableAtRestRecordEncryption { get; init; } = true;
 
     /// <summary>
     /// Gets a value indicating whether batch encryption is enabled during bulk operations.

src/SharpCoreDB/Services/Storage.Append.cs

@@ -1366,9 +1366,34 @@ private bool FlushOverwritePages(string path, long fileLength, int pageBytes, Di
         long position = encrypted ? PersistenceConstants.EncryptedTableMagicLength : 0;
         long fileLength = new FileInfo(path).Length;
 
+        // PERF (encrypted files only): read the file ONCE into a buffer instead of using the
+        // per-record helpers below, which each open a FileStream — two handle open/close pairs PER
+        // RECORD. Every at-rest caller of this method already materialises the whole file
+        // (`ReadBytesWithRecordOffsets`, `DecryptTableFileToPlaintext`, the index build and
+        // compaction), so this adds no new worst case, and it took 1000 repeated full-scan queries
+        // from ~2×N handle opens per query to one read. Plaintext files keep the incremental walk
+        // byte-for-byte (`buffer` stays null), which also keeps arbitrarily large files working.
+        byte[]? buffer = null;
+        if (encrypted && fileLength > 0 && fileLength <= MaxBufferedRecordWalkBytes)
+        {
+            try
+            {
+                using var fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete);
+                buffer = new byte[fs.Length];
+                fs.ReadExactly(buffer);
+            }
+            catch (IOException)
+            {
+                yield break;
+            }
+        }
+
         while (position + 4 <= fileLength)
         {
-            if (!TryReadInt32At(path, position, out int length))
+            bool haveLength = buffer is not null
+                ? TryReadInt32FromBuffer(buffer, position, out int length)
+                : TryReadInt32At(path, position, out length);
+            if (!haveLength)
             {
                 yield break;
             }
@@ -1403,7 +1428,10 @@ private bool FlushOverwritePages(string path, long fileLength, int pageBytes, Di
             }
 
             byte[] payload = new byte[length];
-            if (!TryReadPayloadAt(path, position + 4, payload))
+            bool havePayload = buffer is not null
+                ? TryCopyPayloadFromBuffer(buffer, position + 4, payload)
+                : TryReadPayloadAt(path, position + 4, payload);
+            if (!havePayload)
             {
                 yield break;
             }
@@ -1448,4 +1476,37 @@ private static bool TryReadPayloadAt(string path, long position, byte[] payload)
         fs.Position = position;
         return fs.Read(payload, 0, payload.Length) == payload.Length;
     }
+
+    /// <summary>
+    /// Upper bound for the single-read buffered record walk in <see cref="ReadAllRecords"/>; larger
+    /// at-rest files fall back to the per-record reads (the same 512 MB ceiling
+    /// <c>ReadBytesRange</c> uses). Every at-rest caller already materialises the whole file, so the
+    /// buffered walk is the normal case and the fallback exists only for very large files.
+    /// </summary>
+    private const long MaxBufferedRecordWalkBytes = 512L * 1024 * 1024;
+
+    /// <summary>Reads a 4-byte length prefix out of the buffered record walk. False past the end.</summary>
+    private static bool TryReadInt32FromBuffer(byte[] buffer, long position, out int value)
+    {
+        value = 0;
+        if (position + 4 > buffer.Length)
+        {
+            return false;
+        }
+
+        value = BinaryPrimitives.ReadInt32LittleEndian(buffer.AsSpan((int)position, 4));
+        return true;
+    }
+
+    /// <summary>Copies a record payload out of the buffered record walk. False past the end.</summary>
+    private static bool TryCopyPayloadFromBuffer(byte[] buffer, long position, byte[] payload)
+    {
+        if (position + payload.Length > buffer.Length)
+        {
+            return false;
+        }
+
+        Array.Copy(buffer, position, payload, 0, payload.Length);
+        return true;
+    }
 }

tests/SharpCoreDB.Tests/EncryptionCoverageTests.cs

@@ -15,11 +15,10 @@ namespace SharpCoreDB.Tests;
 
 /// <summary>
 /// Plan §3-1c deliverable 4: a permanent guard for what "encrypted" actually means per configuration.
-/// The audit found that the DEFAULT configuration stores table data (records and the overflow arena) as
-/// PLAINTEXT while still encrypting metadata and transaction files — so this test pins the posture, and
-/// doubles as the tripwire: flipping <c>EnableAtRestRecordEncryption</c> to the default changes the
-/// <see cref="Kind.Default"/> expectation, and that must only happen together with the work package
-/// listed in plan §3-1c deliverable 2.
+/// The shipped default encrypts table data (records and the overflow arena) as well as metadata and
+/// transaction files, so this test pins that posture; it doubles as the tripwire — if the default ever
+/// stops protecting table payloads, the <see cref="Kind.Default"/> case fails, and that may only happen
+/// as a deliberate decision (plan §3-1c deliverable 2).
 /// </summary>
 public sealed class EncryptionCoverageTests : IDisposable
 {
@@ -28,13 +27,13 @@ public sealed class EncryptionCoverageTests : IDisposable
 
     public enum Kind
     {
-        /// <summary>Default config: NoEncryptMode=false, at-rest records off.</summary>
+        /// <summary>Default config: NoEncryptMode=false, at-rest records ON (the shipped default).</summary>
         Default,
 
         /// <summary>The documented raw-speed opt-out.</summary>
         Raw,
 
-        /// <summary>Per-record encryption opted in.</summary>
+        /// <summary>Per-record encryption opted in explicitly.</summary>
         AtRest,
     }
 
@@ -69,7 +68,7 @@ private static bool ContainsMarker(byte[] fileBytes) =>
         fileBytes.AsSpan().IndexOf(Encoding.UTF8.GetBytes(Marker)) >= 0;
 
     [Theory]
-    [InlineData(Kind.Default, true)]
+    [InlineData(Kind.Default, false)]
     [InlineData(Kind.Raw, true)]
     [InlineData(Kind.AtRest, false)]
     public void TableDataFiles_MarkerPresence_MatchesTheDocumentedPosture(Kind kind, bool expectPlaintext)