Merge pull request #391 from MPCoreDeveloper/docs/2.0.0.2-finalize

docs(2.0.0.2): finalize release documentation

Author: MPCoreDeveloper

docs/2.0.0.2_WHAT_CHANGED.md

@@ -1,10 +1,12 @@
 # SharpCoreDB 2.0.0.2 — What changed & the 2.x engine vs the 1.9.x line
 
 - **Release:** `2.0.0.2` (2026-09-04) — 4-part patch release on the 2.0.0.x line.
-- **Quality gate:** full suite **1770 tests / 0 failed**, CI green on Windows/Ubuntu/macOS +
-  CodeQL + SonarCloud + Server Compatibility Smoke for every merged PR.
-- **Contents:** everything merged since 2.0.0.1 — the perf/hardening batch of PRs #376-#388 —
-  plus the consolidated 2.x-vs-1.9.x overview below.
+- **Quality gate:** full core suite **1777 tests / 0 failed**; CI-filtered core suite **1674 / 0**,
+  plus SharpCoreDB.CQRS.Tests 64/64, VectorSearch 143, EF Core 116 and Functional.Linq2DB 24 —
+  validated on Windows/Ubuntu/macOS + CodeQL + SonarCloud + Server Compatibility Smoke.
+- **Contents:** everything merged since 2.0.0.1 — the perf/hardening batch (PRs #376-#388), the
+  release documentation PR #389, and the reopen **data-integrity hardening** PR #390 described
+  below.
 - See `docs/CHANGELOG.md` → `[2.0.0.2]` for the detailed change list and
   `docs/2.0.0.0_WHAT_CHANGED.md` for the full 2.0.0.0 report (this document is the short,
   forward-looking summary).
@@ -15,7 +17,7 @@
 | Step | 1.9.x | 2.x |
 |---|---|---|
 | Record layout | variable-length length-prefixed rows only | **Fixed-width record layout** (constant stride + out-of-line `.ovf` overflow arena) for columnar PK tables; **default for new PK tables** since B7; per-table flag **persisted in metadata** (authoritative on reopen) with **1.x → 2.0 auto-migration** (opt-in via `FixedWidthRecordLayout`) |
-| DELETE durability | logical delete + full-file rewrite at flush | **Commit-time tombstone markers** (in-place negative length-prefix) → durable in O(delete), no full-file rewrite |
+| DELETE durability | logical delete + full-file rewrite at flush | **Commit-time tombstone markers** (in-place negative length-prefix) → durable in O(delete), no full-file rewrite; legacy (1.x) variable-length tables additionally purge remaining versions of a deleted key so stale rows cannot resurrect after reopen |
 | In-place writes | only same-length patches, buffered later | **Buffered in-place overwrites flushed per storage page** (C6); markers also batched per page (C5) |
 | Single-file (`.scdb`) | fixed-layout registry, compression bugs | **Format v2** (block registry/FSM/table directory), crash-safe auto-migration, block-level Brotli/GZip/Zstd, envelope encryption, concurrent-write corruption fix (2.0.0.1) |
 
@@ -46,6 +48,18 @@
   PageBased (not yet OLTP-ready: ~26K UPDATE ops/s vs ~245K on the fixed-width Columnar path);
   PageBased remains explicit opt-in.
 - 4-part patch versioning (`n.n.n.n`) since 2.0.0.0.
+- **Reopen data-integrity fixes (PR #390, found by the new reopen round-trip matrix):**
+  - **Overflow-arena reload** — a valid zero-length record (empty TEXT/BLOB value) was treated as
+    end-of-file by the `.ovf`/data-file loader, silently dropping every later row/block after a
+    reopen; length-0 records are now parsed as valid empty payloads.
+  - **Single-file fixed-width overflow arena** — freed blocks were serialized as zero-filled gaps
+    that misaligned the sequential loader; dead slots are now persisted as negative-length
+    tombstone markers tracked across sessions.
+  - **Legacy (1.x) variable-length DELETE** — only the newest row version was tombstoned, so an
+    older stale UPDATE version won the reopen index rebuild and resurrected the deleted row;
+    DELETE now purges every remaining record of a deleted key (1.x-upgrade path made safe).
+  - Locked in by `ReopenRoundTripMatrixTests` (four storage variants × insert/update/delete ×
+    three reopen rounds) and dedicated empty-value/delete regressions.
 
 ## Benchmark snapshot (fair-PK, tuned harness config, median-of-3, same machine)
 

docs/INDEX.md

@@ -1,6 +1,6 @@
 # SharpCoreDB Documentation Index
 
-**Status:** Active documentation set (`2.0.0.0`)
+**Status:** Active documentation set (`2.0.0.2`)
 
 ## 0. Manual (start here)
 
@@ -23,6 +23,8 @@ guide explaining when SharpCoreDB is fastest.
 
 ## 0b. Release notes
 
+- `2.0.0.2_WHAT_CHANGED.md` — **what's new in 2.0.0.2** (2.x-vs-1.9.x major steps + the hardening
+  and data-integrity batch)
 - `2.0.0.0_WHAT_CHANGED.md` — **what's new in v2.0 vs v1.9** (features + benchmarks vs SQLite/LiteDB)
 
 ## 1. Project Entry Points

docs/manual/upgrade-and-downgrade.md

@@ -7,7 +7,7 @@ Downgrade below this line is **not supported** after the database contains certa
 
 | Scenario | Supported? | Notes |
 |---|---|---|
-| Open a legacy (pre-fixed-width, variable-length) database with the current version | ✅ Yes | Plaintext + encrypted-header legacy files are read natively; a table without `IsFixedWidthRecords` stays variable-length (the persisted flag is authoritative). |
+| Open a legacy (pre-fixed-width, variable-length) database with the current version | ✅ Yes | Plaintext + encrypted-header legacy files are read natively; a table without `IsFixedWidthRecords` stays variable-length (the persisted flag is authoritative). UPDATE/DELETE on such tables is safe in 2.0.0.2: DELETE purges older stale row versions of a deleted key so deleted rows cannot resurrect after a reopen. |
 | Open a current fixed-width Columnar database with the current version after reopen cycles | ✅ Yes | Covered by `FormatCompatPolicyTests` / `FixedWidthMigrationTests`; per-table flag persisted. |
 | Migrate a legacy table to fixed-width | ✅ Yes (opt-in) | `DatabaseConfig.FixedWidthRecordLayout = true` triggers `MigrateToFixedWidth()` on open (never on read-only opens). |
 | Open a database written by the **current** version (contains commit-time tombstone markers) with an **older** version that predates markers | ❌ **Not supported** | Deleted rows are stored as **negative length-prefix markers** (introduced with commit-time tombstones). An older binary that only knows positive length prefixes cannot skip these records and must not be pointed at such a file. |
@@ -32,8 +32,9 @@ Recommended upgrade order:
 ## Verification
 
 - In-repo: `FormatCompatPolicyTests`, `FixedWidthMigrationTests`,
-  `DefaultEngineSelectionTests`, and the full suite (currently 1768+ tests) cover legacy reads,
-  opt-in migration, marker durability across reopen cycles, and the default fast-path engine.
+  `DefaultEngineSelectionTests`, `ReopenRoundTripMatrixTests` and the full suite (currently 1777+
+  tests) cover legacy reads, opt-in migration, marker durability across reopen cycles, and the
+  default fast-path engine.
 - Planned (CI): a true **cross-version** job that writes a database with a pinned older commit and
   reads it with `master` (requires an old-binary generator; tracked as follow-up).
 

src/SharpCoreDB/NuGet.README.md

@@ -22,6 +22,12 @@ SharpCoreDB is a modern, encrypted, file-based database engine with SQL support,
   OLTP-ready); upgrade/downgrade policy documented with format-compat regression tests; a
   pure-default-config benchmark arm (`--pk-default`) and a same-window interleaved A/B mode
   (`--pk-ab`) now guard the numbers.
+- **Reopen data-integrity fixes** — a reopen round-trip hardening sweep fixed three silent-data-loss
+  edge cases: empty TEXT/BLOB values no longer truncate the overflow-arena reload; the single-file
+  fixed-width overflow arena persists freed slots as markers (no misaligned zero gaps); and legacy
+  (1.x-upgrade) variable-length tables purge older row versions when a row is deleted, so deleted
+  rows can no longer resurrect after a reopen. Locked in by the new `ReopenRoundTripMatrixTests`
+  and outbox/CQRS integration coverage.
 - Full report (incl. the 2.x-vs-1.9.x major steps): `docs/2.0.0.2_WHAT_CHANGED.md`.
 
 ## What's New in 2.0.0.1