Skip to content

Commit d8201f5

Browse files
author
MPCoreDeveloper
committed
docs+test: upgrade/downgrade compatibility policy and format-compat regressions
Add docs/manual/upgrade-and-downgrade.md recording the compatibility matrix: new versions read legacy (variable-length, pre-marker) databases; downgrading below the marker-introducing release is NOT supported for databases that contain commit-time tombstone markers (negative length prefixes). Includes the recommended read-only-first upgrade order and notes the planned cross-version CI job. FormatCompatPolicyTests locks in the forward-compatibility guarantees: (1) a legacy variable-length file reads back and accepts current-version marker writes across reopens, (2) commit-time tombstone markers stay stable across reopen cycles while rows appended afterwards coexist.
1 parent 097c51c commit d8201f5

3 files changed

Lines changed: 189 additions & 0 deletions

File tree

‎docs/CHANGELOG.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
99

1010
### Hardening
1111

12+
- **Upgrade/downgrade policy documented + format-compat regression tests** - new
13+
`docs/manual/upgrade-and-downgrade.md` records the compatibility matrix: reading legacy
14+
(variable-length, pre-marker) databases with the current version is supported; opening a database
15+
that already contains commit-time tombstone markers with a version that predates them is **not**
16+
supported (negative length-prefix markers), and the recommended read-only-first upgrade order.
17+
`FormatCompatPolicyTests` locks in the two forward-compatibility guarantees: legacy files read
18+
back and accept marker writes across reopens, and commit-time tombstone markers stay stable
19+
across reopen cycles while rows appended afterwards coexist.
20+
1221
- **Auto engine selection no longer lands on PageBased (production hardening)** - with default
1322
configuration (`StorageEngineType.Auto` + `WorkloadHint.General`) `GetOptimalStorageEngine`
1423
returned PageBased, which is not yet OLTP-ready (measured UPDATE ~26K ops/s vs ~245K ops/s on the
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Upgrade & downgrade compatibility
2+
3+
Status: **backward compatible** (a new version opens databases written by older versions).
4+
Downgrade below this line is **not supported** after the database contains certain on-disk markers.
5+
6+
## Compatibility matrix
7+
8+
| Scenario | Supported? | Notes |
9+
|---|---|---|
10+
| Open a legacy (pre-fixed-width, variable-length) database with the current version | ✅ Yes | Plaintext + encrypted-header legacy files are read natively; a table without `IsFixedWidthRecords` stays variable-length (the persisted flag is authoritative). |
11+
| Open a current fixed-width Columnar database with the current version after reopen cycles | ✅ Yes | Covered by `FormatCompatPolicyTests` / `FixedWidthMigrationTests`; per-table flag persisted. |
12+
| Migrate a legacy table to fixed-width | ✅ Yes (opt-in) | `DatabaseConfig.FixedWidthRecordLayout = true` triggers `MigrateToFixedWidth()` on open (never on read-only opens). |
13+
| Open a database written by the **current** version (contains commit-time tombstone markers) with an **older** version that predates markers | ❌ **Not supported** | Deleted rows are stored as **negative length-prefix markers** (introduced with commit-time tombstones). An older binary that only knows positive length prefixes cannot skip these records and must not be pointed at such a file. |
14+
| Open a fixed-width table with a version that only understands variable-length records | ❌ Not supported | Downgrade requires a migration/export; none is shipped. |
15+
16+
## Downgrade boundary
17+
18+
The on-disk markers (negative length prefixes, written at COMMIT for transactional deletes) make a
19+
database **forward-compatible only**. If you must keep the ability to downgrade, either:
20+
21+
1. Keep a separate pre-upgrade copy of the database, or
22+
2. Do not upgrade software that writes tombstones in place on a database you need to open again
23+
with the old software, or
24+
3. Export/re-import data (SQL dump) instead of copying `.dat`/metadata files across versions.
25+
26+
Recommended upgrade order:
27+
1. Back up the database directory (or single-file database).
28+
2. Open it with the new version once **read-only** first (this never rewrites data).
29+
3. Open read-write and run the normal DML regression/verification.
30+
4. Only then let the new version write to the file.
31+
32+
## Verification
33+
34+
- In-repo: `FormatCompatPolicyTests`, `FixedWidthMigrationTests`,
35+
`DefaultEngineSelectionTests`, and the full suite (currently 1768+ tests) cover legacy reads,
36+
opt-in migration, marker durability across reopen cycles, and the default fast-path engine.
37+
- Planned (CI): a true **cross-version** job that writes a database with a pinned older commit and
38+
reads it with `master` (requires an old-binary generator; tracked as follow-up).
39+
40+
## Changelog
41+
42+
See `docs/CHANGELOG.md` → `[Unreleased]` → **Hardening** for the marker/downgrade notes that
43+
accompanied the tombstone work (PRs #367/#368) and this policy document.
Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
// <copyright file="FormatCompatPolicyTests.cs" company="MPCoreDeveloper">
2+
// Copyright (c) 2026 MPCoreDeveloper. All rights reserved.
3+
// Licensed under the MIT License. See LICENSE file in the project root for full license information.
4+
// </copyright>
5+
namespace SharpCoreDB.Tests;
6+
7+
using Microsoft.Extensions.DependencyInjection;
8+
using SharpCoreDB.Interfaces;
9+
using System;
10+
using System.Collections.Generic;
11+
using System.IO;
12+
using Xunit;
13+
14+
/// <summary>
15+
/// Encoding of the compatibility policy in <c>docs/manual/upgrade-and-downgrade.md</c>:
16+
/// (1) files written in the legacy variable-length format (the pre-marker / pre-fixed-width
17+
/// representation) must read back and keep working across reopen cycles, and (2) commit-time
18+
/// tombstone markers written by the current version must stay stable across repeated reopen cycles
19+
/// and coexist with rows appended afterwards. These are the forward-compatibility guarantees that
20+
/// make downgrade the only unsupported direction.
21+
/// </summary>
22+
public sealed class FormatCompatPolicyTests : IDisposable
23+
{
24+
private readonly DatabaseFactory _factory;
25+
private readonly string _dirPath;
26+
27+
public FormatCompatPolicyTests()
28+
{
29+
var services = new ServiceCollection();
30+
services.AddSharpCoreDB();
31+
_factory = services.BuildServiceProvider().GetRequiredService<DatabaseFactory>();
32+
_dirPath = Path.Combine(Path.GetTempPath(), $"SCDB_FormatCompat_{Guid.NewGuid():N}");
33+
}
34+
35+
public void Dispose()
36+
{
37+
try { if (Directory.Exists(_dirPath)) Directory.Delete(_dirPath, true); } catch { }
38+
}
39+
40+
private static void InsertRange(IDatabase db, int from, int to)
41+
{
42+
var stmts = new List<string>(to - from + 1);
43+
for (int i = from; i <= to; i++)
44+
{
45+
stmts.Add($"INSERT INTO docs VALUES ({i}, 'user{i}', {i * 0.5})");
46+
}
47+
48+
db.ExecuteBatchSQL(stmts);
49+
db.Flush();
50+
}
51+
52+
private static void DeleteRange(IDatabase db, int from, int to)
53+
{
54+
var stmts = new List<string>(to - from + 1);
55+
for (int i = from; i <= to; i++)
56+
{
57+
stmts.Add($"DELETE FROM docs WHERE id = {i}");
58+
}
59+
60+
db.ExecuteBatchSQL(stmts);
61+
db.Flush();
62+
}
63+
64+
[Fact]
65+
public void LegacyVariableLengthFile_ReadsBackAndKeepsWorking_AcrossReopens()
66+
{
67+
// A legacy variable-length database (no fixed-width layout) is byte-compatible with the
68+
// pre-fixed-width / pre-marker format. It must open, round-trip and then accept marker
69+
// writes from the current version.
70+
IDatabase? db = _factory.Create(_dirPath, "pw", isReadOnly: false,
71+
config: new DatabaseConfig { NoEncryptMode = true, AutoFixedWidthRecords = false, FixedWidthRecordLayout = false });
72+
try
73+
{
74+
db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, name TEXT, score REAL)");
75+
InsertRange(db, 1, 300);
76+
Assert.Equal(300, db.ExecuteQuery("SELECT id FROM docs").Count);
77+
}
78+
finally { (db as IDisposable)?.Dispose(); }
79+
80+
// First reopen = "old file read by new version" (no markers written yet).
81+
db = _factory.Create(_dirPath, "pw", isReadOnly: false,
82+
config: new DatabaseConfig { NoEncryptMode = true, AutoFixedWidthRecords = false, FixedWidthRecordLayout = false });
83+
try
84+
{
85+
Assert.Equal(300, db.ExecuteQuery("SELECT id FROM docs").Count);
86+
DeleteRange(db, 1, 40);
87+
Assert.Equal(260, db.ExecuteQuery("SELECT id FROM docs").Count);
88+
}
89+
finally { (db as IDisposable)?.Dispose(); }
90+
91+
// Final reopen: the legacy rows plus current-version tombstone markers coexist.
92+
db = _factory.Create(_dirPath, "pw", isReadOnly: false,
93+
config: new DatabaseConfig { NoEncryptMode = true, AutoFixedWidthRecords = false, FixedWidthRecordLayout = false });
94+
try
95+
{
96+
Assert.Equal(260, db.ExecuteQuery("SELECT id FROM docs").Count);
97+
Assert.Empty(db.ExecuteQuery("SELECT id FROM docs WHERE id = 10"));
98+
Assert.Single(db.ExecuteQuery("SELECT id FROM docs WHERE id = 41"));
99+
}
100+
finally { (db as IDisposable)?.Dispose(); }
101+
}
102+
103+
[Fact]
104+
public void CommitTimeTombstoneMarkers_StableAcrossReopenCycles_AndCoexistWithAppends()
105+
{
106+
IDatabase? db = _factory.Create(_dirPath, "pw", isReadOnly: false, config: new DatabaseConfig());
107+
try
108+
{
109+
db.ExecuteSQL("CREATE TABLE docs (id INTEGER PRIMARY KEY, name TEXT, score REAL)");
110+
InsertRange(db, 1, 1000);
111+
DeleteRange(db, 1, 500);
112+
Assert.Equal(500, db.ExecuteQuery("SELECT id FROM docs").Count);
113+
}
114+
finally { (db as IDisposable)?.Dispose(); }
115+
116+
db = _factory.Create(_dirPath, "pw", isReadOnly: false, config: new DatabaseConfig());
117+
try
118+
{
119+
Assert.Equal(500, db.ExecuteQuery("SELECT id FROM docs").Count);
120+
InsertRange(db, 1001, 1200); // appends after the marker region
121+
Assert.Equal(700, db.ExecuteQuery("SELECT id FROM docs").Count);
122+
DeleteRange(db, 1001, 1100);
123+
}
124+
finally { (db as IDisposable)?.Dispose(); }
125+
126+
db = _factory.Create(_dirPath, "pw", isReadOnly: false, config: new DatabaseConfig());
127+
try
128+
{
129+
Assert.Equal(600, db.ExecuteQuery("SELECT id FROM docs").Count);
130+
Assert.Empty(db.ExecuteQuery("SELECT id FROM docs WHERE id = 1"));
131+
Assert.Empty(db.ExecuteQuery("SELECT id FROM docs WHERE id = 1050"));
132+
Assert.Single(db.ExecuteQuery("SELECT id FROM docs WHERE id = 1101"));
133+
Assert.Single(db.ExecuteQuery("SELECT id FROM docs WHERE id = 1200"));
134+
}
135+
finally { (db as IDisposable)?.Dispose(); }
136+
}
137+
}

0 commit comments

Comments
 (0)