Skip to content

Commit da7c73a

Browse files
author
MPCoreDeveloper
committed
bench(pk): add same-window interleaved A/B harness mode (--pk-ab)
Runs two default-config variants as alternating rep pairs (A1,B1,A2,B2,...) and reports the per-rep paired median ratio B/A per phase, so machine drift hits both arms of a pair and cancels out — the reliable way to attribute default-vs-tuned deltas. Variant config builder factored out of --pk-default (BuildPkDefaultVariantConfig) and passed per arm; arms selectable via SHARPCOREDB_PK_AB_ARM_A/_B, reps via SHARPCOREDB_BENCH_REPS. Smoke (1 rep, pure default vs 'plain'/NoEncryptMode=true, same window): UPDATE 1.41x, DELETE 1.28x, INSERT 1.21x, READ 1.10x (B over A). Documented in docs/benchmarks/default-config-pk.md.
1 parent 9549fb5 commit da7c73a

3 files changed

Lines changed: 152 additions & 25 deletions

File tree

‎docs/CHANGELOG.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
99

1010
### Hardening
1111

12+
- **Same-window interleaved A/B harness (`--pk-ab`) (P3c)** - new harness mode runs two default-
13+
config variants as alternating rep pairs (A1,B1,A2,B2,...) and reports the per-rep paired median
14+
ratio B/A per phase, so machine drift affects both arms of a pair and cancels out. Smoke (1 rep,
15+
pure default vs `plain`/NoEncryptMode=true): UPDATE 1.41x, DELETE 1.28x, INSERT 1.21x, READ 1.10x
16+
in the same window. Use `SHARPCOREDB_PK_AB_ARM_A` / `SHARPCOREDB_PK_AB_ARM_B` /
17+
`SHARPCOREDB_BENCH_REPS`; documented in `docs/benchmarks/default-config-pk.md`.
18+
1219
- **Default-config benchmark follow-up: FullSync hypothesis falsified (P3b)** - single-knob
1320
isolation via `SHARPCOREDB_PK_DEFAULT_VARIANT` (`async`, `bufferedio`, `novalidate`,
1421
`noadaptive`, `hsinsert`, `plain`, `tuned`) disproved the earlier “FullSync dominates the

‎docs/benchmarks/default-config-pk.md‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -39,13 +39,26 @@ Observed (2026-09-04 afternoon):
3939
Machine drift is significant: SQLite's own UPDATE varied 275K-315K across these runs. Single-knob
4040
deltas below ~1.3x are not reliably attributable outside a same-window interleaved A/B.
4141

42+
## Same-window interleaved A/B (--pk-ab)
43+
44+
Run: `dotnet run --project tests/benchmarks/SharpCoreDB.Benchmarks.Comparative -- -c Release -- --pk-ab`
45+
Arm names: `SHARPCOREDB_PK_AB_ARM_A` / `SHARPCOREDB_PK_AB_ARM_B` (defaults: `""` pure default vs
46+
`plain`); reps via `SHARPCOREDB_BENCH_REPS` (default 3). Each rep runs A then B back-to-back and
47+
the per-rep **paired ratio** B/A per phase is reported (median), so slow-machine windows affect
48+
both arms of a pair and cancel out — this is the reliable way to attribute default-vs-tuned deltas.
49+
50+
Preliminary smoke result (1 rep, 2026-09-04, default vs `plain`): UPDATE **1.41x**, DELETE **1.28x**,
51+
INSERT 1.21x, READ 1.10x — direction consistent with the earlier medians, now measured inside a
52+
single window. Re-run with the default 3 reps before quoting final numbers.
53+
4254
## Honest conclusion
4355

4456
1. The earlier claim that the default `WalDurabilityMode.FullSync` dominates the gap is **wrong**
4557
(disproven by the `async` variant). Do **not** implement a “FullSync commit-flush optimization”
4658
based on it.
47-
2. The default path is correct and engages the fast paths; part of the remaining gap correlates
48-
with `NoEncryptMode` (record/at-rest toggles and file-format decisions), part is machine drift.
49-
3. Next step: add a **same-window interleaved A/B** mode to this harness (arms round-robin within
50-
one process) so default-vs-tuned deltas are attributable, then re-open the optimization only on
51-
a measured knob.
59+
2. The default path is correct and engages the fast paths; the `--pk-ab` smoke (same window) shows
60+
`plain` (NoEncryptMode=true) ahead by UPDATE 1.41x / DELETE 1.28x, so the remaining gap
61+
correlates with `NoEncryptMode`; the exact mechanism (record/at-rest toggles and file-format
62+
decisions) still needs a code-level explanation before any change.
63+
3. The same-window A/B tooling now exists; re-run with 3 reps to quantify the paired ratio, then
64+
investigate the `NoEncryptMode` mechanism in code and only then consider a change.

‎tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/Program.cs‎

Lines changed: 127 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,21 @@ static async Task Main(string[] args)
7979
return;
8080
}
8181

82+
// Optional: --pk-ab → same-window interleaved A/B: runs arm A and arm B as alternating
83+
// rep pairs (A1,B1,A2,B2,...) and reports the PER-REP median ratio B/A per phase, so
84+
// machine drift affects both arms of each pair equally. Arms are config variants named by
85+
// SHARPCOREDB_PK_AB_ARM_A / SHARPCOREDB_PK_AB_ARM_B (defaults: pure default vs 'plain').
86+
if (args.Any(a => a.Equals("--pk-ab", StringComparison.OrdinalIgnoreCase)))
87+
{
88+
var engineArgAb = args.FirstOrDefault(a => a.StartsWith("--engine=", StringComparison.OrdinalIgnoreCase));
89+
var engineTypeAb = engineArgAb is not null
90+
&& engineArgAb.Substring("--engine=".Length).Equals("pagebased", StringComparison.OrdinalIgnoreCase)
91+
? SharpCoreDB.Interfaces.StorageEngineType.PageBased
92+
: SharpCoreDB.Interfaces.StorageEngineType.AppendOnly;
93+
RunPkAbComparison(engineTypeAb);
94+
return;
95+
}
96+
8297
// Optional: --engine=appendonly (default) | --engine=pagebased
8398
// PageBased is the v2.0 in-place-update engine (WP10-WP13 storage engine roadmap).
8499
var engineArg = args.FirstOrDefault(a => a.StartsWith("--engine=", StringComparison.OrdinalIgnoreCase));
@@ -395,6 +410,34 @@ static DatabaseConfig BuildConfig(SharpCoreDB.Interfaces.StorageEngineType engin
395410
};
396411
}
397412

413+
/// <summary>
414+
/// Builds the variant DatabaseConfig used by the --pk-default arm. The variant name comes from
415+
/// SHARPCOREDB_PK_DEFAULT_VARIANT or the --pk-ab arm selector; "" is the pure default config.
416+
/// </summary>
417+
private static DatabaseConfig BuildPkDefaultVariantConfig(
418+
SharpCoreDB.Interfaces.StorageEngineType engineType,
419+
string? variant)
420+
{
421+
return variant switch
422+
{
423+
"async" => new DatabaseConfig { StorageEngineType = engineType, WalDurabilityMode = SharpCoreDB.Services.DurabilityMode.Async },
424+
"bufferedio" => new DatabaseConfig { StorageEngineType = engineType, UseBufferedIO = true },
425+
"novalidate" => new DatabaseConfig
426+
{
427+
StorageEngineType = engineType,
428+
SqlValidationMode = SharpCoreDB.Services.SqlQueryValidator.ValidationMode.Disabled,
429+
StrictParameterValidation = false,
430+
},
431+
"noadaptive" => new DatabaseConfig { StorageEngineType = engineType, EnableAdaptiveWalBatching = false },
432+
"hsinsert" => new DatabaseConfig { StorageEngineType = engineType, HighSpeedInsertMode = true },
433+
// "plain" == the tuned harness config with NoEncryptMode=true (BuildConfig default);
434+
// "tuned" == the same knob set but NoEncryptMode=false (isolates that flag).
435+
"plain" => BuildConfig(engineType, fixedWidth: true),
436+
"tuned" => BuildConfig(engineType, fixedWidth: true, noEncrypt: false),
437+
_ => new DatabaseConfig { StorageEngineType = engineType },
438+
};
439+
}
440+
398441
static BenchmarkResult RunSharpCoreDB(SharpCoreDB.Interfaces.StorageEngineType engineType)
399442
{
400443
var dbPath = Path.Combine(Path.GetTempPath(), $"bench-sharpcoredb-{Guid.NewGuid()}");
@@ -862,7 +905,8 @@ data TEXT
862905
static BenchmarkResult RunSharpCoreDBPk(
863906
SharpCoreDB.Interfaces.StorageEngineType engineType,
864907
bool fixedWidth = false,
865-
bool useDefaultConfig = false)
908+
bool useDefaultConfig = false,
909+
string? defaultVariant = null)
866910
{
867911
var dbPath = Path.Combine(Path.GetTempPath(), $"bench-sharpcoredb-pk-{Guid.NewGuid()}");
868912
var result = new BenchmarkResult();
@@ -877,25 +921,9 @@ static BenchmarkResult RunSharpCoreDBPk(
877921
DatabaseConfig config;
878922
if (useDefaultConfig)
879923
{
880-
var variant = Environment.GetEnvironmentVariable("SHARPCOREDB_PK_DEFAULT_VARIANT")?.ToLowerInvariant();
881-
config = variant switch
882-
{
883-
"async" => new DatabaseConfig { StorageEngineType = engineType, WalDurabilityMode = SharpCoreDB.Services.DurabilityMode.Async },
884-
"bufferedio" => new DatabaseConfig { StorageEngineType = engineType, UseBufferedIO = true },
885-
"novalidate" => new DatabaseConfig
886-
{
887-
StorageEngineType = engineType,
888-
SqlValidationMode = SharpCoreDB.Services.SqlQueryValidator.ValidationMode.Disabled,
889-
StrictParameterValidation = false,
890-
},
891-
"noadaptive" => new DatabaseConfig { StorageEngineType = engineType, EnableAdaptiveWalBatching = false },
892-
"hsinsert" => new DatabaseConfig { StorageEngineType = engineType, HighSpeedInsertMode = true },
893-
// "plain" == the tuned harness config with NoEncryptMode=true (BuildConfig default);
894-
// "tuned" == the same knob set but NoEncryptMode=false (isolates that flag).
895-
"plain" => BuildConfig(engineType, fixedWidth: true),
896-
"tuned" => BuildConfig(engineType, fixedWidth: true, noEncrypt: false),
897-
_ => new DatabaseConfig { StorageEngineType = engineType },
898-
};
924+
var variant = defaultVariant
925+
?? Environment.GetEnvironmentVariable("SHARPCOREDB_PK_DEFAULT_VARIANT")?.ToLowerInvariant();
926+
config = BuildPkDefaultVariantConfig(engineType, variant);
899927
}
900928
else
901929
{
@@ -1127,6 +1155,85 @@ static void RunPkDefaultComparison(SharpCoreDB.Interfaces.StorageEngineType engi
11271155
Console.WriteLine($"\nResults saved to: {path}");
11281156
}
11291157

1158+
/// <summary>
1159+
/// P3c: same-window interleaved A/B comparison of two default-config variants. Each rep runs
1160+
/// arm A then arm B back-to-back, and the result reports the per-rep paired ratio B/A per phase
1161+
/// (median), so slow-machine windows affect both arms of a pair and cancel out.
1162+
/// </summary>
1163+
static void RunPkAbComparison(SharpCoreDB.Interfaces.StorageEngineType engineType)
1164+
{
1165+
var armA = Environment.GetEnvironmentVariable("SHARPCOREDB_PK_AB_ARM_A")?.ToLowerInvariant() ?? string.Empty;
1166+
var armB = Environment.GetEnvironmentVariable("SHARPCOREDB_PK_AB_ARM_B")?.ToLowerInvariant() ?? "plain";
1167+
int reps = 3;
1168+
if (int.TryParse(Environment.GetEnvironmentVariable("SHARPCOREDB_BENCH_REPS"), out int envReps) && envReps > 0)
1169+
{
1170+
reps = envReps;
1171+
}
1172+
1173+
Console.WriteLine("╔══════════════════════════════════════════════════════════╗");
1174+
Console.WriteLine("║ Same-window interleaved A/B (default-config variants) ║");
1175+
Console.WriteLine("╚══════════════════════════════════════════════════════════╝");
1176+
Console.WriteLine();
1177+
Console.WriteLine($"Arm A variant: '{armA}' Arm B variant: '{armB}' reps: {reps}");
1178+
Console.WriteLine("(each rep: A then B back-to-back; per-rep ratios cancel drift)");
1179+
Console.WriteLine();
1180+
1181+
var listA = new List<BenchmarkResult>(reps);
1182+
var listB = new List<BenchmarkResult>(reps);
1183+
for (int r = 0; r < reps; r++)
1184+
{
1185+
Console.WriteLine($"── rep {r + 1}/{reps} · A='{armA}' ──");
1186+
listA.Add(RunSharpCoreDBPk(engineType, useDefaultConfig: true, defaultVariant: armA));
1187+
Console.WriteLine($"── rep {r + 1}/{reps} · B='{armB}' ──");
1188+
listB.Add(RunSharpCoreDBPk(engineType, useDefaultConfig: true, defaultVariant: armB));
1189+
}
1190+
1191+
static double Med(IEnumerable<double> xs)
1192+
{
1193+
var sorted = xs.Where(x => x > 0).OrderBy(x => x).ToArray();
1194+
return sorted.Length == 0 ? 0 : sorted[sorted.Length / 2];
1195+
}
1196+
1197+
static int Ops(IEnumerable<BenchmarkResult> runs, Func<BenchmarkResult, int> select)
1198+
{
1199+
var arr = runs.Select(select).Where(x => x > 0).OrderBy(x => x).ToArray();
1200+
return arr.Length == 0 ? 0 : arr[arr.Length / 2];
1201+
}
1202+
1203+
static double Ratio(IEnumerable<BenchmarkResult> a, IEnumerable<BenchmarkResult> b, Func<BenchmarkResult, int> sel)
1204+
{
1205+
var ratios = a.Zip(b, (x, y) => sel(x) > 0 ? sel(y) / (double)sel(x) : 0.0);
1206+
return Med(ratios);
1207+
}
1208+
1209+
var aUpdate = Ops(listA, static r => r.UpdateOpsPerSec);
1210+
var bUpdate = Ops(listB, static r => r.UpdateOpsPerSec);
1211+
var aDelete = Ops(listA, static r => r.DeleteOpsPerSec);
1212+
var bDelete = Ops(listB, static r => r.DeleteOpsPerSec);
1213+
1214+
Console.WriteLine("║ phase │ A median ops/s │ B median ops/s │ median B/A (per rep) ║");
1215+
Console.WriteLine($"║ UPDATE │ {aUpdate,13:N0} │ {bUpdate,14:N0} │ {Ratio(listA, listB, static r => r.UpdateOpsPerSec):F2}x");
1216+
Console.WriteLine($"║ DELETE │ {aDelete,13:N0} │ {bDelete,14:N0} │ {Ratio(listA, listB, static r => r.DeleteOpsPerSec):F2}x");
1217+
Console.WriteLine($"║ INSERT │ {Ops(listA, static r => r.InsertOpsPerSec),13:N0} │ {Ops(listB, static r => r.InsertOpsPerSec),14:N0} │ {Ratio(listA, listB, static r => r.InsertOpsPerSec):F2}x");
1218+
Console.WriteLine($"║ READ │ {Ops(listA, static r => r.ReadOpsPerSec),13:N0} │ {Ops(listB, static r => r.ReadOpsPerSec),14:N0} │ {Ratio(listA, listB, static r => r.ReadOpsPerSec):F2}x");
1219+
1220+
var summary = new Dictionary<string, object>
1221+
{
1222+
["armA"] = armA,
1223+
["armB"] = armB,
1224+
["medianUpdateRatio_B_over_A"] = Ratio(listA, listB, static r => r.UpdateOpsPerSec),
1225+
["medianDeleteRatio_B_over_A"] = Ratio(listA, listB, static r => r.DeleteOpsPerSec),
1226+
["medianInsertRatio_B_over_A"] = Ratio(listA, listB, static r => r.InsertOpsPerSec),
1227+
["medianReadRatio_B_over_A"] = Ratio(listA, listB, static r => r.ReadOpsPerSec),
1228+
};
1229+
1230+
var dir = "results";
1231+
Directory.CreateDirectory(dir);
1232+
var path = Path.Combine(dir, $"pk_ab_{armA}_{armB}_{DateTime.UtcNow:yyyyMMdd_HHmmss}.json");
1233+
File.WriteAllText(path, JsonSerializer.Serialize(summary, new JsonSerializerOptions { WriteIndented = true }));
1234+
Console.WriteLine($"\nResults saved to: {path}");
1235+
}
1236+
11301237
// ══════════════════════════════════════
11311238
// LiteDB
11321239
// ══════════════════════════════════════

0 commit comments

Comments
 (0)