Skip to content

Commit ed70ce6

Browse files
perf(core): automatic hash indexes register on demand (§9 row 5) — arm B/C INSERT reach 1,60x / 1,26x, and two stale-index defects fall out of it
Registration, not building, was the eager half: SqlParser.DDL.cs registered one index per column at CREATE TABLE and EnsureAllRegisteredIndexesLoaded() then loads every registered index before a write on the append/Columnar engines, so the five-column docs shape paid hash remove+add per row on all five columns while the workload filtered on one. - Table.EnsureAutoHashIndexRegistered is the single gate every lookup path now shares (the read path, the position-aware update locate, both delete cores, the parallel scan, both struct-scan fast paths, structured DML); it registers on demand when the table is Columnar and DatabaseConfig.EnableHashIndexes is on, and it takes the write lock only when the caller does not already hold it (the write paths reach it from inside SelectInternal). - SqlParser.DDL.cs registers only a user-declared primary key; the hidden _rowid fallback is never registered (the PK B-tree already serves it). PageBased keeps its pre-load exemption and still auto-creates nothing. information_schema.indexes reads the registered set, so the catalog reports reality without a code change. - HashIndexAutoCreationGateTests rewritten to the new contract (declared PK only, others on the first filtered operation, nothing with the dial off) with the two regressions below added. Measured (this build, D:\scdb-bench-tmp, 3 reps, median with the run's per-rep range; INSERT vs SQLite): arm B batch 0,95x -> 1,60x (1,05-1,92) [same-build dial-off ceiling 1,68x] arm B SQL 0,83x -> 1,05x (0,94-1,26) [1,31x] range straddles 1,00 - not quoted as a win arm C batch 1,26x (1,19-1,54) [1,25x] whole margin captured arm C SQL 0,90x -> 0,98x (0,98-1,29) [0,95x] --auto-index-benefit now reports the first filtered query on its own line, because that is the query that pays for the index: first 35,5-71,0 ms, then 4,7-14,1 us repeated on email/score against 4,8-5,0 ms with the dial off (354x-988x, 3.106x-7.656x), age 31,6x-33,0x at ~1 ms, control unchanged, row counts identical on/off in two runs. Two defects found while implementing, each fixed at the cause and pinned red->green: - An index built after a length-changing UPDATE indexed the superseded record (the Columnar engines leave it in the file; only compaction drops it). WHERE v = 'old1' answered 1 row where SQLite returns 0. The scan's rule now lives in one place, Table.IsCurrentRecordVersion, and both rebuild walks (at-rest and plaintext) apply it too. - the single-row UPDATE's append fallback left the entries of columns the statement did not name pointing at the record it abandoned - 4 entries for 3 rows, and a ghost row once that column's value changed. CaptureHashKeysAt now reads the old keys from the record the append leaves. This is what made SqlInPlaceUpdateTests.SqlUpdate_VariableWidth_Grows... fail on the first suite run, and it passes now on a contract it no longer leaned on. Verification: core suite 1952 / 0 failed / 0 errors; --gate PASSED, 8/8 metrics ok, all eight ahead of the recorded baseline (default INSERT 0,78x, default UPDATE 0,31x); gate run once. No baseline re-recorded, nothing pushed.
1 parent 2c1af9c commit ed70ce6

17 files changed

Lines changed: 753 additions & 100 deletions

‎docs/CHANGELOG.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1717

1818
### Changed
1919

20+
- **A Columnar table now registers an automatic hash index for a column the first time an operation filters on it, instead of registering one per column at `CREATE TABLE` — so a write stops maintaining indexes the workload never uses.** `DatabaseConfig.EnableHashIndexes` (default `true`) had always meant "auto-create a hash index per column": `SqlParser.DDL.cs` registered the primary key *and every other column*, and because `Table.EnsureAllRegisteredIndexesLoaded()` loads every **registered** index before a write on the append/Columnar engines, the five-column `docs` shape paid a hash remove+add per row on all five columns while the workload filtered on one — the measured 20.000 `index-maint` calls per 10.000 updates. Registration is now on demand: a user-declared primary key is still registered up front, every other column registers itself through the single gate `Table.EnsureAutoHashIndexRegistered`, which every lookup path now shares (the read path, the position-aware update locate, both delete cores, the parallel scan, both struct-scan fast paths and structured DML), and the hidden `_rowid` fallback is never registered because the PK B-tree already serves its point lookups. *Building* was already lazy (`Table.EnsureIndexLoaded`), so nothing else in the index lifecycle changed, `HashIndexAutoCreationGateTests` was rewritten to the new contract (primary key eagerly, others on the first filtered operation, none with the dial off) rather than loosened, and `SHARPCOREDB_HASH_INDEXES=0` still turns automatic indexes off entirely. **Measured on arms B and C (`--pk-default-batch` / `--docs-batch`, INSERT vs SQLite, median with the run's own per-rep range): arm B batch 0,95× → 1,60× (1,05–1,92) and arm B SQL 0,83× → 1,05× (0,94–1,26); arm C batch 1,26× (1,19–1,54) and arm C SQL 0,90× → 0,98× (0,98–1,29)** — with same-build `SHARPCOREDB_HASH_INDEXES=0` runs as the ceiling on the day (arm B 1,68× batch / 1,31× SQL, arm C 1,25× batch / 0,95× SQL), i.e. what is left of the margin is the eagerly registered declared-primary-key index. On the query side the cost is now visible instead of hidden: `--auto-index-benefit` reports the **first** filtered query separately, because that is the query that pays for building the index — 20.000 rows, 2.000 point queries per column, two runs: the **first** query costs 35,5–71,0 ms on `email`/`score`/`age` and the warm queries after it cost 4,7–14,1 µs (`email`, `score`) against 4,8–5,0 ms with the dial off (**354×–988×** and **3.106×–7.656×**), with the explicitly indexed control unchanged (63–68 µs first, 5,9–6,3 µs repeated) and identical row counts in both configurations. Core suite **1952 / 0 failed / 0 errors**; `--gate` **PASSED, 8/8 metrics `ok`**, all eight ahead of the recorded baseline (default INSERT 0,78×, default UPDATE 0,31×).
2021
- **The single-file growth minimum is now 1 MiB (was 10 MiB), which takes every small `.scdb` database from 14,7 MB to 6,4 MB.** The 10 MiB minimum was what made a 100-row `.scdb` file 14,7 MB: a fresh file is 1.037 pages and the extension is `max(requiredPages, currentSize / 2, minExtensionPages)`, where 10 MiB is 2.560 pages at a 4 KiB page size — so the minimum won the contest and the first extension landed on 3.597 pages = **14.733.312 B**, the same size at 1 row and at 2.000 rows (measured 2026-09-23, which is what proved it was a floor rather than growth). At the new 1 MiB default the minimum no longer binds (256 pages against the halving term's 518) and the same database lands on **6.369.280 B, −56,8 %**, measured at **1 / 100 / 200 / 500 / 2.000 / 20.000 / 200.000 / 1.000.000 rows** with **byte-identical allocation per row** and no measurable throughput difference — the single-file blocks are compressed, so even a million rows of the benchmark schema stay inside that first extension. Existing databases are unaffected: the value is read **per open**, never changes how a file is read, and the historical 14,7 MB floor is one configuration line away (`SingleFileMinExtensionBytes = 10 * 1024 * 1024`), pinned by `SingleFileFileGrowthTests.ExplicitHistoricalTenMiBMinimum_StillProducesTheOldFloor` so the old behaviour stays reproducible byte for byte.
2122
- **The inline capacity's default is now 24 bytes (was 16), and it is worth 0,63× → 0,87× on the fair-PK INSERT ratio.** `DatabaseConfig.FixedWidthInlineValueBytes` decides how many payload bytes a variable-length column may keep **inside** its fixed-width record slot instead of writing them to the overflow arena, and 16 was two bytes short of the most common short-value shape: on the benchmark schema the 18-byte `user99999@test.com` overflowed for every row whose index is ≥ 1.000, so **99.000 of 100.000 rows paid an arena write each**. At 24 every value in that schema fits inline and `arena-write`, `arena-append` and `encode-scratch` fall to **zero calls**, measured on the harness's own `[diag]` counter and on `--pk-profile-insert`.
2223
- **Measured, shipped default, same machine and session (`REGIME: no SHARPCOREDB_* switches set`), `--pk` median of three runs with each arm's own same-run SQLite arm:** fair-PK INSERT **0,63× → 0,87×** (0,85 / 0,87 / 0,92 across three runs), absolute **130–135K → 162–171K ops/s** (above the plan's 150K acceptance target), READ 1,26×, UPDATE 1,29×, DELETE 1,62× (all ahead, unchanged in direction); the engine's own allocation per inserted row **1.923 → 1.436 B**; `--multirowinsert` **62.668 → 84.263/85.467 rows/s** with 3.555 B/row; `--pk-default` (encrypted pure default) INSERT **0,59–0,61× → 0,68–0,75×** and READ 0,50× → 0,59–0,64×.
@@ -27,6 +28,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
2728

2829
### Fixed
2930

31+
- **A hash index built from the data file after a length-changing UPDATE returned the superseded row, so `CREATE INDEX` (and the on-demand build) could answer a query with a row no scan returns.** The Columnar engines append a new version when an update grows a record and leave the old record in the file — compaction drops it, no tombstone marks it — while the index build walked the file with only a tombstone check, not the version rule the scan applies (the current version is the one the PK index points at). Measured on a 3-row variable-length table: `UPDATE t SET v = 'newvalue1-newvalue1-newvalue1' WHERE id = 1` followed by `CREATE INDEX … ON t(v)` made `SELECT … WHERE v = 'old1'` answer with **1 row where SQLite returns 0**, in both the PK and the PK-less shape, while the same table's scan correctly returned 3 rows. The rule now lives in one method, `Table.IsCurrentRecordVersion`, used by both the Columnar scan and the index rebuild (the at-rest and the plaintext walk), so an index and the scan that replaces it cannot disagree about which rows exist. Red→green: `HashIndexAutoCreationGateTests.LateIndexBuild_AfterLengthChangingUpdate_IgnoresTheSupersededRow` and `LateIndexBuild_OnDemandAfterLengthChangingUpdate_IgnoresTheSupersededRow`, each answering with the superseded row before the fix (live before/after on the same scenario: 1 → 0).
32+
- **The single-row UPDATE's append fallback left the hash-index entries of columns the statement did not name pointing at the record it abandoned — which surfaced as a ghost row once that row's indexed value changed.** `Table.CRUD.cs`'s `UpdateColumnarRow` snapshots the old values of indexed columns only when the statement touches a loaded index (the S2 shortcut, whose reasoning — "every entry still holds the same key at the same position" — is valid only while the record stays in place), and the append fallback moves the record. The abandoned entry is invisible while the indexed value is unchanged (the record it points at still carries that key) and becomes a returned ghost the moment it changes, because then the entry names a key the current row no longer has: on a 3-row variable-length table the loaded index held **4 entries for 3 rows** and `WHERE v = 'old1'` answered with the abandoned record (**1 row where SQLite returns 0**, and no scan shows it). The append path now captures the old keys of every loaded index from the record it is about to leave (`CaptureHashKeysAt`, reusing the bytes already read for the patch attempt). Red→green: `AppendUpdateIndexMaintenanceTests.AppendUpdate_OfAColumnNoIndexNames_LeavesNoEntryForThePositionItAbandoned` (4 entries and the ghost row on the pre-fix build, 3 entries and no ghost now); the pre-existing `SqlInPlaceUpdateTests.SqlUpdate_VariableWidth_GrowsWhenStoredLengthChanges_StillCorrect` failed on the intermediate build that made the per-column index set optional, which is how the defect was found.
3033
- **The coverage tool handed the collector an argument file instead of the test host, because three of the files the coverage tooling leaves in a build directory also end in `.Tests`.** `tools/measure-coverage.ps1` resolved the xUnit v3 host by name suffix alone, and a bin directory that has ever been collected holds `CoverletSourceRootsMapping_<suite>.Tests`, `.msCoverageSourceRootsMapping_<suite>.Tests` and `.msCoverageExtensionSourceRootsMapping_<suite>.Tests` — all matching the suffix, all sorting before `SharpCoreDB.Tests.exe` — so the collector was asked to run one of them and answered `An error occurred trying to start process '…\.msCoverageExtensionSourceRootsMapping_SharpCoreDB.Tests' with working directory '…' The specified executable is not a valid application for this OS platform`, producing no report and no test run (exit 3, twice in a row, until the resolver was read out of the code). The resolver now probes the exact host name (`<suite>.exe` on Windows, `<suite>` elsewhere) and its fallback scan skips dot-prefixed names and `*SourceRootsMapping*`. Red→green: two self-test cases plant all three argument files next to the host; both fail with the pre-fix resolver (`SELFTEST FAIL - 8 passed, 2 failed`, exit 1) and both pass now (`SELFTEST PASS - 10 passed, 0 failed`).
3134
- **A covered line was counted once per class and once per test project's copy of the assembly, so the coverage rate was compared against a threefold denominator.** Cobertura writes a line once for every class that owns it, and every test project's bin directory holds its own copy of every product assembly: counting the raw `<line>` elements made one report's 59515 real lines look like 120419, the core assembly's 91090 lines look like 273270 across three reports, and the merged rate **20.37%** where the same reports merged by line give **45.53%**. The aggregation now keys on `(assembly, source file, line number)`, ORs the hits and counts each line once — which is what codecov does with the uploaded reports — and skips the generated sources `codecov.yml` ignores (`**/*.g.cs`, `**/*.g.i.cs`, `**/*.Designer.cs`, `GlobalUsings.cs`). Red→green: `a line owned by two classes counts once` and `the same file in two reports counts once` fail under the old counting (exit 1) and pass under the fix.
3235
- **A server whose master database cannot be resolved no longer answers with an HTTP 500 plus a stack trace — it resolves the master database once, at startup, with an actionable message.** `TenantCatalogRepository` and `DatabaseGrantsRepository` were registered as singletons whose factory resolved the master database lazily, from `DatabaseRegistry` at *first request*. With `Server:DefaultDatabase` (or the configured system-database name) absent from `Server:Databases`, every request that constructed a controller — the anonymous readiness endpoint `/api/v1/health` included — failed with `InvalidOperationException: Unable to resolve master database for tenant catalog repository.` at error level plus a ~33-line ASP.NET stack, and that endpoint's documented `503 {"status":"starting"}` was unreachable; a request arriving between `app.StartAsync` and the registry's `InitializeAsync` took the same path (two unhandled-exception blocks in the harness's own server log). The startup sequence now initializes the registry, starts `NetworkServer` and resolves both repositories **before Kestrel starts listening**, so the window no longer exists and a misconfigured master database fails exactly once, loudly, at startup: `[FTL] SharpCoreDB Server failed startup validation` with the cause, and a **non-zero exit code** where the process previously exited 0. The three copies of the resolution logic (`Program.cs` twice, `NetworkServer`'s catalog bootstrap) now share one `MasterDatabaseLocator`, whose failure message names the configured system database, the default database, the databases the registry actually holds and whether it finished initializing. `DatabaseRegistry` also reports a configuration error instead of `Sequence contains no elements` when system databases are enabled but `Server:Databases` is empty. Measured live on the built server with a 24-database config whose `DefaultDatabase` is missing: before the fix `/api/v1/health` answered **500 on every probe**, with **10** `Unable to resolve` occurrences and **10** unhandled-exception blocks; after it the process exits **1** without ever opening the port (15/15 probes refused) and reports the misconfiguration once, while a correct config reaches 200 with **0** unresolved blocks and **0** unhandled blocks. Pinned by `MasterDatabaseLocatorTests` (7 facts); server integration suite **205 / 0 failed**, smoke **7/7**, core suite **2049 / 0 failed / 16 skipped**.

0 commit comments

Comments
 (0)