diff --git a/.changeset/25019-sqlite-file-failure-family.md b/.changeset/25019-sqlite-file-failure-family.md new file mode 100644 index 00000000000..a940ce43f6f --- /dev/null +++ b/.changeset/25019-sqlite-file-failure-family.md @@ -0,0 +1,24 @@ +--- +'@objectstack/driver-sql': minor +'@objectstack/service-datasource': minor +--- + +fix(driver-sql): a SQLite database whose header SQLite cannot read fails `connect()` at once, naming the file, and a dangling symbolic link as the database's directory is refused on the boot that writes, as the read-only boot already refused it (#25019) + +Three more SQLite files a declared datasource cannot use, each now classified by the file's own driver at connect: + +- **A database whose header or schema SQLite cannot read** (a file truncated inside its 100-byte header, or short of the pages its schema names). The connection's first statement, `PRAGMA auto_vacuum`, answers `SQLITE_CORRUPT`; `SqlDriver.connect()` logged it as "Failed to set PRAGMA auto_vacuum=INCREMENTAL" and resolved over a file no statement could read. On a development boot, `resolveSqliteDriver` (`@objectstack/service-datasource`) then stepped down to wasm SQLite, whose open renamed the file to `.corrupt-` and wrote an empty database in its place, under a warning that blamed the native addon. `connect()` now refuses it with a `SqliteDatabaseFileError` (`code: 'SQLITE_CORRUPT'`, `filename`, a message naming the file) in every mode, before anything is written, and `resolveSqliteDriver` re-throws it instead of stepping down. A database damaged deeper inside still connects: its first statement answers, and the damage surfaces on the read that reaches it, as before. +- **A dangling symbolic link as the database's directory, on the boot that writes.** `connect()` took `mkdir`'s `EEXIST` on the link for a concurrent creator and resolved; the first statement then failed with a code-less "directory does not exist", and a development boot stepped down to wasm SQLite under the same false native-addon warning. `connect()` now refuses it with `SqliteDatabaseFileError` (`code: 'ENOENT'`), naming the file and the link, which is the error the read-only boot (`sqliteAbsentFile: 'empty-in-memory'`) gives for the same path. +- **A dangling symbolic link as the database file itself, on the read-only boot.** When the directory its chain of links ends in does not exist, the read-only boot read the file absent and reported the datasource connected, while the boot that writes refuses the same path with `SQLITE_CANTOPEN`. The read-only boot now refuses it too (`code: 'ENOENT'`), naming the link's target. When that directory exists, both boots behave as before: the read-only boot opens an empty in-memory database, and the boot that writes creates the file through the link. + +A real database file, an absent file in a directory that exists or can be made, and a native addon that cannot load over a real database keep today's behaviour. + +Clause-②: no (narrowing) + + + +**BREAKING** (`SqlDriver.connect()`'s accept set and the read-only boot's narrow): a connect that resolved over a SQLite database whose header SQLite cannot read, or over a path whose directory is a dangling symbolic link, now rejects; a read-only boot that read a dangling file link absent although its target's directory does not exist now rejects. It ships as `minor` under the launch-window convention for accept-set narrowings: during the launch window a breaking change does not take `major`, so the level does not say it breaks, and breaking-ness is carried by this banner and the disposition above. + +- **Before:** `connect()` resolved; in production the first query failed with `DATABASE_ERROR`, and on a development boot a damaged file was renamed to `.corrupt-` and replaced with an empty database, and a dangling-link directory stepped down to wasm SQLite under a warning that blamed the native addon. The read-only boot reported a dangling file link connected. +- **Now:** `connect()` rejects with `SqliteDatabaseFileError`, `code: 'SQLITE_CORRUPT'` for the damaged file and `code: 'ENOENT'` for the dangling-link directory, naming the file (and the link); the read-only boot rejects the dangling file link with `code: 'ENOENT'`, naming its target. The file is untouched. +- **Fix:** restore the database from a backup, or move the damaged file aside yourself (an absent file is created empty on the next connect); point the dangling link at a directory that exists, or point the datasource at a real path. diff --git a/packages/drivers/driver-sql/src/sql-driver-25019-sqlite-damaged-file-and-dangling-link.test.ts b/packages/drivers/driver-sql/src/sql-driver-25019-sqlite-damaged-file-and-dangling-link.test.ts new file mode 100644 index 00000000000..b08fcd4ad6f --- /dev/null +++ b/packages/drivers/driver-sql/src/sql-driver-25019-sqlite-damaged-file-and-dangling-link.test.ts @@ -0,0 +1,258 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The last members of the SQLite file-failure family: a declared SQLite file + * the datasource cannot use is classified by its own driver at connect, naming + * the file, and nothing is written (#25019). + * + * - A SQLite database whose header or schema SQLite cannot read (a file + * truncated inside its 100-byte header, or short of the pages its schema + * names). The connection's first statement, `PRAGMA auto_vacuum`, answers + * `SQLITE_CORRUPT`, which `connect()` logged and swallowed: it resolved + * over a file no statement could read, and on a development boot the + * step-down then moved the file aside. `SQLITE_CANTOPEN` (#23696) and + * `SQLITE_NOTADB` (#24960) were already refused this way. + * - A dangling symbolic link as the file's directory, on the boot that + * writes. `makeDatabaseDirectory` took `mkdir`'s `EEXIST` on the link for + * a concurrent creator, `connect()` resolved, and the first statement + * failed with a code-less "directory does not exist". The read-only boot + * already refused the same path (#24961), so the two boots disagreed. + * - A dangling symbolic link as the file itself, on the read-only boot: it + * read the file absent although the directory the link points into does + * not exist, where the boot that writes refuses it with `SQLITE_CANTOPEN`. + * + * Every refusal is the driver's own, so the first one is pinned under both + * `NODE_ENV` values. + */ + +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import { + existsSync, lstatSync, mkdirSync, mkdtempSync, openSync, readdirSync, readFileSync, rmSync, symlinkSync, + truncateSync, writeSync, closeSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { SqlDriver, resolveSqliteAbsentFileTarget } from './index.js'; +import { SqliteDatabaseFileError } from './sqlite-database-file.js'; + +const DANGLING_WORDS = (link: string) => `'${link}' is a symbolic link to a path that does not exist`; + +const drivers: SqlDriver[] = []; +function make(filename: string): SqlDriver { + const d = new SqlDriver({ client: 'better-sqlite3', connection: { filename }, useNullAsDefault: true } as any); + drivers.push(d); + return d; +} + +/** `connect()`'s rejection, or a failure saying it resolved. */ +async function refusalOf(driver: SqlDriver): Promise { + try { + await driver.connect(); + } catch (error) { + return error; + } + throw new Error('connect() resolved for a file the datasource cannot use'); +} + +/** What `fn` threw; a failure naming its answer when it threw nothing. */ +function thrownBy(fn: () => unknown): any { + let answer: unknown; + try { + answer = fn(); + } catch (error) { + return error; + } + throw new Error(`answered ${JSON.stringify(answer)} for a file that cannot be created`); +} + +function expectFileRefusal(error: any, filename: string, code: string): void { + expect(error).toBeInstanceOf(SqliteDatabaseFileError); + expect(error.code).toBe(code); + expect(error.filename).toBe(filename); + expect(error.message).toContain(filename); +} + +/** Every path under `root`, each with its kind, so a link, a file or a directory that appears is seen. */ +function tree(root: string): string[] { + return readdirSync(root, { recursive: true }).map(String).sort() + .map((name) => `${name}:${lstatSync(join(root, name)).isSymbolicLink() ? 'link' : lstatSync(join(root, name)).isDirectory() ? 'dir' : 'file'}`); +} + +/** A real SQLite database, written by this driver, holding one row in `t`. */ +async function seedDatabase(filename: string): Promise { + const seed = new SqlDriver({ client: 'better-sqlite3', connection: { filename }, useNullAsDefault: true } as any); + await seed.connect(); + await seed.execute('CREATE TABLE t (a INTEGER)'); + await seed.execute('INSERT INTO t (a) VALUES (7)'); + await seed.disconnect(); +} + +/** + * The refused file holds the bytes it was found with, and nothing is left + * beside it. The header still says WAL (this driver's journal mode), so SQLite + * keeps its `-shm` and an empty `-wal` while the refused connection is open: + * no `.corrupt-*` sibling then, and nothing at all once it is closed. + */ +async function expectLeftAsItWas(driver: SqlDriver, filename: string, before: Buffer): Promise { + expect(readdirSync(dir).filter((name) => name.includes('.corrupt-'))).toEqual([]); + await driver.disconnect(); + expect(readFileSync(filename).equals(before)).toBe(true); + expect(readdirSync(dir)).toEqual(['archive.db']); +} + +let dir: string; +beforeEach(() => { dir = mkdtempSync(join(tmpdir(), 'os-25019-')); }); +afterEach(async () => { + vi.unstubAllEnvs(); + await Promise.all(drivers.splice(0).map((d) => d.disconnect().catch(() => {}))); + rmSync(dir, { recursive: true, force: true }); +}); + +describe('SqlDriver.connect — a declared SQLite file whose header or schema SQLite cannot read (#25019)', () => { + for (const nodeEnv of ['development', 'production'] as const) { + it(`refuses a database truncated inside its header with SQLITE_CORRUPT, naming it, and leaves its bytes untouched (NODE_ENV=${nodeEnv})`, async () => { + const filename = join(dir, 'archive.db'); + await seedDatabase(filename); + truncateSync(filename, 50); + const before = readFileSync(filename); + vi.stubEnv('NODE_ENV', nodeEnv); + + const driver = make(filename); + const error = await refusalOf(driver); + + expectFileRefusal(error, filename, 'SQLITE_CORRUPT'); + expect(error.message).toContain((error.cause as Error).message); + await expectLeftAsItWas(driver, filename, before); + }); + } + + it('refuses a database cut short of the pages its schema names, the same way', async () => { + const filename = join(dir, 'archive.db'); + await seedDatabase(filename); + truncateSync(filename, 4096); + const before = readFileSync(filename); + + const driver = make(filename); + expectFileRefusal(await refusalOf(driver), filename, 'SQLITE_CORRUPT'); + await expectLeftAsItWas(driver, filename, before); + }); + + it('CONTROL: a database damaged deeper inside still connects, and the read that reaches the damage fails', async () => { + // The refusal is the FIRST statement's verdict, never an integrity check: + // here the header and schema read, and one table's page does not. + const filename = join(dir, 'archive.db'); + await seedDatabase(filename); + const probe = make(filename); + await probe.connect(); + const [{ rootpage }] = (await probe.execute("SELECT rootpage FROM sqlite_master WHERE name = 't'")) as Array<{ rootpage: number }>; + const [{ page_size: pageSize }] = (await probe.execute('PRAGMA page_size')) as Array<{ page_size: number }>; + await probe.disconnect(); + const fd = openSync(filename, 'r+'); + writeSync(fd, Buffer.alloc(pageSize, 0xab), 0, pageSize, (rootpage - 1) * pageSize); + closeSync(fd); + + const driver = make(filename); + await driver.connect(); + const failure = await driver.execute('SELECT a FROM t').then( + () => { throw new Error('a read of the damaged page answered'); }, + (error: any) => error, + ); + expect(failure?.cause?.code).toBe('SQLITE_CORRUPT'); + }); +}); + +describe('SqlDriver.connect — a dangling symbolic link as the directory, on the boot that writes (#25019)', () => { + it('refuses it with the read-only boot\'s code, naming the file and the link, and creates nothing through it', async () => { + const link = join(dir, 'link'); + symlinkSync(join(dir, 'nowhere'), link); + const filename = join(link, 'archive.db'); + const before = tree(dir); + + const writing = await refusalOf(make(filename)); + expectFileRefusal(writing, filename, 'ENOENT'); + expect(writing.message).toContain(DANGLING_WORDS(link)); + + // The read-only boot over the same path answers with the same error class, + // code, file and words (#24961). + const readOnly = thrownBy(() => resolveSqliteAbsentFileTarget(filename, 'empty-in-memory')); + expectFileRefusal(readOnly, filename, 'ENOENT'); + expect(readOnly.message).toContain(DANGLING_WORDS(link)); + + expect(lstatSync(link).isSymbolicLink()).toBe(true); + expect(existsSync(join(dir, 'nowhere'))).toBe(false); + expect(tree(dir)).toEqual(before); + }); + + it('refuses a deeper path under the link the same way, naming the link', async () => { + const link = join(dir, 'link'); + symlinkSync(join(dir, 'nowhere'), link); + const filename = join(link, 'sub', 'archive.db'); + const before = tree(dir); + + const writing = await refusalOf(make(filename)); + expectFileRefusal(writing, filename, 'ENOENT'); + expect(writing.message).toContain(DANGLING_WORDS(link)); + expect(tree(dir)).toEqual(before); + }); + + it('CONTROL: a directory that is a link to an existing directory connects, and the file is created in it', async () => { + mkdirSync(join(dir, 'real')); + symlinkSync(join(dir, 'real'), join(dir, 'link')); + const driver = make(join(dir, 'link', 'archive.db')); + await driver.connect(); + expect(await driver.execute('SELECT 1 AS one')).toEqual([{ one: 1 }]); + expect(readdirSync(join(dir, 'real'))).toContain('archive.db'); + }); + + it('CONTROL: an absent file under directories that can be made connects, and both are created', async () => { + const filename = join(dir, 'new', 'deeper', 'archive.db'); + const driver = make(filename); + await driver.connect(); + expect(await driver.execute('SELECT 1 AS one')).toEqual([{ one: 1 }]); + expect(existsSync(filename)).toBe(true); + }); +}); + +describe('a dangling symbolic link as the database file itself (#25019)', () => { + it('the read-only boot refuses it when the directory the chain of links ends in does not exist, as the writing boot does', async () => { + const target = join(dir, 'missing', 'target.db'); + const direct = join(dir, 'archive.db'); + symlinkSync(target, direct); + // The same end, reached through a second link. + symlinkSync(target, join(dir, 'hop.db')); + const chained = join(dir, 'chained.db'); + symlinkSync(join(dir, 'hop.db'), chained); + const before = tree(dir); + + for (const filename of [direct, chained]) { + const readOnly = thrownBy(() => resolveSqliteAbsentFileTarget(filename, 'empty-in-memory')); + expectFileRefusal(readOnly, filename, 'ENOENT'); + expect(readOnly.message).toContain(target); + + // The writing boot refuses the same path with SQLite's own answer. + expectFileRefusal(await refusalOf(make(filename)), filename, 'SQLITE_CANTOPEN'); + } + expect(tree(dir)).toEqual(before); + }); + + it('CONTROL: when that directory exists, the read-only boot reads it absent and writes nothing, and the writing boot creates the file through the link', async () => { + symlinkSync(join(dir, 'target.db'), join(dir, 'archive.db')); + // A relative target is read from the link's own directory. + symlinkSync('relative-target.db', join(dir, 'relative.db')); + const before = tree(dir); + + for (const name of ['archive.db', 'relative.db']) { + expect(resolveSqliteAbsentFileTarget(join(dir, name), 'empty-in-memory')) + .toEqual({ filename: ':memory:', openedEmptyInMemory: true }); + } + expect(tree(dir)).toEqual(before); + + for (const name of ['archive.db', 'relative.db']) { + const driver = make(join(dir, name)); + await driver.connect(); + expect(await driver.execute('SELECT 1 AS one')).toEqual([{ one: 1 }]); + } + expect(existsSync(join(dir, 'target.db'))).toBe(true); + expect(existsSync(join(dir, 'relative-target.db'))).toBe(true); + }); +}); diff --git a/packages/drivers/driver-sql/src/sql-driver.ts b/packages/drivers/driver-sql/src/sql-driver.ts index d70cdc1b6cb..778b08f36ba 100644 --- a/packages/drivers/driver-sql/src/sql-driver.ts +++ b/packages/drivers/driver-sql/src/sql-driver.ts @@ -7061,7 +7061,7 @@ export class SqlDriver implements IDataDriver { * not merely that space hygiene is unavailable. * @throws {SqliteDatabaseFileError} when that first statement's answer is a * verdict on the file itself ({@link SQLITE_FILE_REFUSALS}): it cannot be - * opened, or it is not a SQLite database. + * opened, it is not a SQLite database, or its header or schema is damaged. */ private async applyAutoVacuumIncremental(): Promise { try { @@ -7097,7 +7097,17 @@ export class SqlDriver implements IDataDriver { // used to answer it with wasm SQLite, whose open moved the // operator's file aside and wrote an empty database in its place. // The refusal is issued before anything is written: the file is left - // exactly as it was found. + // exactly as it was found; + // • SQLITE_CORRUPT (#25019) — the file is a SQLite database SQLite + // cannot read: its header or its schema is damaged (a file truncated + // inside its 100-byte header, or short of the pages its schema + // names). At the FIRST statement this is a verdict on the whole + // file, measured: `SELECT 1` answers the same code. A database + // damaged deeper inside answers this read and connects; its damage + // surfaces on the read that reaches it, as before. A development + // boot used to take the same step-down to wasm SQLite as for + // SQLITE_NOTADB, and the same rename; it is refused the same way, + // before anything is written. const refusal = code === undefined ? undefined : SqlDriver.SQLITE_FILE_REFUSALS.get(code); if (refusal !== undefined) { const declared = (this.config as any).connection; @@ -7132,6 +7142,7 @@ export class SqlDriver implements IDataDriver { private static readonly SQLITE_FILE_REFUSALS: ReadonlyMap = new Map([ ['SQLITE_CANTOPEN', 'cannot be opened or created'], ['SQLITE_NOTADB', 'is not a SQLite database, and was left as it is'], + ['SQLITE_CORRUPT', 'is damaged (SQLite cannot read its header or its schema), and was left as it is'], ]); /** What `PRAGMA auto_vacuum` answers for INCREMENTAL (0 is NONE, 1 is FULL). */ diff --git a/packages/drivers/driver-sql/src/sqlite-database-file.ts b/packages/drivers/driver-sql/src/sqlite-database-file.ts index 2bf610f6785..adf3e6b6214 100644 --- a/packages/drivers/driver-sql/src/sqlite-database-file.ts +++ b/packages/drivers/driver-sql/src/sqlite-database-file.ts @@ -1,6 +1,6 @@ // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. -import { accessSync, constants, lstatSync, statSync } from 'node:fs'; +import { accessSync, constants, lstatSync, readlinkSync, realpathSync, statSync } from 'node:fs'; import { mkdir, stat } from 'node:fs/promises'; import { dirname, isAbsolute, resolve } from 'node:path'; @@ -53,6 +53,14 @@ export class SqliteDatabaseFileError extends Error { * the answer. When a file holds the nearest existing name, the operating * system answers for it too: `ENOTDIR` from the first missing level under it, * or `EEXIST` from asking for `dir` itself. + * + * A level whose name a dangling symbolic link holds is refused during the + * walk, with `stat`'s `ENOENT` as its code and the link named (#25019). + * `mkdir` answers such a level `EEXIST`, which the creating loop would take + * for a concurrent creator, and nothing can be created through it: the connect + * used to resolve, and the first statement then failed with a code-less + * "directory does not exist". {@link probeAbsentSqliteDatabaseFile}, the + * read-only twin, refuses the same level with the same words. */ export async function makeDatabaseDirectory(dir: string): Promise { const missing: string[] = []; @@ -63,6 +71,7 @@ export async function makeDatabaseDirectory(dir: string): Promise { break; } catch (e) { if ((e as { code?: unknown } | null)?.code !== 'ENOENT') throw e; + if (isDanglingLink(level)) throw danglingLinkRefusal(level, e); missing.push(level); const parent = dirname(level); if (parent === level) break; @@ -99,6 +108,12 @@ export async function makeDatabaseDirectory(dir: string): Promise { * is the refusal. `existsSync` answers `false` for every one of these, which * is how a read-only boot used to open an empty in-memory database over a * path its writing twin refuses, and report the datasource connected. + * - `ENOENT`, and the file's own name is a dangling symbolic link (#25019): + * the boot that writes creates the file where the chain of links ends, in + * a directory it does not make, so that directory is judged instead: it + * must exist, and allow write and search (`access(W_OK | X_OK)`). Each + * refusal is the answer; otherwise `'absent'`. The writing boot refuses + * the same path with SQLite's `SQLITE_CANTOPEN`. * - `ENOENT`: walk up to the nearest level that exists. A level whose name a * dangling symbolic link holds cannot be made (`mkdir` answers `EEXIST`, * and nothing can be created through it): refused. Otherwise the level is a @@ -119,6 +134,24 @@ export function probeAbsentSqliteDatabaseFile(filename: string): 'exists' | 'abs } catch (e) { if (errorCode(e) !== 'ENOENT') throw new SqliteDatabaseFileError(filename, 'cannot be opened or created', e); } + const linkEnd = danglingLinkEnd(filename); + if (linkEnd !== undefined) { + const where = dirname(linkEnd); + const linked = `cannot be created: it is a symbolic link to '${linkEnd}', whose directory '${where}'`; + try { + // `stat(filename)` answered `ENOENT`, not `ENOTDIR` or `EACCES`, so + // `where` is a directory this process may search when it exists at all. + statSync(where); + } catch (e) { + throw new SqliteDatabaseFileError(filename, `${linked} does not exist`, e); + } + try { + accessSync(where, constants.W_OK | constants.X_OK); + } catch (e) { + throw new SqliteDatabaseFileError(filename, `${linked} does not allow it to be created`, e); + } + return 'absent'; + } const dir = dirname(resolve(filename)); const unmakeable = `cannot be created: its directory '${dirname(filename)}' cannot be made`; let level = dir; @@ -128,9 +161,7 @@ export function probeAbsentSqliteDatabaseFile(filename: string): 'exists' | 'abs break; } catch (e) { if (errorCode(e) !== 'ENOENT') throw new SqliteDatabaseFileError(filename, unmakeable, e); - if (isDanglingLink(level)) { - throw new SqliteDatabaseFileError(filename, `${unmakeable} ('${level}' is a symbolic link to a path that does not exist)`, e); - } + if (isDanglingLink(level)) throw new SqliteDatabaseFileError(filename, `${unmakeable} (${danglingLinkWords(level)})`, e); const parent = dirname(level); if (parent === level) break; level = parent; @@ -154,6 +185,43 @@ function isDanglingLink(path: string): boolean { } } +/** How both boots name a dangling link that holds a level of the database's path. */ +function danglingLinkWords(level: string): string { + return `'${level}' is a symbolic link to a path that does not exist`; +} + +/** + * The writing boot's refusal of a directory level a dangling link holds: the + * link named, and `stat`'s own error as its `code` and `cause`, so + * `SqlDriver.connect()` raises it as the read-only boot does. + */ +function danglingLinkRefusal(level: string, cause: unknown): Error { + const reason = cause instanceof Error ? cause.message : String(cause); + return Object.assign(new Error(`${danglingLinkWords(level)}: ${reason}`), { code: errorCode(cause), cause }); +} + +/** + * Where creating `filename` puts the file when `filename` is a dangling + * symbolic link, which `stat` has already answered `ENOENT` for: the name at the + * end of its chain of links. `undefined` when `filename` is not a link. Each + * relative target is read from the real directory of the link that holds it, + * as the kernel reads it. The chain ends within the kernel's own link limit, + * or `stat` would have answered `ELOOP`; the bound below only restates that. + */ +function danglingLinkEnd(filename: string): string | undefined { + let at = resolve(filename); + for (let hops = 0; hops <= 40; hops++) { + let next: string; + try { + next = resolve(realpathSync(dirname(at)), readlinkSync(at)); + } catch { + return hops === 0 ? undefined : at; + } + at = next; + } + return at; +} + function errorCode(e: unknown): unknown { return (e as { code?: unknown } | null | undefined)?.code; } diff --git a/packages/services/service-datasource/src/__tests__/sqlite-driver-fallback-damaged-file-and-dangling-link.test.ts b/packages/services/service-datasource/src/__tests__/sqlite-driver-fallback-damaged-file-and-dangling-link.test.ts new file mode 100644 index 00000000000..8b8f853ccb2 --- /dev/null +++ b/packages/services/service-datasource/src/__tests__/sqlite-driver-fallback-damaged-file-and-dangling-link.test.ts @@ -0,0 +1,263 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * A declared SQLite datasource whose file SQLite cannot read, or whose + * directory is a dangling symbolic link, is refused at connect, naming the + * file, in development as in production. On a development boot nothing steps + * down to wasm SQLite or the in-memory driver, so the operator's file is + * neither renamed nor replaced, and no "native better-sqlite3 unavailable" + * line blames the addon (#25019). + * + * Before, measured through `os migrate apply --yes` on the card: + * - a database truncated inside its header: `SqlDriver.connect()` swallowed + * the first statement's `SQLITE_CORRUPT`, the probe's `SELECT 1` failed as + * a `DATABASE_ERROR`, and the development step-down answered with wasm + * SQLite, whose open moved the file to `.corrupt-` and + * wrote an empty database in its place; the datasource read `connected`; + * - a dangling link as the directory: `connect()` resolved, the probe failed + * with a code-less `TypeError`, and the step-down answered with wasm SQLite + * under the false native-addon warning, while the read-only boot refused + * the same path. + * + * Real `better-sqlite3`, real wasm SQLite and real files in a scratch + * directory, as in `./sqlite-driver-fallback-file-open-failure.test.ts`, whose + * load-failure seam the controls reuse. + */ + +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { existsSync, lstatSync, mkdtempSync, readdirSync, readFileSync, rmSync, symlinkSync, truncateSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { SqlDriver } from '@objectstack/driver-sql'; +import { + NATIVE_SQLITE_WASM_FALLBACK_WARNING, + resolveSqliteDriver, + type ResolvedSqliteDriver, +} from '../sqlite-driver-fallback.js'; +import { DatasourceConnectionService } from '../datasource-connection-service.js'; +import { createDefaultDatasourceDriverFactory } from '../default-datasource-driver-factory.js'; + +const DANGLING_WORDS = (link: string) => `'${link}' is a symbolic link to a path that does not exist`; + +let dir: string; +const opened: ResolvedSqliteDriver[] = []; + +beforeEach(() => { dir = mkdtempSync(join(tmpdir(), 'os-25019-')); }); + +afterEach(async () => { + vi.restoreAllMocks(); + vi.unstubAllEnvs(); + await Promise.all(opened.splice(0).map((r) => r.driver?.disconnect?.().catch(() => {}))); + rmSync(dir, { recursive: true, force: true }); +}); + +/** A real SQLite database, written by the native driver, holding one row in `t`. */ +async function seedDatabase(filename: string): Promise { + const seed = new SqlDriver({ client: 'better-sqlite3', connection: { filename }, useNullAsDefault: true } as any); + await seed.connect(); + await seed.execute('CREATE TABLE t (a INTEGER)'); + await seed.execute('INSERT INTO t (a) VALUES (7)'); + await seed.disconnect(); +} + +/** A database this driver wrote, truncated inside its 100-byte header. Its bytes. */ +async function headerTruncatedDatabase(filename: string): Promise { + await seedDatabase(filename); + truncateSync(filename, 50); + return readFileSync(filename); +} + +/** `promise`'s rejection; a failure naming the engine when it resolves instead. */ +async function rejectionOf(promise: Promise): Promise { + let resolved: ResolvedSqliteDriver; + try { + resolved = await promise; + } catch (error) { + return error; + } + opened.push(resolved); + throw new Error(`resolved to engine '${resolved.engine}' for a file the datasource cannot use`); +} + +/** The driver's refusal of the file: its own error name, the code, and the file. */ +function expectFileRefusal(error: any, filename: string, code: string): void { + expect(error?.name).toBe('SqliteDatabaseFileError'); + expect(error?.code).toBe(code); + expect(error?.filename).toBe(filename); + expect(String(error?.message)).toContain(filename); +} + +/** The bytes it was found with, and no `.corrupt-*` sibling: nothing moved it aside. */ +function expectNotMovedAside(filename: string, bytes: Buffer): void { + expect(readFileSync(filename).equals(bytes)).toBe(true); + expect(readdirSync(dir).filter((name) => name.includes('.corrupt-'))).toEqual([]); +} + +function connectionService(mode: 'development' | 'production', opts: { sqliteAbsentFile?: 'empty-in-memory' } = {}) { + const registered: string[] = []; + const engine = { + registerDriver: (driver: { name?: string }) => { registered.push(String(driver?.name)); }, + getDriverByName: (name: string) => (registered.includes(name) ? {} : undefined), + }; + const service = new DatasourceConnectionService({ + factory: () => createDefaultDatasourceDriverFactory({ dev: mode === 'development', ...opts }), + engine: () => engine as any, + }); + return { service, registered }; +} + +/** A real `ERR_DLOPEN_FAILED`: Node asked to load, as an addon, a file that is not one. */ +function realDlopenFailure(): Error { + const notAnAddon = join(dir, 'not-an-addon.node'); + writeFileSync(notAnAddon, 'x'); + try { + process.dlopen({ exports: {} }, notAnAddon); + } catch (error) { + return error as Error; + } + throw new Error('process.dlopen loaded a file that is not an addon'); +} + +/** Make the native addon fail to load with `error`, where `better-sqlite3` loads it (the #24950 seam). */ +async function failNativeAddonLoad(error: Error): Promise { + const probe = new SqlDriver({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true } as any); + const clientProto = Object.getPrototypeOf((probe as any).knex.client); + await probe.disconnect(); + expect(Object.prototype.hasOwnProperty.call(clientProto, 'acquireRawConnection')).toBe(true); + vi.spyOn(clientProto, 'acquireRawConnection').mockRejectedValue(error); +} + +describe('a declared SQLite database whose header SQLite cannot read (#25019)', () => { + it('development: resolveSqliteDriver rejects with the driver\'s SQLITE_CORRUPT refusal, never wasm or memory, and leaves the file as it was', async () => { + const filename = join(dir, 'archive.db'); + const bytes = await headerTruncatedDatabase(filename); + const warn = vi.fn(); + + expectFileRefusal(await rejectionOf(resolveSqliteDriver({ filename, dev: true, warn })), filename, 'SQLITE_CORRUPT'); + + expect(warn).not.toHaveBeenCalled(); + // The probe's connection was closed, so SQLite's WAL siblings are gone too. + expect(readFileSync(filename).equals(bytes)).toBe(true); + expect(readdirSync(dir)).toEqual(['archive.db']); + }); + + it('production: returns native unprobed, and its connect() refuses the file', async () => { + const filename = join(dir, 'archive.db'); + const bytes = await headerTruncatedDatabase(filename); + const warn = vi.fn(); + const resolved = await resolveSqliteDriver({ filename, dev: false, warn }); + opened.push(resolved); + expect(resolved.engine).toBe('better-sqlite3'); + + let refusal: unknown; + await resolved.driver.connect().catch((e: unknown) => { refusal = e; }); + expectFileRefusal(refusal, filename, 'SQLITE_CORRUPT'); + expect(warn).not.toHaveBeenCalled(); + expectNotMovedAside(filename, bytes); + }); + + for (const mode of ['development', 'production'] as const) { + it(`DatasourceConnectionService, ${mode}: reports the datasource failed naming the file, registers no driver, and moves nothing aside`, async () => { + vi.stubEnv('NODE_ENV', mode); + const filename = join(dir, 'archive.db'); + const bytes = await headerTruncatedDatabase(filename); + const { service, registered } = connectionService(mode); + + await service.connectDeclared({ + datasources: [{ name: 'archive', driver: 'sqlite', config: { filename }, autoConnect: true }], + }); + + const state = service.getConnectionState('archive'); + expect(state?.availability).toBe('failed'); + expect(state?.reason).toContain(filename); + expect(registered).toEqual([]); + expectNotMovedAside(filename, bytes); + }); + } +}); + +describe('a declared SQLite datasource whose directory is a dangling symbolic link, on the boot that writes (#25019)', () => { + let link: string; + let filename: string; + beforeEach(() => { + link = join(dir, 'link'); + symlinkSync(join(dir, 'nowhere'), link); + filename = join(link, 'archive.db'); + }); + + /** The link still dangles, and nothing was created through it. */ + function expectNothingCreated(): void { + expect(lstatSync(link).isSymbolicLink()).toBe(true); + expect(existsSync(join(dir, 'nowhere'))).toBe(false); + expect(readdirSync(dir)).toEqual(['link']); + } + + it('development: resolveSqliteDriver rejects with the read-only boot\'s refusal, with no step-down and no native-addon warning', async () => { + const warn = vi.fn(); + const writing = await rejectionOf(resolveSqliteDriver({ filename, dev: true, warn })); + expectFileRefusal(writing, filename, 'ENOENT'); + expect(String(writing.message)).toContain(DANGLING_WORDS(link)); + expect(warn).not.toHaveBeenCalled(); + + // The read-only boot over the same path (#24961): the same class, code, file and words. + const readOnly = await rejectionOf(resolveSqliteDriver({ filename, dev: true, warn, sqliteAbsentFile: 'empty-in-memory' })); + expectFileRefusal(readOnly, filename, 'ENOENT'); + expect(String(readOnly.message)).toContain(DANGLING_WORDS(link)); + + expectNothingCreated(); + }); + + for (const mode of ['development', 'production'] as const) { + it(`DatasourceConnectionService, ${mode}: reports the datasource failed naming the file and the link, as the read-only boot does`, async () => { + vi.stubEnv('NODE_ENV', mode); + for (const opts of [{}, { sqliteAbsentFile: 'empty-in-memory' as const }]) { + const { service, registered } = connectionService(mode, opts); + await service.connectDeclared({ + datasources: [{ name: 'archive', driver: 'sqlite', config: { filename }, autoConnect: true }], + }); + + const state = service.getConnectionState('archive'); + expect(state?.availability).toBe('failed'); + expect(state?.reason).toContain(filename); + expect(state?.reason).toContain(DANGLING_WORDS(link)); + expect(registered).toEqual([]); + } + expectNothingCreated(); + }); + } +}); + +describe('CONTROLS: what keeps today\'s behaviour (#25019)', () => { + it('a real database opens native in development, and answers', async () => { + const filename = join(dir, 'archive.db'); + await seedDatabase(filename); + const warn = vi.fn(); + const resolved = await resolveSqliteDriver({ filename, dev: true, warn }); + opened.push(resolved); + expect(resolved.engine).toBe('better-sqlite3'); + expect(await resolved.driver.execute('SELECT a FROM t')).toEqual([{ a: 7 }]); + expect(warn).not.toHaveBeenCalled(); + }); + + it('an absent file in a directory that can be made opens native in development, and is created', async () => { + const filename = join(dir, 'new', 'archive.db'); + const resolved = await resolveSqliteDriver({ filename, dev: true, warn: vi.fn() }); + opened.push(resolved); + expect(resolved.engine).toBe('better-sqlite3'); + expect(existsSync(filename)).toBe(true); + }); + + it('a native addon that cannot load, over a real database, still steps down to wasm SQLite, which reads the same rows', async () => { + const filename = join(dir, 'archive.db'); + await seedDatabase(filename); + await failNativeAddonLoad(realDlopenFailure()); + + const warn = vi.fn(); + const resolved = await resolveSqliteDriver({ filename, dev: true, warn }); + opened.push(resolved); + expect(resolved.engine).toBe('sqlite-wasm'); + expect(warn).toHaveBeenCalledWith(NATIVE_SQLITE_WASM_FALLBACK_WARNING); + expect(await resolved.driver.execute('SELECT a FROM t')).toEqual([{ a: 7 }]); + expect(readdirSync(dir).filter((name) => name.includes('.corrupt-'))).toEqual([]); + }); +}); diff --git a/packages/services/service-datasource/src/sqlite-driver-fallback.ts b/packages/services/service-datasource/src/sqlite-driver-fallback.ts index 165d52e2d85..6e42085d451 100644 --- a/packages/services/service-datasource/src/sqlite-driver-fallback.ts +++ b/packages/services/service-datasource/src/sqlite-driver-fallback.ts @@ -45,10 +45,12 @@ * - **The database FILE cannot be opened or created, or is not a database — * re-thrown, as in production.** A path under a regular file, a parent * directory the driver cannot make, a permission refusal, a directory where - * the file should be, a file whose bytes are not a SQLite database. + * the file should be, a file whose bytes are not a SQLite database, a + * database whose header SQLite cannot read. * `SqlDriver.connect()` names these with the operating system's code * (`EEXIST`, `ENOTDIR`, `ENOENT`, `EACCES`, …) or SQLite's - * `SQLITE_CANTOPEN` / `SQLITE_NOTADB`; see {@link isSqliteFileOpenFailure}. + * `SQLITE_CANTOPEN` / `SQLITE_NOTADB` / `SQLITE_CORRUPT`; see + * {@link isSqliteFileOpenFailure}. * A file that is not a database used to step down to wasm SQLite, whose * open moved the operator's file aside and wrote an empty database in its * place (#24960). Every rung opens @@ -138,9 +140,10 @@ function isEphemeralFilename(filename: string): boolean { } /** - * Is this probe failure the database FILE failing to open or be created, or - * not being a database at all, as opposed to the native addon failing to load - * (#24950, #24960)? Judged on the thrown value's own `code` alone, never on its + * Is this probe failure the database FILE failing to open or be created, not + * being a database at all, or being one SQLite cannot read, as opposed to the + * native addon failing to load (#24950, #24960, #25019)? Judged on the thrown + * value's own `code` alone, never on its * message. `SqlDriver.connect()` is what refuses such a file, with a * `SqliteDatabaseFileError` whose `code` is the answer it got: * @@ -152,7 +155,11 @@ function isEphemeralFilename(filename: string): boolean { * - SQLite's `SQLITE_NOTADB`: the file opens, and its bytes are not a SQLite * database. wasm SQLite reads the same bytes, and its open moves a file it * cannot read aside and starts an empty database in its place, so stepping - * down would replace the operator's file rather than read it. + * down would replace the operator's file rather than read it; + * - SQLite's `SQLITE_CORRUPT` (#25019): the file is a SQLite database whose + * header or schema SQLite cannot read, such as one truncated inside its + * header. The connection's first statement answers it only for damage of + * that kind, and wasm SQLite would move it aside the same way. * * An addon that cannot load never carries any of these. It surfaces at the * probe's `SELECT 1`, whose failure `SqlDriver.execute()` hands back as @@ -167,7 +174,7 @@ function isEphemeralFilename(filename: string): boolean { function isSqliteFileOpenFailure(err: unknown): boolean { const code = (err as { code?: unknown } | null | undefined)?.code; if (typeof code !== 'string') return false; - return /^E[A-Z0-9]+$/.test(code) || code === 'SQLITE_CANTOPEN' || code === 'SQLITE_NOTADB'; + return /^E[A-Z0-9]+$/.test(code) || code === 'SQLITE_CANTOPEN' || code === 'SQLITE_NOTADB' || code === 'SQLITE_CORRUPT'; } /** Best-effort close of a probe driver that failed; its own error is not the answer. */