Skip to content

Commit 2a89eb6

Browse files
committed
feat: retry SQLite writer locks
1 parent a2675a4 commit 2a89eb6

6 files changed

Lines changed: 166 additions & 29 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
## Unreleased
44

5+
- Retry transient SQLite writer acquisition with bounded capped backoff without
6+
replaying transaction callbacks or extending synchronous deadlines.
57
- Expose concise `configure()` and isolated `createRuntime()` entry points;
68
isolated actor turns resolve references from their executing runtime.
79
- Raise structured `MessageFailed` errors for terminal operations and

‎README.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,11 @@ import { sqlite } from "solid-objects/database/sqlite"
106106
import { Counter } from "./counter.js"
107107

108108
const runtime = configure({
109-
database: sqlite({ path: "storage/solid-objects.sqlite3" }),
109+
database: sqlite({
110+
path: "storage/solid-objects.sqlite3",
111+
timeoutMilliseconds: 5_000,
112+
lockRetryAttempts: 10,
113+
}),
110114
authorizeMessage: ({ authorizationContext }) => authorizationContext !== undefined,
111115
authorizeQuery: ({ authorizationContext }) => authorizationContext !== undefined,
112116
authorizeDestroy: ({ authorizationContext }) => authorizationContext !== undefined,
@@ -124,6 +128,11 @@ process.once("SIGINT", () => shutdown.abort())
124128
await runtime.run(shutdown.signal)
125129
```
126130

131+
SQLite serializes access inside one Node process. Across processes, it uses the
132+
native busy timeout and retries transient `BEGIN IMMEDIATE` contention with
133+
short capped backoff. `lockRetryAttempts` bounds those retries; synchronous
134+
invocations remain bounded by their end-to-end `timeoutMilliseconds` deadline.
135+
127136
`configure()` installs this runtime as the default used by `Actor.ref()`. Use
128137
`createRuntime()` when an application needs an isolated runtime and address its
129138
actors through `runtime.ref(ActorClass, actorId)`. Actor code executing in that

‎docs/architecture.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -38,11 +38,13 @@ enqueue transaction when InnoDB chooses it as a deadlock victim. Every adapter
3838
uses database time and the same fencing predicates.
3939

4040
Synchronous invocation carries a monotonic deadline into adapter operations.
41-
SQLite bounds its process-local access queue and busy timeout. PostgreSQL bounds
42-
pool checkout and installs statement and lock timeouts. MySQL bounds pool
43-
checkout and client queries and installs transaction execution and lock-wait
44-
limits. A deadline before enqueue commit produces no durable message. After
45-
commit, timeout diagnostics retain the message reference for later recovery.
41+
SQLite bounds its process-local access queue and busy timeout. Outside a caller
42+
deadline, it retries only acquisition of a failed `BEGIN IMMEDIATE`; it never
43+
replays a transaction callback. PostgreSQL bounds pool checkout and installs
44+
statement and lock timeouts. MySQL bounds pool checkout and client queries and
45+
installs transaction execution and lock-wait limits. A deadline before enqueue
46+
commit produces no durable message. After commit, timeout diagnostics retain
47+
the message reference for later recovery.
4648
Already-running JavaScript actor code is cooperative rather than forcefully
4749
preempted; leases and fenced commits remain authoritative if it outlives the
4850
caller's wait.

‎docs/parity.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -56,15 +56,15 @@ Reference: Ruby `solid_objects` 0.12.0 at commit `a01b6f5`.
5656

5757
## Databases and wake-up
5858

59-
| Capability | Status | TypeScript shape or remaining work |
60-
| ------------------------ | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
61-
| SQLite | Native | Uses built-in `node:sqlite`, serialized process-local access, foreign keys, strict tables, database time, and deadline-bounded access and lock waits. |
62-
| PostgreSQL | Native | Optional `pg` 8.23 peer, bounded pooling, 64-bit schema, row-locked sequences, server checks, and deadline-bounded pool, statement, and lock waits. |
63-
| MySQL | Native | Optional `mysql2` 3.23 peer, bounded pooling, InnoDB schema, row-locked sequences, scoped deadlock retry, and deadline-bounded pool, query, and lock waits. |
64-
| Durable polling fallback | Native | Every role progresses without a notification service. |
65-
| In-process wake-up | Native | A generation-based default adapter prevents claim-to-wait signal loss; commits wake role-specific waiters and polling remains the fallback. |
66-
| PostgreSQL wake-up | Native | `database.wakeUp()` uses one dedicated event-driven client, role-specific `LISTEN/NOTIFY`, generation fencing, reconnectable listeners, and durable polling fallback. |
67-
| Redis wake-up | Native | An optional `redis` peer provides role-specific Pub/Sub over separate lazy publisher/subscriber connections, with bounded failures and durable polling fallback. |
59+
| Capability | Status | TypeScript shape or remaining work |
60+
| ------------------------ | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
61+
| SQLite | Native | Uses built-in `node:sqlite`, serialized process-local access, bounded transient writer retries, foreign keys, strict tables, database time, and deadline-bounded access and lock waits. |
62+
| PostgreSQL | Native | Optional `pg` 8.23 peer, bounded pooling, 64-bit schema, row-locked sequences, server checks, and deadline-bounded pool, statement, and lock waits. |
63+
| MySQL | Native | Optional `mysql2` 3.23 peer, bounded pooling, InnoDB schema, row-locked sequences, scoped deadlock retry, and deadline-bounded pool, query, and lock waits. |
64+
| Durable polling fallback | Native | Every role progresses without a notification service. |
65+
| In-process wake-up | Native | A generation-based default adapter prevents claim-to-wait signal loss; commits wake role-specific waiters and polling remains the fallback. |
66+
| PostgreSQL wake-up | Native | `database.wakeUp()` uses one dedicated event-driven client, role-specific `LISTEN/NOTIFY`, generation fencing, reconnectable listeners, and durable polling fallback. |
67+
| Redis wake-up | Native | An optional `redis` peer provides role-specific Pub/Sub over separate lazy publisher/subscriber connections, with bounded failures and durable polling fallback. |
6868

6969
## Realtime and browser behavior
7070

‎src/database/sqlite.ts‎

Lines changed: 46 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,17 @@
11
import { DatabaseSync } from "node:sqlite"
22
import type { Database, DatabaseConnection, RunResult } from "./types.js"
3-
import { databaseDeadlineError, requireDatabaseDeadlineRemaining } from "./deadline.js"
3+
import {
4+
databaseDeadlineError,
5+
databaseDeadlineRemainingMilliseconds,
6+
requireDatabaseDeadlineRemaining,
7+
} from "./deadline.js"
48
import { DatabaseDeadlineExceeded } from "../errors.js"
59
import { databaseTransactionActive, withDatabaseTransaction } from "./transaction-context.js"
610

711
export interface SQLiteDatabaseOptions {
812
path: string
913
timeoutMilliseconds?: number
14+
lockRetryAttempts?: number
1015
}
1116

1217
class SQLiteConnection implements DatabaseConnection {
@@ -50,9 +55,14 @@ export class SQLiteDatabase implements Database {
5055
private accessTail: Promise<void> = Promise.resolve()
5156
private closed = false
5257
private readonly timeoutMilliseconds: number
58+
private readonly lockRetryAttempts: number
5359

5460
constructor(options: SQLiteDatabaseOptions) {
5561
this.timeoutMilliseconds = options.timeoutMilliseconds ?? 5_000
62+
this.lockRetryAttempts = options.lockRetryAttempts ?? 10
63+
if (!Number.isSafeInteger(this.lockRetryAttempts) || this.lockRetryAttempts < 1) {
64+
throw new TypeError("lockRetryAttempts must be a positive safe integer")
65+
}
5666
this.database = new DatabaseSync(options.path, {
5767
timeout: this.timeoutMilliseconds,
5868
readBigInts: true,
@@ -70,19 +80,7 @@ export class SQLiteDatabase implements Database {
7080
async transaction<Result>(
7181
callback: (connection: DatabaseConnection) => Promise<Result>,
7282
): Promise<Result> {
73-
return withDatabaseTransaction(this, () =>
74-
this.withAccess(async () => {
75-
this.database.exec("BEGIN IMMEDIATE")
76-
try {
77-
const result = await callback(this.databaseConnection)
78-
this.database.exec("COMMIT")
79-
return result
80-
} catch (error) {
81-
this.database.exec("ROLLBACK")
82-
throw error
83-
}
84-
}),
85-
)
83+
return withDatabaseTransaction(this, () => this.withAccess(() => this.runTransaction(callback)))
8684
}
8785

8886
transactionActive(): boolean {
@@ -135,6 +133,35 @@ export class SQLiteDatabase implements Database {
135133
this.database.exec(`PRAGMA busy_timeout = ${this.timeoutMilliseconds}`)
136134
}
137135
}
136+
137+
private async runTransaction<Result>(
138+
callback: (connection: DatabaseConnection) => Promise<Result>,
139+
): Promise<Result> {
140+
await this.beginTransactionWithRetry()
141+
try {
142+
const result = await callback(this.databaseConnection)
143+
this.database.exec("COMMIT")
144+
return result
145+
} catch (error) {
146+
this.database.exec("ROLLBACK")
147+
throw error
148+
}
149+
}
150+
151+
private async beginTransactionWithRetry(): Promise<void> {
152+
let retries = 0
153+
while (true) {
154+
try {
155+
this.database.exec("BEGIN IMMEDIATE")
156+
return
157+
} catch (error) {
158+
if (databaseDeadlineRemainingMilliseconds() !== undefined) throw error
159+
if (!sqliteBusyError(error) || retries >= this.lockRetryAttempts) throw error
160+
retries += 1
161+
await waitForRetry(retries)
162+
}
163+
}
164+
}
138165
}
139166

140167
function waitForAccess(access: Promise<void>, timeoutMilliseconds: number): Promise<void> {
@@ -156,6 +183,11 @@ function waitForAccess(access: Promise<void>, timeoutMilliseconds: number): Prom
156183
})
157184
}
158185

186+
function waitForRetry(attempt: number): Promise<void> {
187+
const milliseconds = Math.min(2 ** (attempt - 1), 250)
188+
return new Promise((resolve) => setTimeout(resolve, milliseconds))
189+
}
190+
159191
export function sqlite(options: SQLiteDatabaseOptions): SQLiteDatabase {
160192
return new SQLiteDatabase(options)
161193
}

‎test/sqlite.test.ts‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
import { afterEach, describe, expect, it } from "vitest"
2+
import { rm } from "node:fs/promises"
3+
import { DatabaseSync } from "node:sqlite"
4+
import { Worker } from "node:worker_threads"
5+
import { sqlite, type SQLiteDatabase } from "../src/database/sqlite.js"
6+
7+
let database: SQLiteDatabase | undefined
8+
let blocker: Worker | undefined
9+
let path: string | undefined
10+
11+
afterEach(async () => {
12+
await blocker?.terminate()
13+
await database?.close()
14+
if (path) await rm(path, { force: true })
15+
database = undefined
16+
blocker = undefined
17+
path = undefined
18+
})
19+
20+
describe("SQLite adapter", () => {
21+
it("retries a transaction after transient writer contention", async () => {
22+
path = `/tmp/solid-objects-sqlite-${crypto.randomUUID()}.sqlite3`
23+
const setup = new DatabaseSync(path)
24+
setup.exec("CREATE TABLE records(id INTEGER PRIMARY KEY)")
25+
setup.close()
26+
blocker = lockDatabase(path)
27+
await nextMessage(blocker)
28+
database = sqlite({ path, timeoutMilliseconds: 1, lockRetryAttempts: 10 })
29+
30+
await database.transaction((connection) =>
31+
connection.run("INSERT INTO records(id) VALUES (?)", [1]),
32+
)
33+
34+
await expect(
35+
database.connection((connection) =>
36+
connection.get<{ count: number | bigint }>("SELECT COUNT(*) AS count FROM records"),
37+
),
38+
).resolves.toEqual({ count: 1n })
39+
})
40+
41+
it("validates its transaction retry budget", () => {
42+
expect(() => sqlite({ path: ":memory:", lockRetryAttempts: 0 })).toThrow(
43+
"lockRetryAttempts must be a positive safe integer",
44+
)
45+
})
46+
47+
it("stops retrying after its transaction retry budget", async () => {
48+
path = `/tmp/solid-objects-sqlite-${crypto.randomUUID()}.sqlite3`
49+
const setup = new DatabaseSync(path)
50+
setup.exec("CREATE TABLE records(id INTEGER PRIMARY KEY)")
51+
setup.close()
52+
blocker = lockDatabase(path, { holdMilliseconds: 1_000 })
53+
await nextMessage(blocker)
54+
database = sqlite({ path, timeoutMilliseconds: 1, lockRetryAttempts: 2 })
55+
56+
const startedAt = performance.now()
57+
await expect(
58+
database.transaction((connection) =>
59+
connection.run("INSERT INTO records(id) VALUES (?)", [1]),
60+
),
61+
).rejects.toThrow("database is locked")
62+
63+
expect(performance.now() - startedAt).toBeLessThan(100)
64+
})
65+
})
66+
67+
function lockDatabase(databasePath: string, options: { holdMilliseconds?: number } = {}): Worker {
68+
return new Worker(
69+
`
70+
const { DatabaseSync } = require("node:sqlite")
71+
const { parentPort, workerData } = require("node:worker_threads")
72+
const database = new DatabaseSync(workerData.databasePath)
73+
database.exec("BEGIN IMMEDIATE")
74+
parentPort.postMessage("locked")
75+
setTimeout(() => {
76+
database.exec("ROLLBACK")
77+
database.close()
78+
}, workerData.holdMilliseconds)
79+
`,
80+
{
81+
eval: true,
82+
workerData: { databasePath, holdMilliseconds: options.holdMilliseconds ?? 25 },
83+
},
84+
)
85+
}
86+
87+
function nextMessage(worker: Worker): Promise<unknown> {
88+
return new Promise((resolve, reject) => {
89+
worker.once("message", resolve)
90+
worker.once("error", reject)
91+
})
92+
}

0 commit comments

Comments
 (0)