Skip to content

fix: serialize SQLite asset writes without multiplying busy waits - #362

Merged
Ed1s0nZ merged 2 commits into
AIPentest:mainfrom
Sycun:codex/fix-328-sqlite-asset-lock
Oct 16, 2026
Merged

Ed1s0nZ merged 2 commits into
AIPentest:mainfrom
Sycun:codex/fix-328-sqlite-asset-lock

Conversation

@Sycun

@Sycun Sycun commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Parallel create_asset calls can fail immediately with database is locked because deferred read-modify-write transactions cannot upgrade their read snapshots to a writer. Start SQLite transactions with _txlock=immediate so competing writers wait before taking a snapshot, using the shared DSN for both application and knowledge databases.

This builds on @OrdinarySF’s #329 and preserves the original authored commit. It retains the proposed 8/4 connection pool and 30-second busy timeout, but removes the outer eight-attempt retry: a persistent lock could otherwise cause roughly four minutes of waiting. SQLite handles the lock wait directly.

Regression coverage now starts 32 workers together and verifies distinct inserts, same-target deduplication and returned IDs, mixed creates/updates, and persisted counts. Both database constructors are also checked for writer reservation at Begin, concurrent WAL reads, and recovery after a competing writer releases its lock.

Validation:

  • Reproduced immediate lock errors on upstream main before applying the fix.
  • go test -race ./internal/database -count=1
  • go test ./internal/database -run 'TestUpsertAssetsConcurrent|TestAssetConcurrentCreateAndUpdate|TestSQLiteBeginReservesWriter' -count=20
  • go test ./internal/app ./internal/handler -count=1
  • git diff --check

Fixes #328. Alternative to #329 with the retry latency tightened and broader regression coverage.

OrdinarySF and others added 2 commits October 16, 2026 12:20
Deep mode fires many create_asset calls at once. UpsertAssets used a
DEFERRED transaction (SELECT then INSERT) on a 25-connection pool, so
writers hit SQLITE_BUSY immediately and returned "database is locked"
in a few milliseconds instead of waiting.

Open SQLite with _txlock=immediate and a 30s busy timeout, shrink the
pool, and retry locked upserts. Add a 32-way concurrent upsert test.
@Ed1s0nZ
Ed1s0nZ merged commit f8cbb5e into AIPentest:main Oct 16, 2026
3 checks passed
@Sycun
Sycun deleted the codex/fix-328-sqlite-asset-lock branch October 16, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deep 并行 create_asset 报 database is locked,5ms 即失败

3 participants