Repository navigation
Deferred foreign-key reset can reject a concurrent Durable Object request after its writes committed #7657
Description
Summary
A deferred foreign-key violation in one SQLite-backed Durable Object request can reset/break the actor in a way that rejects a different concurrent request after the second request's complete write set has persisted.
This was originally reported against Miniflare/D1 in cloudflare/workers-sdk#16071. I re-ran that reproducer against the current published runtime and the rejected+committed outcome is still present.
Current reproduction
Runtime used:
- Node.js
v24.18.0 miniflare@5.20261006.0-alphaworkerd@1.20261006.1
I adapted the public reproducer from workers-sdk#16071 only for Miniflare's current workers: [{ config: ... }] constructor shape; the database/schema/concurrency logic was unchanged.
One 32-pair run produced:
{
"pairs": 32,
"categories": {
"rejected+absent": 5,
"rejected+committed": 5,
"fulfilled+committed": 22
},
"criticalReproduction": "OBSERVED"
}The rejected+committed cases included both coordinator-driven calls and two independent fetches.
For example, in one independent-fetch case the valid and invalid D1 operations both rejected with the same runtime error reference:
Error: internal error; reference = p9lt6srvi719itm7h9vhbtsi
but a subsequent state read showed the valid operation's entire expected revision/event/receipt write set had persisted exactly.
Why this appears to cross into workerd semantics
The current Miniflare D1 implementation wraps a batch in state.storage.transaction(...).
In workerd, deferred FK commit violations are intentionally treated as actor-reset failures. The existing sql-test.js even describes this as:
Test defer_foreign_keys (explodes the DO)
The relevant path appears to be:
ActorSqlite::TxnCommitRegulator::onError()translatesSQLITE_CONSTRAINTat commit into the "Durable Object was reset..." application error.- SQLite transaction/commit failures can break the actor's output gate.
OutputGatethen propagates that failure to pending outputs.- A separate valid request can therefore receive the reset/internal error even though its write set is present after the actor recovers.
The important correctness property is not whether the invalid transaction resets the actor; that may be intentional. The problem is that a request reported as failed can have fully committed state, which makes retry/idempotency behavior ambiguous.
Expected invariant
For a request rejected because a neighboring transaction resets the actor, one of these should hold:
- its writes are absent/rolled back; or
- if its writes were already durably committed, its result should not be retroactively reported as failed because of a different transaction's constraint violation.
A successful request should not end up in a rejected+committed state.
Scope / question
I don't want to paper over this in Miniflare by serializing all D1 requests or running PRAGMA foreign_key_check after every batch; either would hide the underlying ordering problem and has undesirable performance/semantic tradeoffs.
Would the preferred fix be in workerd's handling of deferred-constraint commit failures / output-gate propagation, or is the actor-wide reset behavior intentional enough that Miniflare's D1 shim should isolate D1 transactions from it?
I'm happy to work on the fix and regression coverage once the intended boundary is confirmed.
Related: cloudflare/workers-sdk#16071
Activity
yaohuangguan commentedon Oct 7, 2026
Code-path follow-up after checking current
main(2df1f851). This narrows the possible cross-request propagation path, but does not yet establish the exact interleaving that caused therejected+committedobservations.ActorSqlite::TxnCommitRegulator::onError(), whereSQLITE_CONSTRAINTis translated to the intentional reset/rollback error. The existing SQL test explicitly expects deferred-FK failure to break the DO.OutputGate::lockWhile()then records the failure for waiters.OutputGate.wait(): each actorIoContextregistersabortWhen(actor.getOutputGate().onBroken()), andabortWheninvokesIoContext::abort(). That gives a concrete actor-wide cancellation path potentially affecting a different request whose SQLite transaction already committed.WorkerEntrypoint::request()explicitly gives an output-gate failure precedence over the handler failure. Existing OutputGate unit tests also distinguish earlier vs. later gate waiters, so I would not change global gate ordering based only on the D1 symptom.Suggested focused regression: Run two independently initiated requests against one SQLite-backed actor: A completes its valid SQL transaction, while B triggers a deferred-FK violation and breaks the actor; capture A's commit-completion point, gate-wait/cancellation timing, caller result, and fresh-stub persisted state. The important assertion is the relationship between acknowledged/failed outcome and observed persisted writes, not simply whether B correctly resets the actor.
Maintainer question: Is an error returned to A after its data commits an intentional ambiguous commit outcome of actor-wide reset (which D1 consumers must handle with idempotency/read-back), or should a request whose commit has been confirmed be protected from a later unrelated request's output-gate abort? The first may call for contract/docs changes; the second likely needs a runtime-level fix and a regression test. I would avoid Miniflare-side global serialization or suppression of the reset until that boundary is decided.
yaohuangguan commentedon Oct 7, 2026
Native-runtime follow-up: I reduced the Miniflare/D1 reproducer to a standalone
workerd testfixture, so this does not depend on the D1 shim, Wrangler, or account credentials.Reproducer (two files + run instructions): https://gist.github.com/yaohuangguan/fbb58a56e22058e754cf818ec8d17c69
On
workerd 2026-10-06:A valid request whose entire write set committed must not be reported as rejected(expected RED on the unfixed binary). The fixture uses a fresh stub after the invalid actor resets.One possible fix boundary is
DurableObjectStorage::asyncTransactionImpl()(src/workerd/api/actor-state.c++): the callback-exception path packs failure asAsyncTxnResultwithout breakingblockConcurrencyWhile(), but synchronous exceptions fromtxn->maybeCommit()appear to escape that path. That may break the input gate; the actor-wide output gate then rejects other responses whose commits already completed.I'm prototyping only recoverable synchronous commit errors as ordinary transaction rejections, while preserving the existing reset/failure path for ambiguous asynchronous persistence errors and critical SQLite I/O failures. The native C++ rebuild/regression is not yet complete, so I am not claiming the patch fixes it.
Would you agree that a deferred constraint failure at a savepoint release can be recovered by rolling back that transaction (instead of resetting the actor), provided other commit/IO failures remain fatal? If not, which gate/commit boundary should the regression target?