Skip to content

Resolve the storage driver in a scope of its own - #2031

Merged
ppXD merged 1 commit into
mainfrom
fix/resolve-the-storage-driver-in-its-own-scope
Sep 26, 2026
Merged

ppXD merged 1 commit into
mainfrom
fix/resolve-the-storage-driver-in-its-own-scope

Conversation

@ppXD

@ppXD ppXD commented Sep 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • A healthy agent run failed on a shared DbContext. On codespace-test, a 22-minute PR review agent run landed Failed with "A second operation was started on this context instance before a previous operation completed". Its log streams closed observer-failed-before-terminal and the attempt closed Lost.
  • The two paths that collided. The executor, the log capture bridge and everything the bridge reaches are resolved from one lifetime scope, the Hangfire job's, and the capture loop (AgentRunLogCaptureBridge.CaptureLoopAsync) runs beside the durable runner's drain tick.
    • ArtifactCasRuntimeCoordinator already runs its own statements on contexts of its own. The storage driver broker it was constructed with, however, carried the scope's CodeSpaceDbContext in its profile and credential readers (StorageProfileSnapshotResolver, StorageCredentialSecretResolver).
    • So when a segment append (every 256 KiB of output) resolved its driver while the tick flushed events or wrote the spool offset, two statements ran on one context.
    • When the tick started second, the error escaped the observer and failed the run. When the capture started second, it was swallowed as a transient storage stall, so only one symptom was ever visible.
  • The fix. ArtifactCasRuntimeCoordinator now resolves the broker in a child lifetime scope per open (ArtifactCasRuntimeCoordinator.cs, OpenDriverInOwnScopeAsync). It follows the same child-of-the-owning-scope pattern NodeInvocationExecutor and WorkflowEngine use.
    • Every coordinator entry point resolves the profile revision on its own context before opening a driver, so the broker never needed the caller's transaction.
    • The consumers that do need it (default materialization, route probing) call the broker directly and are unchanged.
    • This covers every capture session — live launch, re-attach re-tail, shutdown fold — and any other coordinator caller running beside its scope.

Test plan

  • New AgentRunExecutorSharedScopeTests (integration, high fidelity). The executor is resolved the way the job scope resolves it, so bridge, log service, coordinator, broker and resolvers are the production graph. The runner is the real LocalProcessRunner and the database is real Postgres. Every earlier capture test resolved the bridge from a different scope, which is why none caught this.
    • How the overlap is forced: an ACCESS EXCLUSIVE lock on storage_credential, held on a separate connection, keeps the broker's credential read in flight while the agent keeps printing. On the append path nothing else reads that table.
    • A pg_stat_activity witness asserts the read really was waiting (fixture check).
    • Live launch path: the run lands its own verdict. Red on main (the run ends Failed with the EF message).
    • Re-attach re-tail: the re-attached run lands its own verdict. Red on main.
  • Full unit suite: 11199 passed, 0 failed.
  • Integration (Artifact, AgentRunLog, Storage, Materialize, Retention, Routed, every AgentRun* suite, RealHarnessExecution): 1210/1210.
  • Full solution build clean.

A healthy agent run - 22 minutes of review work on codespace-test -
landed Failed with "A second operation was started on this context
instance before a previous operation completed".

The executor, the log capture bridge and everything the bridge reaches
are resolved from one lifetime scope, the Hangfire job's, and the
capture loop runs beside the durable runner's drain tick. The CAS
coordinator runs its own statements on contexts of its own, but the
storage driver broker it was handed carried the scope's
CodeSpaceDbContext in its profile and credential readers. So a segment
append resolving its driver while the tick flushed events or wrote the
spool offset put two statements on one context. EF refused whichever
started second: on the capture side that read as a transient storage
stall, on the tick's side it escaped the observer and failed the run
(observer-failed-before-terminal, the attempt closed Lost).

The coordinator now resolves the broker in a child scope per open, so
nothing it reaches shares its caller's context. Every coordinator
entry point resolves the profile revision on its own context first, so
the broker never needed the caller's transaction; the consumers that
do (default materialization, route probing) call the broker directly
and are unchanged.

Every earlier capture test resolved the bridge from a different scope
than the executor, which is why none saw this. The new tests resolve
it the way the job scope does and hold the broker's read in flight.
@ppXD
ppXD merged commit 056d202 into main Sep 26, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant