Skip to content

bb thread stop reports success but leaves a stray daemon turn, so every send is refused ("Refusing to start a competing turn") #3462

Description

@albrand

What happens

A thread gets stuck: every send fails with Refusing to start a competing turn for thread "…" while another turn is active or starting, while the app shows the thread as idle. bb thread stop prints Thread … stopped and exits 0, but nothing changes; the next send is refused again. bb thread retry / tell also report success and are refused afterwards.

Why

The server and the daemon disagree. The server sees idle, so a stop is sent as intent: "release". The daemon still holds an active turn (one that never reported turn/completed), and the release branch deliberately leaves an active turn alone:

  • apps/host-daemon/src/command-dispatch.ts:365: for release, if getActiveTurnId(...) !== null it returns { providerCheckpointId } without stopping anything, and without telling the caller it declined.
  • apps/server/src/services/threads/thread-lifecycle.ts:1400 (releaseIdleThreadRuntime) treats that as success, and apps/cli/src/commands/thread/actions.ts:532 prints "stopped".
  • packages/agent-runtime/src/runtime.ts:764 keeps refusing new turns while the stray turn id is set, and nothing expires it.

Skipping the release is right when a turn legitimately started after the server read the thread (there is a test for exactly that). But the caller cannot tell "released" from "declined", so an explicit user stop can never clear the stuck thread.

Fix (patch below, against main 8d32c61)

  1. The daemon reports the decline: thread.stop gains an optional activeTurnRetained: true when a release leaves an active turn in place (contract field is optional; .strict() schema updated).
  2. releaseIdleThreadRuntime only runs for explicit user actions (the stop route and context clear). When the release comes back activeTurnRetained, it follows up with intent: "interrupt" sent straight to the daemon. It does not go through markThreadStopRequested, because stop.requested is not a valid transition from idle, and idle is exactly the server's view in this state.

Automatic release paths are unchanged, so the race the existing guard protects is still protected.

Tests

  • New: public-thread-stop-runtime.test.ts › "interrupts when the daemon declines a release to keep an active turn". It fails without the server change and passes with it.
  • Updated: command-dispatch.test.ts › "skips a release when a turn started after the server read the thread" now expects activeTurnRetained: true. The stop is still skipped and the turn is still kept.
  • apps/server stop-runtime suite 9/9; apps/host-daemon command-dispatch.test.ts 40/40; typecheck green for server, host-daemon, host-daemon-contract.

Not covered: retry / tell still report delivery: "sent" before the daemon refuses. Once stop can clear the thread, that is recoverable, but they would ideally surface the refusal too.

Patch
diff --git a/apps/host-daemon/src/command-dispatch.test.ts b/apps/host-daemon/src/command-dispatch.test.ts
index bb3a9009..24675d4f 100644
--- a/apps/host-daemon/src/command-dispatch.test.ts
+++ b/apps/host-daemon/src/command-dispatch.test.ts
@@ -1034,7 +1034,10 @@ describe("dispatchCommand", () => {
 
     expect(runtime.stopThread).not.toHaveBeenCalled();
     expect(runtime.getActiveTurnId("thread-1")).toBe("turn-new");
-    expect(result).toEqual({ providerCheckpointId: null });
+    expect(result).toEqual({
+      providerCheckpointId: null,
+      activeTurnRetained: true,
+    });
   });
 
   it("treats thread.stop as successful when no runtime holds the thread", async () => {
diff --git a/apps/host-daemon/src/command-dispatch.ts b/apps/host-daemon/src/command-dispatch.ts
index 44df15ec..b06b1dc9 100644
--- a/apps/host-daemon/src/command-dispatch.ts
+++ b/apps/host-daemon/src/command-dispatch.ts
@@ -365,7 +365,7 @@ const commandHandlers: CommandHandlerMap = {
       if (command.intent === "release") {
         if (entry.runtime.getActiveTurnId(command.threadId) !== null) {
           await options.eventSink.flush();
-          return { providerCheckpointId };
+          return { providerCheckpointId, activeTurnRetained: true };
         }
       } else {
         await entry.runtime.waitForActiveTurn(command.threadId, {
diff --git a/apps/server/src/services/threads/thread-lifecycle.ts b/apps/server/src/services/threads/thread-lifecycle.ts
index 2a8694eb..01780bb1 100644
--- a/apps/server/src/services/threads/thread-lifecycle.ts
+++ b/apps/server/src/services/threads/thread-lifecycle.ts
@@ -1405,7 +1405,7 @@ async function releaseIdleThreadRuntime(
   if (environment === null) {
     return;
   }
-  await runAwaitedThreadStopCommand(deps, {
+  const released = await runAwaitedThreadStopCommand(deps, {
     command: buildThreadStopCommand({
       environmentId: environment.id,
       hostId: environment.hostId,
@@ -1415,6 +1415,30 @@ async function releaseIdleThreadRuntime(
     hostId: environment.hostId,
     threadId,
   });
+  if (released?.activeTurnRetained !== true) {
+    return;
+  }
+  // The server saw an idle thread, but the daemon still holds an active turn:
+  // a turn that never reported completion. Every later send is refused with
+  // "Refusing to start a competing turn", so a release that leaves it in place
+  // strands the thread. The only callers are explicit user stops, so interrupt.
+  deps.logger.warn(
+    { threadId },
+    "Release declined by an active daemon turn; interrupting",
+  );
+  // Not routed through markThreadStopRequested: `stop.requested` is not a valid
+  // transition from idle, and idle is exactly the server's (wrong) view here.
+  // The interrupt goes straight to the daemon, which owns the stray turn.
+  await runAwaitedThreadStopCommand(deps, {
+    command: buildThreadStopCommand({
+      environmentId: environment.id,
+      hostId: environment.hostId,
+      intent: "interrupt",
+      threadId,
+    }),
+    hostId: environment.hostId,
+    threadId,
+  });
 }
 
 async function runAwaitedThreadStopCommand(
@@ -1424,11 +1448,12 @@ async function runAwaitedThreadStopCommand(
     hostId: string;
     threadId: string;
   },
-): Promise<void> {
+): Promise<{ activeTurnRetained?: boolean } | null> {
+  let outcome: { activeTurnRetained?: boolean } | null = null;
   await threadStopRequestDeduper.run(args.threadId, async () => {
     inFlightThreadRpcGuard.claim(args.threadId, "thread.stop");
     try {
-      await runLiveHostCommand(deps, {
+      outcome = await runLiveHostCommand(deps, {
         command: args.command,
         hostId: args.hostId,
         timeoutMs: AWAITED_THREAD_STOP_TIMEOUT_MS,
@@ -1448,6 +1473,9 @@ async function runAwaitedThreadStopCommand(
       inFlightThreadRpcGuard.release(args.threadId, "thread.stop");
     }
   });
+  // Assigned inside the deduped callback, which control-flow narrowing cannot
+  // see, so restate the declared type rather than let it narrow to `null`.
+  return outcome as { activeTurnRetained?: boolean } | null;
 }
 
 export function requestActiveRuntimeThreadStopIfNeeded(
diff --git a/apps/server/test/public/public-thread-stop-runtime.test.ts b/apps/server/test/public/public-thread-stop-runtime.test.ts
index f6746348..2615d745 100644
--- a/apps/server/test/public/public-thread-stop-runtime.test.ts
+++ b/apps/server/test/public/public-thread-stop-runtime.test.ts
@@ -61,6 +61,46 @@ describe("thread runtime stop", () => {
     });
   });
 
+  it("interrupts when the daemon declines a release to keep an active turn", async () => {
+    await withTestHarness(async (harness) => {
+      // The server believes the thread is idle, but the daemon still holds a
+      // turn that never reported completion. Left in place, every later send
+      // is refused with "Refusing to start a competing turn".
+      const { thread } = seedThreadFixture(harness, {
+        thread: { status: "idle", visibility: "hidden" },
+      });
+      const responsePromise = harness.app.request(
+        `/api/v1/threads/${thread.id}/stop`,
+        { method: "POST" },
+      );
+      const release = await waitForQueuedCommand(
+        harness,
+        ({ command }) =>
+          command.type === "thread.stop" &&
+          command.threadId === thread.id &&
+          command.intent === "release",
+      );
+      await reportQueuedCommandSuccess(harness, release, {
+        providerCheckpointId: null,
+        activeTurnRetained: true,
+      });
+
+      const interrupt = await waitForQueuedCommand(
+        harness,
+        ({ command }) =>
+          command.type === "thread.stop" &&
+          command.threadId === thread.id &&
+          command.intent === "interrupt",
+      );
+      await reportQueuedCommandSuccess(harness, interrupt, {
+        providerCheckpointId: null,
+      });
+
+      const response = await responsePromise;
+      expect(response.status).toBe(200);
+    });
+  });
+
   it("settles background commands terminated by an idle runtime release", async () => {
     await withTestHarness(async (harness) => {
       const { environment, thread } = seedThreadFixture(harness, {
diff --git a/packages/host-daemon-contract/src/commands.ts b/packages/host-daemon-contract/src/commands.ts
index ce0474b6..b93b8677 100644
--- a/packages/host-daemon-contract/src/commands.ts
+++ b/packages/host-daemon-contract/src/commands.ts
@@ -1205,6 +1205,10 @@ const turnSubmitResultSchema = z.object({
 const threadStopResultSchema = z
   .object({
     providerCheckpointId: z.string().min(1).nullable(),
+    // A `release` stop leaves an active turn alone. Reporting that lets the
+    // caller tell "released" from "declined" instead of reporting success for
+    // a thread that is still wedged on a turn the server believes is over.
+    activeTurnRetained: z.boolean().optional(),
   })
   .strict();
 const emptyCommandResultSchema = z.object({});

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    confirmed-reproBug reproduced again from a clean trusted checkout; see linked reporthostHost daemon, process lifecycle, memory, event loopthreadsTurns, timeline, messaging, forks

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions