bug(turns): Gemini/Antigravity batch output is lost on turn timeout #411

Problem Summary

When users run complex workbench turns with deep-reasoning Gemini/Antigravity models and large workspace context, the turn can exceed the default 120-second engine budget and end with:

{
  "code": "turn_timeout",
  "message": "the turn exceeded its time budget and was terminated; partial output was preserved",
  "retryable": true
}

In the observed failures, the desktop UI receives no model.delta events and displays no partial response. The message therefore claims that partial output was preserved even when the turn record contains no output.

Root Cause

The failure is caused by the interaction of batch CLI output, adapter-side buffering, and an absolute turn timeout.

1. Gemini/Antigravity runs in batch JSON mode

In apps/server/bin/qoder-engine.mjs:

args = ["--prompt", prompt, "--output-format", "json", "--mode=plan", "--disable-slash-commands"];

The Gemini branch likewise uses --output-format json.

The adapter accumulates the child process's stdout and does not emit the response as model.delta until the process closes successfully:

let stdout = "";
child.stdout.on("data", (chunk) => {
  stdout += String(chunk);
});

child.on("close", async (code) => {
  // ...
  const result = JSON.parse(stdout);
  emit({ type: "model.delta", runId, timestamp: now(), text: result.response });
});

Consequently, the control plane cannot preserve visible response text produced before a timeout unless the CLI and adapter expose it incrementally.

2. The turn driver enforces an absolute timeout

DigitalEmployeeCliDriver defaults to 120_000 ms (configurable through ORG_WORKBENCH_TURN_TIMEOUT_MS). Its timer is wall-clock based and is not reset by engine activity:

const timer = setTimeout(() => {
  if (settled) return;
  timedOut = true;
  acceptingOutput = false;
  stdoutBuffer = "";
  terminateChild();
}, this.timeoutMs);

If the batch CLI has not completed by then, the process group is terminated and the result is classified as status: "indeterminate", code: "turn_timeout".

3. The partial-output message is unconditional

apps/server/src/routes/turns.ts always maps turn_timeout to:

the turn exceeded its time budget and was terminated; partial output was preserved

However, extractPartialOutput() returns undefined when the event list contains no model.delta. The message does not reflect whether any partial output actually exists.

Observed Evidence

From local turn records under .digital-employee/workbench/sessions/conversations/:

1. Simple prompt ("hi") with a large injected context

   • Input context: 96,564 tokens.
   • Recorded output-token usage: 2,045 tokens, including model reasoning usage.
   • Duration: 41.4 seconds.
   • Visible result: one short greeting.

2. Cross-repository planning request

   • Involved six PRs/issues across digital-employee and roleweave, plus CI and design constraints.
   • run.started: 08:57:46.545Z.
   • Terminated: 08:59:41.703Z (~115.2 seconds after run.started).
   • Recorded events: one run.started, zero model.delta events.

The ~115-second event interval is compatible with a 120-second outer timeout because the timeout starts before the adapter emits run.started; process startup and context preparation consume part of the budget.

These records prove that no visible partial output reached RoleWeave. They do not establish how much final-answer text, as distinct from hidden reasoning tokens, had already been generated inside the provider.

Proposed Remediation

1. Stream observable CLI events

• Use --output-format stream-json where supported.
• Parse JSONL incrementally in qoder-engine.mjs.
• Map assistant message chunks to model.delta as they arrive.
• Map terminal result/error and usage events to the existing engine.v1 protocol.
• Verify Gemini CLI and Antigravity independently; do not assume that both clients expose identical flags or event schemas.
• Keep a safe fallback or return an attributable compatibility error when the installed client does not support the required streaming protocol.

2. Use bounded activity-aware timeouts

• Introduce an idle timeout that is reset only by valid, observable engine progress.
• Keep a longer maximum wall-clock timeout so a stuck or trickle-output process cannot run indefinitely.
• Do not treat adapter-generated heartbeats alone as evidence of model progress.
• Keep ORG_WORKBENCH_TURN_TIMEOUT_MS (or equivalent separate idle/max settings) configurable.

Streaming improves preservation of visible output, but it does not guarantee frequent events while a provider is producing hidden reasoning tokens. Timeout defaults therefore need to accommodate long first-token latency as well as ongoing streamed output.

3. Report the actual preservation outcome

• If extractPartialOutput() returns text, report that partial output was preserved.
• If no delta exists, report that the turn ended before any visible output could be committed.
• Apply the same conditional wording to cancellation where appropriate.

Acceptance Criteria

• Supported Gemini/Antigravity clients are consumed incrementally instead of waiting for a complete batch JSON response.
• Client capability and event-schema handling are covered separately for Gemini CLI and Antigravity.
• Assistant text received before timeout or cancellation is emitted as model.delta, displayed in the UI, and persisted in the turn record.
• Timeout handling distinguishes observable inactivity from the maximum allowed wall-clock duration.
• Timeout defaults allow realistic long-context reasoning and first-token latency without permitting unbounded execution.
• Timeout and cancellation messages state that partial output was preserved only when partial output actually exists.
• Tests cover: streamed deltas followed by timeout, timeout before the first delta, cancellation after partial output, unsupported streaming capability, and maximum-duration termination.