🛬 fix: Settle Background Tool Results Before the Server Shuts Down

[danny-avila]

Summary

Restarting the API loses every background tool task that is still running or still persisting its result. Graceful shutdown drains generations, but nothing drains background tasks: their abort signal is not tied to anything shutdown aborts, and they die with the process. Their automatic completion deliveries then dead-letter as BACKGROUND_TOOL_PRODUCER_LOST once the producer lease expires, which cannot be retried, so the agent never learns what happened. On the demo, one routine config restart lost five in-flight results this way.

There is a second gap: once shutdown begins, even a task that finishes on its own cannot record its result. renewProducerLease and persistBackgroundToolResult go through requireMethods(), which throws as soon as shutdown starts, so a completed result is reported as an ambiguous write failure and also dead-letters later.

This PR drains background tasks within the shutdown budget:

• New background dispatches are refused once shutdown starts, with a shutting_down rejection that tells the model to run the call in the foreground or retry after the restart. Replays of an existing task and launches that already hold a capacity permit still resolve.
• Producer lease renewal and receipt writes stay available during shutdown, like the existing cleanup writes (retire, purgeUser).
• After the generation job manager finalizes interrupted turns, each task whose result is not yet durable gets the time left to settle on its own. Tasks still running are then aborted with a shutdown reason and get a short grace to settle. Any task still without a durable result has one written for it: its own result if it finished, otherwise an interrupted failure. The agent then wakes with "Background task was interrupted because its server shut down before it finished. Run it again if you still need the result." instead of a dead letter.
• endpoints.agents.backgroundTasks.shutdownInterruptGraceMs (default 5000, 0 to 60000) sets that grace.

Clustered workers measure the drain against the primary's force-exit deadline, the same budget the generation job manager already uses. The deadline and elapsed-time fallback are calculated by the typed shutdown service; the worker entrypoint supplies only its primary deadline and force-exit interval.

How it works

SIGTERM
  pre-drain  (110)  background task admission     # closeAdmission(): new dispatches get shutting_down
  pre-drain  (100)  agent trigger delivery engine # stops the engine; producer writes stay available
  HTTP server drains; running tasks keep working
  post-drain (100)  generation job manager        # interrupted turns become terminal
  post-drain  (95)  background tasks              # drainForShutdown(deadline = budget - 2s)
                      wait for tasks to settle     until deadline - flush reserve - grace
                      interrupt still-running      abort with the shutdown reason
                      wait                         until deadline - flush reserve (2s)
                      flush the rest               write the best durable receipt
  post-drain  (90)  subagent task store, caches, telemetry

Each task registers a shutdown handle with the registry when it is dispatched. The durable receipt write is now a single flight per task, so a shutdown flush and the task's own late settlement share the first write: neither can store a contradicting receipt, and retireFailedPersistence never retires a delivery that already holds one. A returned false is not durable: the task stays tracked until the parent-message projection finishes, and a shutdown reports the result unconfirmed if neither write succeeds.

backgroundTaskRegistry.trackShutdown(task, {
  settled,                  // resolves when the receipt is durable (or the task ends without one)
  interrupt: (reason) => …, // backgroundAbortSource = 'shutdown'; abort the invocation
  flush: async (reason) => …, // receipt from the settled result, else an interrupted failure
});

Code tasks write their receipt only after the file harvest, which can wait on the dispatch turn. The flush covers that case by writing the already settled result directly.

Type of change

• Bug fix

Testing

Automated tests:

• background.shutdown.spec.ts (new):
  • admission closes for new tasks while replays and pre-reserved permits still resolve
  • the drain waits for tasks that settle on their own
  • it interrupts running tasks and flushes only those that did not settle
  • it flushes a settled task whose result is not durable yet, without aborting it
  • it stops at the deadline and reports unconfirmed results
  • registration order, phases and budget, the configured grace, and a missing budget; admission runs before delivery-engine shutdown at priority 100
• handlers.shutdown.spec.ts (new), through the real tool execute handler:
  • a tool that honors the abort delivers the shutdown reason as its receipt
  • a tool that ignores the abort gets an interrupted receipt, and its late result neither rewrites nor retires it
  • a finished code task whose harvest is still waiting gets its own result stored
  • a dispatch after admission closes is refused without invoking the tool
  • a false receipt waits for the projection fallback, and failed receipt plus failed projection is reported unconfirmed
• service.delivery.spec.ts: lease renewal and receipt writes still work after the delivery engine stops, while enqueue stays refused.
• config.spec.ts: the new default, a configured value, and out-of-range values.
• index.spec.js, experimental.spec.js, app/shutdown.spec.ts: the drain is registered at startup, and the typed cluster budget uses the primary deadline, an elapsed-time fallback, and the tighter worker limit.
• Existing suites unchanged: handlers.background.spec.ts, background.spec.ts, backgroundCompletionWakeup.spec.ts, service.delivery.spec.ts (290 passing together).
• npx tsc --noEmit in packages/api; node scripts/static-checks.mts --full --against origin/dev (TypeScript and config migration tests included) passes.

Not yet run: a live restart of a server with a running background task. That is the next check.

Screenshots / recordings

No user-facing change beyond the delivered error text for an interrupted task.

Risk / compatibility

• The drain runs after the generation job manager, whose settlement budget keeps a 10 s reserve (3 s in cluster mode). In practice background tasks get that reserve minus 2 s for the tasks that follow, plus all the time they already ran during the HTTP drain. In cluster mode that is short enough that tasks still running are interrupted almost immediately and receive an interrupted receipt.
• A task without a durable delivery (poll-only, ephemeral) has nothing to persist. It is aborted like the rest and its result is lost with the process, as before.
• Shutdown still cannot recover a remote workspace command that keeps running on the worker after LibreChat exits. Reattaching to it needs asynchronous execution on the Code API (Command execution time is bounded by the HTTP request lifetime code-interpreter#255).
• The new config field is optional with a default, so existing librechat.yaml files are unaffected.

Checklist

• I reviewed my own changes
• Relevant tests have been added or updated
• Existing relevant tests pass
• The change does not introduce new warnings or errors
• User-facing or complex behavior is documented where necessary
• Required dependency changes have been merged/published
• Required documentation PR: N/A

[danny-avila]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-25T18:26:35.662153Z	d5d330f	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d5d330f240

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[lia-by-librechat]

Review handoff for head 72d1e7734fc2ca7141e052c1074294317ede0224: failed receipt writes now remain tracked until the projection fallback finishes, admission closes ahead of delivery shutdown, and cluster deadline policy lives in the TypeScript shutdown service. Focused tests, typecheck, build, and CI are running for this head.

[lia-by-librechat]

Review handoff for exact remote head 94ea0211be5d4874915f0d33b2644a54f9862efb: all three Codex findings are addressed, with regression coverage for failed receipt writes, admission order, and typed cluster budget calculation. The additional commit makes the cluster wiring assertion formatting-safe. Focused API tests (132 passing), clustered/standard registration tests, and static checks passed; CI is rerunning on this head.