[analisaperlengkapan]

Issue Number

Closes #673

Why

When many automations fire at once (several 5-minute cron schedules sharing a
timestamp, or a burst of webhook events), the runs that hit a limit are lost
rather than queued. _execute_run marked an organization concurrency-limit 429
(ConcurrencyLimitReachedError) or a daily-sandbox-quota 429
(DailySandboxLimitReachedError) as terminal SKIPPED via mark_run_terminal.
SKIPPED is terminal, so the run is never retried: an event-triggered run
consumes its event with nothing to show, and a cron run misses its fire for a
whole interval.

This implements the bounded queue the issue asks for: a run that cannot start
now is deferred and started later instead of being dropped.

Summary

• Defer instead of drop. A run blocked by a transient limit returns to
  PENDING with deferred_until set from a bounded exponential backoff
  (AUTOMATION_DEFER_BACKOFF_BASE_SECONDS, capped at the max age). The dispatch
  poll skips a PENDING run whose deferred_until is still in the future.
• Bounded deferral. Past AUTOMATION_DEFER_MAX_AGE_SECONDS (default 6h,
  measured from the run's creation) the run is marked SKIPPED with a
  defer_expired status_detail, distinct from concurrency_limit /
  daily_sandbox_limit, so the outcome stays truthful. Setting the age to 0
  disables deferral and restores today's immediate-skip behavior.
• Admission gate. AUTOMATION_MAX_CONCURRENT_RUNS caps runs in flight: the
  dispatcher admits only up to the cap per poll and leaves the rest PENDING
  (the table is the queue) until a slot frees. Unset (0) preserves today's
  behavior. Ordering stays FIFO by created_at, unchanged from the current poll.
• Schema. automation_runs gains deferred_until and deferral_count
  (migration 038, one head, PostgreSQL and SQLite aligned). The requeue write
  is guarded by status == RUNNING, so a run that already reached a terminal
  state (callback, cancel, watchdog) is never resurrected and cannot
  double-execute. deferral_count lives on the row because status_detail is
  cleared when the run goes RUNNING.
• Emits an automation_run_deferred telemetry event with the retry timing, and
  an automation_run_skipped event with skip_reason: defer_expired on expiry.
• AutomationRunResponse returns deferred_until so a client can show a queued
  run's retry time.

Hardening after review

A follow-up commit closes five gaps in the first pass, each with a test that
fails without the fix:

• Admission is serialized across replicas. Counting RUNNING and then
  claiming rows under SKIP LOCKED let two replicas read the same count under
  read-committed isolation and claim disjoint rows, so combined admissions could
  exceed the cap. A transaction-scoped advisory lock
  (pg_advisory_xact_lock) now spans the count and the commit; SQLite, which
  runs single-process, skips it (the same pattern as conversations.py).
• Aged deferred runs are reaped even at capacity. Expiry used to run only on
  a dispatch attempt, but a poll that finds every slot occupied returns before
  claiming, so a deferred run could sit PENDING forever. An independent sweep
  now expires aged deferred runs on every poll, regardless of free capacity.
• A failed requeue write fails the run instead of stranding it.
  requeue_deferred_run used to return False on a write error, which the
  caller could not tell apart from a lost terminal race, leaving the run
  RUNNING with no sandbox until the watchdog. It now raises
  DeferredRunRequeueError; False means only "the guarded update matched no
  row".
• Backoff cannot overshoot the deadline. The delay is bounded by the time
  left before the max age (max_age - age), not by the age limit itself, so a
  run near the deadline is not rescheduled to start after it.
• Disabling deferral keeps the original reason. With the max age at 0 the
  run is skipped with its original concurrency_limit / daily_sandbox_limit
  status kind and skip reason, so existing consumers are unchanged.

A second follow-up closes two gaps in the expiry sweep itself, also each with a
test that fails without the fix:

• Expiring a subject run releases its subject atomically. A deferred run has
  no started_at (the requeue clears it), so a terminal run that kept its
  subject still answered continue_conversation and swallowed the next event
  for that subject into a run that would never dispatch. The sweep now stamps
  subject_released_at in the same guarded UPDATE that sets SKIPPED, so the
  terminal status and the release cannot be observed apart.
• The sweep emits the skip event. It runs on every poll, including one that
  finds every slot occupied and returns before the dispatcher's own skip
  handler, so those expirations were missing from telemetry. It now captures
  automation_run_skipped with skip_reason: defer_expired for each row the
  guarded update actually changed, attributed to the run and its automation
  (the SELECT returns the rows with the automation eager-loaded, and the
  UPDATE ... RETURNING id bounds the events to rows really updated).

How to Test

uv run python -m pytest tests/ — full suite passes (2444 passed, 7 skipped).
New unit tests cover the capacity gate (admit up to capacity, count existing
RUNNING, empty at capacity, deferred runs excluded then eligible after
deferred_until), defer/requeue (returns to PENDING, clears
started_at/timeout_at, is idempotent, never resurrects a terminal run),
backoff progression and its deadline bound, deadline expiry (including the
capacity-independent sweep, subject release, and the skip event), the
cross-replica admission lock, the failed-requeue path, and count_running_runs.
An end-to-end routing test in test_conversations.py reproduces the swallow:
an expired deferred run releases its subject, so the next thread event starts a
fresh run instead of folding into the dead one.

uv run ruff check, uv run ruff format --check and uv run pyright on the
changed files are clean. Migration 038 verified as the single Alembic head.

Notes

• The deferral is only wired into the two transient-limit branches named in the
  issue. A permanent failure still marks the run FAILED unchanged.
• Non-goals, matching the issue's scope: per-automation concurrency caps /
  overlap guards (Add a minimal per-automation overlap guard (skip when a run is already in-flight) #218), hierarchical admission limits (Implement hierarchical concurrency limits for automation runs #72), event-inflow
  backpressure (Implement event queue management with backpressure strategies #74), and Canvas UI work (#18576) — this ships the service-side
  signal only.
• The defer_expired distinction matters for operators: concurrency_limit /
  daily_sandbox_limit now mean "waiting to retry", while defer_expired means
  "gave up after the deadline".

This pull request was created by an AI agent (OpenHands) on behalf of the repository maintainers.