Cancel abandoned phase executions (production CPU-leak root cause)#1257
Merged
Conversation
…ute comparison Adds an observational BlindPay payin FX quote (BRL -> USDT/USDB) alongside the existing SquidRouter/Avenia/Nabla route quotes in compareRoutesUpfront, to evaluate whether BlindPay is a cheaper source of stablecoin over time. - New apps/rebalancer/src/services/blindpay: FX-rate client + types - fetchBlindpayShadowQuoteUsdc returns a USDC-equivalent (6dp) quote, or null when unconfigured or below BlindPay's 5 BRL minimum - Shadow only: logged in the route comparison but never added to candidates, so it is never actually routed through - Optional BLINDPAY_* config; documented in .env.example
…orts waitUntilTrueWithTimeout raced its poll loop against a timeout without cancelling it, so every timed-out call leaked a loop polling forever (the production CPU-leak mechanism). The timeout now aborts the loop via an AbortController, and waitUntilTrue/checkEvmBalancePeriodically/ checkEvmNativeBalancePeriodically/checkEvmBalanceForToken accept an optional AbortSignal so callers can cancel abandoned waits. Error messages and BalanceCheckError semantics are unchanged.
The processor's 10-minute execution timeout used Promise.race, which abandons handler.execute without stopping it. Abandoned executions kept their polling loops running forever and piled up with every retry and recovery-worker pass until the CPU pegged (production incidents Jun 26 - Jul 8); a pegged event loop then slowed Postgres SCRAM handshakes enough for Supavisor to fail new connections with EAUTHTIMEOUT. Abandoned executions could also perform late side effects (e.g. Avenia ticket creation) concurrently with the live retry. The processor now aborts each timed-out execution via an AbortSignal threaded through PhaseHandler.execute/executePhase. MAX_EXECUTION_TIME_MS is env-overridable (PHASE_PROCESSOR_MAX_EXECUTION_TIME_MS) so the regression test doesn't wait 10 minutes. Spec updated accordingly.
The Avenia balance wait ran 30 minutes per execution, which always outlived the processor's 10-minute execution timeout - so the 30-minute payment-window cancellation (isPaymentTimeoutReached) never ran for recovered ramps and never-paid onramps churned indefinitely. The wait now runs in 5-minute chunks (like the EVM balance check) and surfaces each chunk timeout as a recoverable error, letting the wall-clock payment window cancel the ramp. Both waits also accept the processor's AbortSignal. Spec updated accordingly.
With pool min 0 / idle 10s every quiet period dropped all connections, so each request burst re-ran the full SCRAM handshake through the Supabase pooler (Supavisor), which fails with EAUTHTIMEOUT when the event loop is busy. Keep 2 warm connections and a 60s idle timeout.
…otification Threads the observational BlindPay shadow quote from compareRoutesUpfront through the preflight quotes and rebalance state into the Base completion Slack message, rendered as a 'BlindPay shadow (not routed)' line with the delta vs the executed route. Absent when BlindPay is unconfigured.
✅ Deploy Preview for vortex-sandbox ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview for vortexfi ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
ebma
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses a production CPU leak by ensuring phase executions and polling helpers are cancellable (so timed-out work doesn’t continue running in the background), and improves operational resilience with a warmer DB connection pool. It also adds an optional “BlindPay shadow quote” to the rebalancer for observational pricing comparisons.
Changes:
- Add
AbortSignalsupport to shared polling helpers (waitUntilTrue*, EVM balance polling) and introduce an abortablesleep. - Update API phase processor to abort timed-out executions via
AbortSignal, plus add regression tests and adjust BRLA onramp mint polling to run in shorter chunks. - Add optional BlindPay integration to rebalancer (types + minimal client + shadow-quote plumbing), and adjust DB pool settings to keep warm connections.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/shared/src/services/evm/balance.ts | Make periodic EVM balance checks abortable and non-overlapping via abortable sleep. |
| packages/shared/src/helpers/functions.ts | Add abortable sleep and propagate AbortSignal through polling helpers; fix timeout race leak. |
| packages/shared/src/helpers/functions.test.ts | Add regression tests ensuring polling stops after timeout/abort. |
| docs/security-spec/05-integrations/brla.md | Update BRLA onramp mint documentation to reflect chunked waits + cancellation behavior. |
| docs/security-spec/03-ramp-engine/state-machine.md | Document processor timeout abort behavior and env override for tests. |
| apps/rebalancer/src/utils/config.ts | Add optional BlindPay config variables. |
| apps/rebalancer/src/services/stateManager.ts | Persist BlindPay shadow quote in rebalance state. |
| apps/rebalancer/src/services/blindpay/types.ts | Add BlindPay API request/response types. |
| apps/rebalancer/src/services/blindpay/blindpayApiService.ts | Introduce BlindPay API client (needs request timeout). |
| apps/rebalancer/src/rebalance/usdc-brla-usdc-base/steps.ts | Fetch BlindPay shadow quote during route comparison and log deltas (observational only). |
| apps/rebalancer/src/rebalance/usdc-brla-usdc-base/notifications.ts | Include BlindPay shadow quote line in completion notifications. |
| apps/rebalancer/src/rebalance/usdc-brla-usdc-base/index.ts | Thread BlindPay shadow quote through state and Slack notification payload. |
| apps/rebalancer/src/index.ts | Include BlindPay shadow quote in policy evaluation metadata. |
| apps/rebalancer/.env.example | Document optional BlindPay env vars. |
| apps/api/src/config/database.ts | Keep a small warm DB pool (min: 2, idle: 60s) to reduce SCRAM handshake churn. |
| apps/api/src/api/services/phases/phase-processor.ts | Abort timed-out phase executions and allow env override of execution timeout. |
| apps/api/src/api/services/phases/phase-processor.cancellation.integration.test.ts | Add integration test proving abandoned executions are aborted (needs test isolation fixes). |
| apps/api/src/api/services/phases/handlers/brla-onramp-mint-handler.ts | Pass abort signal into Avenia/EVM polling and chunk Avenia wait to fit processor timeout. |
| apps/api/src/api/services/phases/base-phase-handler.ts | Thread abort signal through PhaseHandler.execute and executePhase. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+8
to
+24
| export function sleep(ms: number, signal?: AbortSignal): Promise<void> { | ||
| return new Promise((resolve, reject) => { | ||
| if (signal?.aborted) { | ||
| reject(abortReason(signal)); | ||
| return; | ||
| } | ||
| const onAbort = () => { | ||
| clearTimeout(timer); | ||
| reject(abortReason(signal as AbortSignal)); | ||
| }; | ||
| const timer = setTimeout(() => { | ||
| signal?.removeEventListener("abort", onAbort); | ||
| resolve(); | ||
| }, ms); | ||
| signal?.addEventListener("abort", onAbort, { once: true }); | ||
| }); | ||
| } |
Comment on lines
+37
to
+52
| const response = await fetch(url, { | ||
| body: JSON.stringify(input), | ||
| headers: { | ||
| Accept: "application/json", | ||
| Authorization: `Bearer ${blindpayApiKey}`, | ||
| "Content-Type": "application/json" | ||
| }, | ||
| method: "POST" | ||
| }); | ||
|
|
||
| if (!response.ok) { | ||
| throw new Error(`BlindPay request failed with status '${response.status}'. Error: ${await response.text()}`); | ||
| } | ||
|
|
||
| return (await response.json()) as PayinFxRateResponse; | ||
| } |
Comment on lines
+48
to
+52
| beforeAll(async () => { | ||
| await setupTestDatabase(); | ||
| await resetTestDatabase(); | ||
| phaseRegistry.registerHandler(hangingHandler); | ||
| }); |
…on gap Review feedback on #1257: register the abort listener and re-check signal.aborted afterwards — a listener added to an already-aborted signal never fires, so this ordering leaves no path where an abort is missed. Adds a test for the pre-aborted case.
…ation test Review feedback on #1257: a non-numeric PHASE_PROCESSOR_* env value parsed to NaN, and setTimeout(..., NaN) fires immediately — a misconfigured deploy would time out every phase instantly. Fall back to the defaults for non-finite or non-positive values (tested). The cancellation test now snapshots and restores the env overrides and the shadowed nablaSwap registry entry in afterAll, per the repo's restore convention.
Review feedback on #1257: fetch had no timeout, so a stalled BlindPay request could hang the Promise.allSettled upstream and stall the whole rebalance run — cap it at 30s via AbortSignal.timeout. Also adds client tests (configuration gating, request shape incl. the timeout signal, non-OK handling), which lifts the rebalancer function coverage back above the CI ratchet floor (49.57% -> 53.45%).
…hrows Review feedback on #1257: handler.execute was invoked eagerly inside the Promise.race array literal, so a handler that throws synchronously would propagate before .finally was attached — clearTimeout never ran and the orphaned timeout promise later rejected with nobody listening. Wrap the call in Promise.resolve().then(...) so sync throws become a rejection of the race and cleanup runs reliably. Regression test asserts no unhandledRejection fires after the timeout window.
PR #1255 merged a mis-formatted translations file; every PR merge ref now fails the format check. Formatted with biome.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Production (
vortex-productionon Render) repeatedly climbed to 100% CPU over hours and needed manual restarts (incidents Jun 26, Jun 30, Jul 1, Jul 7–8). Once pegged, the busy event loop slowed Postgres SCRAM handshakes enough that the Supabase pooler (Supavisor) failed new connections with(EAUTHTIMEOUT) timeout while waiting for message, breaking quotes and API-key validation.Root cause (diagnosed from Render metrics + logs):
brlaOnrampMint; the recovery worker re-processes them every 5 minutes.Promise.race, which abandonshandler.executewithout stopping it.waitUntilTrueWithTimeouthad the same flaw — its poll loop (waitUntilTrue, awhile(true)) ran forever once the race settled.Bonus hazard: abandoned executions could later perform real side effects (Avenia
createPayInQuote/createPixOutputTicket) concurrently with the live retry. And because the Avenia wait ran 30 min per execution — always outliving the 10-min processor timeout — the 30-minute payment-window cancellation effectively never ran for recovered ramps.Fix (one commit per concern)
waitUntilTrueWithTimeoutaborts its poll loop when the timeout fires;waitUntilTrue/checkEvmBalancePeriodically/checkEvmNativeBalancePeriodically/checkEvmBalanceForTokenaccept an optionalAbortSignal. Error messages andBalanceCheckErrorsemantics unchanged; newsleep(ms, signal)helper exported.AbortSignalthreaded throughPhaseHandler.execute/executePhase(optional param — existing handlers unchanged).MAX_EXECUTION_TIME_MSis env-overridable for tests.min: 2, idle: 60s(wasmin: 0, idle: 10s) to stop paying a full Supavisor SCRAM handshake on every quiet-period burst.docs/security-spec(state-machine.md, brla.md) updated in the commits that change the documented behavior.Testing
packages/shared/src/helpers/functions.test.ts: proves the poll loop stops after timeout/abort (fails against the old implementation).apps/api/.../phase-processor.cancellation.integration.test.ts: real-DB test with a hanging handler — asserts every execution receives an aborted signal after the processor gives up and that polling stops (verified to fail without the signal pass-through).Deploy note
After deploy, stuck-but-live ramps will churn at most ~35 minutes before being cancelled by the payment window, and abandoned executions no longer accumulate — CPU should stay flat. The EAUTHTIMEOUT errors should disappear with it; if any remain, they'd point at pooler-side load, not the service.