fix(trpc): move a boot to a new lock id before its renewal check by superset-home · Pull Request #9225 · ParadevAI/Paradev

superset-home · 2026-11-01T16:14:05Z

Problem

When Autumn has a short outage, a cloud-workspace boot can get stuck in its settle loop. Every settle of that boot fails with 409 lock_already_exists, every 30 minutes plus QStash retries, until the box stops. While it is stuck, the boot never renews, so its spend cap is not checked. In Sentry (API-3W and API-3X, about 150 events in total), each 409 run starts right after a burst of Autumn 503s (10-28 06:00 UTC and 10-30 06:00 UTC). One run went on for about 17 hours.

Root cause

settleBoot renews by calling autumnClient.check with lock id <bootId>:<lockSeq+1>, and it saved lockSeq only after the check returned. If Autumn creates the lock but the response fails (the 503s), lockSeq stays where it was. Every retry then asks for the same lock id, and Autumn answers a repeated lock id with 409 lock_already_exists.

Fix

Save lockSeq + 1 in the same write that records the billed window (finalizedThrough, lockOpen: false), before the renewal check. A retry after a failed check then asks for a fresh lock id and gets Autumn's real allowed/refused answer. A lock the failed attempt may have created is left to expire, and the SDK documents on Lock.expiresAt that expiry "releases the held balance". authorizeBoot's fail-open path already accepts the same kind of orphaned lock.

The first revision of this PR treated the 409 as a granted renewal. Review rejected that, because it assumed a refused check never records its lock id, which nobody has verified. This revision removes it and makes no assumption about Autumn's refusal behaviour.

Verification

$ bunx biome check packages/trpc/src/router/cloud-workspace/meter.ts packages/trpc/src/router/cloud-workspace/meter.test.ts
Checked 2 files in 44ms. No fixes applied.

$ cd packages/trpc && bun run typecheck
$ tsc --noEmit --emitDeclarationOnly false
(no errors)

# before the fix, with the lockSeq assertion added to the failed-renewal test
$ bun test src/router/cloud-workspace/meter.test.ts
(fail) settleBoot > a retry after a failed renewal asks for a new lock ...
 26 pass
 1 fail

# after the fix
$ bun test src/router/cloud-workspace/
 44 pass
 0 fail
Ran 44 tests across 3 files.

Not in scope

This fixes only the renewal 409 loop. An earlier triage of this issue found a second path from the same outage that this PR does not change: if a finalize confirm returns 503 after Autumn applied it, the retried finalize gets 400 "lock not found". recordUsage reports that and falls back to track, which can bill that window twice. Fixing that needs an answer from Autumn on what a 503 means for finalize and whether Idempotency-Key is honoured there.

Review notes

Antagonist review: the first revision (treating the 409 as a granted renewal) got DO NOT MERGE, because it rested on an unverified claim about Autumn. This revision got MERGE.
The Greptile finding about JSON spacing in the 409 matcher refers to code from the first commit, which this revision removed. It does not apply to the current diff.

This is billing code, so a human must review and merge it.

Refs API-3W

… every retry A renewal check that failed after Autumn created the lock (a 503 during an Autumn outage) left lockSeq unadvanced. Each retry asked for the same lock id, got 409 lock_already_exists, and threw, so the boot never renewed and its spend cap was not checked until the box stopped. A refused check reserves nothing, so an existing lock was granted: treat lock_already_exists on the renewal as that grant and carry on. Refs API-3W

coderabbitai · 2026-11-01T16:14:39Z

Walkthrough

settleBoot now persists the incremented lock sequence while recording settled usage, before checking session end or attempting renewal. The test expects the closed lock to retain sequence 1 after a failed renewal.

Changes: Boot settlement lock sequence

Layer / File(s)	Summary
Persist and verify the lock sequence `packages/trpc/src/router/cloud-workspace/meter.ts`, `packages/trpc/src/router/cloud-workspace/meter.test.ts`	`settleBoot` saves the incremented `lockSeq` with the settled usage before the renewal branch. The test expects `lockSeq: 1` after failed renewal.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: harshithmullapudi

Merge Risk: Low · up to 401c9

This fixes boots that failed renewal on every retry after an Autumn outage. If Autumn does create a lock before returning an error, that lock could hold balance until it expires, which could delay renewal for customers with little remaining balance. That behavior is not confirmed, so it is a bounded risk to confirm rather than a blocker.

Pre-merge checks: 5

Check name	Status	Explanation
Title check		The title clearly and concisely describes the main change: assigning a new lock ID before the renewal check.
Docstring Coverage		Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description check		The description clearly explains the problem, root cause, fix, verification steps, scope limits, and review context. It does not use the template's exact headings and omits the checklist, but it is ot…

Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 2 files.

Full details: Description check

Explanation

The description clearly explains the problem, root cause, fix, verification steps, scope limits, and review context. It does not use the template's exact headings and omits the checklist, but it is otherwise complete and technically specific.

Finishing Touches

Generate docstrings

Commit to this branch
Create a new PR

Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

greptile-apps · 2026-11-01T16:16:02Z

Confidence Score: 4/5

[Medium impact] The change is limited to ongoing cloud-workspace billing settles.

The PR appears safe to merge, with a non-blocking recommendation to make the error-code match independent of JSON spacing.

Findings

JSON spacing can break recovery ▶

Fix with agent prompt

### Issue 1
packages/trpc/src/router/cloud-workspace/meter.ts:132
The new match depends on exact JSON spacing rather than the value of `code`. If Autumn returns `{"code": "lock_already_exists"}`, this check misses the same error and renewal keeps failing on every retry. Parse the body safely and compare `code` directly, while keeping the `AutumnError` and 409 checks. Add a test with whitespace to guard against harmless response-format changes.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR lets settleBoot recover a renewal lock that Autumn created before an earlier response failed.

Cloud-workspace retries keep a renewal lock that Autumn already created.

_{Reviews (1) · Last reviewed commit: "fix(trpc): keep a renewal lock Autumn al..." · Reviewed by Greptile}

greptile-apps · 2026-11-01T16:16:05Z

+	return (
+		error instanceof AutumnError &&
+		error.statusCode === 409 &&
+		error.body.includes('"code":"lock_already_exists"')


JSON spacing can break recovery

The new match depends on exact JSON spacing rather than the value of code. If Autumn returns {"code": "lock_already_exists"}, this check misses the same error and renewal keeps failing on every retry. Parse the body safely and compare code directly, while keeping the AutumnError and 409 checks. Add a test with whitespace to guard against harmless response-format changes.

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/trpc/src/router/cloud-workspace/meter.ts Line: 132 Comment: **JSON spacing can break recovery** The new match depends on exact JSON spacing rather than the value of `code`. If Autumn returns `{"code": "lock_already_exists"}`, this check misses the same error and renewal keeps failing on every retry. Parse the body safely and compare `code` directly, while keeping the `AutumnError` and 409 checks. Add a test with whitespace to guard against harmless response-format changes. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

github-actions · 2026-11-01T16:16:49Z

🚀 Preview Deployment

🔗 Preview Links

Service	Status	Link
Database (Neon)	✅	View Branch
API (Vercel)	✅	Open Preview
Web (Vercel)	✅	Open Preview
Marketing (Vercel)	✅	Open Preview
Docs (Vercel)	✅	Open Preview

Preview updates automatically with new commits

Review found the earlier fix relied on an unverified Autumn claim: that a refused check never records its lock id. Drop the 409 classifier. Save lockSeq+1 in the same write that records the billed window, before the renewal check. A retry after a failed check then asks for a fresh lock id, so it gets a real allowed/refused answer. A lock the failed attempt left behind expires and releases its hold (documented on Lock.expiresAt). Refs API-3W

expo · 2026-11-01T16:17:51Z

Preview update published for development build eca66a32 (fingerprint e65ce637d4ee). Open the build on your phone and scan:

Update group ba8cc08d

cubic-dev-ai

2 issues found across 2 files

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/trpc/src/router/cloud-workspace/meter.test.ts">

<violation number="1" location="packages/trpc/src/router/cloud-workspace/meter.test.ts:368">
P3: The title promises a retry with a new lock id, but the test never runs a retry or checks the lock id. Either add a second `settleBoot` call that asserts the `check` call's `lock.lockId` differs from the first attempt, or rename the test to match what it asserts (the persisted `lockSeq: 1`).</violation>
</file>

<file name="packages/trpc/src/router/cloud-workspace/meter.ts">

<violation number="1" location="packages/trpc/src/router/cloud-workspace/meter.ts:418">
P3: `nextSeq` also advances for ended sessions, which never request another lock. If `recordActiveCpu` fails, retries advance `lockSeq` again and send different lock-sequence metadata under the same `bootId:active-cpu` idempotency key; only advance it for a running session.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant Settle as settleBoot()
    participant DB as Cloud Workspace DB
    participant Autumn as Autumn API
    participant Meter as meter.ts

    Note over Settle,Autumn: Boot settle loop (runs every 30 min + QStash retries)
    
    Settle->>Settle: Read boot record (lockSeq, lockOpen, spend data)
    Settle->>Autumn: check lock <bootId>:<lockSeq+1>
    alt Lock granted
        Autumn-->>Settle: Lock acquired
        Settle->>DB: Read session window
        DB-->>Settle: Session data
        Settle->>Settle: Compute gbMinutes usage
        Settle->>DB: Update boot (finalizedThrough, lockOpen=false, lockSeq+1)
        Note over Settle,DB: lockSeq advanced in same write as billed window
        Settle->>Autumn: Confirm usage
    else Lock refused (409 lock_already_exists)
        Autumn-->>Settle: 409 error
        Settle-->>Settle: Retry with FAILOVER
        Note over Settle,Autumn: Autmn 503 during check may create lock but response fails
        Settle->>DB: Update boot (finalizedThrough, lockOpen=false, lockSeq+1)
        Note over Settle,DB: lockSeq advanced even on failed renewal - retry gets fresh lock id
        Settle->>Autumn: check lock <bootId>:<lockSeq+2>
        alt Success
            Autumn-->>Settle: New lock granted
            Settle->>Autumn: Continue renewal
        else Refused
            Autumn-->>Settle: 409 again
            Settle-->>Settle: Escalate error / leave orphaned lock to expire
        end
    end
    Settle-->>Settle: Continue until session ends

_{Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic}

cubic-dev-ai · 2026-11-01T16:20:43Z

 	});

-	test("a settle whose next lock fails keeps the billed window, so a retry does not confirm it again", async () => {
+	test("a settle whose next lock fails keeps the billed window and moves to a new lock id for the retry", async () => {


P3: The title promises a retry with a new lock id, but the test never runs a retry or checks the lock id. Either add a second settleBoot call that asserts the check call's lock.lockId differs from the first attempt, or rename the test to match what it asserts (the persisted lockSeq: 1).

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/trpc/src/router/cloud-workspace/meter.test.ts, line 368: <comment>The title promises a retry with a new lock id, but the test never runs a retry or checks the lock id. Either add a second `settleBoot` call that asserts the `check` call's `lock.lockId` differs from the first attempt, or rename the test to match what it asserts (the persisted `lockSeq: 1`).</comment> <file context> @@ -365,12 +365,13 @@ describe("settleBoot", () => { }); - test("a settle whose next lock fails keeps the billed window, so a retry does not confirm it again", async () => { + test("a settle whose next lock fails keeps the billed window and moves to a new lock id for the retry", async () => { renewalError = new Error("autumn down"); await expect(settleBoot("boot_1")).rejects.toThrow("autumn down"); </file context>

Suggested change

test("a settle whose next lock fails keeps the billed window and moves to a new lock id for the retry", async () => {

test("a settle whose next lock fails keeps the billed window and persists the advanced lock seq", async () => {

cubic-dev-ai · 2026-11-01T16:20:43Z

 			amount: gbMinutes({ ms: end - from, memoryMb: session.memoryMb }),
 			user,
 		});
+		const nextSeq = boot.lockSeq + 1;


P3: nextSeq also advances for ended sessions, which never request another lock. If recordActiveCpu fails, retries advance lockSeq again and send different lock-sequence metadata under the same bootId:active-cpu idempotency key; only advance it for a running session.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/trpc/src/router/cloud-workspace/meter.ts, line 418: <comment>`nextSeq` also advances for ended sessions, which never request another lock. If `recordActiveCpu` fails, retries advance `lockSeq` again and send different lock-sequence metadata under the same `bootId:active-cpu` idempotency key; only advance it for a running session.</comment> <file context> @@ -415,9 +415,15 @@ export async function settleBoot( amount: gbMinutes({ ms: end - from, memoryMb: session.memoryMb }), user, }); + const nextSeq = boot.lockSeq + 1; + // Autumn answers a repeated lock id with 409, so a retry after a failed renewal needs a fresh one. await db </file context>

Suggested change

const nextSeq = boot.lockSeq + 1;

const nextSeq = boot.lockSeq + (session.endedAt === null ? 1 : 0);

coderabbitai

Additional comments: 1

packages/trpc/src/router/cloud-workspace/meter.ts-425-425: 1

425-425: Major · Stability & Availability · Heavy lift

Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

Reconcile the failed renewal lock before requesting another.

If Autumn creates lock 1 but returns 503, the retry requests lock 2 and leaves lock 1 holding balance for up to 70 minutes. (docs.useautumn.com) If the customer has enough balance for one reservation but not both, the retry is refused and the workspace can stop after the 10-minute grace period. Reconcile or release the uncertain lock before requesting a fresh reservation.

Review info

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: bc1a5e31-15db-442b-b2da-d7235858da30

Commits

Reviewing files that changed from the base of the PR and between 8b52324 and 401c9cf.

Files selected for processing: 2

packages/trpc/src/router/cloud-workspace/meter.test.ts
packages/trpc/src/router/cloud-workspace/meter.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

superset-home deployed to preview November 1, 2026 16:14 — with GitHub Actions Active

greptile-apps Bot reviewed Nov 1, 2026

View reviewed changes

superset-home deployed to preview November 1, 2026 16:17 — with GitHub Actions Active

superset-home changed the title ~~fix(trpc): keep a renewal lock Autumn already took instead of failing every retry~~ fix(trpc): move a boot to a new lock id before its renewal check Nov 1, 2026

superset-home deployed to preview November 1, 2026 16:17 — with GitHub Actions Active

superset-home requested a review from saddlepaddle November 1, 2026 16:19

cubic-dev-ai Bot reviewed Nov 1, 2026

View reviewed changes

coderabbitai Bot reviewed Nov 1, 2026

View reviewed changes

Navigation Menu

Repository navigation

fix(trpc): move a boot to a new lock id before its renewal check - #9225

fix(trpc): move a boot to a new lock id before its renewal check#9225
superset-home wants to merge 2 commits into
mainfrom
sentry/api-3w-renewal-lock-retry

superset-home commented Nov 1, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Nov 1, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Nov 1, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Nov 1, 2026

Uh oh!

github-actions Bot commented Nov 1, 2026

Uh oh!

expo Bot commented Nov 1, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	test("a settle whose next lock fails keeps the billed window and moves to a new lock id for the retry", async () => {
	test("a settle whose next lock fails keeps the billed window and persists the advanced lock seq", async () => {

	const nextSeq = boot.lockSeq + 1;
	const nextSeq = boot.lockSeq + (session.endedAt === null ? 1 : 0);

Conversation

superset-home commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Root cause

Fix

Verification

Not in scope

Review notes

Uh oh!

coderabbitai Bot commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Uh oh!

greptile-apps Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Nov 1, 2026

🚀 Preview Deployment

🔗 Preview Links

Uh oh!

expo Bot commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

superset-home commented Nov 1, 2026 •

edited

Loading

coderabbitai Bot commented Nov 1, 2026 •

edited

Loading

greptile-apps Bot commented Nov 1, 2026 •

edited

Loading

expo Bot commented Nov 1, 2026 •

edited

Loading