fix: fail over to other regions when Cloud rejects a connection with 403

[xianshijing-lk]

Part of CLT-3325. Companion PRs: client-sdk-js#2097, client-sdk-flutter#1200.

Problem

LiveKit Cloud enforces project-level region pinning by returning 403 on the RTC paths (/rtc, /rtc/validate) when a project is not allowed in the region the client geo-routed to. /settings/regions is deliberately excluded from that gate so the client can discover its allowed regions and connect there — per the server-side comment, "allow other APIs to pass because clients will permanently give up if they fail."

Swift blocked that recovery twice. HTTP.requestValidation maps any 4xx except 404/429 to .validation, and connectWithCloudRegionFailover short-circuited on it:

if let liveKitError = error as? LiveKitError, liveKitError.type == .validation {
    // Don't retry other regions for validation errors.
    throw liveKitError
}

and even without that, isRetryableForRegionFailover admitted only .network / .timedOut. So a pinned project that geo-routed to a disallowed region gave up before ever fetching the region list.

Fix

1. Carry the HTTP status on LiveKitError as a new optional statusCode, set where HTTP.requestValidation classifies the response.
2. Admit a 403 in isRetryableForRegionFailover.
3. Drop the .validation short-circuit in connectWithCloudRegionFailover, which is now redundant — .validation without a 403 still returns false and throws exactly as before.
4. Report region exhaustion as a nil result from RegionManager.resolveBest rather than a thrown .regionManager, so connectWithCloudRegionFailover rethrows the connection error it is already holding instead of replacing it. A throw from resolveBest still means a genuine settings fetch / HTTP / parse failure and propagates unchanged.
5. Keep failed regions excluded across a settings refresh in applyFetchedRegions, matching updateFromServerReportedRegions.
6. Cap the failover loop at 10 attempts, so termination does not rest on an invariant about how remaining is maintained.

Net behaviour change: 403 → retry other regions; 401 stays terminal (no other region will accept the same token), as does the 404 that surfaces as .serviceNotFound for the v1 → v0 path fallback.

Why status and not the error message

The server's body for this case is "project not allowed in this region.", but that is an unversioned human-readable string, and today it reaches Swift only embedded in message as "HTTP 403: ...". Matching either would mean five SDKs carrying identical literals forever, and a server-side copy edit would silently break already-shipped clients — worst of all on iOS, where a released app cannot be hot-patched. Hence the structured statusCode.

The cost of not discriminating is bounded: if a 403 really was a permissions failure, every region attempt fails the same way and the original error still surfaces, one region lookup later.

That last part was not true as originally written, and review caught it (thanks @pblazej, Devin). Exhaustion threw .regionManager("No more remaining regions.") from inside the same catch, discarding the 403 — so a permission failure came back less informative than before this PR. Item 4 above is what actually makes the sentence hold — and follows rust-sdks, whose livekit-signaling fallback tracks last_err across region attempts and never synthesizes an exhaustion error. A settings fetch/parse failure surfaces the connection error too, matching rust's explicit choice not to mask it. Items 5 and 6 address a related pre-existing gap the wider retry set makes easier to hit: applyFetchedRegions refilled remaining on every refresh, so once per-region failure outlasts cacheInterval the loop could never exhaust and Room.connect would hang rather than fail.

API impact

None. statusCode is internal, and the public LiveKitError initializer keeps its exact signature; a separate internal initializer records the status.

I first added a defaulted statusCode: parameter to the existing public init, assuming that was source-compatible — it is for callers, but the Swift API digester reports the changed signature as a removed declaration, and Check Public API failed on it. Fixed in 376d8da. Exposing statusCode publicly would be useful but is a separate decision that shouldn't ride along in a bugfix.

Cross-SDK status

This bug is not universal. Rust and Android already fail over on any non-cancellation error and are unaffected; JS, Flutter and Swift all bail. Agents SDKs route through the Rust core, so agents are unaffected.

Testing

xcodebuild build -scheme LiveKit -destination 'platform=macOS' succeeds. New test regionPinning403IsRetryableButOtherValidationFailuresAreNot covers 403 / 401 / 404; the existing regionManagerShouldRetryConnection is unchanged and still passes. RegionManagerTests passes in full (6 tests).

LiveKitCoreTests run against a local livekit-server --dev leaves 3 failures — defineAndGetSchema, publishWithFrameMetadata, concurrentPush — all data-track schema tests, which AGENTS.md documents as requiring enable_participant_data_blob: true; the livekit-server build available locally rejects that config key. I was not able to baseline these against main (the run exceeded my time budget), so I am flagging rather than asserting they are pre-existing — though this change only touches error classification on the region-failover path and has no plausible connection to data-track schemas.

Open question

How often the RTC 403 actually fires for pinned projects has not been confirmed with the Cloud team — geo-routing may normally land clients in-region, making this an edge case (bad GeoDNS, anycast flap, VPN). The opt-in prepareConnection() warm-up also does region selection up front and would mask it for apps that call it. The fix is correct either way and works against today's servers, but severity is unconfirmed.

🤖 Generated with Claude Code

[github-actions]

⚠️ This PR does not contain any files in the .changes directory.

[xianshijing-lk]

@pblazej , what do you think about 10 attempt cap ? or we should do less?

[pblazej]

10 is fine — the server already truncates to 6 (len(res.Regions) >= 6 in DefaultRegionMap.GetRegionSettings, a hard constant rather than a per-geography thing), so the cap never fires in practice; I checked four Cloud projects and got exactly 6 each time. I would not go lower: 10 leaves headroom for the prepared-region path (initialRegion is resolved before the loop, so it effectively allows 11) and for that 6 being raised later, whereas anything at or below it turns the safety net into a functional limit.

[pblazej]

LGTM, test flakiness addressed elsewhere.