From a23b790441e7c2f854cb04ef43e8cc386f9c6d8e Mon Sep 17 00:00:00 2001 From: Dante Date: Mon, 26 Oct 2026 21:55:42 +0000 Subject: [PATCH] fix(cloud): retry startup remote config instead of falling back to dev Firebase (#21981) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Why IR-153: production users were intermittently signed in against the **dev** Firebase project (`dreamboothy-dev`) and production then rejected every request (`expected dreamboothy but got dreamboothy-dev`), showing a logout / "Couldn't load your workspace" until a reload. RUM, last 30 days on `cloud.comfy.org`: 119 sessions loaded the dev auth iframe (vs 17,850 on prod), with 41 Google and 19 password sign-in sessions sent to dev, mostly on slow connections. The dev project's sign-in allowlist rejects non-staff, so customers saw a failed sign-in rather than dev data. Once the page has the right config, product use and billing are unaffected. ## Root cause - `getFirebaseConfig()` uses the server's `firebase_config`, else `BUILD_TIME_CONFIG` (`src/config/firebase.ts`). - Cloud bundles (including the prod variant in `Comfy-Org/cloud` `frontend-asset-predeploy.yml`) are built without `USE_PROD_CONFIG`, so `BUILD_TIME_CONFIG` is the dev project. - At startup the anonymous `/features` read has a 5s timeout; on failure `remoteConfigState` becomes `'error'` and `firebaseIdentity` (which only guards `'unloaded'`) initializes from that dev default. - Firebase initializes once, so the later authenticated refresh / 10‑minute poll cannot repair the page. ## How On cloud, startup now retries the anonymous `/features` read with capped backoff (1s → 10s) behind the existing splash until the server names the Firebase project (`loadRemoteConfigUntilReady`). A successful read is not enough on its own: `firebase_config` is optional in the response contract, so a 200 that omits it would otherwise end bootstrap with no server config and let `getFirebaseConfig()` fall through to the build-time default — the same `dreamboothy-dev` outcome. The loop therefore also requires `hasServerFirebaseConfig()`, which counts an explicit `firebase_env: 'dev'` as server-named, since that is a server instruction rather than a build-time guess. `isCloud` is the bundle flavour, not "served by the Cloud API": a cloud bundle can be served by a local ComfyUI, whose `/features` answers 200 and never carries `firebase_config`. Waiting indefinitely there never starts the app — every cloud Playwright shard timed out that way at `59e93a79` — so the two cases are split on `isLoopbackHost(window.location.hostname)`. A deployed host keeps waiting rather than start Firebase on the bundled project; a loopback harness, whose server has no project to send, stops after one further read. A read that *fails* still retries in both, which is the shape the incident had. The first project-less read reports `startup_remote_config_without_firebase_project` (tagged `localCloud`), so a deployed Cloud omitting the field is visible while it waits rather than silent. Firebase therefore only ever starts from the server's `firebase_config`; there is no build-time fallback on cloud. A long outage shows up as a `timed_out` bootstrap report with `startup/remote-config` pending. Local and desktop keep the single read and their build-time defaults, since their server does not send `firebase_config`. Alternative considered: building the prod bundle with `USE_PROD_CONFIG=true`. That prevents the wrong project on prod but still boots on a guessed config when the server does not answer, and staging bundles keep the dev guess. ## AS IS / TO BE AS IS (`origin/main`, `pnpm dev:cloud` → testcloud, `/api/features` aborted): one request, never retried, never answered; the login page renders anyway and Firebase is initialized from the build-time default `dreamboothy-dev`. AS IS TO BE (PR branch, same setup, `/api/features` aborted twice then let through): the splash holds during retries #1 and #2 (1s, 2s backoff), request #3 answers 200, then the login page loads with Firebase on the server's `firebase_config`. testcloud's own `firebase_config` is `dreamboothy-dev`, so the project name matches AS IS here; the difference is that it now comes from the server rather than the build default. On prod the server returns `dreamboothy`. TO BE splash during retries TO BE loaded ## Tests `refreshRemoteConfig.test.ts`: - abort → 503 → success resolves with the server config after three reads. Fails when the loop returns after the first read. - a 200 without `firebase_config` keeps retrying and resolves on the next read that carries one. Fails with `expected "vi.fn()" to be called 2 times, but got 1 times` when the `hasServerFirebaseConfig()` gate is removed. - a 200 carrying only `firebase_env: 'dev'` resolves on the first read, pinning that branch against an accidental narrowing. - on a deployed host (`cloud.comfy.org`), repeated 200s without a project never resolve the bootstrap promise, so Firebase initialization is unreachable. Fails when the loopback guard is dropped. - on a loopback host, the same responses stop after one further read, which is what keeps the cloud e2e harness booting. (cherry picked from commit 9c78428dfe90c0f6a1103c83fee0a8d4ee5168c1) --- src/config/firebase.ts | 8 ++ src/main.ts | 5 +- .../remoteConfig/refreshRemoteConfig.test.ts | 97 +++++++++++++++++++ .../remoteConfig/refreshRemoteConfig.ts | 49 ++++++++++ 4 files changed, 157 insertions(+), 2 deletions(-) diff --git a/src/config/firebase.ts b/src/config/firebase.ts index 90ad7e92e2b..37f91bc5909 100644 --- a/src/config/firebase.ts +++ b/src/config/firebase.ts @@ -36,3 +36,11 @@ export function getFirebaseConfig(): FirebaseOptions { if (remoteConfig.value.firebase_env === 'dev') return DEV_CONFIG return BUILD_TIME_CONFIG } + +/** Whether {@link getFirebaseConfig} names the project from the server rather than the build. */ +export function hasServerFirebaseConfig(): boolean { + return ( + Boolean(remoteConfig.value.firebase_config) || + remoteConfig.value.firebase_env === 'dev' + ) +} diff --git a/src/main.ts b/src/main.ts index ffa6a59324e..222dace8454 100644 --- a/src/main.ts +++ b/src/main.ts @@ -44,9 +44,10 @@ const hasHostTelemetryBridge = Boolean(window.__comfyDesktop2?.Telemetry) async function loadRemoteConfigBeforeFirebase(): Promise { await bootstrapTracer.settle('startup/remote-config', async () => { - const { refreshRemoteConfig } = + const { loadRemoteConfigUntilReady, refreshRemoteConfig } = await import('@/platform/remoteConfig/refreshRemoteConfig') - await refreshRemoteConfig({ useAuth: false }) + if (isCloud) await loadRemoteConfigUntilReady() + else await refreshRemoteConfig({ useAuth: false }) }) } diff --git a/src/platform/remoteConfig/refreshRemoteConfig.test.ts b/src/platform/remoteConfig/refreshRemoteConfig.test.ts index e9bc5287da6..93adf878669 100644 --- a/src/platform/remoteConfig/refreshRemoteConfig.test.ts +++ b/src/platform/remoteConfig/refreshRemoteConfig.test.ts @@ -5,6 +5,7 @@ import { api } from '@/scripts/api' import { invalidateRemoteConfig, + loadRemoteConfigUntilReady, refreshRemoteConfig } from './refreshRemoteConfig' import { @@ -478,4 +479,100 @@ describe('refreshRemoteConfig', () => { expect(remoteConfigErrorStatus.value).toBeNull() }) }) + + describe('loadRemoteConfigUntilReady', () => { + const BOOTSTRAP_RELEASE_MS = 30_000 + + it('retries until /features loads instead of leaving startup on build-time defaults', async () => { + vi.useFakeTimers() + const serverConfig = { firebase_config: { projectId: 'dreamboothy' } } + vi.mocked(global.fetch) + .mockRejectedValueOnce(new DOMException('Aborted', 'AbortError')) + .mockResolvedValueOnce(mockErrorResponse(503, 'Service Unavailable')) + .mockResolvedValueOnce(mockSuccessResponse(serverConfig)) + + const loading = loadRemoteConfigUntilReady() + await vi.runAllTimersAsync() + await loading + + expect(global.fetch).toHaveBeenCalledTimes(3) + expect(remoteConfigState.value).toBe('anonymous') + expect(remoteConfig.value).toEqual(serverConfig) + vi.useRealTimers() + }) + + it('keeps retrying a 200 that omits firebase_config instead of booting on the build default', async () => { + vi.useFakeTimers() + const serverConfig = { firebase_config: { projectId: 'dreamboothy' } } + vi.mocked(global.fetch) + .mockResolvedValueOnce( + mockSuccessResponse({ subscription_required: true }) + ) + .mockResolvedValueOnce(mockSuccessResponse(serverConfig)) + + const loading = loadRemoteConfigUntilReady() + await vi.runAllTimersAsync() + await loading + + expect(global.fetch).toHaveBeenCalledTimes(2) + expect(remoteConfig.value).toEqual(serverConfig) + vi.useRealTimers() + }) + + it('stops waiting on a loopback host, whose server never names a project', async () => { + vi.useFakeTimers() + vi.stubGlobal('location', { hostname: '127.0.0.1' }) + vi.mocked(global.fetch).mockResolvedValue( + mockSuccessResponse({ subscription_required: true }) + ) + + const loading = loadRemoteConfigUntilReady() + await vi.runAllTimersAsync() + await loading + + expect(global.fetch).toHaveBeenCalledTimes(2) + vi.unstubAllGlobals() + vi.useRealTimers() + }) + + it('never reaches Firebase startup on deployed cloud that keeps answering without a project', async () => { + vi.useFakeTimers() + vi.stubGlobal('location', { hostname: 'cloud.comfy.org' }) + vi.mocked(global.fetch).mockResolvedValue( + mockSuccessResponse({ subscription_required: true }) + ) + const reachedFirebaseStartup = vi.fn() + + void loadRemoteConfigUntilReady().then(reachedFirebaseStartup) + await vi.advanceTimersByTimeAsync(5 * 60_000) + + expect(reachedFirebaseStartup).not.toHaveBeenCalled() + expect(vi.mocked(global.fetch).mock.calls.length).toBeGreaterThan(5) + + vi.mocked(global.fetch).mockResolvedValue( + mockSuccessResponse({ firebase_config: { projectId: 'dreamboothy' } }) + ) + await vi.advanceTimersByTimeAsync(BOOTSTRAP_RELEASE_MS) + expect(reachedFirebaseStartup).toHaveBeenCalledOnce() + + vi.unstubAllGlobals() + vi.useRealTimers() + }) + + it('accepts a 200 that names the dev project through firebase_env', async () => { + vi.useFakeTimers() + const serverConfig = { firebase_env: 'dev' as const } + vi.mocked(global.fetch).mockResolvedValue( + mockSuccessResponse(serverConfig) + ) + + const loading = loadRemoteConfigUntilReady() + await vi.runAllTimersAsync() + await loading + + expect(global.fetch).toHaveBeenCalledTimes(1) + expect(remoteConfig.value).toEqual(serverConfig) + vi.useRealTimers() + }) + }) }) diff --git a/src/platform/remoteConfig/refreshRemoteConfig.ts b/src/platform/remoteConfig/refreshRemoteConfig.ts index 1a437869e3c..d661e252632 100644 --- a/src/platform/remoteConfig/refreshRemoteConfig.ts +++ b/src/platform/remoteConfig/refreshRemoteConfig.ts @@ -1,6 +1,11 @@ +import { delay } from 'es-toolkit' + import { resolveUnifiedWebSession } from '@comfyorg/account-core/webSessionFlag' +import { hasServerFirebaseConfig } from '@/config/firebase' import { isCloud } from '@/platform/distribution/types' +import { reportError } from '@/platform/telemetry/reportError' +import { isLoopbackHost } from '@/utils/hostWhitelist' import { authenticatedRemoteConfigState, @@ -250,3 +255,47 @@ export async function refreshRemoteConfig( activeRefreshControllers.delete(controller) } } + +const BOOTSTRAP_RETRY_BASE_MS = 1_000 +const BOOTSTRAP_RETRY_MAX_MS = 10_000 +const BOOTSTRAP_UNNAMED_PROJECT_READS = 1 + +function servesLocalCloud(): boolean { + return isLoopbackHost(window.location.hostname) +} + +/** + * Blocks until the server names the Firebase project; a loopback harness, + * whose server never sends one, stops after a further read instead. + */ +export async function loadRemoteConfigUntilReady(): Promise { + let unnamedProjectReads = 0 + for (let attempt = 0; ; attempt++) { + await refreshRemoteConfig({ useAuth: false }) + if (remoteConfigState.value !== 'error') { + if (hasServerFirebaseConfig()) return + unnamedProjectReads++ + if (unnamedProjectReads === 1) { + reportError( + new Error( + 'Startup read answered without naming the Firebase project' + ), + { + surface: 'workspace', + errorType: 'startup_remote_config_without_firebase_project', + level: 'warning', + context: { localCloud: servesLocalCloud() } + } + ) + } + if ( + servesLocalCloud() && + unnamedProjectReads > BOOTSTRAP_UNNAMED_PROJECT_READS + ) + return + } + await delay( + Math.min(BOOTSTRAP_RETRY_BASE_MS * 2 ** attempt, BOOTSTRAP_RETRY_MAX_MS) + ) + } +}