Always send Access-Control-Allow-Origin so CDN-cached responses stay usable

[joreilly]

Summary

Fixes the web client intermittently showing no data — empty conference list, empty schedule — with no error in the UI, recovering on its own after a while.

Cause. Apollo sends persisted queries as GETs, and those responses are public, max-age=1800, so Cloud CDN caches them. The cache key is {protocol, host, query string, conference header} and does not include Origin (backend/terraform/main.tf, the graphql backend service's cache_key_policy). CorsWebFilter correctly follows the CORS spec and only emits Access-Control-Allow-Origin when the request has an Origin — which browsers send and the mobile apps do not.

So whichever client warms a cache entry decides, for the next 30 minutes, whether browsers can read it:

Cache entry warmed by	Cached copy has CORS header	Browser result
A browser (sends Origin)	yes	works
A mobile app / server (no Origin)	no	response blocked, silent empty screen

The mobile apps poll far more often than anyone opens the web client, so they usually win that race — which is why the web client is the visible victim of traffic it doesn't generate, and why it comes and goes on a ~30 minute cycle.

Measured on production before the fix:

# APQ GET, hash registered -> served from CDN
x-cache-hit: hit          age: 136      cache-control: public, max-age=1800
(no access-control-allow-origin)                      <-- browsers blocked

# APQ GET, hash not registered -> PersistedQueryNotFound, goes to origin
x-cache-hit: miss         cache-control: no-store
access-control-allow-origin: *                        <-- falls back to POST, works

From the page context the request fails with TypeError: Failed to fetch — a CORS block, not an HTTP error the app can surface, which is why it renders as a silent empty list rather than an error view.

I also confirmed the entry is sticky regardless of caller: after warming GetConferenceData with an Origin request, the cached copy returned access-control-allow-origin: * even to a request sending no Origin. Nothing distinguishes the conference-list query from the per-conference one; both are equally exposed.

Fix. The CORS policy is * for every caller anyway, so emit the header unconditionally. Every cached copy is then valid for every client, with no cache fragmentation. The filter only fills in a header CorsWebFilter did not already set, so genuine Origin requests keep their spec-compliant handling.

Adding Origin to the CDN cache key would also work, but fragments the cache per origin and leaves the no-Origin variant still missing the header.

Test plan

• :backend:service-graphql:compileKotlin — success
• CI: Backend Test (:backend:service-graphql:build)
• Post-deploy verification (the part that actually proves it):

# Must now show access-control-allow-origin even with NO Origin header.
# Use a fresh query string so it is a cache miss, then repeat for the cached copy.
curl -sD - -o /dev/null 'https://confetti-app.dev/graphql?query=%7Bconferences%7Bid%7D%7D' \
  | grep -iE 'access-control-allow-origin|x-cache-hit'

Not verified locally

I could not run the service on this machine — bootRun fails with IllegalStateException: no credentials found for firebase_service_account_key.json, and I did not work around that. So the header behaviour is verified by reading the filter, not by executing it.

backend/service-graphql currently has no test sources, and only testImplementation(libs.junit) — no spring-test/WebTestClient. A regression test for this is worth having (the bug is subtle and invisible in the UI), but it means adding test dependencies and a test source set to a module that has neither, so I left that as a separate call rather than bundling it in.

🤖 Generated with Claude Code

https://claude.ai/code/session_01E3Syr6Ss5YAKH69qjbVUe4

[joreilly]

@martinbonnin again this is Claude's findings so not certain there isn't better way. These were options it proposed

`Option 1 — always emit the header at the origin (what I implemented in #1917)

Make the backend send Access-Control-Allow-Origin: * even when the request has no Origin header. Since the policy is * for everyone anyway, whichever client warms a cache entry now produces a copy that's valid for all callers.

• One bean in DefaultApplication.kt, no infrastructure change
• No cache fragmentation — still one cached copy per {protocol, host, query string, conference}
• Fixes the no-Origin response itself, so it's correct regardless of what the CDN does

Option 2 — add Origin to the CDN cache key

Add "Origin" to include_http_headers in the graphql backend service's cache_key_policy in backend/terraform/main.tf (currently ["conference"]), so browser and non-browser responses cache separately.

• Also a one-line change, but in terraform rather than code
• Fragments the cache per distinct origin value for no real benefit, given the policy is *
• Doesn't fix the underlying response — the no-Origin variant still lacks the header; it just stops browsers being served that variant

I recommended Option 1 because Option 2 papers over the response being wrong rather than correcting it, and it makes caching less effective. That said, they aren't mutually exclusive — if you wanted belt-and-braces, Option 2 on top would isolate browser traffic even if the origin behaviour regressed later.
`

[joreilly]

a bit more background (I asked Claude why this had started happening).....again if correct! Seemed to be triggered by deployment yesterday:

"
backend-deploy.yml runs terraform:apply before the Cloud Run steps, so the terraform breakage was silently freezing backend deployments too — not just infrastructure. Fixing it released 20 days of accumulated change in one go.

What that deploy carried. The backend's own source is byte-identical between 512cf2c and 8bf79ee — the only diffs are backend/terraform/main.tf and the version catalog. The relevant catalog change is Apollo 5.0.1 → 5.1.0 (#1868, also merged yesterday).

Why that flips the bug on — two things reset the CDN key space simultaneously:

1. A fresh Cloud Run revision starts with an empty InMemoryPersistedDocumentCache. Every client has to re-register its hashes, so the whole APQ hit/miss pattern restarts from zero.
2. Apollo 5.1.0 changed the request URL. Its requests carry "clientLibrary":{"name":"apollo-kotlin","version":"5.1.0"} in the extensions — the older deployed bundle's requests don't. The CDN cache key includes the query string, so 5.1.0 clients hash to entirely new cache entries.

Result: every entry got re-warmed from scratch yesterday, by whichever client asked first. The Android and iOS apps poll constantly and never send Origin, so they won most of those races — writing header-less copies that browsers then can't read.

Before yesterday, the key space had been stable for weeks, and entries that happened to be warmed correctly kept getting refreshed in a working state.

Confidence: the mechanism is measured and certain. The trigger — redeploy plus key-space reset — is strongly supported by the timeline and the observed URL difference, but I can't prove which client warmed which entry historically, since Cloud CDN doesn't expose that.

Two implications worth noting:

• The bug was latent for as long as APQ + CDN + this CORS filter have coexisted (the CDN config dates to 2024). It needed a cache-key reset plus unlucky warming to surface. It could have appeared at any earlier redeploy, and would likely recur on future ones even without my changes.
• It also means Fix Backend Deploy: pin google-beta and set the LB scheme explicitly #1908 didn't break anything — it revealed this by restoring a deployment pipeline that had been silently stuck. Worth knowing that the backend was running three-week-old code the whole time.
• "

[martinbonnin]

Wow, thanks for opening that. TIL 👍

[martinbonnin]

I have just invalidated the whole CDN
but not sure it'll save us a lot of time compared to just waiting for regular expiration (~30min).