Repository navigation
feat: Run Roach as a remote proxy on GCP with GCS recordings - #1
Merged
Merged
Conversation
dcramer
marked this pull request as ready for review
October 9, 2026 18:31
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
![sentry-warden[bot]](https://avatars.githubusercontent.com/in/2755838?s=60&v=4){kind=small}
sentry-junior
Bot
force-pushed
the
poc/remote-service
branch
2 times, most recently
from
October 9, 2026 18:46
5811a2c to
c694a42
Compare
Replace the shared Node proxy service with a remote store. Each CI run keeps its own local proxy, and only the recordings move to a Worker. This needs no raw TCP ingress and no shared HTTPS interception. The recorder now uses a RecordingStore: the file store (as before) or the remote store, which calls the Worker. The Worker keeps each recording in R2 and one D1 row with its last use and the hash of each request part. Miss diagnosis no longer reads every recording on the first miss. The Worker finds the closest recording with one indexed D1 query on the part hashes, and returns the parts that differ. Recordings expire by last use, not by upload: a replay refreshes the row at most once a day, and a daily cron deletes rows and objects that nobody used for RECORDING_TTL_DAYS. Tenants have hashed tokens and see only their own recordings. tests/worker.test.ts runs the real wrangler.jsonc with local R2 and D1 through createTestHarness: record and replay through a proxy, misses, tenant isolation, size limits, and expiry. Co-Authored-By: David Cramer <david@sentry.io>
sentry-junior
Bot
force-pushed
the
poc/remote-service
branch
from
October 9, 2026 19:03
c694a42 to
693377f
Compare
sentry-junior
Bot
changed the title
A failed session lists the keys that it recorded before. keysOf read every entry of the recordings directory as a rule directory, so a file such as .DS_Store failed with ENOTDIR and ended the session with HTTP 409. It now reads only directories.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
The Worker now keeps recordings only in R2. A 30-day R2 lifecycle rule expires them, so the D1 index, the closest route, and the cron trigger are gone. The closest recording was only a hint to debug a miss; the file store still gives it. Each read and write sends the Sentry metric roach.recording with the tenant, rule, key, run, and result. The proxy sends its run in the X-Roach-Run header from the new store.run config. Grouping by key and run shows recordings that only one run uses. Co-Authored-By: David Cramer <david@sentry.io>
Used files list keys with '/', but readdir returns paths with the platform separator. On Windows every recording looked unused and prune deleted all of them.
Move the part comparison back into request-key.ts, since the Worker no longer imports it. RequestParts moves to store.ts so the Worker can still typecheck store.ts without Node types. Drop the atomic file writes, which no reader needed. Fix comments and docs that still described the D1 index and expiry by last use. Co-Authored-By: David Cramer <david@sentry.io>
sentry-junior
Bot
changed the title
Roach is now a hosted man-in-the-middle proxy. Clients point HTTPS_PROXY at it and trust its CA; they install nothing else. Cloudflare Workers cannot accept CONNECT, so the Worker store is gone. The service holds many runs at once. Each proxied request names its run in Proxy-Authorization. Replay runs are public, so fork CI can replay; any mode that writes needs the tenant token. Recordings live in a GCS bucket per tenant prefix and expire 30 days after they are written. Each read and write sends a roach.recording count to Sentry with the tenant, key, run, and result. deploy/gcp is the production setup: a GCS bucket with the TTL rule, the CA and tenant tokens, the service config in Secret Manager, one COS VM, and an SSL proxy load balancer with a managed certificate. The Image workflow builds the container and pushes it to ghcr on main, and runs terraform validate. A request body over 64 MiB now gets a 413 instead of a reset socket. Co-Authored-By: David Cramer <david@sentry.io>
The 64 MiB limit also applied to recorded upstream responses, so a large response failed with a 413 that blamed the client. Only client request bodies now have the limit.
The control API sent 500 for every error, also for a body over the limit or a body that is not JSON. It now sends the status of the error, with its message. Any other error is still a 500.
sentry-junior
Bot
changed the title
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 5e48d71. Configure here.
Anyone can start a replay run without a tenant token. A request that no rule matched still went live, so such a run was an open proxy to every allowed origin. A run without the token now refuses those requests with 403, so it never sends anything live.
Anyone can start a replay run without a token, and each run stays in memory for up to 6 hours. So repeated calls could fill the memory of the shared process. A tenant can now have 50 such runs open; another one gets HTTP 429 until one ends. Runs with the tenant token have no cap.
A body such as `null` parsed as JSON, and the caller then read a field of it. That threw a TypeError, so the control API answered 500. A body that is not a JSON object now gets HTTP 400.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Run Roach as a hosted man-in-the-middle proxy. Junior and other projects point
HTTPS_PROXYat it and trust its CA certificate. They install nothing else.Cloudflare Workers can't accept an incoming
CONNECT, so the service runs on one GCP VM behind an SSL proxy load balancer. Recordings go to GCS. This replaces the R2 Worker store from earlier commits on this branch.How it works
POST /__roach/runsand gets back a run with its own id and token. Each proxied request names its run inProxy-Authorization, so many CI jobs can share the service at once.replayrun, so fork PRs can replay. Every other mode writes, so it needs the tenant's token. The service checks this, not the CI workflow. The config stores only the SHA-256 of each token.<tenant>/in the bucket. A GCS lifecycle rule deletes each one 30 days after it was written.roach.recordingcount to Sentry, withtenant,rule,key,runandresult. Group bykeyand countrunto find recordings that only one run uses.Production setup
deploy/gcp/is Terraform for everything: the bucket and its expiry rule, the CA, one token per tenant, the service config in Secret Manager, a Container-Optimized OS VM, and the load balancer with a Google-managed certificate.roach.tfvars.examplealready lists Junior's origins and value patterns. The README "Deploy" section has the steps. The newImageworkflow builds the image on PRs and pushesghcr.io/getsentry/roachonmain. It also runsterraform validate.Tests
tests/service.test.ts: tenants, public replay, concurrent runs, redaction, the body limit on requests but not on responses, and 400/413 from the control API.tests/deployed.test.ts: runs the service the way production does. It starts from the CLI with a config file and a fixed CA, behind a TLS front end like the load balancer. A fake GCS server, metadata server and Sentry server stand in for the real ones. Child processes set only the run's proxy variables. It checks record with a token, replay for a fork, and the Sentry counts.Known limits
replaymode, a request for an expired recording fails.terraform applyagainst a real project.via David Cramer.
--
View Junior Session [Sentry]