Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .changeset/member-directory-auth-cutover.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
"@executor-js/cloud": patch
"@executor-js/api": patch
"@executor-js/host-selfhost": patch
---

Cloud now authorizes every protected request against the local membership mirror through the shared `MemberDirectory` seam: the per-request org membership check, the admin gates on the account and admin planes, the org switcher's organization list, and the free-organization limit all read the mirror instead of calling WorkOS. WorkOS is now a write target and an event source only. The seam gains `membershipsOf(accountId)` and `membershipById(organizationId, membershipId)` on both hosts.

The mirror is trusted only while it is **ready**: the backfill has written every organization and the Events reconciler has drained the stream within the last ten minutes (both recorded on the `workos_sync` row). Until then the membership check falls back to WorkOS, exactly as before, so a member the backfill has not written yet is not locked out and a member revoked while the reconciler was down is not let in. The deploy runs `scripts/ensure-workos-mirror-ready.ts` after the migrations: it runs the backfill if needed, drains the events stream itself if the reconciler has not recently (so the gate never waits on a cron this same deploy ships), and fails the deploy if the mirror is still not ready. An organization the mirror does not hold at all (one that predates the mirror and nobody has signed in to since) is resolved from WorkOS on demand for a caller WorkOS confirms as its member, so CLI and MCP tokens naming such an organization are not refused. Deleting an organization now cancels billing before deleting the WorkOS organization, and a retry after a partial deletion is admitted from the mirror even while the mirror is not ready.

**Ops step (cloud):** add the `WORKOS_API_KEY` secret to the `production` GitHub environment so the deploy gate can run the backfill.
10 changes: 10 additions & 0 deletions .changeset/member-directory-readers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
"@executor-js/cloud": patch
"@executor-js/api": patch
"@executor-js/react": patch
"@executor-js/sdk": patch
---

Member lists, the admin users page, and seat counts on cloud now read from the local membership mirror through the shared `MemberDirectory` seam instead of fanning out one WorkOS read per member. The admin users page gains an email/name search.

**Deploy prerequisite (cloud):** `bun run --cwd apps/cloud db:backfill-workos-mirror:prod` must complete before this build is deployed, and its printed membership count should match WorkOS. Until the backfill has stamped the mirror's marker, seat reporting to Autumn is skipped with a warning (never a partial count) and member lists show only members who have signed in since the mirror shipped.
7 changes: 7 additions & 0 deletions .changeset/member-directory-reconciler.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"@executor-js/cloud": patch
---

The cloud membership mirror is now reconciled from the WorkOS Events API: an every-minute cron replays user, organization-membership, and organization events from a persisted cursor, so changes made in the WorkOS dashboard (a removed member, a role edit, a profile update) reach the mirror without anyone signing in. A signed webhook at `/api/webhooks/workos` pokes the same reconciler so those changes land in seconds, and `bun run --cwd apps/cloud db:drain-workos-events:prod` runs the same replay out-of-band until the stream is drained.

**Ops steps (cloud):** set the webhook signing secret with `wrangler secret put WORKOS_WEBHOOK_SECRET`, then register `https://executor.sh/api/webhooks/workos` as a webhook endpoint in the WorkOS dashboard for the `user.*`, `organization_membership.*`, `organization.updated`, and `organization.deleted` events. Until the secret is set the route answers 503 and the cron alone keeps the mirror current.
5 changes: 5 additions & 0 deletions .changeset/mirror-readiness-removal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@executor-js/cloud": patch
---

`authorizeOrganization` now reads the local membership mirror unconditionally: the per-request readiness check (`MirrorReadiness`) and its live WorkOS `listUserMemberships` fallback are gone from the request path entirely. The backfill is complete and permanent, and an organization that predates the mirror is still covered by the existing on-demand scan (`ensureOrganizationBackfilled`). A stalled reconciler is now an operational alert instead of a per-request fallback: after each run, the cron checks the mirror's `drained_at` heartbeat and, if it has fallen behind the lag budget, logs a structured error and reports it to Sentry. The deploy gate (`scripts/ensure-workos-mirror-ready.ts`) is unchanged — it still refuses to ship while the mirror is unready — and `drained_at` keeps being written by every reconciler run.
5 changes: 5 additions & 0 deletions .changeset/oauth-background-resource-lifetime.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@executor-js/api": patch
---

Keep request resources alive until background OAuth tool discovery finishes, so slow cloud connections can publish their tools after the callback returns.
5 changes: 5 additions & 0 deletions .changeset/restart-hosted-invitation-login.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@executor-js/cloud": patch
---

Restart hosted invitation logins that return without state, while keeping authorization codes bound to the browser that started the login.
28 changes: 28 additions & 0 deletions .claude/skills/prod-telemetry/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,34 @@ join the same traces via traceparent).
`execute`/`execute-action` calls `mcp.execute.code` (the script itself,
capped at 10k chars — cloud-only content capture; local/self-host
telemetry never records content).
- `auth.authorize_organization` — every membership authorization. Reads the
local membership mirror unconditionally; there is no per-request readiness
check and no WorkOS fallback, so this span carries no readiness attribute.
The mirror's write spans are `workos_mirror.<op>`; the reconciler run is
`workos_events.sync`. `workos_sync.drained_at` in the prod DB is the
reconciler heartbeat, and a stalled reconciler now raises its own error
from the cron (see below) rather than showing up as a fallback here.

**Recipe — reconciler heartbeat (ticks should land roughly every minute; a
gap wider than the 10-minute lag budget means the cron alert should already
have fired — see `workos_events: reconciler stale` below):**

```apl
['executor-cloud']
| where _time > ago(1h) and name == "workos_events.sync"
| summarize n = count() by bin(_time, 1m)
| sort by _time desc
```

**Recipe — stale-reconciler alerts (should be empty; each row is one paging
event):**

```apl
['executor-cloud']
| where _time > ago(1d) and ['status.message'] contains "workos_events: reconciler stale"
| project _time, trace_id, msg = tostring(['status.message'])
| sort by _time desc
```

**Recipe — error signatures by class (the daily-digest query):**

Expand Down
13 changes: 13 additions & 0 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,19 @@ jobs:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}

# The build below authorizes every request from the local membership
# mirror. This runs the mirror backfill if it has not completed, drains
# the WorkOS events stream itself if the reconciler has not recently
# (it does not wait on the cron, which this same deploy may be the one
# to ship), and FAILS the deploy if the mirror is still not ready — see
# scripts/ensure-workos-mirror-ready.ts.
- name: Backfill and verify the membership mirror
run: bun run scripts/ensure-workos-mirror-ready.ts
working-directory: apps/cloud
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}
WORKOS_API_KEY: ${{ secrets.WORKOS_API_KEY }}

deploy-cloud:
name: Deploy cloud
runs-on: blacksmith-4vcpu-ubuntu-2404
Expand Down
30 changes: 30 additions & 0 deletions apps/cloud/drizzle/0018_member_directory_mirror.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
CREATE TABLE "membership_tombstones" (
"membership_id" text PRIMARY KEY NOT NULL,
"account_id" text NOT NULL,
"organization_id" text NOT NULL,
"deleted_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
CREATE TABLE "workos_sync" (
"id" text PRIMARY KEY NOT NULL,
"cursor" text,
"updated_at" timestamp with time zone DEFAULT now() NOT NULL
);
--> statement-breakpoint
ALTER TABLE "accounts" ADD COLUMN "email" text;--> statement-breakpoint
ALTER TABLE "accounts" ADD COLUMN "first_name" text;--> statement-breakpoint
ALTER TABLE "accounts" ADD COLUMN "last_name" text;--> statement-breakpoint
ALTER TABLE "accounts" ADD COLUMN "avatar_url" text;--> statement-breakpoint
ALTER TABLE "accounts" ADD COLUMN "workos_updated_at" timestamp with time zone;--> statement-breakpoint
ALTER TABLE "accounts" ADD COLUMN "last_sign_in_at" timestamp with time zone;--> statement-breakpoint
ALTER TABLE "memberships" ADD COLUMN "membership_id" text;--> statement-breakpoint
ALTER TABLE "memberships" ADD COLUMN "role" text DEFAULT 'member' NOT NULL;--> statement-breakpoint
ALTER TABLE "memberships" ADD COLUMN "status" text DEFAULT 'active' NOT NULL;--> statement-breakpoint
ALTER TABLE "memberships" ADD COLUMN "workos_updated_at" timestamp with time zone;--> statement-breakpoint
ALTER TABLE "memberships" ADD COLUMN "deleted_at" timestamp with time zone;--> statement-breakpoint
ALTER TABLE "membership_tombstones" ADD CONSTRAINT "membership_tombstones_account_id_accounts_id_fk" FOREIGN KEY ("account_id") REFERENCES "public"."accounts"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
ALTER TABLE "membership_tombstones" ADD CONSTRAINT "membership_tombstones_organization_id_organizations_id_fk" FOREIGN KEY ("organization_id") REFERENCES "public"."organizations"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
CREATE INDEX "membership_tombstones_organization_id_idx" ON "membership_tombstones" USING btree ("organization_id");--> statement-breakpoint
CREATE INDEX "accounts_email_lower_idx" ON "accounts" USING btree (lower("email"));--> statement-breakpoint
CREATE UNIQUE INDEX "memberships_membership_id_unique" ON "memberships" USING btree ("membership_id");--> statement-breakpoint
CREATE INDEX "memberships_organization_id_idx" ON "memberships" USING btree ("organization_id");
21 changes: 21 additions & 0 deletions apps/cloud/drizzle/0019_workos_mirror_sync_state.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
-- Backfill completeness per organization (`organizations.backfilled_at`: when
-- its membership list was last fully scanned from WorkOS), the organization
-- tombstone (`organizations.deleted_at`: kept by the local purge so a delayed
-- login cannot re-mint a deleted organization), the organization name stamp
-- (`organizations.workos_updated_at`: a name write stamped earlier is refused,
-- so a delayed login cannot revert a rename), and on the "events" row of
-- `workos_sync` the Events API replay boundary (`range_start`) the
-- reconciler's first run reads from plus the backfill completion mark
-- (`backfill_completed_at`) the authorization path checks before it trusts
-- the mirror over WorkOS. A database with no organizations has nothing to
-- backfill, so seed both there (fresh dev, test, and e2e databases); a
-- database that already holds organizations gets them from the backfill
-- script (scripts/backfill-workos-mirror.ts).
ALTER TABLE "organizations" ADD COLUMN "backfilled_at" timestamp with time zone;--> statement-breakpoint
ALTER TABLE "organizations" ADD COLUMN "deleted_at" timestamp with time zone;--> statement-breakpoint
ALTER TABLE "organizations" ADD COLUMN "workos_updated_at" timestamp with time zone;--> statement-breakpoint
ALTER TABLE "workos_sync" ADD COLUMN "range_start" timestamp with time zone;--> statement-breakpoint
ALTER TABLE "workos_sync" ADD COLUMN "backfill_completed_at" timestamp with time zone;--> statement-breakpoint
INSERT INTO "workos_sync" ("id", "cursor", "range_start", "backfill_completed_at", "updated_at")
SELECT 'events', NULL, now(), now(), now()
WHERE NOT EXISTS (SELECT 1 FROM "organizations");
1 change: 1 addition & 0 deletions apps/cloud/drizzle/0020_workos_sync_drained_at.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE "workos_sync" ADD COLUMN "drained_at" timestamp with time zone;
Loading
Loading