Skip to content

Add operator-only Convex function to clear app and Better Auth tables - #43

Merged
ethan-br merged 1 commit into
mainfrom
t3code/add-convex-clear-all-tables
Sep 27, 2026
Merged

ethan-br merged 1 commit into
mainfrom
t3code/add-convex-clear-all-tables

Conversation

@ethan-br

Copy link
Copy Markdown
Owner

Summary

  • Add an internal Convex action that deletes every app table and all ten Better Auth component tables in bounded batches.
  • Require an explicit DELETE_ALL_DATA argument and return deletion counts by table.
  • Document the operator command and verify batching, complete coverage, and safe reruns.

Checks

  • npm run check (Chromium and Firefox builds; 165 tests passed)

The reset is intentionally destructive and is only callable by a deployment operator through npx convex run.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

The reset is correct in shape and the test is strong, but it cannot clear a deployment that holds real bookmark data: batchSize bounds each mutation by document count, not bytes, and collections/backups embed whole snapshot trees.

Reviewed changes

  • convex/maintenance.ts (new) — adds clearAll, an internalAction gated on confirm: v.literal('DELETE_ALL_DATA'), that walks every table in convex/schema.ts through a new clearAppBatch internalMutation and then all ten Better Auth component tables through components.betterAuth.adapter.deleteMany, returning per-table deletion counts.
  • tests/maintenance.test.ts (new) — one convex-test end-to-end test that seeds app rows and 101 backups / 101 rateLimit rows, asserts the exact counts, pins the reported key set against schema.tables plus betterAuth.schema.tables, checks every table is empty afterwards, and asserts a rerun reports all zeros.
  • docs/auth-setup.md — new "Clearing a deployment" section with the npx convex run maintenance:clearAll '{"confirm":"DELETE_ALL_DATA"}' command, deployment-selection flags, the destructive-scope list, and the "stop clients from writing" / "rerun to finish" guidance.
  • convex/_generated/api.d.ts — registers the new maintenance module in the generated API surface, correctly sorted between http and sync.

⚠️ A batch of 100 snapshot documents can blow Convex's 16 MiB transaction read limit

clearAppBatch deletes up to batchSize documents per mutation, and collections / backups each embed a complete bookmark tree, so the batch is bounded by count while the limit is by bytes. This fails deterministically on the same documents, which means the documented "run the same command again to finish" recovery never converges.

Technical details
# Bound the reset batch by bytes, not document count

## Affected sites
- convex/maintenance.ts:62 — `ctx.db.query(table as TableNames).take(batchSize)` reads up to 100 whole documents in one mutation.
- convex/maintenance.ts:7 — `const batchSize = 100;`
- packages/model/index.ts:65 — `MAX_NODES = 2000`, and `validateTree` allows up to 600 KB of `JSON.stringify(nodes)` per snapshot, so a single `collections`/`backups` document can be ~0.6 MB.
- convex/sync.ts:92, convex/sync.ts:274 — both `backups` inserts store the full `nodes` array, and nothing prunes the table, so the oldest snapshots (the largest) sit in the first batch.
- docs/auth-setup.md:73 — "if it is interrupted, run the same command again to finish" is not true for a limit error; the same first 100 documents fail again every time.
- tests/fixtures/auth.ts:7 — `convexTest(schema, modules)` with no `transactionLimits`, so no test in the repo can currently observe a Convex transaction limit.

## Reproduction
Convex caps a transaction at 16 MiB of data read (docs.convex.dev/production/state/limits);
16,777,216 / 100 documents is ~168 KB average per document, roughly 600 bookmarks.
Under `convexTest({ schema, modules, transactionLimits: true })` with the Better Auth
component registered, inserting 100 `backups` of 700 nodes each (~206 KB per document,
well inside the app's own 2,000-node / 600 KB cap) and then calling
`internal.maintenance.clearAppBatch` with `{ table: 'backups' }`:

    clearAppBatch threw: Error: Read too much data in a single function execution
    (limit: 16777216 bytes). This is a Convex limit: https://docs.convex.dev/production/state/limits

## Blast radius
`collections`, `devices` and `operations` are cleared before `backups`, so the reset aborts
with the first three tables already gone — a half-cleared deployment, and the operator is
left with the first 100 backup documents undeletable by this tool.

## Required outcome
- `clearAppBatch` must stop taking documents once the transaction is close to Convex's
  16 MiB read budget, so it still makes progress on any mix of tiny and 600 KB documents
  and still returns a non-zero count while rows remain.
- Add a regression test that observes the limit. It needs
  `convexTest({ schema, modules, transactionLimits: true })`; the shared fixture in
  `tests/fixtures/auth.ts` builds the instance without limits, so the test either needs
  its own instance or the fixture needs an option.
- Correct the recovery sentence in `docs/auth-setup.md:73` so it does not promise that a
  rerun fixes every kind of interruption.

## Suggested approach
Track the bytes as you delete and return early, e.g. accumulate
`getConvexSize(doc)` (or read `ctx.meta.getTransactionMetrics().bytesRead.remaining`) and
break out of the loop before exceeding a budget well under 16 MiB — the outer
`do { ... } while (batch > 0)` in `clearAll` then keeps draining the table. A budget of a
few MiB still allows at least one document per call given the 1 MiB per-document cap, so
the loop cannot stall.

## Open questions for the human
- Is this tool expected to work on production data, or only on throwaway dev deployments?
  If only dev, a note in `docs/auth-setup.md` about the size limit may be enough instead
  of a byte-aware batch.

ℹ️ Nitpicks

  • docs/auth-setup.md:73 lists what the reset removes but skips verification, twoFactor and the four OAuth tables, so the enumeration reads as exhaustive. "accounts, sessions, verification and two-factor records, OAuth rows and rate-limit data" would match what the action actually clears.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using Big Pickle (free) | 𝕏

Comment thread convex/maintenance.ts
args: { table: v.string() },
handler: async (ctx, { table }) => {
if (!(table in schema.tables)) throw new Error(`Unknown app table: ${table}`);
const docs = await ctx.db.query(table as TableNames).take(batchSize);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

batchSize bounds this mutation by document count, but collections and backups each embed a whole snapshot tree, so the batch is bounded by bytes in practice. packages/model's own validateTree allows 2,000 nodes / 600 KB per snapshot, and Convex caps a transaction at 16 MiB read — about 168 KB per document at 100 docs, so any deployment whose snapshots average ~600 bookmarks makes this call throw Read too much data in a single function execution.

Two consequences beyond the throw: collections/devices/operations are already gone by then, so the reset stops half-finished, and the same first 100 documents fail on every rerun, which makes the documented "run the same command again to finish" recovery in docs/auth-setup.md:73 unreachable. Bound the batch by bytes instead (details in the review body).

@ethan-br
ethan-br merged commit 46a46db into main Sep 27, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant