Repository navigation
Conversation
…SQLite-safe) First phase of the SQLite -> Supabase Postgres migration: dialect-neutral rewrites that run identically on SQLite >=3.35. - 17 cursor.lastrowid reads become INSERT ... RETURNING id fetches (users.db async stores x7, parses x3, zones curation x4, raids x3), each with its paired -- :name SQL block. - INSERT OR REPLACE x3 -> ON CONFLICT ... DO UPDATE (census character_aas/gear_sets, shared _meta upsert) — also fixes the delete-then-reinsert column-reset footgun. - INSERT OR IGNORE x9 -> ON CONFLICT DO NOTHING (zones x5, downloads, capped favorites, grant_role, erasure tombstone); rowcount semantics preserved where callers check them. - gitignore .mcp.json (local MCP config carries the Supabase token). No behaviour change; full suite green (2119). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…+ deps
parse_detail_views_total{age_bucket} counts detail-page opens by the
encounter's age (<7d / 7-30d / 30-90d / >90d) — the data that will
decide whether attack/damage-type detail older than N days can take a
rolling retention window on Postgres (user decision: Supabase Pro now,
revisit pruning once this histogram has months of data).
Also lands backend/pg.py (DSN resolution DATABASE_URL -> SUPABASE_DB_URL
-> POSTGRES_CONNECTION_STRING; async+sync psycopg pools opened in the
app lifespan; pool-if-open-else-direct checkout contextmanagers) and the
psycopg[binary]/psycopg-pool dependencies — the P1 connection seam.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
backend/pg_migrate.py: ordered db/migrations/*.sql applier with a public.schema_migrations ledger, advisory-locked, per-file transactions; retarget() rewrites a family file's two-line header at a scratch schema — the test-isolation mechanism (validated: users_s1 provisions all 25 tables from the same file). db/migrations/0001_users.sql: the users.db schema translated AND reviewed, not mechanically ported — identity bigints, epoch-bigint defaults, deferrable FKs to users(discord_id) (erasure becomes one SET CONSTRAINTS ALL DEFERRED transaction), attendance zones/uploaders to jsonb (exact key-exists erasure probe verified), item_watch index re-scoped to (world, guild_name), redundant token-hash index dropped, idx_claims_primary made partial, idempotent seeds section (re-run by the scratch-schema reset). Verified end-to-end against local PostgreSQL 17. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…int[] User-approved review changes: partial unique index enforcing ONE approved claim per (world, lower(character_name)) at the database level (verified: case-variant duplicate blocked with UniqueViolation; local dev data audits clean — prod audit runs at rehearsal via the copy script), and raid_slots.days becomes a real integer[] (API already speaks lists; the store/poller code change lands with the P1 raid_schedule swap; the copy script splits prod CSV). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sers schema
PgStoreBase/SchemaBound seams in db_catalogue; every store's _db() now
checks out a pooled psycopg connection scoped by SET search_path. SQL
sidecars: %s params, strftime -> extract(epoch), COLLATE NOCASE ->
lower()=lower(%s), {placeholders} IN-lists -> = ANY(%s), RETURNING ids
read as dict rows. attendance swaps BEGIN IMMEDIATE for
pg_advisory_xact_lock + jsonb zones/uploaders (Json writes, parsed
reads); recruitment logo round-trips bytea via bytes(); raid_slots.days
rides integer[] through api/raid_schedule + raid_live; raid_planning
keeps NULLS FIRST ordering parity. Route layer: role_requests 409 now
catches psycopg UniqueViolation, bot guild_context degrades on
psycopg.Error. Verified by live smoke of all stores on local PG17
(fixed two conversion bugs it caught: a dropped select_claim_user_and_world
block and %s-in-comment placeholder miscounts in claims.sql).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nsaction sqlite users.db connection replaced by pg.connection + search_path; the PRAGMA foreign_keys=OFF ordering dance becomes SET CONSTRAINTS ALL DEFERRED inside a single transaction. attendance_sessions.uploaders is probed with the jsonb ?-operator instead of LIKE and written back via Json(); per-table existence checks dropped (migrations guarantee the schema). Test seam is users_schema= (was users_path=). The parses half stays SQLite until P2 unifies both halves into one transaction. Verified by a live seed-erase-verify smoke on local PG17. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tartup App lifespan now opens the psycopg pools and applies pending migrations (run_sync keeps the runner off the loop; ledger + advisory lock make overlapping deploy containers safe), and closes the pools at shutdown. The bot's defensive users_db.init_db() is gone — pre-lifespan queries fall back to direct connections and guild_context degrades on psycopg.Error. main.py selects the selector event-loop policy on Windows before asyncio.run (psycopg async can't ride ProactorEventLoop). Verified by booting the real lifespan against local PG17: migrations, /api/health, a store call over the pool, clean pool shutdown. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…readers converted tests/fixtures/pg.py: provision_for_session points pg.dsn() at the local TEST database (never the .env Supabase DSN), creates it if missing, rebuilds session schemas from db/migrations; SchemaLeaser pools scratch schemas (users_s<pid>_N) built by retargeting the same 0001 file, reset on release with one TRUNCATE + seeds re-run; users_schema fixture + pg_conn seeding helper. point_users_db_at now takes a schema. Facade: init_db/DB_PATH and the SQLite schema/migrations/_assertions machinery deleted (tests/server/test_db_migrations.py with them). Remaining raw users.db readers converted to the pg seam: characters + guild fallback prefix searches, the character-cache prewarm, the users/claims Prometheus counts (file-size gauge drops users until the P2 pg_total_relation_size split), seed_raid_kills primary lookup. test_guild_settings.py converted as the model file — 11/11 green on PG. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Users-family tests now run against real PostgreSQL; tests/fixtures/pg.py reads TEST_DATABASE_URL and provisions schemas from db/migrations/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion runner tests/test_sql_blocks.py pins every _SQL[...] reference to an existing sidecar block (the dropped-block class) and rejects format-field comments that survive parsing inside PG-dialect block bodies (psycopg counts %s inside comments). tests/test_pg_migrate.py enforces the two-line family header convention retarget() depends on, and exercises the ledger: idempotent re-runs, per-file rollback leaving failures unrecorded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The deletions staged for the test-infrastructure commit were silently unstaged by an intervening stash cycle — landing them now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…abase provision_for_session drops + rebuilds the session and scratch schemas, so two pytest invocations sharing one database clobbered each other (roving UndefinedTable failures — the old per-PID tmpdir layout was parallel-safe, TEST-039). A process-lifetime advisory lock now queues a second session until the first exits. Also: psycopg.sql via explicit import (stubs don't re-export it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…full suite green All 25 users-family test files now run on leased scratch schemas / pg_conn seeding (three parallel conversion agents + verification pass). Highlights: PG actually enforces the users FKs SQLite never did, so store tests seed their user rows; erasure tests drive the new users_schema= seam (parses half stays SQLite); servers/xpac tests adapt to the migration-seeded Varsoon+Wuoshi baseline; the legacy SQLite-ALTER regression tests became information_schema invariants; guild_context's missing-table degrade is simulated with a nonexistent schema. Route fix the conversion surfaced: attendance list/detail still json.loads'd jsonb zones/uploaders (TypeError 500 once non-empty) — now uses the parsed values. test_raid_live fixtures carry days as int[]; the role-requests 409 test raises UniqueViolation. Full suite: 2149 passed, 0 failed, on local PostgreSQL 17. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
STATIC_PULSE_INTERVALS override in spell_effects (cleric AA — applies to Templar AND Inquisitor), applied at both rotation call sites; measured live at 7.1s on both classes. (Simulator work predating the Postgres branch; its test rode along in the suite-conversion commit.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rovisioning 0002_parses.sql: encounters/combatants/damage_types/attack_types/ ingest_log/tamper_reports translated per the type policy (identity bigints, epoch bigints, double precision, 0/1 ints, payload_json stays byte-exact text). Review items: detail_pruned_at on encounters + partial sweep index (tiered retention: curated raid 30d / group-instance 14d / other 7d); rankings cover becomes key(encounter_id, damage DESC) + INCLUDE payload with aggressive combatants autovacuum (re-benchmark at rehearsal); new (world, lower(guild_name), started_at DESC) index for the NOCASE rewrites. Test provisioning now derives family schemas from the migration headers, so new family files re-provision cleanly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pg.getconn()/putconn() give the catalogue families a caller-owned sync checkout (pool-if-open-else-direct). PgCatalogue.init_db() keeps its name but creates nothing — it hands back a PgConnProxy scoped to the family schema: sqlite3.Connection surface (execute/commit/close, txn- scope `with`), close() returns to the pool, dict rows, fetchval() for the old positional fetchone()[0] sites. Preserves the ~60 caller-owns-connection sites' shape across parses/census/zones/raids. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion helpers
db.sql is DML-only now (DDL owns 0002_parses.sql; the SQLite schema/
ALTER/rebuild machinery is deleted): named inserts use %(param)s against
the models' as_db_params dicts, IN-list {placeholders} composition is
replaced by = ANY(%s) (the 500-id chunking in get_combatants_for_
encounters is gone), instr->strpos, IFNULL->COALESCE, COUNT aliases.
db.py: PgCatalogue base (init_db returns a pooled schema-scoped proxy),
dict rows, explicit commits replace sqlite `with conn:` scopes, and the
new tiered-retention helpers (select_detail_prune_candidates /
prune_encounter_detail stamping detail_pruned_at).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…unified Every parses consumer now rides the pooled schema-scoped proxy: parses package routes (list/ingest/delete/tamper/export), rankings' kills rebuild, zones progress stitches (zones.db side stays SQLite), admin sanitize/tamper panels, attendance reconstruct (COLLATE NOCASE -> lower() on the new index), bot parse_posts, dev seed scripts. path guards deleted; IN-list chunking -> ANY; instr->strpos; IFNULL-> COALESCE; dict rows throughout; PG requires the derived-table alias SQLite tolerated in list_encounters_recent. cleanup.py gains the tiered detail-retention sweep (user decision): curated raid 30d / curated group-instance 14d / other 7d via _classify_zone; attack_types+damage_types dropped, detail_pruned_at stamped, encounters+combatants forever. Erasure is now ONE transaction spanning users+parses schemas (search_path switch mid-txn). Metrics: parses counts via PG + a pg_schema_size_bytes gauge per family (the Supabase tier-gate measurement). PgConnProxy.close() rolls back implicit read transactions (silences psycopg_pool's warning). Parses test fixtures lease scratch schemas from 0002. Verified by a live store + tiered-retention smoke and the app lifespan boot on local PG17. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ail-pruned notice 0003_census.sql: reviewed census schema (jsonb data_json, text_pattern_ops prefix indexes, DISTINCT ON latest-member-counts index). CensusStore on PgCatalogue: Json() writes, parsed jsonb reads (per-row json.loads gone, cls extracted in SQL), DISTINCT ON replaces the bare-column-MAX idiom; path guards dropped at the five census consumers. census_schema leased fixture + conftest rewiring. Erasure reads the parses schema at call time (test-pointable symmetry). All 21 parses test files converted by three agents + verified: schema-shape tests became information_schema invariants, ingest's shared-connection tricks dropped (READ COMMITTED), delete's ANY() wire shape asserted. New tests/server/parses/ test_detail_retention.py pins the 30/14/7 tier sweep. ParsePage shows a "breakdown pruned" notice when detail_pruned_at is set (450/450 vitests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
zones: builder-assigned ids stay plain bigint PKs; curator-managed encounter/mob rows get identity ids; featured-raid trio + _meta ported; flags stay 0/1 integers. raids: all ids identity (scrape+curator inserts); markdown blobs stay text; ACT trigger/spell-timer tables ported column-for-column (absolute_/fill_color quirks kept); strftime defaults -> extract(epoch). Both verified to apply cleanly via retarget to throwaway schemas on local PG17. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scripts/migrate_to_postgres.py: per family one transaction (TRUNCATE RESTART IDENTITY CASCADE -> COPY in topo-sorted FK order -> setval identities -> verify counts -> COMMIT). Columns/types introspected from information_schema; transforms by type (days CSV -> integer[], jsonb validated + shipped as text, bytea passthrough). SQLite-side FK orphan anti-joins quarantine rows to reports/ and fail without --allow-orphans; duplicate approved claims abort before any write; the tiered detail retention is applied AT LOAD (attack/damage_types rows past their zone tier never reach Postgres; encounters stamped detail_pruned_at) — the step that turns 6.8GB of SQLite into the tier-gate measurement. Target host confirmation required (--yes to skip). JSON run report. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… suite green Eleven files converted by two agents + verified: census_schema replaces every store.path/DB_PATH re-point (the shared instance re-point makes route patches unnecessary), schema-shape tests became information_schema invariants, top_n seeds satisfy the now-enforced combatants FK via stub encounters, dead parses_db.path patches dropped from admin tests. Full suite: 2153 passed, 0 failed on local PostgreSQL 17. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…g_dump 0006_lockdown (-- global): RLS enabled on every family table + guarded revokes from Supabase's anon/authenticated — the auto REST API can't read the data even though public stays the only exposed (and empty) schema. railway.toml startCommand drops litestream entirely (deleted litestream.yml): mkdir only the read-only mirror dirs, exec main.py; lifespan migrations make deploy = migrate. .env.example: DATABASE_URL chain + TEST_DATABASE_URL documented; retired DB path vars removed. New nightly pg-backup workflow: pg_dump -Fc of the five schemas to the repurposed R2 bucket with 30-day in-action pruning. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ipts ZoneCatalogue/RaidCatalogue on PgCatalogue (schemas zones/raids); the act/* and raid_strategies routes, rankings' zones reads, api/zones mob-title matching (= ANY replaces the chunked IN loop), metrics raids counts (the _DBCollector sqlite conn-cache is gone — nothing used it), and the dev scripts all ride the pooled proxy. build_zones_db.py now upserts zones by stable id and prunes removed names with a printed cascade warning, so metadata refreshes become a script run against DATABASE_URL and curator boss rosters survive (the manual-volume-upload workflow dies). NULLS FIRST/LAST pinned where SQLite and PG disagree (featured lanes, expansion_year sorts). 0004 fix: featured_raid_zones position/category (post-launch SQLite ALTERs my translation missed — caught by the conversion agent). app.py startup parses checkout now closes (leaked one sync-pool slot). SchemaBound gains the no-op clear_caches for PathBound parity. Verified by zones/raids live smokes, repo-wide pyright 0 errors, and the app lifespan boot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…est fixtures Key-files table, env vars (DATABASE_URL chain, retired DB_*_PATH rows, R2 as Actions secrets), backups (nightly pg_dump + restore drill), zone-refresh and raid-seed workflows (script runs against DATABASE_URL — the manual volume uploads are gone), erasure/cleanup/store rows updated. tests/fixtures/zones_raids_db.py: leased zones_schema/raids_schema fixtures, re-exported via conftest. app.py: the startup parses checkout closes (pooled-slot leak the zones agent caught). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…LIES GREEN Twelve files converted by two agents + verified: catalogue-base tests re-based onto a still-SQLite catalogue (RecipeCatalogue) so the SQLite bases stay covered for items/spells/recipes; retired one-time SQLite fixups (zones name normalisation, ACT editor-parity backfill) become descendants pinning their surviving invariants (init_db never mutates rows; legacy alias resolution); "missing DB file" soft-degrades become the documented UndefinedTable contract; explicit identity ids use OVERRIDING SYSTEM VALUE. Full suite: 2153 passed, 0 failed on local PostgreSQL 17 — users, parses, census, zones and raids all on Postgres. ruff + pyright clean, app lifespan boot green, frontend tsc clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…needs it The detail-retention tiers classify each encounter's zone via _classify_zone, which reads the zones schema; copying parses first would classify every kill as "other" (7-day tier) against stale/empty zones data and over-prune curated raid detail at load. Also gitignore /restore/ and /reports/ (restored prod replicas + orphan quarantines — real user data, never committed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…6s -> 0.9s Rehearsal #1 against real prod data (15k winning encounters) exposed a per-encounter zones_db.find_by_name round trip in _resolve_boss (and the same latent pattern in _classify_zone's miss path) — invisible under in-process SQLite, 18ms each on Postgres. One query now builds a process-cached {name_lower/alias_lower: canonical} map (ZoneCatalogue.canonical_name_map), consumed by both sites and cleared by invalidate_zones_cache with the other zone caches. End-to-end Wuoshi rebuild on the real 6.8GB dataset: 10,212 kills in 0.9s. Tests re-point their fakes at the new seam. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each rebuild reads ~40 MB of combatants from Postgres (measured on prod data); a 60s SWR ttl under constant traffic could exceed Pro's 250 GB/month egress. At 10 min: worst-case ~175 GB, realistic 30-60 GB incl. nightly pg_dump. SWR semantics unchanged — boards still serve instantly; fresh uploads appear within <=10 min (user-approved trade). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plain `uvicorn backend.server.app:app` on Windows (uvicorn 0.52 picks Proactor without --reload) left the async pool unable to create any connection — every checkout timed out after 30s and the site rendered as a blank page. open_pools now raises a crisp error naming the three working entry points instead. Production Linux unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh planner statistics at commit time — the first post-cutover queries (the rankings rebuild over millions of just-loaded rows) must not plan against empty stats while autovacuum catches up. (supabase-postgres-best-practices: monitor-vacuum-analyze.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The single family-wide transaction accumulated WAL across the whole 3.25M-row parses load and filled a freshly-upgraded Supabase project's small provisioned disk mid-COPY (DiskFull at attack_types row 1.34M). Per-table commits let checkpoints recycle WAL between tables; idempotency is unchanged (every run starts by truncating the family, and within-family FK order keeps committed parents valid). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolutions: litestream.yml stays deleted (the census-replication change it carried lives on main, where it already deployed and did its job — the census replica is in R2); railway.toml keeps the Postgres start command. Merged lockfile verified: multidict 6.9.1 (CVE-2026-104874), fastapi 0.142.2, python-dotenv 1.2.4, ruff 0.16.10 alongside psycopg. Full suite on the merge: 2153 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Moves the five writable data families (users / parses / census / zones / raids) from Railway-PVC SQLite to one Supabase Postgres database with one schema per family.
backend/pg.py(lifespan-owned psycopg pools, schema viaSET search_pathat checkout) andbackend/pg_migrate.py(raw-SQL runner overdb/migrations/, deploy = migrate)PgStoreBase/PgCataloguebases —init_db()keeps its name but returns a pooled schema-scoped connection proxy; thebackend.server.dbfacade re-exports are unchangedtests/fixtures/pg.pyscratch-schema leases; CI gains a postgres:17 servicepg_dump→ R2 via GitHub Actions (pg-backup.yml)Merging this to main is the production cutover (env flip + final data copy happen in the maintenance window per the runbook).
🤖 Generated with Claude Code