Skip to content

feat(intents,solvers,archival): token FKs, replica routing, cold storage, keyset pagination - #578

Merged
NanaKhadija1980j merged 2 commits into
stellar-vortex-protocol:mainfrom
Kaminar-i:feat/410-411-412-413-token-fk-replicas-pagination-archival
Oct 2, 2026
Merged

NanaKhadija1980j merged 2 commits into
stellar-vortex-protocol:mainfrom
Kaminar-i:feat/410-411-412-413-token-fk-replicas-pagination-archival

Conversation

@Kaminar-i

@Kaminar-i Kaminar-i commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Closes #410
Closes #411
Closes #413
Closes #412

This PR delivers four high-priority backend improvements: normalizing intent token data into foreign keys, adding read-replica routing for read-heavy endpoints, building cold-storage archival for terminal intents, and standardizing keyset pagination across the remaining list endpoints.


#410: Normalise intent token data into foreign keys

Problem: Intents stored srcToken/dstToken as JSONB blobs, duplicating registry data, resisting indexing, and allowing references to tokens that don't exist or whose decimals have since changed.

Changes:

  • Added src_token_id/dst_token_id foreign keys on intents, plus immutable src_decimals/dst_decimals snapshot columns captured at creation.
  • Added a backfill that resolves existing JSON blobs to tokens via (address, chain); unresolved rows are reported rather than dropped.
  • Kept the API response shape unchanged by joining at read time.
  • Added an index supporting "volume by token by day" queries, benchmarked under 100 ms on 5M intents.
  • Left the JSON columns in place, to be removed in a follow-up contract migration.

#411: Read-replica routing for read-heavy endpoints

Problem: Solver polling of /intents/open, leaderboard, and stats traffic was competing with latency-critical accept/fill writes on the primary database.

Changes:

  • Added DATABASE_REPLICA_URLS config; PrismaService now exposes both primary and replica clients.
  • Added a @ReadReplica() decorator (and repository-level option) to select the client per query.
  • Added replica-lag measurement via pg_last_xact_replay_timestamp; replicas beyond MAX_REPLICA_LAG_MS are bypassed in favor of the primary.
  • Added read-your-writes support: requests carrying a recent-write token read from the primary.
  • Added a docker-compose profile with a primary + streaming replica for CI integration testing.

#413: Cold-storage archival of terminal intents

Problem: INTENT_RETENTION_DAYS existed, but deleting old data destroyed protocol history needed for analytics, disputes, and governance, while keeping everything hot bloated the primary.

Changes:

  • Added a daily archival job that exports terminal intents (filled/cancelled/expired/slashed) older than retention, along with their audit entries, to partitioned Parquet files (date=YYYY-MM-DD/).
  • The job writes a manifest with row counts and SHA-256 checksums, and only deletes from Postgres after the upload is verified.
  • Added scripts/restore-archive.ts --date to restore an archived date into a staging schema.
  • Made the bucket/endpoint configurable, and verified it works against MinIO locally.
  • Used a streaming, cursor-based export to avoid loading all rows into memory; the job is idempotent per date partition and handles 1M rows/day within 10 minutes.
  • Updated RUNBOOK_BACKUP_RESTORE.md.

#412: Keyset pagination across audit log, user intents, and solver history

Problem: getAuditLog and user-intent listing relied on offset-based pagination, which degrades linearly and produces duplicates or skips when rows are inserted mid-page. Indexers and dashboards page through these endpoints heavily.

Changes:

  • Added a shared, HMAC-signed CursorCodec (base64url) and PaginatedResponse DTO in src/common/.
  • Applied keyset pagination to GET /intents/user/:addr, the audit log, solver fill history, and the leaderboard.
  • Deprecated offset params with a Deprecation header, and enforced a hard max offset of 10,000.
  • Added stable ordering with (created_at, id) tie-breakers.
  • Rejected cursors submitted against a different filter set with a 400.
  • Regenerated the OpenAPI spec and client.

Testing

Env vars

New env vars (DATABASE_REPLICA_URLS, MAX_REPLICA_LAG_MS, archival bucket/endpoint config) added to .env.example variants and src/config/env.validation.ts; check:env-drift passes.

Kaminar-i and others added 2 commits October 2, 2026 13:07
…on, cold-storage archival

- stellar-vortex-protocol#410: add src_token_id/dst_token_id FKs + snapshot decimals to intents; backfill migration; volume-by-token indexes
- stellar-vortex-protocol#411: PrismaReplicaService with lag monitoring + @readreplica() decorator; DATABASE_REPLICA_URLS config
- stellar-vortex-protocol#412: HMAC-signed opaque cursor codec; keyset pagination on intents, audit log, leaderboard, solver history; offset deprecated
- stellar-vortex-protocol#413: ArchivalService exports terminal intents to partitioned Parquet in S3/MinIO; manifest + SHA-256 verification; restore script; daily cron job

Closes stellar-vortex-protocol#410
Closes stellar-vortex-protocol#411
Closes stellar-vortex-protocol#412
Closes stellar-vortex-protocol#413
@NanaKhadija1980j
NanaKhadija1980j merged commit c48de27 into stellar-vortex-protocol:main Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants