feat(intents,solvers,archival): token FKs, replica routing, cold storage, keyset pagination - #578
Merged
Conversation
…on, cold-storage archival - stellar-vortex-protocol#410: add src_token_id/dst_token_id FKs + snapshot decimals to intents; backfill migration; volume-by-token indexes - stellar-vortex-protocol#411: PrismaReplicaService with lag monitoring + @readreplica() decorator; DATABASE_REPLICA_URLS config - stellar-vortex-protocol#412: HMAC-signed opaque cursor codec; keyset pagination on intents, audit log, leaderboard, solver history; offset deprecated - stellar-vortex-protocol#413: ArchivalService exports terminal intents to partitioned Parquet in S3/MinIO; manifest + SHA-256 verification; restore script; daily cron job Closes stellar-vortex-protocol#410 Closes stellar-vortex-protocol#411 Closes stellar-vortex-protocol#412 Closes stellar-vortex-protocol#413
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #410
Closes #411
Closes #413
Closes #412
This PR delivers four high-priority backend improvements: normalizing intent token data into foreign keys, adding read-replica routing for read-heavy endpoints, building cold-storage archival for terminal intents, and standardizing keyset pagination across the remaining list endpoints.
#410: Normalise intent token data into foreign keys
Problem: Intents stored
srcToken/dstTokenas JSONB blobs, duplicating registry data, resisting indexing, and allowing references to tokens that don't exist or whose decimals have since changed.Changes:
src_token_id/dst_token_idforeign keys on intents, plus immutablesrc_decimals/dst_decimalssnapshot columns captured at creation.(address, chain); unresolved rows are reported rather than dropped.#411: Read-replica routing for read-heavy endpoints
Problem: Solver polling of
/intents/open, leaderboard, and stats traffic was competing with latency-critical accept/fill writes on the primary database.Changes:
DATABASE_REPLICA_URLSconfig;PrismaServicenow exposes both primary and replica clients.@ReadReplica()decorator (and repository-level option) to select the client per query.pg_last_xact_replay_timestamp; replicas beyondMAX_REPLICA_LAG_MSare bypassed in favor of the primary.#413: Cold-storage archival of terminal intents
Problem:
INTENT_RETENTION_DAYSexisted, but deleting old data destroyed protocol history needed for analytics, disputes, and governance, while keeping everything hot bloated the primary.Changes:
date=YYYY-MM-DD/).scripts/restore-archive.ts --dateto restore an archived date into a staging schema.RUNBOOK_BACKUP_RESTORE.md.#412: Keyset pagination across audit log, user intents, and solver history
Problem:
getAuditLogand user-intent listing relied on offset-based pagination, which degrades linearly and produces duplicates or skips when rows are inserted mid-page. Indexers and dashboards page through these endpoints heavily.Changes:
CursorCodec(base64url) andPaginatedResponseDTO insrc/common/.GET /intents/user/:addr, the audit log, solver fill history, and the leaderboard.Deprecationheader, and enforced a hard max offset of 10,000.(created_at, id)tie-breakers.Testing
npm run lint,npm run typecheck,npm test, andnpm run test:e2eall pass.Env vars
New env vars (
DATABASE_REPLICA_URLS,MAX_REPLICA_LAG_MS, archival bucket/endpoint config) added to.env.examplevariants andsrc/config/env.validation.ts;check:env-driftpasses.