Skip to content

fix(database): SQL live-update queue, PK, and topology interventions - #1607

Merged
kkopanidis merged 2 commits into
feat/database-realtime-sqlfrom
cursor/sql-cdc-interventions-4407
Sep 13, 2026
Merged

kkopanidis merged 2 commits into
feat/database-realtime-sqlfrom
cursor/sql-cdc-interventions-4407

Conversation

@kkopanidis

@kkopanidis kkopanidis commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Intervention into #1602. Do not restack or merge #1602 into #1601. Do not merge #1600 / #1601 / #1602. Replay this delta after the Mongo restack.

Independent grok of #1602 @ 30314798 found P1s: trim-before-publish, overlapping handleChange, hardcoded _id, autoincrement holes. This PR closes those plus the P2 ops gaps, then retargets leftover same-name #1602 triggers.

Queue (P1)

  • Serialize handleChange. Persist resume after successful bus + socketPush (at-least-once duplicates OK).
  • Trim changelog only after Redis ack. Skip-path tokens still persist so unopted-in rows can be deleted.
  • Emit failure stops the stream and retries from the last good token (no persist).
  • Postgres/MySQL consume rows older than 750ms so a later autoincrement id is less likely to hide an uncommitted lower id. That lag is a timestamp filter, not a commit watermark: a transaction open longer than 750ms after a later row’s insert can still leave a hole. SQLite lag is 0 (serialized writers).
  • Snapshot MAX(id) before attaching new triggers. Do not jump to live MAX(id) on a leftover Mongo token.

Triggers / topology (P1/P2)

  • Triggers and per-table Postgres functions use SequelizeSchema.idField, not a virtual _id. NULL PKs skip the log insert instead of aborting DML.
  • Reconcile retargets leftover same-name cnd_rt_* triggers: if the existing function/body/PK does not match, DROP then CREATE once. Name equality is not “already correct.” DROP leftovers only on opt-out / global disable. Do not create the changelog table when realtime is globally off.
  • checkTopology probes SELECT 1 and Postgres LISTEN on a dedicated session client.
  • Redis keys are realtime:sql:resumeToken / realtime:sql:change-stream:leader. parseSqlResumeId is wired into watch().
  • NOTIFY during drain coalesces (dirty flag) instead of waiting for the 2s fallback.

Semantics copied from Mongo #1605

  • Fail closed on CMS-read deny (already on subscribe).
  • Keep membership when authorization is UNAVAILABLE; removeUser only on deny.
  • Do not add rebacCache.ts (that lands from the Mongo restack).

Docs / tests

README and status say this is an internal change queue, not native CDC (privileges, session-mode LISTEN / pooler, write amplification, no TRUNCATE, 750ms lag window). Tests cover persist-after-emit, emit-failure, UNAVAILABLE membership, leftover Mongo tokens, leftover #1602 _id triggers retargeted onto sku, NULL PK skip after retarget, disable-without-breaking-DML, prepare({ ensureLog: false }), LISTEN failure, notify-during-drain, and prepare() watermark start.

Independent grok of this intervention @ 38d81100: no P0/P1/P2. Snyk inherited from parents is not a merge bar.

Open in Web Open in Cursor 

Persist resume tokens after a successful emit, serialize consume, and
trim the changelog only after Redis ack. Triggers use the schema PK,
LISTEN is probed, Redis keys are SQL-scoped, and opt-out no longer
recreates live triggers.
Same-name cnd_rt_* triggers from #1602 still pointed at
conduit_realtime_capture / inline _id. DROP and CREATE when the
function or body does not match the physical PK so custom-PK DML
no longer aborts. Document the 750ms occurred_at lag window.
@kkopanidis
kkopanidis marked this pull request as ready for review September 13, 2026 15:15
@kkopanidis
kkopanidis merged commit b79d4b8 into feat/database-realtime-sql Sep 13, 2026
12 checks passed
@kkopanidis
kkopanidis deleted the cursor/sql-cdc-interventions-4407 branch September 13, 2026 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants