feat(database): stream live document updates over admin sockets - #1601
Open
JohnChantz wants to merge 10 commits into
Open
JohnChantz wants to merge 10 commits into
JohnChantz wants to merge 10 commits into
Conversation
This was referenced Sep 11, 2026
Contributor
Author
kkopanidis
added this pull request to stack #1603
September 11, 2026 17:43
MongoDB change streams require a replica set, so local Compose now initializes a single-node rs0. Schemas can opt in via realtime.enabled.
Mongo 4.4 keyfiles reject hyphens, and rs.status() returns ok:0 instead of throwing, so Compose never elected a primary for replicaSet URIs.
Stop duplicate admin/router deliveries and apply handshake auth on connect. Retry change-stream leadership after lock races.
kkopanidis
force-pushed
the
feat/database-realtime-mongo
branch
from
September 11, 2026 17:44
ef112bf to
3458413
Compare
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| function extractDocumentId(id: unknown): string | null { | ||
| if (id === undefined || id === null) return null; | ||
| if (typeof id === 'string' || typeof id === 'number') return String(id); | ||
| if (typeof id === 'object' && id !== null && 'toHexString' in id) { |
| const hex = (id as { toHexString: () => string }).toHexString(); | ||
| return typeof hex === 'string' && hex.length > 0 ? hex : null; | ||
| } | ||
| if (typeof id === 'object' && id !== null && 'toString' in id) { |
This was referenced Sep 12, 2026
cursor Bot
pushed a commit
that referenced
this pull request
Sep 13, 2026
…, recovery) Rebase #1601 onto current Event Relays Hermes and close the stack-review P1s: watch $match/$project, serialized handleChange, persist resume token after successful emit, keep token on CursorKilled 237, hermes isSocketHandshake, restore /database/ Redis on recover, and shutdown the watch plus leader lock. Co-authored-by: Konstantinos Kopanidis <kkopanidis@users.noreply.github.com>
…, recovery) (#1605) * fix(router): Event Relays interventions (EventBus, Hermes, relay manager) (#1604) * fix(router): harden event relays for HA and auth lifetime EventBus uses a single Redis message dispatcher with subscriberId maps, SIGTERM/SIGINT shutdown, and subscribe-after-ACK. Hermes installs engine middleware once, skips empty-room namespace broadcasts, demotes hot-path logs, and re-authenticates recovered /events/ subscriptions. EventRelayManager gains coalesced periodic reconcile, compiled templates, inbound caps, TTL-cached emit-time ReBAC, room eviction, backpressure, preview API, and metrics/docs aligned with the interventions plan. * fix(router): address #1604 re-review P1 and P2 follow-ups Recovery re-auth keeps per-user subscription state across disconnect; emit-time ReBAC distinguishes unavailable vs deny; EventBus drops process signal handlers and adds subscribeAck; manager subscribes only after Redis ACK and broadcasts evictRelayIds on refresh; Hermes backpressure uses local sockets without emit acks; cache is bounded; preview caps sample JSON; handshake matcher rejects ticket/sid/non-namespace paths; socket middleware rebind avoids duplicate registration on first sockets enable. * fix(hermes): restore production tsc for engine middleware chain Type the Socket.IO engine middleware runner as Express NextFunction so recursive callbacks match registerGlobalMiddleware. Use Logger.info for socket trace lines (IConduitLogger has no debug). * fix(router): import Express Response for socket global middleware _rebindSocketGlobalMiddlewares passes handlers typed with fetch Response because Response was not imported from express, failing tsc against Hermes registerSocketGlobalMiddleware. * fix(router): quit EventBus on module shutdown signals Stop event relays and call grpcSdk.bus.quit() from Router.shutdown(), registered on SIGTERM/SIGINT with process exit so Redis teardown is not left to SDK signal handlers. Tighten registerGlobalMiddleware typing and trim handshake helper comment (deslop). * fix(router): pass-3 recovery, scoped emit, and subscription hygiene Recovery re-auth keeps membership on Authorization UNAVAILABLE and only leaves on deny; re-check subscriptions for restored er: rooms via room map. Prune user subscriptions on disconnect when no other socket holds them. Scope relay emits to receivers in the target room; use Engine.IO writeBuffer/writable for backpressure. Drop hot-path Hermes socket info logs. * fix(router): pass-4 recovery map, backpressure, emit tests Store relay subs on socket.data; resolve recovered rooms from context and TTL room map without last-writer userId. Prune room map when unused on disconnect. Backpressure uses writeBuffer depth only. Add Hermes/router tests for room-scoped emit and queue metric. * fix(router): clear eventRelaySubs from socket.data on recovery deny * fix(router): clear socket.data subs on recovery fail-closed leave * fix(router): address CodeFactor findings on event relay validation Split validateEventRelayInput into per-field parsers to reduce complexity. Silence unused-parameter lint in EventBus test FakeRedis stub. * fix(database): Mongo live-update interventions (watch, token, tickets, recovery) Rebase #1601 onto current Event Relays Hermes and close the stack-review P1s: watch $match/$project, serialized handleChange, persist resume token after successful emit, keep token on CursorKilled 237, hermes isSocketHandshake, restore /database/ Redis on recover, and shutdown the watch plus leader lock. Co-authored-by: Konstantinos Kopanidis <kkopanidis@users.noreply.github.com> * style: prettier on Mongo intervention tests * fix(database): stop watch on emit failure; keep ReBAC membership Failed socketPush/bus no longer advances the resume token. ReBAC unavailable no longer removeUser. Auth middleware 401s a realtime ticket on POST /realtime/ticket. Drop/rename/invalidate reopen the watch. ---------
|
|
||
| getLocalRoomUserIds(namespace: string, room: string): Promise<string[]> { | ||
| return ( | ||
| this._socketRouter?.getLocalRoomUserIds(namespace, room) ?? Promise.resolve([]) |
|
|
||
| getLocalRoomsWithPrefix(namespace: string, prefix: string): Promise<string[]> { | ||
| return ( | ||
| this._socketRouter?.getLocalRoomsWithPrefix(namespace, prefix) ?? |
This was referenced Sep 13, 2026
…1606) * fix(router): Event Relays interventions (EventBus, Hermes, relay manager) (#1604) * fix(router): harden event relays for HA and auth lifetime EventBus uses a single Redis message dispatcher with subscriberId maps, SIGTERM/SIGINT shutdown, and subscribe-after-ACK. Hermes installs engine middleware once, skips empty-room namespace broadcasts, demotes hot-path logs, and re-authenticates recovered /events/ subscriptions. EventRelayManager gains coalesced periodic reconcile, compiled templates, inbound caps, TTL-cached emit-time ReBAC, room eviction, backpressure, preview API, and metrics/docs aligned with the interventions plan. * fix(router): address #1604 re-review P1 and P2 follow-ups Recovery re-auth keeps per-user subscription state across disconnect; emit-time ReBAC distinguishes unavailable vs deny; EventBus drops process signal handlers and adds subscribeAck; manager subscribes only after Redis ACK and broadcasts evictRelayIds on refresh; Hermes backpressure uses local sockets without emit acks; cache is bounded; preview caps sample JSON; handshake matcher rejects ticket/sid/non-namespace paths; socket middleware rebind avoids duplicate registration on first sockets enable. * fix(hermes): restore production tsc for engine middleware chain Type the Socket.IO engine middleware runner as Express NextFunction so recursive callbacks match registerGlobalMiddleware. Use Logger.info for socket trace lines (IConduitLogger has no debug). * fix(router): import Express Response for socket global middleware _rebindSocketGlobalMiddlewares passes handlers typed with fetch Response because Response was not imported from express, failing tsc against Hermes registerSocketGlobalMiddleware. * fix(router): quit EventBus on module shutdown signals Stop event relays and call grpcSdk.bus.quit() from Router.shutdown(), registered on SIGTERM/SIGINT with process exit so Redis teardown is not left to SDK signal handlers. Tighten registerGlobalMiddleware typing and trim handshake helper comment (deslop). * fix(router): pass-3 recovery, scoped emit, and subscription hygiene Recovery re-auth keeps membership on Authorization UNAVAILABLE and only leaves on deny; re-check subscriptions for restored er: rooms via room map. Prune user subscriptions on disconnect when no other socket holds them. Scope relay emits to receivers in the target room; use Engine.IO writeBuffer/writable for backpressure. Drop hot-path Hermes socket info logs. * fix(router): pass-4 recovery map, backpressure, emit tests Store relay subs on socket.data; resolve recovered rooms from context and TTL room map without last-writer userId. Prune room map when unused on disconnect. Backpressure uses writeBuffer depth only. Add Hermes/router tests for room-scoped emit and queue metric. * fix(router): clear eventRelaySubs from socket.data on recovery deny * fix(router): clear socket.data subs on recovery fail-closed leave * fix(router): address CodeFactor findings on event relay validation Split validateEventRelayInput into per-field parsers to reduce complexity. Silence unused-parameter lint in EventBus test FakeRedis stub. * feat(database): restack Mongo live updates onto current Event Relays Replay the Mongo delta onto feat/router-event-relays (9e6356e). Keep #1604 Hermes (engine.use, er: recovery, writeBuffer) and graft only the port-keyed adapter plus register-before-initSockets. Do not take old #1601 Socket.ts wholesale. Co-authored-by: Konstantinos Kopanidis <kkopanidis@users.noreply.github.com> * fix(database): fence change-stream leadership and expire recovery Redis keys Lock renew failure bumps a generation so a draining watch cannot emit. Do not open a watch unless the leader lock extends after acquire. Arm TTL on realtime socket/doc keys when a recoverable disconnect never recovers; persist on restore. CMS-read deny fails closed at emit without extra ReBAC round-trips and keeps membership; UNAVAILABLE keeps membership. Cover dropDatabase reopen. Co-authored-by: Konstantinos Kopanidis <kkopanidis@users.noreply.github.com> * fix(database): serialize leader acquire so concurrent reconcile still watches Two overlapping ensureLeader calls both bumped the lock generation while the first openStream was still reading the resume token, so the watch never attached. Hold an acquiring flag so only one opener runs. ---------
Create a merge commit so #1600 (9e6356e) is an ancestor of this branch. Do not squash: squash of #1605/#1606 dropped that ancestry and left #1601 CONFLICTING. # Conflicts: # libraries/hermes/src/Socket/Socket.ts # libraries/hermes/src/Socket/isSocketHandshake.test.ts # libraries/hermes/src/interfaces/Socket.ts Co-authored-by: Konstantinos Kopanidis <kkopanidis@users.noreply.github.com>
Live updates notify only: change streams start at the end of the oplog. Do not persist Redis resume tokens or restore on CursorKilled. Leader restart or cursor drop is a gap; clients refetch. Co-authored-by: Konstantinos Kopanidis <kkopanidis@users.noreply.github.com>
This was referenced Sep 13, 2026
Watch is pipeline-only (db.watch(pipeline)). Strip unused change-stream _id/clusterTime fields and test resume-token fixtures. Co-authored-by: Konstantinos Kopanidis <kkopanidis@users.noreply.github.com>
…sume-ab98 fix(database): merge-commit restack onto #1600 and drop Mongo resume
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce?
Does this PR introduce a breaking change?
Local Compose Mongo now runs as a single-node replica set (
rs0) so change streams work. Existing deployments are unchanged; localDB_CONN_URImust includereplicaSet=rs0.The PR fulfills these requirements:
mainbranchfix #xxx, where "xxx" is the issue number)If adding a new feature, the PR's description includes:
Other information:
Split from #1594 (2/3). Stacked on #1600 (event relays). SQL live updates follow in a stacked PR.
Schemas can opt in to live document updates via
realtime.enabledplusmodelOptions.conduit.realtime.enabled. MongoDB change streams require a replica set, so local Compose initializes a single-noders0. Events arechangemetadata only (no document fields); consumers refetch through authorized REST. Admin clients usePOST /realtime/ticketfor a 30-second handshake token so session JWTs / masterkey are not sent from the browser.Hermes isolates Redis adapter streams by port so admin and router Socket.IO servers do not deliver the same change twice, and handshake auth runs on connect.
Test plan
modules/database/src/realtime/__tests__/, excluding integration): coordinator, authorize, topology, subscriptions, rooms, normalize, statuspackages/core/src/admin/realtime/*.test.ts): ticket, handshake, namespaceapplySocketGlobalMiddlewarestestsdocker compose --profile mongodb up: Mongo becomes replica setrs0;DB_CONN_URIincludesreplicaSet=rs0realtime.enabled, opt a schema in, subscribe on/database/, mutate a document, confirm achangeevent with no document payloadPOST /realtime/ticket(session JWT / masterkey not sent from the browser)