Repository navigation
Change data capture for EloqDoc: log scan, schema-op timestamps, cluster gate, multi-node fixes - #577
Merged
Merged
Conversation
…og.proto Wire contract for EloqDoc change data capture (docs/cdc-change-stream.md in the EloqDoc repo). Messages and RPCs only; no handler implements them yet. - CdcScan: reads a log group's committed data records and its DDL store as one merged, timestamp-ordered sequence, with a continuation that resumes both streams exactly after a page. The continuation byte layout is frozen here because EloqDoc embeds it in the resume tokens it hands to clients. - CheckSchemaOpsInFlight: read-only poll the cluster gate's pause uses to finish its drain; it also reports the group's highest DDL timestamp, from which CDC bootstrap picks its starting position. - ClusterGateLogMessage (log content kind 6): pauses or resumes the cluster gate. Gate transactions are pinned to log group 0; the pause never expires, only a resume clears it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Implements CdcScan, declared in log.proto by the previous commit, and the
CDC retention floor (EloqDoc docs/cdc-change-stream.md §4; plan PR 3: A2,
A4, A6, A7, A11).
CdcScan returns a log group's committed data records merged by timestamp
with its DDL store (empty until the store exists), data first at one
timestamp: only records at or below arrival_complete_ts (min_ckpt_ts_),
cut near max_bytes but never empty while a record remains, with a
continuation that resumes both streams exactly and from_ts to start at a
timestamp. The core is the new cdc_scan.{h,cpp}: continuation codec, the
page merge over two CdcStreamCursors, and a cursor over the 20-byte data
keys shared by both RocksDB log states. Leader only: a follower, or a
leader whose term moved while the page was built, answers NotLeader with
the leader's address; a cloud leader not yet in sync answers with its own.
The memory state fails the RPC. wait_ms is ignored until the long poll
exists.
Two log-state details the scan needs:
- RocksDB state: scans register on an atomic counter and check a flag;
closing db_ (snapshot load, shutdown) clears the flag and polls with
bthread_usleep until the counter drains, so a scan begun before a
step-down can never read a closed DB and the FSM bthread never blocks a
worker. A shared_ptr like the cloud state's would not do: the reopen of
the same directory fails while the old DB still holds RocksDB's
per-process path lock.
- RocksDB-Cloud state: mem_db_in_sync_ speaks only for the current term's
DB container, so the scan reads the leader term before checking it and
again after the page, discarding the page if the term moved; it waits
up to ~0.5 s for the flag instead of refusing at once.
Retention floor: neither purge (PurgingSstFiles, MoveFilesToArchive) now
removes a file holding a key at or above min_ckpt_ts minus
txlog_cdc_retention_seconds, a new gflag / [local] ini key read in
log_init.cpp, launch_sv.cpp and AttachLog. It defaults to 0, which keeps
both purges exactly as before, so products without CDC are unaffected.
Tests: CdcScan-Test (15 cases: codec, the merge at every page size, the
RocksDB state, the purge floor, snapshot reinstall under concurrent scans)
built by tx_service/tests and the standalone log-service project; two
cases in log_server_rocksdb_cloud_tests.cpp run against MinIO in the
log-service workflow. docs/10-log-service.md documents the RPC boundary,
the setting and the snapshot contents per log state.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rom its prepare log Plan PR 4 (EloqDoc docs/cdc-change-stream.md §4.4(a) and (b); plan B1 and the add-index items). Checkpoint bound (B1). TxLockInfo::wlock_ts_ is now the latest write-lock grant, not the first, and a new holds_ckpt_wlock_ marks a tx that holds a write lock that must keep the checkpoint below it: on a data table or a catalog entry (TxLockInfo::WriteLockHoldsBackCheckpoint), not on range, bucket or cluster-config entries, which split and migration hold for long stretches. CcShard::OnWriteLockDowngraded, called from the single downgrade site, clears the flag once no such write lock remains. ActiveTxMinTs (checkpoint) and ActiveTxMaxTs (standby subscription barrier) read only flagged txs. So a DDL's catalog write lock keeps ckpt_ts, and with it the log group's min_ckpt_ts_, below the DDL's log record, which EloqDoc's change data capture relies on. Add index order. Recovery returns a node to the state right after the prepare log, where the dirty schema is installed and DML writes the new index, so the index's kv table must exist by then; the old order created it after the install. - The kv table of each new index is created before the prepare log, under the prepare-stage write lock, through a new generic DataStoreHandler::CreateKvTable(table_name, version) (the DSS client writes the initial range record; its CreateTable RPC is a no-op). One attempt, no retry: it blocks DML and holds the checkpoint back, and a failure only fails the command. - If the creation or the prepare log fails, the op drops the tables (DropKvTable) and fails with no log. An orphan after a coordinator failure is accepted: no schema refers to it. - kv_create_index_op_ is gone; the range-id sequence is seeded after the cluster-config unlock, before the build. - The new schema reaches the kv catalog once, after the commit log, at the op's timestamp D (kv_update_schema_image_op_, catalog only, retried until it succeeds). It stays under the commit-stage write lock, so a later DDL on the table cannot write its catalog row first: the DSS RocksDB backends apply writes in arrival order, and a drop table deletes the row, leaving nothing to compare a late write against. - A build constraint violation is the only rollback after the prepare log. Catalog entry commit ts. PrepareCommit under a write lock now installs D as the catalog cc entry's commit ts (TemplateCcMap, the entry marked Normal), and replaying an index op's prepare-stage log does the same (CatalogCcMap). A catalog read enters the read set at the entry's commit ts, so every DML that sees the dirty schema commits above D on every node; before, only the node clocks ordered it, and a DML below D lost its index write on replay. Test support. Three fault points, one per add-index failure path: alter_table_index_create_kv_table_fail, alter_table_index_prepare_log_fail and alter_table_index_update_schema_image_fail (one-shot). FaultInject::CompiledIn() reports whether the fault points are compiled in, from the data substrate's own compilation, so an API layer's test command can refuse to arm a fault that could never fire. AsyncOp::SetBlockingWork runs blocking work on the op's worker thread and finishes hd_result_ with the CcErrorCode it returns; the new kv table ops use it. Docs: 03 (registry, §10), 04 (SetBlockingWork), 05 (catalog entry commit ts), 07 (§3.2, §5.4), 08 (§6 add index order and its cost), 09. Tests: CkptLockRegistry-Test (9 cases). Unit suite 85/85. With a Debug EloqDoc build: index builds under concurrent writes, kill -9 during a build (recovered from the prepare and PrepareData logs), a too-long key (rollback after the prepare log), and each fault point, through the new EloqDoc eloq_fault jstests. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SplitFlushRangeOp's ds_upsert_range_op_ captured the split's range list in a mutable op_func_ and moved it into the worker thread. When UpsertRanges failed, RetrySubOperation ran op_func_ again with the moved-from, empty list, and DataStoreServiceClient::UpsertRanges returns true for an empty list, so the split completed without writing its new ranges (or the old range's updated slice sizes) to the range table. A restart soon after the split hides it, since replay of the split's log restores the ranges, but the range table stays wrong. UpsertRanges only reads its ranges, so DataStoreHandler::UpsertRanges and its two implementations now take them by const reference, and the op runs through AsyncOp::SetBlockingWork with a work that passes its captured ranges without moving them, so every retry writes the full list. Adds split_flush_upsert_ranges_fail, a one-shot fault point that fails one upsert before it writes, and a log line with the number of ranges each attempt writes. Tested on a Debug EloqDoc build: 350,000 documents of about 1 KB split one range with the fault armed; the retry wrote 2 ranges (the old code wrote 0), and every document was found after the split and after a restart. Unit suite 85/85. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
EloqDoc's change-stream cursor (EloqDoc plan PR 5, the scan client C4) sends CdcScan to every log group's leader. TxLog now exposes that, so the cursor shares the tx service's leader cache instead of keeping its own. - TxLog::CdcScan sends the scan asynchronously to the group's cached leader (LogAgent::CdcScan, a LogService_Stub call) and reports the node ID it sends to. The node ID is written before the call is sent, since the closure may run on another thread before the stub returns. With no channel for the group the controller fails, the closure runs before CdcScan returns, and no node ID is reported. - TxLog::LogNodeByAddress returns the node ID configured with the host:port of a NotLeader redirect among the group's node IDs, matched as RefreshLeader matches braft's leader. The log service formats a redirect as a hostname or an IP address, as its raft configuration does, so a caller compares a redirect with the scan's target by node ID rather than as a string, and follows it to another node with the existing UpdateLeaderCache. - The TxLog defaults fail the scan and find no node ID, for a log without change data capture; EloqLogAgent forwards both to its LogAgent. LogAgentCdc-Test covers the lookup (the three node IDs of a group; eight malformed or unmatched addresses and an unknown group find nothing) and CdcScan's closure running exactly once, with no node ID for an unknown group and the cached leader's node ID for an unreachable one. No log server runs in it. docs/10-log-service.md §9 describes both calls. Unit suite 89/89. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
InitializeTxService read checkpointer_interval as a signed flag or ini value and stored it unsigned, so -1 became an interval of about 136 years and the periodic checkpoints stopped without an error; a change stream opened on such a node waited out its open timeout and failed blaming cluster start-up. A zero interval ran checkpoint rounds back to back, keeping about two cores busy on an idle node. - A negative checkpointer_interval is an error: the data substrate refuses to start. - A value from 0 to 4 is raised to 5 s, with a warning. - Checkpointer::CheckpointInterval() returns the interval the checkpointer runs with, so an engine can name it; EloqDoc puts it in a change stream's ChangeStreamStartNotReady message. Checked on a single node: -1 refuses to start with "Invalid `checkpointer_interval` -1: it must not be negative"; 0 logs the warning and runs rounds every 4.5-6 s at the normal idle CPU. docs/01 and docs/07 state the rule; the tx_service_init.cpp line references in docs/07 and docs/08 follow the moved definitions. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Plan PR 7 (EloqDoc docs/cdc-change-stream.md §4.4(b), §10; plan B2-B5, B7, B8 and the B6 tests). Commit-stage timestamp (B2, B4). CREATE, DROP and TRUNCATE TABLE hold only a write intent across their prepare log, so DML of the old table can commit above the prepare-stage timestamp D. Once the op holds the commit-stage write lock, UpsertTableOp::AssignCommitStageTs re-assigns its commit timestamp P above every DML the lock drained; the commit log, the DDL record and PostCommit carry P. CatalogEntry::CommitDirtySchema promotes at P, and core 0 rebuilds a truncate's schema at P before any core moves its cc maps (CatalogCcMap::CommitNodeLevelSchema). Tables keep their own versions: SchemaOp::ResolveSchemaImages builds the new schema at D before anything is logged and keeps its image, in which the engine records each table's version. CatalogFactory::CreateTableSchema now documents that contract, and TableKeySchemaTs reads a version the image does not record as 0, the unknown timestamp (was 1). Timestamps above readers and the lock grant. AcquireAllCc reports max(last read ts, local time), so a schema op's timestamps are above its catalog write lock's grant, and so above any checkpoint taken while it holds the lock. EloqKV's fast catalog read publishes the reader's commit ts in the shard's last read ts when it releases the lock (LocalCcHandler::ReleaseCatalogRead), as PostReadCc does for a regular read lock. Create's kv work around its logs (B3). The kv tables (CreateKvTable, base and indexes, initial ranges at D) and the sequence table are written after the prepare log, under the write intent. A failure there is not retried: the create gives up (AbandonCreate drops what it created, one attempt) and fails with a commit log at timestamp 0 and DATA_STORE_ERROR. The catalog row is written after the commit log, at P (UpsertTable with op type Update, the catalog-only write). TRUNCATE. On shared storage it empties the table under the write lock before its commit log (the cc maps, the kv table through DataStoreHandler::DropKvTable, the sequences), each step retried, and writes only its catalog row after it; a participant that recovers from the commit log and takes writes into the emptied table finds no later step that wipes them. Without shared storage each node group still truncates its own store after the commit log. Data store failures (B7). A failure before the op's point of no return (the commit log of UpsertTableOp, the prepare log of UpsertTableIndexOp) rolls the op back; one after it is retried until it succeeds, ddl_kv_retry_interval_seconds apart (new setting, 30 s by default), with a warning naming the table and the step (SchemaOp::ArmKvRetryWait). TRUNCATE retries the steps it runs before its commit log. Rolled-back add index (B8). A build that breaks a constraint takes the commit-stage lock and commits at P = max(D + 1, lock results) with a rolled-back mark in its commit log (SchemaOpMessage.rolled_back). Its second phase is the new PostWriteType::Rollback (CommitType Rollback = 5 on the wire), which discards the dirty schema and moves the entry to P; after the commit log the index's kv tables are dropped and the current schema's catalog row is written at P. Commit timestamp 0 now occurs only with Rollback or DowngradeLock: PrepareCommit and PostCommit assert above 0, OnCommittedUpdate runs only for non-meta tables, and a Rollback with nothing installed carries no record. UpsertTableOp as a coroutine. The op moves to tx_upsert_table_operation.cpp and runs as a stackful coroutine (boost::context, a 128 KB stack): the first Forward() starts it and later ones resume it. Its path is written flat in three segments cut at the two log records (RunUntilPrepareLog, RunUntilCommitLog, RunAfterCommitLog), where recovery enters it (UpsertTableOp::Entry). Retries are loops; a step that finishes inside Process() does not suspend; a fault-point Await fails a step without running it; AwaitUpsertKvTable names the store operation it asks for. Cluster config lock. UpsertTableOp and UpsertTableIndexOp hold the cluster config read lock from their first step through their last post write all, then release it, so every step they send to all node groups reaches the same node groups. Recovery (B5). The log state keeps a commit log's timestamp and its rolled-back mark (LogState::UpsertSchemaOp). From a commit log an op resumes at P; from a prepare log it re-runs its steps and re-derives P. Commit-stage replay now restores the coordinator's write lock on the schema (a write intent while a range split holds the catalog read lock), so no newer DDL runs before the recovered op's steps after the commit log; participants restore none, and a post write finding no lock of its tx on the entry finishes at once. Commit-stage replay also loads the catalog cc entry at the committed version, and RollBackSchema and CommitNodeLevelSchema accept an entry that replay already committed. The interplay with a range split recovered on the same table is deferred: docs/range-split-schema-op-recovery.md. Fault points and tests. New fault points: upsert_table_update_sequence_fail, and upsert_table_prepare_log_fail through the fault-point Await; during_post_write_all's actions now test Rollback (rollback_panic, rollback_sleep). LogStateSchemaOp-Test: the rolled-back mark survives in the log state. CcEntry-Test: VersionedPayload's copy-on-write install. Docs: 01 (the setting), 03, 04, 05, 06, 07, 08, README, and the new range-split-schema-op-recovery.md. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…a recovered DDL Plan PR 8 (EloqDoc docs/cdc-change-stream.md §4.2, §4.3, §4.5, §4.6, §5.8; plan A3, A5, A8). DDL history (A3). The log state keeps an in-memory history of EloqDoc's schema ops for the CDC scan: one record per create, drop and index op, two for an add index whose build broke a constraint, the second, at its commit log's timestamp, carrying the rolled-back mark. An index op's record goes in when its prepare log is applied, at that log's timestamp; a create's or drop's when its commit log is applied, at the commit log's timestamp; an op rolled back at timestamp 0 leaves none. ALTER, TRUNCATE, other engines and the logical alters inside DML are not recorded. Records are keyed by their position, be64 ddl_ts | be64 txn (CdcDdlPosition), the form the continuation already carries. A scan copies the records it needs under the shared lock (CdcDdlHistoryCursor, from LogState::CdcDdlStream), which replaces the empty DDL stream in CdcScanRocksDB. The RocksDB and cloud states save the history in the raft snapshot, in a file of its own (cdc_ddl_history) so the memory state's snapshot format is unchanged; a snapshot without the file loads an empty history, and the loader bounds every record length by the file's end. The history is trimmed by the data log's purges, by their floor, when they run (TrimCdcDdlHistory, from PurgingSstFiles and the archive mover), not at every checkpoint: with the default retention window of 0 a checkpoint-time trim would drop each record as soon as a scan could return it. Long poll (A5). A CdcScan whose page holds nothing emittable and whose request sets wait_ms is parked (CdcLongPoll) and completed exactly once by the first of: an advance of arrival_complete_ts, applied with an UpdateCkptTs entry (the page is scanned again); its deadline (scanned again); the client's cancellation (NotifyOnCancel, registered before the park); the node's step-down or shutdown (NotLeader). Completions run on bthreads of their own; the list is guarded by a plain mutex, and the poll's destructor waits for every prepared request and deadline timer. At most cdc_scan_max_parked (gflag, 1024) scans are parked per group, each for at most cdc_scan_max_park_ms (gflag, 60 s), since a client that gives up on a call cancels nothing on the server. Commit-stage replay. The coordinator's node group restores a write intent, not a write lock, on the catalog entry of a schema op recovered from its commit log. The entry already holds the committed schema, so the op's effect is in place for everyone else, and the intent still keeps a newer DDL out until the op's PostCommit or Rollback. With a write lock, data-log replay of the table failed InitCcm (READ_CATALOG_CONFLICT) and dropped its records, and the op, whose PostWriteAll needs the leader term that only the end of replay brings, could not release the lock before replay ended: a rolled-back add index or an ALTER recovered from its commit log lost the documents written since the last checkpoint. The range-split special case at the commit stage goes with it (docs/range-split-schema-op-recovery.md). Cloud state (A8). test_cdc_ddl_history_cloud_state: the cloud leader's scan merges the DDL history with the data in its DB, and a replica that installs the leader's raft snapshot serves the same records once it leads the next term. test_cdc_retention_floor_cloud_state also checks that the archive mover trims the history by its floor. Both run in the log-service workflow against MinIO. Tests: LogStateSchemaOp-Test (appends, rollbacks, filters, copy by position, purge-time trim, snapshot round trip, a corrupt record length), CdcScan-Test (the merged scan and its paging, snapshot reinstall, the RocksDB purge trimming by its floor), CdcLongPoll-Test (each source, racing sources, destruction). log.proto, cdc_scan.h and docs 07, 10 and the range-split doc are updated. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… locks For EloqDoc's plan B6 (ii) (docs/cdc-change-stream.md §4.4(b), F7): a test must commit DML while a drop is between its prepare log and its commit-stage write locks, and that window is too short to hit without a hold. upsert_table_hold_before_commit_lock, in UpsertTableOp::RunUntilCommitLog after the dirty schema is installed, holds create, drop, truncate and ALTER there while it stays armed (UpsertTableOp::HoldBeforeCommitLock). It arms upsert_table_held_before_commit_lock, which the test polls to learn the op is held, and waits on kv_retry_wait_op_, idle before the commit log, whose worker thread polls the hold every 100 ms for at most 60 s; the op's coroutine is suspended meanwhile, so its TxProcessor keeps serving other requests, which the SLEEP action, a sleep() on that thread, would not. It disarms the marker when released. Fault-injection builds only. While armed it holds every table op that reaches it, of any table, which its Doxygen comment warns: a test arms it around one DDL and disarms it in a finally. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
For EloqDoc's change data capture bootstrap (docs/cdc-change-stream.md §6.2; plan D1-D3, D5): while a consumer copies the catalog, a paused cluster gate keeps new DDL out. This change has the carrier, the check and the log side; the gate op that pauses and resumes, and its replay on the tx side, come next, together. - ClusterConfigRecord carries an optional ClusterGate (mode, paused) through every copy and its serialization, behind a has-bit after the configs. A record that owns no configs, as a gate's does, now copies without dereferencing null. - ClusterConfigCcMap::Execute(PostWriteAllCc&): a record carrying a gate never updates Sharder, whose version its commit timestamp always exceeds; it sets the gate in the entry's payload in place, and only while the request's tx holds the write lock, so a retried old resume cannot reopen a gate a newer pause closed. - ClusterConfigCcMap::Execute(AcquireAllCc&) reports max(LastReadTs, Now()), as every other write-lock grant does, so a cluster scale or gate op commits above each node group's clock at the grant. It reported LastReadTs alone, despite its comment. - UpsertTableOp and UpsertTableIndexOp check the gate where they take the cluster config read lock, their only admission point: a paused gate releases the lock and fails the op with the new TxErrorCode::CLUSTER_GATED before any log. An op past that step, or a recovered one, is never refused. A pause needs no drain poll: its write lock waits for every admitted op, which holds the read lock through its last post write. - Log service: the cluster_gate_log apply case sets a per-group slot (value, version, txn), newest entry wins, never purged; the RocksDB and cloud states save it in its own snapshot file (the memory state does not); a replay list carries it as one item (ReplayClusterGateMsg), which the tx side does not install yet; a retried write log and RecoverTx look it up by tx, so orphan lock recovery leaves a logged gate tx's cluster config write lock to its coordinator's replay instead of clearing it on a group that has not installed the gate. Tests: ClusterGate-Test (record round trips and copies; the map's gate-only post write all stores the gate and releases the lock without touching Sharder, and a stale one stores nothing) and LogStateClusterGate-Test (newest wins, never purged, one replay item, snapshot round trip and corrupt files). docs/03, 04, 06, 10. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
For EloqDoc's change data capture bootstrap (docs/cdc-change-stream.md §6.2; plan D4, D6): the transaction that pauses or resumes the cluster gate, and its replay. - ClusterGateTxRequest runs a ClusterGateOp (tx_cluster_gate_operation.cpp): a read of the record for write on this node group (a write intent, which schema ops' read locks pass), where a pause of a paused gate fails with the new CLUSTER_GATE_HELD before anything else is locked; write intents on every node group with OCC, which only another gate op or a cluster scale can fail; write locks with Locking, upgrading the intents, which wait for every schema op holding the read lock (the drain); one cluster_gate_log entry; the post write all of a record carrying the new gate. The result is the op's commit timestamp, above every schema op committed before. The request is the whole transaction: the op commits or aborts it. Ops are pooled in LocalCcShards, like cluster scale's: a remote response that arrives after an op finished still reads the cc handler results the op owns. - The log entry carries the terms of the node groups that granted the lock. Unlike cluster scale, the gate writes no log before it locks, so a group that failed over between granting the lock and the entry would come back with the old value and no lock, and the install would skip it; the log service refuses the entry instead. - Any failure before the log (an OCC conflict, a node group without a leader, a log refused on a node group's term) releases the locks at tx_op_failed_ts_, backs off 1 to 3 seconds with jitter and starts over from the read. Nothing waits while holding the entry anywhere, so gate ops and cluster scale cannot deadlock: deadlock detection runs only on request, and AcquireAllOp never requests it. After the log the op rolls forward, retrying the install a second apart. The fault point cluster_gate_intent_conflict_once forces one conflict, for tests on a single node group. - Replay: RecoveryService replays log group 0's gate slot after cluster scale's item and before any schema op (ReplayLogCc::SetClusterGate). The map sets the gate in the payload in place, keeping the configs scale's replay cached there, and raises the entry's commit timestamp only if the slot's is larger; a corrupt slot fails the replay. On the slot's coordinator node group a recovery tx (RecoverClusterGate) re-sends the install under the same tx, with no lock or intent: nothing follows the log but the install, which stores and releases only where the tx still holds the lock. Holding the lock during replay would deadlock with the schema-op recovery that reads the cluster config next. - ClusterConfigCcMap::Execute(AcquireAllCc&) reports the entry's lock address in local_cce_addr_ on a first execution too. It reported 0, so AcquireAllOp's DedupRead never matched a read of the cluster config the tx made before, and cluster scale (and the gate, which reads first) ended every commit releasing that read with a version mismatch. - ClusterConfigCcMap no longer calls OnCommittedUpdate in its post write, cluster scale's replay or the gate's replay: it keeps data-key statistics, which a meta map does not take part in (AdjustDataKeyStats already returned early for one). - ReadLocalOperation can read for write (is_for_write_). - TransactionExecution holds the composite op of the tx request it runs in one std::variant slot (composite_op_, CompositeOpSlot<Op>()) instead of a unique pointer per op type: a txm runs one tx request at a time. Reset clears the slot. The gate request's result reuses uint64_resp_. - LocalCcShards guards all its op pools (and the sk generator pool) with one mutex, op_pool_mux_, instead of one per pool: each user holds it only to take or return one object. Tests: ClusterGate-Test (the replay installs in place and keeps the larger commit timestamp; a corrupt slot fails the replay). docs/04, 06, 07. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
For EloqDoc's change data capture (docs/cdc-change-stream.md §6.2, §8;
plan A9, D8):
- cluster_gate_paused{ng_id}, on the node meter: 1 while the node group's
cluster config map holds a paused cluster gate, set by the gate's post
write all and by its replay (ClusterConfigCcMap::ReportClusterGate),
and 0 once the node stops leading the group, after CcNode's leader
stop has cleared the group's maps: a gate install queued before the
clear runs before it on core 0 and may report 1, and none can report
later. No pause ends on its own, so its maximum over the
cluster is what a gate-age alert watches.
- cdc_arrival_complete_ts{lg_id}, per log group on each replica: the
group's arrival-complete timestamp in microseconds, set when it
advances, when a snapshot is loaded, and once when metrics are enabled
(the group serves before they exist). The apply thread keeps the value
in an atomic, and a mutex orders the reports, so the first one cannot
land after a newer one and leave the gauge behind.
- cdc_scan_parked_requests{lg_id}, per log group: the CdcScan requests
the leader holds parked, through a new size observer on CdcLongPoll,
called under the poll's lock on every change of the count.
The two log group gauges are on with enable_metrics and
enable_log_service_metrics. The log server starts before the metrics
registry exists, so DataSubstrate::Start calls the new
LogServer::EnableMetrics after InitializeMetrics; each LogInstance makes
its meter (node labels plus lg_id), registers both gauges and only then
publishes it to the threads already serving. LogServiceImpl keeps the
registry and labels, so the groups a later AttachLog adds get their
meters too; one mutex orders EnableMetrics' walk of the replica map
against AttachLog's and DetachLog's changes to it. The log service uses
eloq_metrics' header-only Meter against the registry passed in, so
logservice gains the include directory but no library, in both its
build files: build_eloq_log_service.cmake and the standalone
eloq_log_service/CMakeLists.txt that CI configures directly.
Tests: CdcLongPoll-Test (the observer sees every change of the parked
count, not a request completed before it is parked, and nothing once
cleared). Docs: 01 (start order), 06 (the gauge), 10 (the log group
metrics; the build section's source list, which lacked the CDC sources,
and the ini reader's path).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
For EloqDoc's change data capture (plan F6), bringing the docs in line with what PRs 4, 7, 10 and 11 changed: - 03: schema ops rely on the exclusive section; index ops choose their commit timestamp at prepare, while create, drop and truncate choose it again under their commit-stage write lock (AssignCommitStageTs). A tx's wlock_ts_ bounds its commit timestamp only for DML. - 07: the cluster_gate_log record (log group 0's slot, never purged); the gate's replay step, after cluster scale's and before any schema op; the gate's recovery (ReplayClusterGate, RecoverClusterGate); and how a rolled-back add index replays (the old schema at its commit-stage timestamp, only its catalog row written). Line numbers in the replay section, which had drifted, give way to function names. - tx_operation.h: write_time_'s comment says what a rolled-back add index writes at (its commit-stage timestamp), not the formula it once used. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Found and verified with EloqDoc's multi-node tests (cmake/tests/multinode/), on clusters of three node groups run as separate processes. Log replay: - A recovering node group asks each log group for its replay once, at its leader start (Sharder::RequestLogReplay, queued to RecoveryService). Each send is one bounded attempt (LogAgent::ReplayLogOnce: a leader lookup of at most 1 s, an RPC with a 2 s deadline). TxLog::ReplayLog takes one log group and returns whether its leader accepted; the host manager no longer sends replay requests (NotifyLogReplay removed). - RecoveryService's watchdog asks again a log group that owes the replay and has had no recovering stream for 5 s since the last send, reusing the first request's start timestamp. It visits only the node groups this node is recovering (candidate_replay_watch_). - A log group asked again replaces the node group's idle shipping agent, which stops at its next message (LogShippingAgent::IsTerminating). - Fault point replay_drop_stream_once. ELOQ_TX_FAULT_INJECT and ELOQ_DSS_FAULT_INJECT arm fault points from a process's start. Schema op recovery (docs/range-split-schema-op-recovery.md): - After add index's last prepare data record (last key +inf), a participant restores the write lock, unless a split of its own holds the table's catalog read lock (ReplayLogCc::RangeSplitting); before it, a write intent. - Fault point alter_table_index_hold_before_commit_lock takes hold_s. Data loss: FillDataLogRequest logged a transaction's writes to a splitting range's new range to the new owner only for the first such key, so the new owner lost the rest if it restarted before checkpointing them. Every key is now logged to every node group it is written to, once per node group. Reads: ReadOperation and BatchReadOperation no longer assert that a retried read of a hash-partitioned table ran its lock step: a bucket already known (FastToGetBucket) runs none. Snapshot transactions start at max(ts base, wall clock), advancing the base (CcShard::NewTx). Leader caches: - Every leader update carries its term. A lookup asks the host manager for the leader, which braft names without a term, then asks that node for its term (Sharder::FetchLeaderTerm, FetchNodeInfo's new leader_term); a node that does not lead the group answers none and the lookup changes nothing. The cache applies an update only above its cached term, leader and term together under ng_leader_mux_. - Every newer term goes to the data store (DataStoreHandler:: OnLeaderChanged), also when the leader is the same node. The data store service client points the group's data shard at the leader with the term as the shard's version, and refuses a move below the shard's version, checked against the slot it replaces (UpgradeShardVersion, UpdatePrimaryNodeIfNotOlder). Data store service: - A request for a shard in Starting waits in one list that the transition out of Starting drains (WaitWhileStarting, RunStartingWaiters). - The client retries a failed request at once only if the shard is local or its channel is healthy (CanRetryShardNow), and fails fast otherwise. BatchWriteRecords reports a network failure instead of success. - EloqStore: each data shard's instance owns only its node group's partitions, set before every start (SetPartitionBuckets), replacing the process-level filter. - ckpt_delay_time_'s comment says what the delay does: it keeps recently updated entries in memory and holds back no version. Docs: 03, 06, 07, 09, 10, README and range-split-schema-op-recovery.md. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The CDC branch's documentation was written against the module docs before main consolidated them (5c0be9f). Rebasing kept main's versions of the numbered docs; this brings the CDC branch's current-architecture claims into them, in their present-tense form with source map rows: - 01: the in-process log server receives the metrics registry after start. - 03: lock grants report timestamp bounds; catalog readers publish their commit timestamps; the lock registry records each transaction's latest write-lock grant and whether it bounds the checkpoint; snapshot transactions start at the later of the timestamp base and the wall clock. - 04: schema operations' points of no return, store-step retries and commit-stage timestamps; the cluster gate operation. - 05: the catalog entry's commit timestamp, rolled-back operations, and resolved table versions in schema images. - 06: term-ordered leader updates and lookups; the cluster gate. - 07: the checkpoint as a commit watermark; forwarded keys logged to every NG; bounded replay requests; restored schema-op locks by role; MVCC persistence and the checkpoint delay as a cache residency choice (main's text tied the delay to version visibility, which the code does not do). - 08: a split that fails before its log; the index build's kv table, point of no return and kv catalog write. - 09: CreateKvTable; Starting waits; retry only when reachable; data shards bound to NG leaders by term; per-data-shard EloqStore instances and their partition filter. - 10: the CdcScan RPC, DDL history, long poll and metrics; the CDC retention floor; the cluster gate record and slot; replay requests and shipping replacement. - README: the range-split recovery design record. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A node leading several node groups ran one EloqStore per data shard, each with its own buffer pool, fd budget and local cache quota, so the process's memory could not be budgeted as one. EloqStore now hosts several data store shards on one store (eloqdata/eloqstore PR 502), and the data store service uses that. - The factory owns the one store, built and started (StartInstance) with the first data store it makes, and stopped with the factory. Its metrics are initialized once there. - Opening a data shard attaches it at its term (EloqStore::AttachShard; ExpiredTerm fails the open as a failed start did) and closing it detaches it (DetachShard), while the other shards keep serving. Every request the shard's data store sends names the shard (KvRequest::SetDsShard), the scan-and-delete loop's follow-up requests included, and ReloadData checks the shard's term. - The store's partition filter takes the data store shard id and asks that shard's data store, which still keeps its buckets (SetPartitionBuckets, set before each attach). The data stores register with the factory for that, and DataStoreService destroys them before its members go, since the factory would otherwise be destroyed first. - The per-shard ds_<shard> directories are gone: one store, one local cache. StartService no longer refuses more than one shard. - The EloqStore submodule moves to the PR 502 commit, and its new source joins build_eloq_store.cmake. docs/09 describes the one store. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The task carries its data store shard and term and the worker thread publishes them at every task switch; the per-partition binding table is gone (eloqdata/eloqstore PR 502). No change on this side: the service already stamps every request with its shard. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The one-shot upsert_table_create_kv_table_fail fired on whichever create reached it first. The server creates its feature document beside the first collections, so on a fast machine that create consumed the fault a test had armed for its own create, which then committed normally and was rolled forward (DDL recovery scenario vi on the VM lane). The fault now honors the table_name parameter the fault entry already parses; without it, it fires as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The crash-after-prepare-log and crash-after-commit-log faults fired on whichever upsert-table op reached them first. The server creates its feature document beside a database's first collections, so a crash fault a test armed right after creating a collection could kill the node on that create's commit log instead, leaving the test's DDL at its prepare log (the DDL recovery scenarios on the VM lane, 2 of 5 runs). UpsertTableOp now fires its action faults through TriggerTableFault, which honors the table_name parameter like the kv-table-create failure does; without the parameter they fire as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two test files stopped compiling on this branch, so CI's unit job failed at its build step. FetchRecordCc-Test still called store() on the node group leader term cache, which the multi-node recovery fixes made a plain array; log_snapshot_test called the LogInstance constructor without the cdc_retention_seconds argument the CDC log scan added. Both now match the code, and the unit job builds and passes all of its tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pass that ended with nothing evictable waited for the next flush to start another, and a request that parked again after its release could be released into the same full heap. The cleaner is now a single request per shard that stays in use while it is queued, running or deferred. A pass starts at the LRU head and evicts in bounded batches while less than 5% of the shard limit is free. It releases parked requests in batches of 20 below the hard limit. Once a request has parked since the pass started or last released, the room handed out is used up: the next release waits until the pass has reclaimed shard_clean_release_kb more (a tx service gflag and [local] config key, default 256 KB, zero meaning any amount). At the LRU tail whatever was reclaimed goes out; an execution that starts at the tail reclaims nothing and releases nothing, so a request that keeps parking cannot keep the pass alive. A pass that ends with requests parked aborts the requests that opted into OOM failure when the heap is full or nothing was freed. When it reclaimed less than one release-worth it asks for a checkpoint and backs off: the timer thread retries a deferred cleaner after 100 ms, doubling to 1.6 s, and the backoff clears when the cleaner goes idle. A flush or range split completion runs a deferred cleaner at once, or marks a running pass to run once more; nothing rewinds a pass. The timer thread walks the shards on every tick, so it starts once they exist, and a worker that exits at shutdown frees its shard under the timer's mutex: otherwise the walk could reach a destroyed shard and crash a clean shutdown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The header relied on another include for its integer types. Including <cstdint> directly lets it compile on its own, whatever includes it first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The branch pinned a commit of the pull request's branch, one short of what merged into EloqStore main (eloqdata/eloqstore PR 502). The last commit runs the term file's compare-and-swap as a task, detaches a shard by the partition filter, and keeps one archive schedule per attached shard. Its new attach-shard task source joins the EloqStore source list the data substrate builds from. Otherwise no change on this side: the data store already sets the partition filter, which a second shard now requires, and a shard's data store keeps answering it until after its detach returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The repository's format check runs clang-format-18 over every file; three files the CDC branch added or changed did not pass it. Whitespace and the order of one include block only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Review skippedToo many files! This PR contains 124 files, which is 24 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.
 |
5 tasks done
main's snapshot hardening (#555) closes and checks every snapshot file and publishes no file list if a write failed. The CDC DDL history and cluster gate files, added on this branch before that, were written unchecked: a failed write, a full disk say, still published a snapshot whose follower would load a truncated history. Both log states now check them as they do their other files, and the RocksDB state syncs them as it does 'data'. log_snapshot_stream_test (built for the RocksDB-Cloud log state) expected the snapshot files of main, without the two CDC files, and failed in CI. It now expects them and injects a write failure into each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With ELOQ_MODULE_ENABLED, EloqStore's shards run on the bthread workers as a brpc module, so a request to EloqStore completes only once workers run. Since the EloqStore pin moved to the shard support as merged, attaching a data shard in cloud mode is such a request: the term file's compare-and-swap runs as a task on a worker shard. A serving node has started the workers by then, its brpc servers and clients having run bthreads, but a bootstrap (EloqKV's --bootstrap, before its Redis server starts) has started none and waited on the attach forever. EloqKV's multi-node scenarios on EloqStore's cloud mode then started their cluster without the bootstrap's tables, and the first FLUSHALL crashed node 0 on a catalog record with no schema. DataSubstrate::Start now runs one bthread to completion before the storage handler starts, which starts the workers when none runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| /** | ||
| * @param source Where Seek() copies the records from. | ||
| */ | ||
| explicit CdcDdlHistoryCursor(Source source) : source_(std::move(source)) |
There was a problem hiding this comment.
[cpplint] reported by reviewdog 🐶
Add #include for move [build/include_what_you_use] [4]
| Completion completion; ///< Runs once, when the request ends. | ||
| std::atomic<bool> completed{false}; ///< Taken by the source that wins. | ||
| bool parked{false}; ///< In the poll's list. | ||
| std::list<std::shared_ptr<Request>>::iterator position; ///< Its entry. |
There was a problem hiding this comment.
[cpplint] reported by reviewdog 🐶
Add #include for list<> [build/include_what_you_use] [4]
| return nullptr; | ||
| } | ||
|
|
||
| void CdcLongPoll::Finish(const std::shared_ptr<Request> &request, Wake why) |
There was a problem hiding this comment.
[cpplint] reported by reviewdog 🐶
Add #include for shared_ptr<> [build/include_what_you_use] [4]
| std::string position = | ||
| from_ts == 0 ? std::string() : CdcTimestampPosition(from_ts); | ||
| start->data = position; | ||
| start->ddl = std::move(position); |
There was a problem hiding this comment.
[cpplint] reported by reviewdog 🐶
Add #include for move [build/include_what_you_use] [4]
| uint64_t arrival_complete_ts, | ||
| const CdcScanRequest &request, | ||
| CdcScanResponse *response, | ||
| std::string *error) |
There was a problem hiding this comment.
[cpplint] reported by reviewdog 🐶
Add #include for string [build/include_what_you_use] [4]
| CdcTimestampPosition(3)}; | ||
| const CdcScanResponse page = ScanPage(data, {}, start, 10, 1000); | ||
| CHECK(page.data_size() == 0); | ||
| CHECK(page.ddl_size() == 0); |
There was a problem hiding this comment.
[cpplint] reported by reviewdog 🐶
Consider using CHECK_EQ instead of CHECK(a == b) [readability/check] [2]
| const CdcScanResponse page = ScanPage(data, {}, start, 10, 1000); | ||
| CHECK(page.data_size() == 0); | ||
| CHECK(page.ddl_size() == 0); | ||
| CHECK(page.arrival_complete_ts() == 10); |
There was a problem hiding this comment.
[cpplint] reported by reviewdog 🐶
Consider using CHECK_EQ instead of CHECK(a == b) [readability/check] [2]
|
|
||
| // No bound at all yet: still an empty, well-formed page. | ||
| const CdcScanResponse unbounded = ScanPage(data, {}, {}, 0, 1000); | ||
| CHECK(unbounded.data_size() == 0); |
There was a problem hiding this comment.
[cpplint] reported by reviewdog 🐶
Consider using CHECK_EQ instead of CHECK(a == b) [readability/check] [2]
| // No bound at all yet: still an empty, well-formed page. | ||
| const CdcScanResponse unbounded = ScanPage(data, {}, {}, 0, 1000); | ||
| CHECK(unbounded.data_size() == 0); | ||
| CHECK(unbounded.arrival_complete_ts() == 0); |
There was a problem hiding this comment.
[cpplint] reported by reviewdog 🐶
Consider using CHECK_EQ instead of CHECK(a == b) [readability/check] [2]
| // Below from_ts nothing is returned; above the bound nothing yet. | ||
| std::vector<FakeRecord> data{Rec(2, "a"), Rec(6, "a")}; | ||
| CdcScanResponse page = ScanPage(data, {}, start, 5, 1000); | ||
| CHECK(page.data_size() == 0); |
There was a problem hiding this comment.
[cpplint] reported by reviewdog 🐶
Consider using CHECK_EQ instead of CHECK(a == b) [readability/check] [2]
liangjchen
added a commit
to eloqdata/eloqdoc
that referenced
this pull request
Oct 29, 2026
eloqdata/tx_service#577 was squash-merged onto tx_service main. The merged commit's tree is identical to the pull request head this branch pointed at, so no file changes; the pointer now names a commit on main instead of one only on the pull request's branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
liangjchen
added a commit
to eloqdata/eloqdoc
that referenced
this pull request
Oct 29, 2026
… reference consumer (#506) Change data capture: change streams, cluster gate, CDC consumer (#506) Change data capture for EloqDoc: $changeStream served from the log service instead of an oplog, a cluster gate that holds DDL still while a consumer copies the catalog, and a reference consumer that copies a cluster into another MongoDB and then follows its stream. Design in docs/cdc-change-stream.md, plan in docs/cdc-implementation-plan.md, operator runbook in docs/cdc-operations.md. The data substrate side is eloqdata/tx_service#577; the submodule points at its merged commit. Change streams - aggregate with $changeStream at collection, database or deployment level, then getMore: insert, replace and delete events with full documents; create, drop and index DDL events; collection-level invalidate. Updates are reported as replace (no updateDescription). - The server merges the log groups by commit timestamp before emitting. - Resume tokens and postBatchResumeToken on every reply, the opening one included; resumeAfter and startAfter resume exactly across pages, log groups and leader changes. The legacy shell's cursor tracks tokens, and the PyMongo, Node.js and Java drivers' watch() work unchanged. - New server parameters cdc_open_wait_ms (default 30 s) and cdc_scan_max_bytes. New error codes ChangeStreamStartNotReady (5000), ChangeStreamHistoryLost (286) and ChangeStreamFatalError (280). Cluster gate - Admin command {eloqClusterGate: {op: "pause" | "resume"}}, needing the new eloqClusterGate action (held by clusterManager, so clusterAdmin and root). While paused, every schema change fails with ClusterGatePaused (5001). Reference consumer - tools/cdc_consumer/ bootstraps an empty sink under the gate, then follows the deployment stream, keeping its position in the sink's admin.__eloq_cdc_progress. Fixes the work found outside CDC - KeyString decoded a decimal with an integer part above 2^53 one unit low. _id point lookups decoded numbers as int32. - Indexes under construction: updates left stale keys, a sparse build aborted the server on a document without the field, and multikey documents wrote only their first key. - Unique index builds over existing documents are refused (CannotCreateIndex): the tx service's build does not detect duplicates. - View definitions are read from system.views on every lookup, so a view change on one worker thread or node is visible to the others. - Write-conflict retries yield the coroutine instead of sleeping the brpc worker. - Startup skips a collection dropped between listing and lookup. - A create refused for its options (a validator in local, admin or config or on a system collection, or an unknown validationLevel or validationAction) no longer leaves its table behind; before, the server could not restart afterwards. This was a bug on main too. Tests and CI - New jstest suites eloq_cdc and eloq_fault (the latter only with the new WITH_FAULT_INJECT CMake option), runtime targets for the drivers, DDL recovery, the cluster gate and the consumer, and multi-node tests in cmake/tests/multinode/ (not run by CI). - The SCons jobs' jstest server runs with test commands enabled. Behavior changes to note: unique index builds on non-empty collections now fail; every write-conflict retry loop yields instead of sleeping; views are read per lookup. With txlog_cdc_retention_seconds at its default 0, a consumer behind the checkpoint when a purge runs gets ChangeStreamHistoryLost and must re-bootstrap. Verified: GitHub CI on this head (CMake and both SCons jobs: unit tests, all four jstest suites, TPC-C, drivers, DDL recovery, cluster gate, consumer); the multi-node tests, local and cloud mode, passed on a VM before the create-option fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
EloqDoc is gaining change data capture: MongoDB-style change streams served from the log service, and a reference consumer that copies a cluster into another MongoDB. The design and its plan live in the EloqDoc repo (
docs/cdc-change-stream.md,docs/cdc-implementation-plan.md); the EloqDoc side is eloqdata/eloqdoc#506, which bumps its submodule to this branch. This PR is everything CDC needs from the data substrate, plus the fixes that EloqDoc's new multi-node tests found along the way, the move to one EloqStore per process, and a rework of shard memory eviction.The branch is 27 commits on top of current
main(no rebase needed). Each commit message is written to be read on its own; the history follows the plan's PR sequence, and every plan PR with a data substrate part has it here as one or more commits.Behavior before and after
Log service
CdcScanRPC: a log group's committed data records and its DDL history, merged in timestamp order, paged with an exact continuation, leader only (NotLeaderwith a redirect). Long polls when idle (cdc_scan_max_parked,cdc_scan_max_park_ms).txlog_cdc_retention_seconds(default 0) keeps SST files the purges would otherwise remove. At 0 both purges behave exactly as before.cdc_arrival_complete_ts{lg_id}andcdc_scan_parked_requests{lg_id}.Schema operations (affects every product)
max(last read ts, local time), so a schema op's timestamps are above its catalog write lock's grant; EloqKV's fast catalog read publishes its commit timestamp on release.ddl_kv_retry_interval_seconds(default 30 s) with a warning.holds_ckpt_wlock_).Cluster gate
CLUSTER_GATED. The pause drains admitted DDL through the cluster config read lock and returns a timestamp above all of them; replay reinstates it. Gaugecluster_gate_paused{ng_id}.Recovery and routing (found by multi-node tests)
TxLog::ReplayLogchanged accordingly (one log group, one attempt).UpsertRangesretries with the full range list (it retried with an empty list and silently wrote nothing).EloqStore
AttachShard/DetachShard, eloqdata/eloqstore PR 502), with one buffer pool, fd budget and local cache instead of one per shard. Pinned to the merged commit on EloqStoremain.AttachShard.Shard memory eviction
shard_clean_release_kbmore (default 256 KB; 0 means any amount); at the LRU tail whatever was reclaimed goes out.Smaller
checkpointer_intervalrefuses to start; 0 to 4 is raised to 5 s.table_name).CircularQueueincludes<cstdint>.Compatibility. The proto changes are additive. No setting changes behavior at its default except eviction pacing (a behavior change by design). See the risk section for downgrade and for EloqKV/EloqSQL.
Implementation
eloq_log_service/src/cdc_scan.cpp,include/cdc_long_poll.h,include/log_state.hand both RocksDB log states: the scan, its continuation codec and the merge; the DDL history and its snapshot file; the long poll; the retention floor inPurgingSstFilesandMoveFilesToArchive. The RocksDB state drains in-flight scans before closing its DB.tx_service/src/tx_operation.cpp(UpsertTableOp::AssignCommitStageTs,AbandonCreate),tx_index_operation.cpp,include/cc/template_cc_map.h(AcquireAllCc,PrepareCommit),catalog_cc_map.h: commit-stage timestamps, exact recovery from each log, kv work around the logs.docs/range-split-schema-op-recovery.mdrecords the lock a recovered op restores when a range split runs on the same table.tx_service/src/tx_cluster_gate_operation.cpp,include/cc/cluster_config_cc_map.h: the gate op, its admission check in the schema ops, its replay.tx_service/tx-log-protos/log_agent.cpp,sharder.cpp,tx_execution.cpp: bounded replay requests, term-ordered leader caches, the forwarded-key logging fix.store_handler/eloq_data_store_service/eloq_store_data_store*: one store, shards attached by term, the partition filter answered by each shard's data store.tx_service/src/cc/cc_req_misc.cpp(ShardCleanCc::Execute),cc_shard.{h,cpp},local_cc_shards.cpp: the eviction pass, the deferred retry on the timer thread, and the timer walk's shutdown ordering.docs/: the numbered architecture docs describe the result in present tense (01, 03 to 10).Design decisions and alternatives
shard_clean_release_kb.Test plan
Commands and results:
cpplint: on the current head the Reviewdog job ran and posted 29 inline comments, all in the
log service's CDC files: 21 Catch2
CHECK(a == b)assertions it takes for glog'sCHECK, 5missing standard includes (
<utility>,<list>,<memory>,<string>), 2using namespacedirectives in tests, and
<filesystem>flagged as an unapproved C++17 header. On the previoushead the job failed before linting: GitHub's API refuses this PR's diff (over its 20,000-line
limit) and reviewdog's fallback
git fetchfailed in the shallow checkout. Run locally with theworkflow's flags and filter over every changed file, cpplint reports 83 findings on lines this
branch adds: 35 Catch2 checks, 30 include order, 12 missing includes, 4
using namespace, 2C-style casts. None were changed in this PR.
Known flakes seen during these runs, not caused by this branch: a restarted node whose node group is led elsewhere can fail its first transaction (
TX_INIT_FAIL) before it follows the group (voter takeover, occasionally; reruns pass); EloqStore's cloud mode occasionally reports a corrupted manifest after akill -9.Risk assessment
ReplayLogforwarder. EloqKV builds against the data substrate's own log wrapper; its CI matrix passes against this branch (test plan).Rollback plan
Revert this PR (and the EloqDoc PR that bumps to it). A cluster that has already logged a cluster gate entry or saved a snapshot with DDL history should not be rolled back to an older binary without checking that its replay tolerates them.
Reviewer guide
tx_service/src/tx_operation.cppandtx_index_operation.cpp: the commit-stage timestamp, the point of no return, and what each log recovers to.docs/range-split-schema-op-recovery.mdis the companion.tx_service/include/cc/template_cc_map.h:AcquireAllCc's reported timestamp andPrepareCommit's catalog commit timestamp.eloq_log_service/src/cdc_scan.cppandinclude/log_state.h: the merge, the continuation (frozen; EloqDoc embeds it in resume tokens), the DDL history, the retention floor.tx_service/src/tx_cluster_gate_operation.cpp: the drain argument in its header comment.tx_service/src/tx_execution.cpp(FillDataLogRequest): the forwarded-key data loss fix.tx_service/src/cc/cc_req_misc.cpp(ShardCleanCc::Execute): the eviction rules, documented above the function.Follow-up work
ddl_dmlmode beyond parsing (EloqDoc plan §11).🤖 Generated with Claude Code