fix(router): cap AnnounceFile tx_ids fan-out - #431
Merged
Conversation
`on_announce_file` expands one gossip message into `msg.tx_ids.len()` separate `send_to_sync` calls. `AnnounceFile.tx_ids` is an uncapped Vec<TxID> and nothing on the receive path checked its length: the only bound was GOSSIP_MAX_SIZE_POST_MERGE, applied to the *decompressed* size, so one accepted 10 MB message can carry ~262k TxIDs and enqueue all of them on the unbounded sync channel. The per-peer PubsubRateLimiter does not help - it counts messages, not their contents - so the router amplifies rather than paces. A small snappy payload that expands to the 10 MB limit is enough. Reject anything above `max_announce_file_tx_ids` (default 256) and penalise the sender. Honest senders bound their own lists with `batcher_file_capacity`, which the shipped configs set to 10, so the default leaves a 25x margin over real traffic while cutting the worst case by three orders of magnitude. It is configurable for deployments that batch more aggressively. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A per-node threshold is the wrong shape for this. Rejecting a gossip message also stops its propagation and scores the sender down, so nodes disagreeing on the limit would penalise honest peers and make propagation depend on which node received the message. An operator on 16 and one on 256 do not compose into anything coherent. It also had the escape-hatch argument backwards: a knob that can be set lower is a footgun, and one set higher only helps if the whole network raises it together. `GOSSIP_MAX_SIZE_POST_MERGE` - the only other bound on this same field - is a plain const for exactly this reason. Replace the config field with `MAX_ANNOUNCE_FILE_TX_IDS`. Changing it now means a coordinated upgrade, which is the honest requirement. Handle the case that motivated the knob at the correct end instead: a deployment batching above the limit is not something peers should absorb by loosening their own validation, it is a misconfiguration. Reject `batcher_file_capacity > MAX_ANNOUNCE_FILE_TX_IDS` at startup so it fails once, locally, with a clear message - rather than silently earning peer penalties across the network. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #423
on_announce_fileexpands one gossip message intomsg.tx_ids.len()separate consumers.tx_idsis an uncappedVec<TxID>and nothing on the receive path checked its length — the only bound wasGOSSIP_MAX_SIZE_POST_MERGE= 10 MB, applied to the decompressed size, so a small snappy payload can carry ~262k TxIDs.The sharp edge: quadratic memory in the file location cache
AnnounceFile.tx_idsis a plainVecwith noArc, so each clone deep-copies the whole list. N clones each holding N entries is O(N²) memory from a single message. At 40 bytes perTxID:max_entries_total = 1_000_000does not help: the loop inserts N distinct tx_ids, sototal_announcements == Nafter one message, well under the cap — no eviction fires. The limit counts announcements, not bytes, and each announcement here is up to 10 MB.One signed message within the protocol's own size limit therefore OOMs the node, with no sustained bandwidth required.
Secondary: sync channel fan-out
Each tx_id also becomes its own
SyncMessage::AnnounceFileGossipon an unbounded channel, andPubsubRateLimitermeters messages rather than their contents. Milder than it looks — with the defaultsync_file_on_announcement_enabled = falsethe drain early-returns after a hashmap lookup, so the queue keeps up under one peer. It matters mainly when that setting is on, where each message costs an inlineget_tx_statusRocksDB read.Change
Reject anything above
MAX_ANNOUNCE_FILE_TX_IDS(256) and penalise the sender.This is a const, not a config field. Rejecting a gossip message also stops its propagation and scores the sender down, so nodes disagreeing on the threshold would penalise honest peers and make propagation depend on which node received the message.
GOSSIP_MAX_SIZE_POST_MERGE— the only other bound on this same field — is a plain const for the same reason. Changing it means a coordinated network upgrade.256 leaves a 25x margin over
batcher_file_capacity = 10in the shipped configs, and caps the cache cost at roughly 2.6 MB per message.A deployment batching above the limit is a misconfiguration, not something peers should absorb by loosening their own validation, so
batcher_file_capacity > MAX_ANNOUNCE_FILE_TX_IDSnow fails at startup — once, locally, with a clear message — instead of silently earning peer penalties network-wide.Scope
This bounds the blast radius; it does not fix the quadratic clone itself. That needs the cache to share one
Arc<SignedAnnounceFile>across entries rather than cloning per tx_id — filed as #437.Verification
cargo test -p router: 21 passed, includingtest_on_pubsub_announce_file_oversized_tx_ids, which assertsRejectand that nothing reached the sync layer.cargo fmt --all -- --checkandcargo clippy -- -D warningsboth clean.🤖 Generated with Claude Code
This change is