Skip to content

fix(router): cap AnnounceFile tx_ids fan-out - #431

Merged
0g-peterzhb merged 2 commits into
mainfrom
@peter/fix-announce-file-fanout
Aug 21, 2026
Merged

fix(router): cap AnnounceFile tx_ids fan-out#431
0g-peterzhb merged 2 commits into
mainfrom
@peter/fix-announce-file-fanout

Conversation

@0g-peterzhb

@0g-peterzhb 0g-peterzhb commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Closes #423

on_announce_file expands one gossip message into msg.tx_ids.len() separate consumers. tx_ids is an uncapped Vec<TxID> and nothing on the receive path checked its length — the only bound was GOSSIP_MAX_SIZE_POST_MERGE = 10 MB, applied to the decompressed size, so a small snappy payload can carry ~262k TxIDs.

The sharp edge: quadratic memory in the file location cache

// node/file_location_cache/src/file_location_cache.rs:306-309
let mut cache = self.cache.lock();
for tx_id in announcement.tx_ids.iter() {
    cache.insert(*tx_id, announcement.clone());   // deep copy, once per tx_id
}

AnnounceFile.tx_ids is a plain Vec with no Arc, so each clone deep-copies the whole list. N clones each holding N entries is O(N²) memory from a single message. At 40 bytes per TxID:

tx_ids allocated
1,000 ~40 MB
10,000 ~4 GB
262,000 ~2.7 TB

max_entries_total = 1_000_000 does not help: the loop inserts N distinct tx_ids, so total_announcements == N after one message, well under the cap — no eviction fires. The limit counts announcements, not bytes, and each announcement here is up to 10 MB.

One signed message within the protocol's own size limit therefore OOMs the node, with no sustained bandwidth required.

Secondary: sync channel fan-out

Each tx_id also becomes its own SyncMessage::AnnounceFileGossip on an unbounded channel, and PubsubRateLimiter meters messages rather than their contents. Milder than it looks — with the default sync_file_on_announcement_enabled = false the drain early-returns after a hashmap lookup, so the queue keeps up under one peer. It matters mainly when that setting is on, where each message costs an inline get_tx_status RocksDB read.

Change

Reject anything above MAX_ANNOUNCE_FILE_TX_IDS (256) and penalise the sender.

This is a const, not a config field. Rejecting a gossip message also stops its propagation and scores the sender down, so nodes disagreeing on the threshold would penalise honest peers and make propagation depend on which node received the message. GOSSIP_MAX_SIZE_POST_MERGE — the only other bound on this same field — is a plain const for the same reason. Changing it means a coordinated network upgrade.

256 leaves a 25x margin over batcher_file_capacity = 10 in the shipped configs, and caps the cache cost at roughly 2.6 MB per message.

A deployment batching above the limit is a misconfiguration, not something peers should absorb by loosening their own validation, so batcher_file_capacity > MAX_ANNOUNCE_FILE_TX_IDS now fails at startup — once, locally, with a clear message — instead of silently earning peer penalties network-wide.

Scope

This bounds the blast radius; it does not fix the quadratic clone itself. That needs the cache to share one Arc<SignedAnnounceFile> across entries rather than cloning per tx_id — filed as #437.

Verification

cargo test -p router: 21 passed, including test_on_pubsub_announce_file_oversized_tx_ids, which asserts Reject and that nothing reached the sync layer. cargo fmt --all -- --check and cargo clippy -- -D warnings both clean.

🤖 Generated with Claude Code


This change is Reviewable

`on_announce_file` expands one gossip message into `msg.tx_ids.len()`
separate `send_to_sync` calls. `AnnounceFile.tx_ids` is an uncapped
Vec<TxID> and nothing on the receive path checked its length: the only
bound was GOSSIP_MAX_SIZE_POST_MERGE, applied to the *decompressed*
size, so one accepted 10 MB message can carry ~262k TxIDs and enqueue
all of them on the unbounded sync channel.

The per-peer PubsubRateLimiter does not help - it counts messages, not
their contents - so the router amplifies rather than paces. A small
snappy payload that expands to the 10 MB limit is enough.

Reject anything above `max_announce_file_tx_ids` (default 256) and
penalise the sender. Honest senders bound their own lists with
`batcher_file_capacity`, which the shipped configs set to 10, so the
default leaves a 25x margin over real traffic while cutting the
worst case by three orders of magnitude. It is configurable for
deployments that batch more aggressively.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A per-node threshold is the wrong shape for this. Rejecting a gossip
message also stops its propagation and scores the sender down, so nodes
disagreeing on the limit would penalise honest peers and make
propagation depend on which node received the message. An operator on 16
and one on 256 do not compose into anything coherent.

It also had the escape-hatch argument backwards: a knob that can be set
lower is a footgun, and one set higher only helps if the whole network
raises it together. `GOSSIP_MAX_SIZE_POST_MERGE` - the only other bound
on this same field - is a plain const for exactly this reason.

Replace the config field with `MAX_ANNOUNCE_FILE_TX_IDS`. Changing it
now means a coordinated upgrade, which is the honest requirement.

Handle the case that motivated the knob at the correct end instead: a
deployment batching above the limit is not something peers should absorb
by loosening their own validation, it is a misconfiguration. Reject
`batcher_file_capacity > MAX_ANNOUNCE_FILE_TX_IDS` at startup so it
fails once, locally, with a clear message - rather than silently earning
peer penalties across the network.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@0g-peterzhb
0g-peterzhb merged commit 67f7a3f into main Aug 21, 2026
6 checks passed
@0g-peterzhb
0g-peterzhb deleted the @peter/fix-announce-file-fanout branch August 21, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

One AnnounceFile gossip message can enqueue ~262k sync messages

1 participant