Skip to content

events: make the event upload rate configurable as max_upload_rate_mbps - #12471

Open
kbhargava-jump wants to merge 5 commits into
mainfrom
kbhargava/event-upload-rate
Open

kbhargava-jump wants to merge 5 commits into
mainfrom
kbhargava/event-upload-rate

Conversation

@kbhargava-jump

Copy link
Copy Markdown
Contributor

No description provided.

Copilot AI balanced review requested due to automatic review settings October 21, 2026 19:34
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
 ┌─ ⚡ PERF · ef8dba2 vs main@5ca81ea ─────────────────────────────────
 │ SUITE                               BASELINE          NEW         Δ
 │ replay tps, mainnet               28,537 tps   28,420 tps  ·  -0.41%
 │ bench tps, localnet              669,742 tps  670,697 tps  ·  +0.14%
 │ snapshot load, testnet                3.78 s       3.79 s  ·  +0.25%
 │ mem total, mainnet                146.36 GiB   146.36 GiB  ·   0.00%
 │ mem total, testnet                 87.25 GiB    87.25 GiB  ·   0.00%
 │ mem total, ag mainnet             147.17 GiB   147.17 GiB  ·   0.00%
 │ clean compile, firedancer             3.43 s       3.45 s  ·  +0.33%
 │ binary size, firedancer             59.94 MB     59.95 MB  ·  +0.01%
 ├─────────────────────────────────────────────────────────────────────
@@ 0 REGRESSIONS · 0 WARNINGS · 0 IMPROVED · 8 NOISE @@
 └─────────────────────────────────────────────────────────────────────
history · 5 pushes
 ┌─ HISTORY · Δ vs main, per push, newest first ──────────────────────────────────
 │ HEAD         TPS    BENCH     SNAP    MEM·M    MEM·T     AG·M  COMPILE   BINARY
 │ ef8dba2   -0.41%   +0.14%   +0.25%    0.00%    0.00%    0.00%   +0.33%   +0.01%
 │ 7d53855   +0.17%   -0.70%   -0.45%    0.00%    0.00%    0.00%   +0.95%   +0.01%
 │ 212a159        …        …        …    0.00%    0.00%    0.00%        …   +0.01%
 │ ae1a589   +0.48%   -0.23%   -0.18%    0.00%    0.00%    0.00%   +1.05%   +0.01%
 │ ccabb71   +0.64%   +0.31%   -1.05%    0.00%    0.00%    0.00%   +0.18%   +0.01%
 └────────────────────────────────────────────────────────────────────────────────

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Low-rate pacing and stalled stream rotation can cause unnecessary repeated polling work.

2 open issues
What changed in this PR

Makes telemetry upload bandwidth configurable through max_upload_rate_mbps, retaining the 32 Mbps default.

Changes:

  • Adds configuration parsing, JSON output, and topology wiring.
  • Updates client pacing and introduces bounded backlog draining.
  • Adapts existing tests to the new constructor parameter.
File Description
src/​disco/​topo/​fd_topo.h Stores the event upload rate.
src/​disco/​events/​test_event_client.c Updates constructors and default-rate references.
src/​disco/​events/​fd_event_tile.c Passes the configured rate to the client.
src/​disco/​events/​fd_event_client.h Extends the constructor and exposes the default.
src/​disco/​events/​fd_event_client.c Implements configurable pacing and backlog draining.
src/​app/​shared/​fd_config.h Adds the upload-rate setting.
src/​app/​shared/​fd_config_parse.c Parses the setting.
src/​app/​shared/​fd_config_json.c Serializes the setting and updates the layout assertion.
src/​app/​firedancer/​topology.c Converts Mbps to bytes per second.
src/​app/​firedancer/​config/​default.toml Documents the setting and default.

Review effort: Balanced

Comment thread src/disco/events/fd_event_client.c Outdated
Comment thread src/disco/events/fd_event_client.c Outdated
Copilot AI balanced review requested due to automatic review settings October 21, 2026 21:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unchecked rate conversion can cause division by zero, and explicit non-default rates lack pacing tests.

3 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Add pacing tests for explicit rates and fractional refill timing

src/​disco/​events/​test_event_client.c:268

All three test constructors pass 0UL, so the pacing tests exercise only the 32 Mbps default. They do not verify that an explicit rate is honored or exercise refill/deadline rounding when nanoseconds per byte is fractional. Parameterize this helper and add pacing cases below and above the default (for example, 3 and 64 Mbps), checking token accrual and wake-up timing. Keep a zero-rate case to verify the default behavior.

Review effort: Balanced

Comment thread src/disco/events/fd_event_client.c
Copilot AI balanced review requested due to automatic review settings October 21, 2026 22:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Accepted upload rates can overflow signed deadline arithmetic, causing undefined behavior in pacing.

1 open issue
3 resolved since last review

Review effort: Balanced

Comment thread src/disco/events/fd_event_client.c Outdated
Copilot AI balanced review requested due to automatic review settings October 21, 2026 22:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The accepted upload-rate range permits signed overflow in the pacing deadline calculation.

1 open issue

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 21, 2026 23:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The multi-send polling and transport-flushing changes need runtime validation under TLS and socket backpressure before final human approval.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants