Benchmark stress phase stalls at 0 TPS on `main` — distributed reproduction case (fixed by #50)

[keanji-x]

Summary

When reproducing the Gravity benchmark with gravity_bench built from main, funding completes but the stress phase never makes progress — the Monitor sits at Progress 0/N, TPS 0.0, In Progress 1, Processing 0 indefinitely (observed >15 min with no advance, no error in the log). The chain is healthy and producing blocks the whole time.

Root cause is the connection leak + faucet U256 underflow fixed in #50 (fix/connection-leak-and-faucet-underflow). main already carries the fee fixes (#46/#48, 50 Gwei min-base-fee alignment) but not #50, and #50 is the one that unblocks the stress phase. Building the bench from the #50 branch is required to get a TPS number.

Environment / topology (distributed)

Run on the gravity-internal-test gcloud nodes, as the gravity user:

• Cluster machine gravity-test-node-1 (internal 10.128.0.51): 3 genesis validators on 8545/8546/8547, single-machine deployment, RPC bound 0.0.0.0, txpool_max_account_slots = 1024.
• Bench machine gravity-test-node-2 (internal 10.128.0.50): gravity_bench only, bench_config nodes → http://10.128.0.51:8545|8546|8547 (chain_id 1337).

"Distributed" = the load generator is on its own box; the validator cluster itself stays single-machine.

Reproduction

1. Cluster (node1) — build gravity_node and gravity_cli (MODE=quick-release), then bring up the 3-validator cluster with txpool_max_account_slots = 1024 and 0.0.0.0 RPC. Verify all three RPCs advance, and that node2 can reach http://10.128.0.51:8545.
2. Bench (node2) — git clone --branch fix/connection-leak-and-faucet-underflow … gravity_bench; bash setup.sh (Uniswap repos + npm @openzeppelin + python venv — deploy.py compiles the Uniswap sources even with enable_swap_token=false); cargo build --release.
3. bench_config.toml: nodes = node1 internal IP :8545/8546/8547, target_tps=9000, num_accounts=10000 (or 100000), num_senders=600, duration_secs=120, enable_swap_token=false.
4. rm -f deploy.json snapshot.json accounts.txt; source venv/bin/activate; cargo run --release -- --config bench_config.toml.
5. Read sustained TPS from the TPS cell in ./log.<ts>.log once Progress advances.

Expected vs actual

	main	#50 branch
funding (Ready Accounts)	reaches target	reaches target
stress Progress	stuck 0/N	advances
TPS	0.0 (stall)	sustained throughput

Other gotchas hit while reproducing (cluster-side, not gravity_bench bugs)

• reth default txpool.max-account-slots=16 throttles 100k-account funding to ~100 min → set 1024.
• cluster/init.sh needs gravity_cli built (not just gravity_node).
• cluster/deploy.sh prompts interactively if a data dir exists → wipe before deploy in non-interactive runs.

Ask

Merge #50 (or cut a release/tag off it) so a from-main benchmark run reaches the stress phase. A short "distributed reproduction" note in the README/docs would also help the next person.

[keanji-x]

Single-host reproduction + verified root cause

Follow-up: reproduced the stall on a single machine (not just the distributed gcloud setup), and traced the root cause through the code. Posting both because the single-host repro proves the bug is in gravity_bench logic, not the topology, and the symptom is sharper than first described.

Single-host repro (the bug is not topology-dependent)

One box, no Docker, 3 genesis validators + 1 VFN on 127.0.0.1 (benchmark_doc_repro harness, SDK_DIR = a local prebuilt gravity-sdk). gravity_bench built from main (0713edc, pre-#50), num_accounts=10000, num_senders=600, target_tps=9000, duration_secs=120, enable_swap_token=false.

The chain is healthy and producing blocks the whole time. The bench:

1. Contract deploy: OK.
2. Funding: Ready Accounts reaches 10.0K — looks complete.
3. Stress wedges. Sharper than "stuck at 0/N": Progress briefly advances (10 → 12 → 21 → 46/110, TPS peaks ~2.7), then freezes at 46/110 with TPS 0.0 for the rest of the run. Pool Pending = 0, Pool Queued stuck at 9, pending txns all time out, Send Failures = 0, Exec Failures = 0.

Independent cross-check (chain-derived, eth_getBlockByNumber over the last 120 s): tps=4.4 (528 txs across blocks 9741–10268) — i.e. ~0 effective throughput against a 9000 target.

Smoking gun (full bench log — not the Monitor table)

The Monitor table is silent, but the full log shows the real failure:

amount_per_recipient=11579208923731619542357098500868790785326998466564056403485596400791312963
ERROR gravity_bench::actors::consumer::actor: Permanent error for txn …:
  insufficient funds for gas * price + value: have 0
  want 1157920892373161954235709850086879078532699846656405640404126640079131296300, marking as failed (no retry)

That ~2²⁵⁶ "want" value is the faucet U256 underflow manifesting.

Root cause

A. Faucet U256 underflow (primary, directly evidenced). src/txn_plan/constructor/faucet.rs:78 and :103:

let amount_for_leaves = faucet_balance - total_cost;       // :78  (cascade mode)
let amount_for_leaves = faucet_balance - total_gas_cost;   // :103 (direct mode)

ruint/U256 operators wrap, so when the configured faucet balance is below the budgeted cost this does not error — it wraps to a near-2²⁵⁶ amount. Every recipient is then asked to receive an impossible amount, so funding txns are rejected insufficient funds (have 0) and accounts end up unfunded. Their later stress txns can't pay → sit in txpool queued forever → never promote to pending → Pool Queued 9, TPS 0.

B. Connection churn under load via tokio::time::timeout. src/eth/eth_cli.rs:388 (send_raw_tx), :156 (get_pending_txn_count), :448 (get_latest_txn_count) all wrap the reqwest/alloy HTTP send in tokio::time::timeout(Duration::from_secs(10), …). Cancelling an in-flight HTTP future drops it mid-read and leaks/discards the socket under high concurrency. Each sender holds an OwnedSemaphorePermit (src/actors/consumer/actor.rs:179/190/212, num_senders plumbing from src/main.rs:465) for the full send, so once sends stop completing the senders stop making forward progress — Progress freezes.

C. Why it looks silent (accounting amplifier). src/actors/consumer/actor.rs:277-305: permanent errors (insufficient funds, gas limit exceeded, …) log error! … marking as failed, but then dispatch SubmissionResult::NonceTooLow { …placeholder 0… }. src/actors/monitor/txn_tracker.rs:307 handles NonceTooLow by inserting a (fake) PendingTxInfo into pending_txns instead of counting a send failure, and the producer unlocks the account on that result (src/actors/producer/producer_actor.rs:433). Net effect: Ready Accounts shows 10.0K and Send Failures shows 0 even though funding actually failed — the failure only appears as ERROR lines in the full log, never in the headline Monitor table.

How #50 fixes it

git diff origin/main..origin/fix/connection-leak-and-faucet-underflow confirms:

• faucet.rs: adds assert!(faucet_balance >= total_cost, …) / assert!(faucet_balance >= total_gas_cost, …) before the subtraction → the underflow can no longer wrap silently.
• main.rs: queries the real on-chain faucet balance and computes effective_faucet_eth, clamping the distribution against reality before building the ETH faucet plan; also drops nonce-init fanout 1024 → 16 (RPC-overload hardening).
• eth_cli.rs: replaces the tokio::time::timeout wrappers with reqwest's own .timeout(Duration::from_secs(10)), which times out without dropping/leaking the socket.

Residual concern (not fixed by #50)

The accounting amplifier (C) is untouched: permanent errors are still mapped to SubmissionResult::NonceTooLow and still tracked as pending rather than as a hard send failure. If any other permanent error class (e.g. gas limit exceeded) shows up later, the Monitor table will again read Send Failures = 0 and wait on tx hashes that were never accepted. Worth surfacing these in the Monitor as real failures.

Minor: bench ignores duration_secs when wedged

In the stalled state the bench did not stop at duration_secs=120 — it hung indefinitely (observed ~48 min) because the frozen in-flight plan never completes. A wedged run should still honor the duration cap and exit.