fix(zero): size-aware predicate move timeout and rebalancer backoff

[matthewmcneely]

Summary

Zero cancels every predicate move at a hardcoded 120 minutes (predicateMoveTimeout in dgraph/cmd/zero/tablet.go), so moves of large tablets can never complete. The report in #9784 is a 20GB predicate that needed about 16 hours. This constant has been raised once before for the same reason, from 20 minutes to 2 hours in #6388, so any fixed wall-clock value is eventually wrong for someone's data size.

The failure mode is worse than a cancelled move. While a move runs, Zero aborts every commit on the predicate (blockTablet, enforced in oracle.go). On timeout, all streamed work is discarded, and the auto-rebalancer deterministically re-picks the same largest tablet every --rebalance_interval (default 8m). A cluster with one oversized predicate ends up in a permanent loop: two hours of write aborts and streaming, cancel, discard, repeat.

Changes

• Scale the move timeout with tablet size. moveTimeout() grants at least size / 256 KiB-per-second, using the larger of on-disk and uncompressed bytes, with the previous 2h constant as the floor. Tablets under ~1.8GiB behave exactly as before; the 20GB tablet from Unblock large (>15GB) predicate moves by removing hardcoded 2-hour timeout #9784 gets ~23h.
• Back off after a failed move. A failure that ran at least a minute puts the tablet on a cooldown that doubles per consecutive failure (1h up to 24h) and is never shorter than the failed attempt itself. chooseTablet skips tablets on cooldown, a successful move clears the state, and manual moves via /moveTablet bypass the backoff.
• Log the computed timeout in the move announcement line so operators can see the budget each move was given.

A configurable flag was considered and dropped. A floor-style flag reads ambiguously against a "timeout" name, and a cap-style flag recreates the original bug; the size-derived value needs no configuration.

Why 256 KiB/s

The receive path applies the whole stream through serial 32MB Raft proposals (batchAndProposeKeyValues in worker/predicate_move.go): each chunk is replicated to quorum, written to the Raft WAL, and fully applied before the next is proposed. Sustained rates in the field are therefore low; the #9784 report works out to ~0.35 MB/s on-disk. The floor rate is deliberately conservative. The throughput problem itself is out of scope here and tracked in #9790 (pipeline the receive path) and #9791 (two-phase move to shrink the commit-abort window).

Testing

• New unit tests for the timeout computation and backoff bookkeeping in dgraph/cmd/zero/tablet_test.go (TestMoveTimeout, TestMoveCooldown, TestMoveBackoff).
• go build, gofmt, and trunk pass.
• Existing move coverage (systest/group-delete TestNodes, systest/integration2 TestUniqueMultipleGroups) moves small tablets, so the 2h floor applies and behavior there is unchanged.

Fixes #9784

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}