Repository navigation
fix: alert on unsendable cancels and stop paying 2 indexers - #737
Merged
MoonBoi9001 merged 7 commits intoOct 9, 2026
Conversation
A cancel the chain client refused to send, such as gas over the cap or a signer out of funds, counted as an outage, so it was retried forever without raising agreement_cancel_stuck. The chain is read just before each send, so every failed send now counts except an unchecked receipt.
Agreements marked Cancelling were only ever finished by a sweep inside the chain listener, which config can turn off, leaving them cancelling for good. The sweep now runs on its own every 5 minutes whatever the listener does, and a stop request cuts a slow sweep short.
When the cancel of an agreement whose indexer stopped serving it failed, a replacement was queued anyway, so dipper paid both indexers until a retry landed the cancel. The replacement now waits until the chain shows the old one ended, and the cancel retry queues it once it ends it.
When a replacement was accepted, an old agreement in a status dipper can't mark Cancelling, such as Unresponsive with its offer still open, got no on-chain cancel, so the indexer could accept it. It now gets one if the chain shows it live, and its pending row stays for retry if that fails.
A stale agreement whose cancel failed was replaced only if the cancel retry ended it, so one the chain listener ended first was never replaced. The agreement now records that a replacement is due, and the cancel retry queues it once the agreement has ended, whatever ended it, and only once.
A replaced agreement that can't be marked cancelling had its cancel resent every sweep even when the contract refused it or it mined without ending the agreement, costing gas each time with no alert. Those now raise agreement_cancel_stuck once and stop; a failed read or send still retries.
MoonBoi9001
marked this pull request as ready for review
October 8, 2026 20:44
That change cancelled a replaced Unresponsive agreement on-chain, assuming its offer could still be open. An agreement is only marked Unresponsive when the gRPC proposal fails, and dipper puts an offer on-chain only after the indexer accepts that proposal, so no such offer exists.
MoonBoi9001
added this pull request to stack #741
October 9, 2026 17:57
MoonBoi9001
removed this pull request from stack #741
October 9, 2026 17:58
MoonBoi9001
added this pull request to stack #742
October 9, 2026 17:58
MoonBoi9001
removed this pull request from stack #742
October 9, 2026 17:58
MoonBoi9001
added this pull request to stack #743
October 9, 2026 17:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A cancel that can never be sent now counts toward the stuck alert, and an agreement whose indexer stopped serving it is replaced only once it can't be paid, whatever ends it. The cancel retry also runs as its own service rather than inside the optional chain listener, which config can turn off.