Skip to content

Let indexing recover by itself after a database hiccup - #74

Merged
minuszka merged 3 commits into
defcon-project:mainfrom
minuszka:fix/a2b-sync-lock-expiry
Nov 14, 2026
Merged

minuszka merged 3 commits into
defcon-project:mainfrom
minuszka:fix/a2b-sync-lock-expiry

Conversation

@minuszka

Copy link
Copy Markdown
Contributor

If the database failed at the wrong moment, the indexer could stay blocked until the next restart, and a database reconnect started a second copy of the indexer and of the Telegram bot.

  • The indexing lock now expires: if its holder has shown no sign of life for 10 minutes (each indexed block and each step of an address rebuild counts), the next run takes it over and first rebuilds the address balances. Before, a lock left behind by a failed run made every later run fail until a restart.
  • When recording an indexing error fails as well, the original error is still logged and the run ends normally.
  • Starting the indexer or the Telegram bot a second time, as happens after a database reconnect, no longer starts a second copy.
  • Checked with database tests for an expired lock, a recent lock, a failed error record, a long rebuild and repeated starts; each part of the fix was undone once to confirm that a test fails without it.

If indexing a block stopped halfway, for example on a database error, the transactions already stored were treated as done when the block was indexed again, and their balance changes were silently lost.

- The indexer now marks a block as in progress before its first write and clears the mark after its last. If anything fails in between, the next run first rebuilds the address balances from the stored transactions.
- A normal shutdown now waits up to 15 seconds for the block being indexed and stops between blocks, so a restart during a deploy no longer leaves half a block behind. If the block does not finish in time, startup recovery rebuilds as before.
- Each new block costs two small extra writes to the sync state.
- Checked with database tests that interrupt indexing at each step (a transaction write, the balance write, the block write, a shutdown) and compare every address balance with the expected one; each part of the fix was undone once to confirm that a test fails without it.
If the database failed at the wrong moment, the indexer could stay blocked until the next restart, and a database reconnect started a second copy of the indexer and of the Telegram bot.

- The indexing lock now expires: if its holder has shown no sign of life for 10 minutes (each indexed block and each step of an address rebuild counts), the next run takes it over and first rebuilds the address balances. Before, a lock left behind by a failed run made every later run fail until a restart.
- When recording an indexing error fails as well, the original error is still logged and the run ends normally.
- Starting the indexer or the Telegram bot a second time, as happens after a database reconnect, no longer starts a second copy.
- Checked with database tests for an expired lock, a recent lock, a failed error record, a long rebuild and repeated starts; each part of the fix was undone once to confirm that a test fails without it.
@minuszka
minuszka marked this pull request as ready for review November 14, 2026 06:35
@minuszka
minuszka merged commit 463a4f7 into defcon-project:main Nov 14, 2026
1 check passed
@minuszka
minuszka deleted the fix/a2b-sync-lock-expiry branch November 14, 2026 09:54
minuszka added a commit to minuszka/explorer that referenced this pull request Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant