Skip to content

Keep address data complete while the address index is rebuilt - #76

Merged
minuszka merged 5 commits into
defcon-project:mainfrom
minuszka:fix/a2d-address-rebuild-swap
Nov 14, 2026
Merged

minuszka merged 5 commits into
defcon-project:mainfrom
minuszka:fix/a2d-address-rebuild-swap

Conversation

@minuszka

Copy link
Copy Markdown
Contributor

Rebuilding the address data (after an interrupted indexing run, and once after the previous change) first emptied it, so the rich list, the address pages and the address statistics showed incomplete data until the rebuild finished.

  • The rebuild now writes into a separate collection with the same indexes and swaps it in only when it is complete, so visitors keep seeing the previous, complete data in the meantime.
  • If a rebuild fails, the previous data stays as it was, the unfinished copy is removed, and the rebuild is tried again on the next run.
  • Checked with database tests that read the address data and the rich list in the middle of a rebuild, make a rebuild fail, and check the indexes of the swapped-in data; each part of the change was undone once to confirm that a test fails without it.

If indexing a block stopped halfway, for example on a database error, the transactions already stored were treated as done when the block was indexed again, and their balance changes were silently lost.

- The indexer now marks a block as in progress before its first write and clears the mark after its last. If anything fails in between, the next run first rebuilds the address balances from the stored transactions.
- A normal shutdown now waits up to 15 seconds for the block being indexed and stops between blocks, so a restart during a deploy no longer leaves half a block behind. If the block does not finish in time, startup recovery rebuilds as before.
- Each new block costs two small extra writes to the sync state.
- Checked with database tests that interrupt indexing at each step (a transaction write, the balance write, the block write, a shutdown) and compare every address balance with the expected one; each part of the fix was undone once to confirm that a test fails without it.
If the database failed at the wrong moment, the indexer could stay blocked until the next restart, and a database reconnect started a second copy of the indexer and of the Telegram bot.

- The indexing lock now expires: if its holder has shown no sign of life for 10 minutes (each indexed block and each step of an address rebuild counts), the next run takes it over and first rebuilds the address balances. Before, a lock left behind by a failed run made every later run fail until a restart.
- When recording an indexing error fails as well, the original error is still logged and the run ends normally.
- Starting the indexer or the Telegram bot a second time, as happens after a database reconnect, no longer starts a second copy.
- Checked with database tests for an expired lock, a recent lock, a failed error record, a long rebuild and repeated starts; each part of the fix was undone once to confirm that a test fails without it.
The transaction count at the top of an address page could be lower than the real number. During normal indexing, an address that took part in several transactions of the same block was counted once for that block, while a rebuild of the address data counted every transaction.

- Normal indexing now counts each transaction, as a rebuild does; both use the same code.
- To correct the counts already stored, the first start after this update rebuilds the address data once and records that it has done so; later starts do not rebuild. On the live database the rebuild takes a while, and address data may be incomplete while it runs.
- Checked with database tests that index the same small chain normally and by rebuild and compare every address field, and that the one-time rebuild runs exactly once; each part of the change was undone once to confirm that a test fails without it.
Rebuilding the address data (after an interrupted indexing run, and once after the previous change) first emptied it, so the rich list, the address pages and the address statistics showed incomplete data until the rebuild finished.

- The rebuild now writes into a separate collection with the same indexes and swaps it in only when it is complete, so visitors keep seeing the previous, complete data in the meantime.
- If a rebuild fails, the previous data stays as it was, the unfinished copy is removed, and the rebuild is tried again on the next run.
- Checked with database tests that read the address data and the rich list in the middle of a rebuild, make a rebuild fail, and check the indexes of the swapped-in data; each part of the change was undone once to confirm that a test fails without it.
@minuszka
minuszka marked this pull request as ready for review November 14, 2026 10:03
@minuszka
minuszka merged commit d8d8ba1 into defcon-project:main Nov 14, 2026
1 check passed
@minuszka
minuszka deleted the fix/a2d-address-rebuild-swap branch November 14, 2026 10:14
minuszka added a commit to minuszka/explorer that referenced this pull request Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant