Skip to content

Commit 4f5e58c

Browse files
meiji163grodowski
andauthored
Panic if InitiateHeartbeat exhausts retries to avoid looping infinitely (#1594)
* Panic if InitiateHeartbeat exhausts retries to avoid looping infinitely. Based on experience, if the writer database fails inbeetween the copy & cutover stages (e.g. during cutover pause), the heartbeat writes will fail and stop, then leading to throttled state and an infinite loop of throttler.shouldThrottle(). Since this state is irrecoverable, make the heartbeat writer panic if retries are exhausted, so that the migration can fail and be restarted later. * Add sysbench localtest (#1590) * add sysbench localtest * fix table name * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * ensure cleanup --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * add toxiproxy option for localtests (#1591) --------- Co-authored-by: Jan Grodowski <jan.grodowski@shopify.com>
1 parent 6b676f1 commit 4f5e58c

1 file changed

Lines changed: 1 addition & 0 deletions

File tree

go/logic/applier.go

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -575,6 +575,7 @@ func (this *Applier) InitiateHeartbeat() {
575575
continue
576576
}
577577
if err := injectHeartbeat(); err != nil {
578+
this.migrationContext.PanicAbort <- fmt.Errorf("injectHeartbeat writing failed %d times, last error: %w", numSuccessiveFailures, err)
578579
return
579580
}
580581
}

0 commit comments

Comments
 (0)