Repository navigation
fix: retry with real backoff, and only for replayable methods - #109
Merged
Merged
Conversation
Two defects in the retry configuration that sit next to the timeout, both
reachable by every consumer of the SDK.
1. The retry interval was ten nanoseconds
SetCommonRetryFixedInterval(10)
10 is an untyped constant, so it converts to time.Duration(10) - ten
NANOseconds, not ten seconds. The three retries therefore fired back to back
with no pause at all. That does nothing for an endpoint that is slow or
briefly unavailable; it just multiplies the request count by four, and with no
jitter every caller retries in lockstep.
Replaced with req/v3's own backoff, which is exponential with jitter:
SetCommonRetryBackoffInterval(httpRetryIntervalMin, httpRetryIntervalMax) // 1s -> 8s
The three magic numbers are now named constants so the unit is explicit and
the next edit cannot silently reintroduce a bare 10.
2. Mutating requests were retried after a timeout
req/v3's default condition is `needRetry := err != nil` - retry on any
transport error, for any method. A POST that times out is the dangerous case:
the server may well have accepted it and still be working, so re-issuing
creates a duplicate. For CreateBlockVolume that is a duplicate volume, billed
and orphaned.
retryOnlyIdempotent now gates it. The method list follows Go's own
net/http Request.isReplayable(): GET, HEAD, OPTIONS, TRACE. PUT and DELETE are
idempotent by HTTP definition but are deliberately left out - a detach or
delete that timed out and is then re-issued can still land on IaaS state that
is mid-transition. Callers that want to retry a mutating call should do it at
a layer that can re-read state first and decide.
The condition preserves the previous behaviour in every other respect: still
only on transport errors, never on an HTTP status code, and context
cancellation still short-circuits ahead of it.
Worst case for a single call is unchanged in shape - up to 4 attempts for a
replayable method, so 4 x the timeout plus roughly 15s of backoff. Bounding
that properly is the caller's job, via its own context or WithTimeout().
Tests: pkg vngcloud/client
- truth table over all methods, retried vs not, plus the success case
- resp == nil and resp.Request == nil must not panic; RetryConditionFunc is
called before a response exists
- the interval constants are a real duration, ordered, and the whole retry
sequence stays short relative to one timeout
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two defects in the retry configuration that sit next to the timeout, both reachable by every consumer of the SDK.
The retry interval was ten nanoseconds
SetCommonRetryFixedInterval(10)
10 is an untyped constant, so it converts to time.Duration(10) - ten NANOseconds, not ten seconds. The three retries therefore fired back to back with no pause at all. That does nothing for an endpoint that is slow or briefly unavailable; it just multiplies the request count by four, and with no jitter every caller retries in lockstep.
Replaced with req/v3's own backoff, which is exponential with jitter:
The three magic numbers are now named constants so the unit is explicit and the next edit cannot silently reintroduce a bare 10.
req/v3's default condition is
needRetry := err != nil- retry on any transport error, for any method. A POST that times out is the dangerous case: the server may well have accepted it and still be working, so re-issuing creates a duplicate. For CreateBlockVolume that is a duplicate volume, billed and orphaned.retryOnlyIdempotent now gates it. The method list follows Go's own net/http Request.isReplayable(): GET, HEAD, OPTIONS, TRACE. PUT and DELETE are idempotent by HTTP definition but are deliberately left out - a detach or delete that timed out and is then re-issued can still land on IaaS state that is mid-transition. Callers that want to retry a mutating call should do it at a layer that can re-read state first and decide.
The condition preserves the previous behaviour in every other respect: still only on transport errors, never on an HTTP status code, and context cancellation still short-circuits ahead of it.
Worst case for a single call is unchanged in shape - up to 4 attempts for a replayable method, so 4 x the timeout plus roughly 15s of backoff. Bounding that properly is the caller's job, via its own context or WithTimeout().
Tests: pkg vngcloud/client