Skip to content

fix: retry with real backoff, and only for replayable methods - #109

Merged
vks-team merged 1 commit into
mainfrom
tytv2/retry-backoff-and-idempotency
Aug 28, 2026
Merged

vks-team merged 1 commit into
mainfrom
tytv2/retry-backoff-and-idempotency

Conversation

@vks-team

Copy link
Copy Markdown
Contributor

Two defects in the retry configuration that sit next to the timeout, both reachable by every consumer of the SDK.

  1. The retry interval was ten nanoseconds

    SetCommonRetryFixedInterval(10)

10 is an untyped constant, so it converts to time.Duration(10) - ten NANOseconds, not ten seconds. The three retries therefore fired back to back with no pause at all. That does nothing for an endpoint that is slow or briefly unavailable; it just multiplies the request count by four, and with no jitter every caller retries in lockstep.

Replaced with req/v3's own backoff, which is exponential with jitter:

SetCommonRetryBackoffInterval(httpRetryIntervalMin, httpRetryIntervalMax)  // 1s -> 8s

The three magic numbers are now named constants so the unit is explicit and the next edit cannot silently reintroduce a bare 10.

  1. Mutating requests were retried after a timeout

req/v3's default condition is needRetry := err != nil - retry on any transport error, for any method. A POST that times out is the dangerous case: the server may well have accepted it and still be working, so re-issuing creates a duplicate. For CreateBlockVolume that is a duplicate volume, billed and orphaned.

retryOnlyIdempotent now gates it. The method list follows Go's own net/http Request.isReplayable(): GET, HEAD, OPTIONS, TRACE. PUT and DELETE are idempotent by HTTP definition but are deliberately left out - a detach or delete that timed out and is then re-issued can still land on IaaS state that is mid-transition. Callers that want to retry a mutating call should do it at a layer that can re-read state first and decide.

The condition preserves the previous behaviour in every other respect: still only on transport errors, never on an HTTP status code, and context cancellation still short-circuits ahead of it.

Worst case for a single call is unchanged in shape - up to 4 attempts for a replayable method, so 4 x the timeout plus roughly 15s of backoff. Bounding that properly is the caller's job, via its own context or WithTimeout().

Tests: pkg vngcloud/client

  • truth table over all methods, retried vs not, plus the success case
  • resp == nil and resp.Request == nil must not panic; RetryConditionFunc is called before a response exists
  • the interval constants are a real duration, ordered, and the whole retry sequence stays short relative to one timeout

Two defects in the retry configuration that sit next to the timeout, both
reachable by every consumer of the SDK.

1. The retry interval was ten nanoseconds

    SetCommonRetryFixedInterval(10)

10 is an untyped constant, so it converts to time.Duration(10) - ten
NANOseconds, not ten seconds. The three retries therefore fired back to back
with no pause at all. That does nothing for an endpoint that is slow or
briefly unavailable; it just multiplies the request count by four, and with no
jitter every caller retries in lockstep.

Replaced with req/v3's own backoff, which is exponential with jitter:

    SetCommonRetryBackoffInterval(httpRetryIntervalMin, httpRetryIntervalMax)  // 1s -> 8s

The three magic numbers are now named constants so the unit is explicit and
the next edit cannot silently reintroduce a bare 10.

2. Mutating requests were retried after a timeout

req/v3's default condition is `needRetry := err != nil` - retry on any
transport error, for any method. A POST that times out is the dangerous case:
the server may well have accepted it and still be working, so re-issuing
creates a duplicate. For CreateBlockVolume that is a duplicate volume, billed
and orphaned.

retryOnlyIdempotent now gates it. The method list follows Go's own
net/http Request.isReplayable(): GET, HEAD, OPTIONS, TRACE. PUT and DELETE are
idempotent by HTTP definition but are deliberately left out - a detach or
delete that timed out and is then re-issued can still land on IaaS state that
is mid-transition. Callers that want to retry a mutating call should do it at
a layer that can re-read state first and decide.

The condition preserves the previous behaviour in every other respect: still
only on transport errors, never on an HTTP status code, and context
cancellation still short-circuits ahead of it.

Worst case for a single call is unchanged in shape - up to 4 attempts for a
replayable method, so 4 x the timeout plus roughly 15s of backoff. Bounding
that properly is the caller's job, via its own context or WithTimeout().

Tests: pkg vngcloud/client
- truth table over all methods, retried vs not, plus the success case
- resp == nil and resp.Request == nil must not panic; RetryConditionFunc is
  called before a response exists
- the interval constants are a real duration, ordered, and the whole retry
  sequence stays short relative to one timeout
@vks-team
vks-team merged commit c4a606c into main Aug 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant