[ubaskota]

Issue #, if available:

Description of changes:
Implement AdaptiveRetryStrategy that extends StandardRetryStrategy with client-side rate limiting using token bucket and CUBIC congestion control algorithm.

Key Changes:

• Add AdaptiveRetryStrategy class with rate limiter integration
• Fail the operation when a send token can't be acquired within 30s, without retrying or consuming an attempt
• Integrate rate limiting hooks in RequestPipeline._retry() method
• Add thread-safe async lock to RequestRateTracker.measure_rate()
• Apply rate limiting to non-retryable calls
• Surface the previous service error when a send-token timeout happens before a retry
• Propagate send-token failures to input/duplex stream callers so they don't hang
• Create a new asyncio.Condition per event loop so a client can be reused across asyncio.run() calls
• Add comprehensive unit and functional tests for adaptive retry behavior

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[kuhe]

Add why/when to use Adaptive over Standard.

[ubaskota]

Added

[kuhe]

I suggest two tests:

Sequence test

At the unit test level, a sequencing test with a no-jitter version of the backoff computation that always returns the middle value.

The sequence can be minimally described as rate limiting duration, response code, backoff duration, terminal.

Example (not real backoff rates):

# rate limiting, response status, backoff, terminal
(0, 429, 50, false),
(0, 429, 150, false),
(0, 429, 500, true)

Convergence tests

For adaptive in particular, other tests aside if the server only lets through successful responses at 10 per second, that's what the request rate should converge to. Have a client attempt to make 100 requests/s, and exit the test successfully if it converges below e.g. 15 r/s.

This is a local version of what exists in the external test suites.

[ubaskota]

Added both in TestAdaptiveRetryPipeline.

[Alan4506]

In the spec, I don't see a token-acquisition timeout should be treated as a retryable error. The spec says the following:

• If a token is not available from the token bucket, the client can either fast fail the request or block until a token becomes available. This behavior must be configurable in the client.
• If there is sufficient capacity (tokens) available, GetSendToken() immediately returns. If there is not sufficient capacity, GetSendToken() will either sleep a certain amount of time until the rate limiter can retrieve a token from its token bucket or raise an exception indicating there is insufficient capacity.

Other smithy SDKs (Go/Java/Kotlin/Rust/TS) all block until a token is available, without counting the wait as an attempt (Go also offers fast-fail). Should we do the same here?

[ubaskota]

Updated. A send-token timeout is no longer treated as a retryable error, and it doesn't use up an attempt. The request now waits up to 30 seconds for a token and then fails the operation. If the timeout happens before a retry, the caller gets the previous service error, such as the 429, with the timeout attached as its cause.

We kept a bounded wait on purpose instead of blocking forever. The token bucket doesn't serve waiters in order, so a new request can grab a token before one that has already been waiting. When many requests compete for tokens, an unbounded wait could leave some requests waiting forever. We decided to hold off on adding fair ordering until there's a real need for it. Once that's in place, we can switch to blocking until a token is available, like the other SDKs do. Making the choice between blocking and failing fast configurable will be handled in a follow-up.

[Alan4506]

Thanks, your approach and rationale make sense to me. But just curious, is there a specific reason for 30 seconds rather than 60 or some other value? Not a blocker though.

[ubaskota]

Just a judgment call. The minimum fill rate is 0.5 tokens per second, so a request waits at most 2 seconds for a token when the bucket is empty. A request that has waited 30 seconds means at least ~15 other requests got tokens before it, which points to heavy contention or starvation, so it should fail rather than keep waiting.

[kuhe]

for convergence, as long as the initial rate measures outside of the desired range, the test can exit successfully as soon as it enters the correct range after an initial maybe 4s lockout, rather than running a full 30s or more, to save time for the runner.

Because it is one of multiple mutually-supporting tests, it doesn't need to be so thorough as to test the range over the full 30s+

[ubaskota]

It doesn't take 30s in real time. The test uses a simulated clock, so the full 120 simulated seconds run in about 0.2s.