Skip to content

Retry object deployment after temporary API errors #295 - #510

Merged
NowinskiK merged 1 commit into
developfrom
issue-295-deploy-retry
Oct 16, 2026
Merged

NowinskiK merged 1 commit into
developfrom
issue-295-deploy-retry

Conversation

@NowinskiK

Copy link
Copy Markdown
Member

Closes #295. Approach agreed in this comment on #295.

Problem

Every object was deployed with a single API call and no retry, so one temporary time-out failed the whole publish. In the issue's log the request failed after almost exactly 100 seconds, which is the client-side HttpClient timeout. Retrying is safe because these calls create or update (HTTP PUT).

Changes

  • Test-AdfTransientError (new, private): decides whether an error is worth retrying, checking inner exceptions too.
    • Retried: time-outs (including TaskCanceledException and TimeoutException), HTTP 429 / TooManyRequests, HTTP 5xx, "concurrent operations".
    • Not retried: everything else (BadRequest, NotFound, policy, authorization...).
  • Invoke-AdfWithRetry (new, private): runs the call and retries temporary errors after 2, 4, 8... seconds, with a warning for each retry. Other errors, or the last failed attempt, are passed on.
  • Deploy-AdfObjectOnly: all 10 Az calls are wrapped and called with -ErrorAction Stop. This covers both publish methods: AzResource (New-AzResource, the default and the cmdlet in the issue's log) and AzDataFactory (Set-AzDataFactoryV2*). Global parameters are unchanged.
  • New publish option MaxRetries ([Int32], default 3, 0 = off). Documented in README.md, docs/ADVANCED/CMDLET_REFERENCE.md and docs/GUIDE/PUBLISH_OPTIONS.md.
  • Starting/stopping triggers is unchanged; it has its own retry.

Example output with two simulated time-outs:

Start deploying object: [pipeline].[PL_Wait5sec] (0 dependency/ies)
WARNING: Deployment of [pipeline].[PL_Wait5sec] failed with a temporary error: Operation failed because a request timed out. Retry 1 of 3 in 2 seconds...
WARNING: Deployment of [pipeline].[PL_Wait5sec] failed with a temporary error: Operation failed because a request timed out. Retry 2 of 3 in 4 seconds...
Finished deploying object: [pipeline].[PL_Wait5sec]

⚠️ Behaviour change

The Az cmdlets write non-terminating errors. Under the default $ErrorActionPreference = 'Continue', a failed deployment until now only printed a red error: the publish carried on and the object was marked as deployed (#472 covered terminating errors only). Now any failure that is not retried, or is still failing after the last retry, stops the publish whatever the caller's preference is. This is what #472 intended and has its own changelog line under "Fixed". Pipelines that relied on such errors being ignored will now fail.

Tests

New test/Invoke-AdfWithRetry.Tests.ps1, written before the fix:

  • 18 cases telling temporary errors from permanent ones
  • 7 for the retry loop: success, retry then success, retries running out, the 2/4/8-second waits, a permanent error, MaxRetries = 0, the module's own throw
  • 3 for Deploy-AdfObjectOnly with each publish method. Before the fix they failed as expected: no retry, and a permanent error did not stop the publish under Continue.
  • 1 for the option's default

Locally with Pester 6.2.0, 82/82 passed under both Stop and Continue: the new file plus the #472 error-handling, publish options, DryRun and trigger tests.

Changelog

Two lines under [Unreleased]: "Added" for the retry and MaxRetries, "Fixed" for the behaviour change.

🤖 Generated with Claude Code

Deploy-AdfObjectOnly wraps every Az call in Invoke-AdfWithRetry, which
retries time-outs, throttling (429), server errors (5xx) and concurrent
operations, waiting 2, 4, 8... seconds. Permanent errors are not retried.
New publish option MaxRetries (default 3, 0 = off).

Az calls now use -ErrorAction Stop, so a failed deployment stops the
publish whatever $ErrorActionPreference is, instead of being skipped and
the object marked as deployed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@NowinskiK
NowinskiK merged commit 58f414c into develop Oct 16, 2026
2 checks passed
@NowinskiK NowinskiK linked an issue Oct 16, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No retries on REST API time-outs

1 participant