Skip to content

fix: retry ECR push on IAM propagation access-denied errors - #169

Merged
BSick7 merged 1 commit into
masterfrom
fix/iam-propagation-retry
Sep 1, 2026
Merged

BSick7 merged 1 commit into
masterfrom
fix/iam-propagation-retry

Conversation

@BSick7

@BSick7 BSick7 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Problem

During an initial infra launch/push/deploy, an app can fail to push because the image_pusher IAM identity is created seconds before the push runs. AWS IAM is eventually consistent, so ECR rejects the freshly-created identity even though its permissions are correctly defined:

operation error ECR: GetAuthorizationToken, exceeded maximum number of attempts, 5, ...
api error AccessDeniedException: User: arn:aws:sts::...:assumed-role/pusher-.../...-enigma-runner
is not authorized to perform: ecr:GetAuthorizationToken on resource: *
because no identity-based policy allows the ecr:GetAuthorizationToken action

The existing ECR client retryer (5 attempts, ~1 minute window) retried this exact error but gave up before IAM finished propagating.

Changes

  • New aws/iampropagation package:
    • IsPropagationError classifies AWS access-denied errors (smithy AccessDenied/AccessDeniedException/InvalidClientTokenId/UnrecognizedClientException, plus the plain-text denied: ... not authorized to perform errors surfaced by docker push).
    • Retry retries the whole operation with exponential backoff (2s → 30s) for up to 3 minutes, failing immediately on any other error. Each attempt logs to the user's stderr so they can see we're waiting on IAM propagation and why:
      Access denied while pushing the image to ECR.
      If this app's infrastructure was just launched, its IAM permissions were created moments ago and AWS IAM is eventually consistent. Waiting for AWS to propagate IAM permissions.
      Retrying in 2s (attempt 1, giving up in 3m0s)...
      
  • ECR pusher: Push retags once, then runs auth + docker push inside the retry loop (covers both a GetAuthorizationToken denial and a mid-push layer-upload denial). ListArtifactVersions gets the same wrapper since the launch path calls it in the same race window.
  • Removed the narrower AccessDeniedException retryer from the ECR client so retry loops don't nest.

Trade-off

A genuinely misconfigured pusher identity now fails after ~3 minutes instead of ~1, but with explicit logging of what it was waiting for, and the final error explains both possibilities.

Follow-up

Port the same wrapper to GAR/ACR/S3/GCS/Blob pushers (same class of propagation race on other clouds).

During an initial infra launch, the image_pusher IAM identity is created
seconds before the push runs. AWS IAM is eventually consistent, so
authenticating with ECR (or the docker push itself) can fail with
access-denied until the identity's permissions propagate.

Add aws/iampropagation, which retries an operation with exponential
backoff for up to 3 minutes when it fails with an access-denied error,
logging each attempt to stderr so the user can see we're waiting on IAM
propagation and why. Wrap ECR Push and ListArtifactVersions with it,
replacing the narrower AccessDeniedException retryer on the ECR client
(5 attempts / ~1m window) that the observed failure outlasted.
@BSick7
BSick7 requested a review from ssickles September 1, 2026 02:59
@BSick7
BSick7 merged commit d30c9ab into master Sep 1, 2026
1 check failed
@BSick7
BSick7 deleted the fix/iam-propagation-retry branch September 1, 2026 03:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants