Skip to content

Cut AWS integration suite runtime and stop credentials expiring mid-run - #342

Merged
nuwang merged 1 commit into
mainfrom
aws-integration-test-runtime
Aug 1, 2026
Merged

Cut AWS integration suite runtime and stop credentials expiring mid-run#342
nuwang merged 1 commit into
mainfrom
aws-integration-test-runtime

Conversation

@nuwang

@nuwang nuwang commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

The AWS cloud integration job took ~59 minutes and started failing its last
three tests with RequestExpired
(run). The
OIDC session defaults to one hour and the credentials reach tox as static
environment variables, so botocore cannot refresh them mid-run. The run before
it took 56 minutes and cleared the deadline by four.

Raising the session lifetime alone would have hidden the real problem, so this
also fixes what made the suite slow. Investigation notes below.

Changes

Request a 3 hour OIDC session instead of the 1 hour default.

⚠️ Requires an IAM change before merge. The role's
MaxSessionDuration defaults to 3600s and AssumeRoleWithWebIdentity fails
outright if asked for more, so merging this without the IAM change turns a
late-run failure into an immediate one. Already applied to
cloudbridge-github-actions (MaxSessionDuration now 10800).

Memoise the AWS VM type catalogue per availability zone. EC2 has no
server-side paging for instance types, so AWSVMTypeService.list()
materialises the whole catalogue and pages it client-side — and it previously
refetched that catalogue on every call: one DescribeInstanceTypeOfferings
walk plus a DescribeInstanceTypes call per 100 types, ~14 API calls to return
one page. Walking every page was therefore quadratic in API calls.

Measured against us-east-1a: 1343 instance types, and at the tests'
default_result_limit of 5 that is 269 pages × 16 calls ≈ 4300 API calls.
End-to-end through the moto-backed provider, a full 255-page walk:

Before After
API calls 3570 14
Types returned 1273 unique 1273 unique

This was the single biggest contributor: test_vm_types_standard alone took
~55 minutes (3289s in the failing run, 3014s before it).

Poll Route53 record changes every 5s instead of boto3's 30s default,
keeping the same ~30 minute ceiling. Record create and delete block on the
resource_record_sets_changed waiter. Instrumented against real Route53, the
pattern was identical every time — poll, PENDING, sleep 30s, poll, INSYNC:

Wall clock 126.9s
In AWS API calls 5.9s (4.6%)
Sleeping 121.0s (95.4%)

Changes reach INSYNC well inside 30s, so all 120s was poll granularity across
the test's 4 record changes.

Testing

  • New unit tests for both fixes, no cloud credentials needed:
    tests/test_aws_vm_types.py (catalogue is fetched once per zone, contents
    unchanged, keyed by zone) and tests/test_aws_dns_waiters.py (drives the
    real botocore waiter against a simulated clock, so it asserts how long we
    would sleep without sleeping; includes guards that faster polling does not
    return early or shrink the ceiling).
  • Full moto-backed suite green: 113 passed, 6 skipped.
  • flake8 and mypy clean.
  • Both fixes verified against real AWS.

Known gaps

Two things this PR does not resolve, recorded so they are not lost:

  • test_create_and_list_image takes ~55 minutes and reproduces outside CI
    (a local run exceeded 75 minutes). Ruled out: CreateImage/wait_till_ready
    (the AMI reaches available in seconds) and image list/iter (the account
    holds one self-owned AMI). It never reaches the launch-from-image step, so
    the time is inside check_standard_behaviour. One contributor is confirmed —
    AWSImageService.find searches all public AMIs because check_find does not
    scope by owner, and an unscoped tag:Name DescribeImages measures 11.4s
    against 0.32s scoped — but that is ~12s, not 45 minutes. Not yet root-caused.
  • The DNS test is 9-20× slower in CI than locally (1128s and 3231s in CI
    versus 127s solo and 157s under 5 workers locally). Parallel contention was
    the hypothesis and it is disproved. The remaining variable is how long
    Route53 actually takes to report INSYNC, which is server-side. The polling
    fix helps in both regimes but will not by itself close a 20× gap.

Separately, the failed run leaked resources, because cleanup handlers also
need working credentials: instance i-096aa37d487803314 has been running since
07:34 on 2026-07-31, plus ami-07d70bca45ccaf5d1, a snapshot and a volume.
These need clearing by hand; the session-lifetime change prevents a recurrence.

The AWS cloud integration job took ~59 minutes and started failing its last
tests with RequestExpired. The OIDC session defaults to one hour and the
credentials reach tox as static environment variables, so botocore cannot
refresh them mid-run. Cleanup handlers need working credentials too, so the
failing tests also leaked the instances and images they had created.

Request a 3 hour OIDC session instead of the 1 hour default. This requires
the IAM role's MaxSessionDuration to permit the longer session.

Memoise the AWS VM type catalogue per availability zone. EC2 has no
server-side paging for instance types, so list() materialises the whole
catalogue and pages it client-side, and it previously refetched that
catalogue on every call -- one DescribeInstanceTypeOfferings walk plus a
DescribeInstanceTypes call per 100 types. That made walking a full listing
quadratic in API calls: the 1343 types offered in us-east-1a, paged at the
tests' result limit of 5, cost ~4300 calls where 14 suffice. This was the
bulk of the runtime; test_vm_types_standard alone took ~55 minutes.

Poll Route53 record changes every 5s rather than boto3's 30s default, while
keeping the same ~30 minute ceiling. Measured against Route53, changes
reached INSYNC inside the first poll interval every time, so a test making
four record changes spent 120s asleep for work that finished in seconds.
@nuwang
nuwang temporarily deployed to cloud-integration July 31, 2026 14:24 — with GitHub Actions Inactive
@nuwang
nuwang temporarily deployed to cloud-integration July 31, 2026 14:24 — with GitHub Actions Inactive
@nuwang
nuwang had a problem deploying to cloud-integration July 31, 2026 14:24 — with GitHub Actions Failure
@nuwang
nuwang temporarily deployed to cloud-integration July 31, 2026 14:24 — with GitHub Actions Inactive
@nuwang
nuwang had a problem deploying to cloud-integration July 31, 2026 20:01 — with GitHub Actions Failure
@nuwang
nuwang had a problem deploying to cloud-integration August 1, 2026 05:46 — with GitHub Actions Failure
@nuwang
nuwang temporarily deployed to cloud-integration August 1, 2026 15:29 — with GitHub Actions Inactive
@nuwang
nuwang merged commit c29ae4a into main Aug 1, 2026
22 of 25 checks passed
@nuwang
nuwang deleted the aws-integration-test-runtime branch August 1, 2026 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant