Skip to content

Use resourcetype params for AWS resource enumeration - #65

Open
bencehezso wants to merge 1 commit into
mainfrom
task/update_aws_resource_enumeration
Open

Use resourcetype params for AWS resource enumeration#65
bencehezso wants to merge 1 commit into
mainfrom
task/update_aws_resource_enumeration

Conversation

@bencehezso

Copy link
Copy Markdown
Contributor

Summary

The AWS scanner derived each boto3 call by splitting resourcetype.code on . and requiring exactly four parts: AWS.<service>.<operation>.<resultKey>. That format cannot express a nested result path or extra API arguments, so any catalogue row needing either was silently skipped. The master data now carries a params JSON column holding the call spec explicitly, and the scanner prefers it.

{
  "service": "cloudfront",
  "operation": "list_distributions",
  "result_path": ["DistributionList", "Items"],
  "kwargs": {"OwnerIds": ["self"]}
}

What changes

  • Call spec resolutionresolve_aws_call_spec() takes params when it carries both service and operation, otherwise derives from the code. Unknown params keys are ignored, since master data ships ahead of app releases, and rows for the other cloud carry unrelated params, so usability is detected by probing for the keys rather than testing for a non-empty object.
  • Code fallback relaxed — from exactly 4 parts to >= 4, so both paths agree on what a valid code is. This alone fixes CloudFront and Pinpoint even without params.
  • Result path walk — applied per page when paginating and to the response when not. A missing intermediate key yields nothing for that page rather than raising; a final value that is not a list yields nothing.
  • Skips are no longer silent — 2-part service placeholders (AWS.iam, AWS.cleanrooms, 46 rows) log at DEBUG. Anything that parses as neither is catalogue corruption and logs a WARNING; this is the previously commented-out warning, restored with the corrected condition.
  • Item cap — 50,000 per resource type. Some list operations return every publicly shared resource in the region when called without a filter, which would otherwise stall the scan and blow up the raw-data file. Hitting the cap stops pagination for that row and logs a WARNING naming the service and operation.

@bencehezso bencehezso self-assigned this Aug 12, 2026
@bencehezso bencehezso added the enhancement New feature or request label Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant