Skip to content

fix: added new variable to support use existing SQS queue for s3 logs bucket data event #334

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
githubjianli merged 7 commits into from
Jan 22, 2026
Merged

fix: added new variable to support use existing SQS queue for s3 logs bucket data event #334

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
9911187
fix: added new variable to support use existing SQS queue for s3 logs…
JianLi-Expedia Jan 21, 2026
fd3f972
Decouple the ´apiary_log_bucket´ variable for s3 logs and s3 logs hiv…
Jan 22, 2026
03d9a83
Revert "Decouple the ´apiary_log_bucket´ variable for s3 logs and s3 …
Jan 22, 2026
c48a065
fix: update condition
githubjianli Jan 22, 2026
e8c319c
Added flag to enable or disable s3 access logging of the buckets
Jan 22, 2026
6ba996f
Merge branch 'feature/support_ext_s3_log_queue' of https://github.com…
Jan 22, 2026
6e22e95
fix: update s3 bucket logging configuration
JianLi-Expedia Jan 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

## [7.12.6] - 2026-01-22
### Added
- Added var `apiary_managed_s3_logs_queue_arn` to support use existing SQS queue for s3 logs bucket data event.

## [7.12.5] - 2025-11-12
### Added
- Added var `lf_catalog_data_location_access_producer_arns` that is used to give `DATA_LOCATION_ACCESS` permission in LakeFormation on s3 locations of all databases.
Expand Down
1 change: 1 addition & 0 deletions VARIABLES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
| apiary\_governance\_iamroles | AWS IAM governance roles allowed read and tagging access to managed Apiary S3 buckets. | `list(string)` | `[]` | no |
| apiary\_log\_bucket | Bucket for Apiary logs.If this is blank, module will create a bucket. | `string` | `""` | no |
| apiary\_log\_prefix | Prefix for Apiary logs. | `string` | `""` | no |
| apiary\_managed\_s3\_logs_queue\_arn | Apiary Managed S3 Logs SQS Queue ARN. | `string` | `""` | no |
| apiary\_managed\_schemas | List of maps, each map contains schema name from which S3 bucket names will be derived, and various properties. The corresponding S3 bucket will be named as apiary\_instance-aws\_account-aws\_region-schema\_name. | `list(map(string))` | `[]` | no |
| apiary\_producer\_iamroles | AWS IAM roles allowed write access to managed Apiary S3 buckets. | `map(any)` | `{}` | no |
| apiary\_rds\_additional\_sg | Comma-separated string containing additional security groups to attach to RDS. | `list(any)` | `[]` | no |
Expand Down
1 change: 1 addition & 0 deletions common.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ locals {
create_sqs_data_event_queue = contains([for schema in local.schemas_info : lookup(schema, "enable_data_events_sqs", "0")], "1") ? true : false
enable_apiary_s3_log_management = var.apiary_log_bucket == "" ? true : false
enable_apiary_s3_log_hive = var.apiary_log_bucket == "" && var.enable_apiary_s3_log_hive ? true : false
create_sqs_s3_logs_queue = var.apiary_managed_s3_logs_queue_arn == "" && local.enable_apiary_s3_log_management ? true : false
apiary_s3_logs_bucket = local.enable_apiary_s3_log_management ? "${local.apiary_bucket_prefix}-s3-logs" : ""
apiary_s3_hive_logs_bucket = local.enable_apiary_s3_log_management ? "${local.apiary_s3_logs_bucket}-hive" : ""
apiary_system_bucket = "${local.apiary_bucket_prefix}-${replace(var.system_schema_name, "_", "-")}"
Expand Down
2 changes: 1 addition & 1 deletion s3-other.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ resource "aws_s3_bucket_notification" "apiary_managed_logs_bucket" {
bucket = aws_s3_bucket.apiary_managed_logs_bucket[0].bucket

queue {
queue_arn = aws_sqs_queue.apiary_managed_logs_queue[0].arn
queue_arn = local.create_sqs_s3_logs_queue ? aws_sqs_queue.apiary_managed_logs_queue[0].arn : var.apiary_managed_s3_logs_queue_arn
events = ["s3:ObjectCreated:*"]
}
}
Expand Down
13 changes: 8 additions & 5 deletions s3.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,6 @@ resource "aws_s3_bucket" "apiary_data_bucket" {
tags = merge(tomap({"Name"=each.value["data_bucket"]}),
var.apiary_tags,
jsondecode(lookup(each.value, "tags", "{}")))

logging {
target_bucket = local.enable_apiary_s3_log_management ? aws_s3_bucket.apiary_managed_logs_bucket[0].id : var.apiary_log_bucket
target_prefix = "${var.apiary_log_prefix}${each.value["data_bucket"]}/"
}
}

resource "aws_s3_bucket_versioning" "apiary_data_bucket_versioning" {
Expand Down Expand Up @@ -188,6 +183,14 @@ resource "aws_s3_bucket_notification" "data_queue_events" {
}
}

resource "aws_s3_bucket_logging" "apiary_bucket" {
for_each = {
for schema in local.schemas_info : "${schema["schema_name"]}" => schema
}
bucket = each.value["data_bucket"]
target_bucket = local.enable_apiary_s3_log_management ? aws_s3_bucket.apiary_managed_logs_bucket[0].id : var.apiary_log_bucket
target_prefix = "${var.apiary_log_prefix}${each.value["data_bucket"]}/"
}

resource "aws_s3_bucket_metric" "paid_metrics" {
for_each = var.enable_s3_paid_metrics ? {
Expand Down
2 changes: 1 addition & 1 deletion sns.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ POLICY
}

resource "aws_sqs_queue" "apiary_managed_logs_queue" {
count = local.enable_apiary_s3_log_management ? 1 : 0
count = local.create_sqs_s3_logs_queue ? 1 : 0
name = "${local.instance_alias}-s3-logs-queue"
tags = var.apiary_tags

Expand Down
6 changes: 6 additions & 0 deletions variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1251,6 +1251,12 @@ variable "additional_s3_log_buckets" {
default = []
}

variable "apiary_managed_s3_logs_queue_arn" {
description = "Apiary Managed Log SQS Queue ARN"
type = string
default = ""
}

variable "hms_ro_k8s_log4j_properties" {
description = "Custom Log4j properties for apiary readonly metastore."
type = string
Expand Down
Loading