Architecture Diagram

Below is the architecture diagram showing the interaction between S3, Lambda, and DynamoDB with the necessary IAM roles:

AWS E-commerce Order Processing Solution Infrastructure Setup with Terraform

Overview

This project can be used as an automated solution for processing e-commerce orders using AWS services. The architecture ensures secure storage, real-time processing, and efficient monitoring of order data. The setup includes an S3 bucket for data storage, a Lambda function for data processing, and a DynamoDB table for data persistence. Additionally, it configures necessary IAM roles and policies, logging, and monitoring using CloudWatch and SNS.

Table of Contents

• Prerequisites
• Setup Instructions
• Terraform Configuration Details
  • S3 Bucket
  • S3 Bucket Notification
  • IAM Roles and Policies
  • Lambda Function
  • DynamoDB Table
  • Logging and Monitoring
• Testing and Verification
• Cleaning Up
• Contributing
• License

Prerequisites

Before you begin, ensure you have the following installed:

• Terraform (version 0.12 or later)
• AWS CLI (configured with appropriate access)
• An AWS account with permissions to create the necessary resources

Setup Instructions

1. Clone the Repository:

   git clone https://github.com/dhamsey3/aws-data-automation.git

2. Initialize Terraform:

   terraform init

3. Review and Modify Configuration:

   • Open the main.tf file and review the configuration.
   • Modify any parameters (such as resource names, tags, etc.) as needed.

4. Apply the Terraform Configuration:

   terraform apply

   • Review the plan and type yes to apply.

Terraform Configuration Details

S3 Bucket

• Resource: aws_s3_bucket.top_secret_data
• Configuration:
  • Bucket name: top-secret-data
  • ACL: private
  • Versioning: enabled
  • Server-side encryption: AES256
  • Lifecycle rule: Expire objects after 90 days

S3 Bucket Notification

• Resource: aws_s3_bucket_notification.top_secret_notification
• Configuration:
  • Triggers the Lambda function process_order_function on object creation events with prefix input/

IAM Roles and Policies

• IAM Role: aws_iam_role.lambda_execution
  • Allows Lambda service to assume the role
• Policies:
  • aws_iam_policy.lambda_logs_policy: Allows logging to CloudWatch
  • aws_iam_policy.lambda_s3_dynamodb_policy: Allows access to S3 bucket, DynamoDB table, and invoking Lambda functions

Lambda Function

• Resource: aws_lambda_function.process_order_function
• Configuration:
  • Function name: process-order-function
  • Runtime: python3.8
  • Handler: lambda_function.lambda_handler
  • Memory size: 256 MB
  • Timeout: 15 seconds
  • Source code: lambda/lambda_function.zip
  • Environment variables: TABLE_NAME set to the DynamoDB table name

DynamoDB Table

• Resource: aws_dynamodb_table.table
• Configuration:
  • Table name: my-table
  • Billing mode: PAY_PER_REQUEST
  • Primary key: id (string)
  • Server-side encryption: enabled

Logging and Monitoring

• CloudWatch Log Group: aws_cloudwatch_log_group.lambda_log_group
  • Log group for the Lambda function with a 14-day retention
• CloudWatch Alarm: aws_cloudwatch_metric_alarm.lambda_error_alarm
  • Alarm for Lambda function errors, sends notifications to an SNS topic
• SNS Topic: aws_sns_topic.lambda_notifications
  • Topic for notifications
• SNS Subscription: aws_sns_topic_subscription.lambda_notifications_email
  • Email subscription for the SNS topic

Testing and Verification

After applying the Terraform configuration, you can verify the setup as follows:

1. Upload a Test File to S3:

   • Upload a file to the top-secret-data bucket with the prefix input/.
   • Verify that the Lambda function is triggered and processes the file.

2. Check DynamoDB Table:

   • Verify that the processed data is correctly stored in the DynamoDB table.

3. Monitor Logs and Alarms:

   • Check the CloudWatch log group for logs generated by the Lambda function.
   • Ensure there are no errors and that the CloudWatch alarm is not triggered.

Cleaning Up

To clean up and remove all resources created by this Terraform configuration:

terraform destroy