Skip to content

About

No description, website, or topics provided.

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

🚀 Cloud Order System — AIOps + MLOps Production Project

CI Pipeline Python 3.11 Terraform License: MIT

A production-grade AIOps + MLOps platform that generates e-commerce order logs, trains an anomaly detection model using Isolation Forest, serves predictions via FastAPI, and deploys infrastructure using Terraform on AWS.


📋 Table of Contents


🏗️ Architecture Overview

┌─────────────────────────────────────────────────────────────────────┐
│                        AWS Cloud (VPC)                               │
│                                                                       │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────────┐      │
│  │ Data         │───▶│ ML Training  │───▶│ S3 Model Store   │      │
│  │ Generator    │    │ Pipeline     │    │ (Versioned .pkl) │      │
│  │ (JSON Logs)  │    │ (IsoForest)  │    └────────┬─────────┘      │
│  └──────────────┘    └──────────────┘             │                  │
│         │                                          │                  │
│         ▼                                          ▼                  │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────────┐      │
│  │ Log Storage  │    │ FastAPI      │◀───│ Model Loader     │      │
│  │ (JSONL files)│    │ Inference    │    │ (Local / S3)     │      │
│  └──────────────┘    │ Service      │    └──────────────────┘      │
│                       │ :8000        │                               │
│                       └──────┬───────┘                               │
│                              │                                        │
│                              ▼                                        │
│                       ┌──────────────┐    ┌──────────────────┐      │
│                       │ Prometheus   │───▶│ CloudWatch       │      │
│                       │ :9090        │    │ Alarms & SNS     │      │
│                       └──────────────┘    └──────────────────┘      │
│                                                                       │
└─────────────────────────────────────────────────────────────────────┘

Data Flow

  1. Data Generator produces structured JSON logs simulating e-commerce order events
  2. Training Pipeline reads logs, extracts features, trains Isolation Forest model
  3. Trained model is saved as versioned .pkl and uploaded to S3
  4. Inference Service loads model from S3 or local storage
  5. Clients send order event features to /predict endpoint
  6. Prometheus scrapes metrics (latency, anomaly count, errors)
  7. CloudWatch triggers alarms on CPU spikes, error rates, and anomaly surges

📁 Project Structure

cloud-order-system/
│
├── services/
│   ├── inference-service/       # FastAPI anomaly detection API
│   │   ├── main.py              # FastAPI app with /predict, /health, /reload
│   │   ├── metrics.py           # Prometheus metric definitions
│   │   ├── model_loader.py      # Model I/O (local + S3 fallback)
│   │   ├── requirements.txt
│   │   └── Dockerfile
│   │
│   └── data-generator/          # Structured JSON log generator
│       ├── app.py               # Event generation with anomaly injection
│       ├── requirements.txt
│       └── Dockerfile
│
├── ml/
│   ├── training-pipeline/       # ML training code
│   │   ├── train.py             # Feature extraction + Isolation Forest training
│   │   └── requirements.txt
│   └── models/                  # Trained model artifacts (.pkl)
│
├── infrastructure/
│   └── terraform/               # AWS infrastructure as code
│       ├── main.tf              # Provider and backend config
│       ├── vpc.tf               # VPC, subnets, route tables
│       ├── ec2.tf               # Application server instance
│       ├── security_groups.tf   # Network access control
│       ├── iam.tf               # IAM roles and policies
│       ├── s3.tf                # Model storage bucket
│       ├── cloudwatch.tf        # Monitoring alarms
│       ├── variables.tf         # Input variables
│       ├── outputs.tf           # Output values
│       └── user_data.sh         # EC2 bootstrap script
│
├── observability/
│   └── prometheus.yml           # Prometheus scrape configuration
│
├── .github/
│   └── workflows/
│       └── ci.yml               # GitHub Actions CI pipeline
│
├── docker-compose.yml           # Multi-service local orchestration
└── README.md                    # This file

🧩 Components

Data Generator Service

  • Generates realistic e-commerce order events (placed, payment, shipping, errors)
  • Injects anomalies at configurable rate (default 5%): latency spikes, amount spikes, error bursts
  • Outputs structured JSON logs to files (.jsonl) and stdout
  • Anomaly types: latency_spike, amount_spike, error_burst, quantity_spike

ML Training Pipeline

  • Reads .jsonl log files and extracts numerical features
  • Features: latency_ms, order_amount, quantity, http_status, hour_of_day
  • Trains sklearn.ensemble.IsolationForest with configurable hyperparameters
  • Saves versioned model bundles (model + scaler + metadata) as .pkl
  • Uploads models to S3 with encryption

Inference Service (FastAPI)

  • /predict — Single event anomaly detection
  • /predict/batch — Batch prediction (up to 1000 events)
  • /health — Health check with model status
  • /reload — Hot-reload model without restart
  • /metrics — Prometheus metrics exposition
  • /docs — Auto-generated Swagger UI

Prometheus Metrics

Metric Type Description
inference_request_latency_seconds Histogram Request processing time
inference_anomaly_count_total Counter Total anomalies detected
inference_error_count_total Counter Total errors by type
inference_predictions_total Counter Total predictions (normal/anomaly)
inference_model_load_seconds Gauge Model load duration
inference_active_requests Gauge Currently active requests

🚀 Quick Start

Prerequisites

  • Python 3.11
  • Docker & Docker Compose
  • AWS CLI (for S3 operations)

Local Development

# 1. Start all services
docker-compose up -d

# 2. Check service status
docker-compose ps

# 3. Generate training data (wait ~60 seconds)
# The data-generator is already running and producing logs

# 4. Train the model (run inside the container or locally)
docker exec -it data-generator ls /data/logs/
# Or run locally:
pip install -r ml/training-pipeline/requirements.txt
export DATA_DIR=./data/logs
export MODEL_DIR=./ml/models
python ml/training-pipeline/train.py

# 5. Test the inference service
curl http://localhost:8000/health

curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{
    "latency_ms": 150.0,
    "order_amount": 79.99,
    "quantity": 2,
    "http_status": 200,
    "hour_of_day": 14
  }'

# 6. Test with anomalous data
curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{
    "latency_ms": 2500.0,
    "order_amount": 50000.0,
    "quantity": 300,
    "http_status": 503,
    "hour_of_day": 3
  }'

# 7. View Prometheus metrics
curl http://localhost:8000/metrics

# 8. Open Prometheus UI
# Navigate to http://localhost:9090

# 9. View API documentation
# Navigate to http://localhost:8000/docs

☁️ Infrastructure Deployment

Terraform Setup

cd infrastructure/terraform

# Initialize Terraform
terraform init

# Review planned changes
terraform plan -var="environment=dev" -var="alarm_email=your@email.com"

# Apply infrastructure
terraform apply -var="environment=dev" -var="alarm_email=your@email.com"

# Get outputs
terraform output

Infrastructure Components

Resource Purpose
VPC Isolated network with public/private subnets across 2 AZs
EC2 (t3.medium) Application server running Docker containers
Security Groups Allows ports 8000 (API), 9090 (Prometheus), 22 (SSH)
IAM Role EC2 access to S3 (models) and CloudWatch (metrics)
S3 Bucket Versioned ML model storage with encryption & lifecycle policies
CloudWatch Alarms CPU > 80%, error rate > 50/5min, anomaly spike > 100/5min
SNS Topic Alarm notification delivery (email, Slack, PagerDuty)

🔄 CI/CD Pipeline

The GitHub Actions pipeline runs on every push to main/develop and all PRs:

┌──────┐    ┌──────┐    ┌──────┐    ┌──────────┐
│ Lint │───▶│ Test │───▶│Build │    │ Security │
│flake8│    │pytest│    │Docker│    │  safety  │
│black │    │      │    │      │    │          │
└──────┘    └──────┘    └──────┘    └──────────┘
                                    ┌──────────┐
                                    │Terraform │
                                    │ validate │
                                    └──────────┘

Jobs:

  1. Lint — flake8 style checks, Black formatting, isort import ordering
  2. Test — pytest for all services
  3. Build — Docker image compilation with Buildx caching
  4. Security — Dependency vulnerability scanning
  5. Terraform — terraform validate and terraform fmt checks

📊 Monitoring & Observability

Prometheus

  • Scrapes inference service every 15 seconds
  • Stores 15 days of metric history
  • Access UI at http://<host>:9090

Useful PromQL Queries

# Request rate (requests per second)
rate(inference_predictions_total[5m])

# 95th percentile latency
histogram_quantile(0.95, rate(inference_request_latency_seconds_bucket[5m]))

# Anomaly rate percentage
rate(inference_anomaly_count_total[5m]) / rate(inference_predictions_total[5m]) * 100

# Error rate
rate(inference_error_count_total[5m])

CloudWatch Alarms

Alarm Threshold Action
CPU High > 80% for 5 min Scale up / investigate
Status Check Failed Any failure Instance recovery
Error Rate > 50 errors / 5 min Check logs, rollback
Anomaly Spike > 100 anomalies / 5 min Investigate incident

📈 Scaling Considerations

Horizontal Scaling Path

Current: Single EC2 + Docker Compose
   │
   ▼
Phase 2: EC2 Auto Scaling Group + ALB
   │  - Multiple EC2 instances behind a load balancer
   │  - Auto-scale based on CPU / request count
   │
   ▼
Phase 3: ECS Fargate
   │  - Serverless containers (no EC2 management)
   │  - Per-service scaling (inference can scale independently)
   │
   ▼
Phase 4: EKS (Kubernetes)
      - Full orchestration with K8s
      - Horizontal Pod Autoscaler
      - Service mesh (Istio) for traffic management
      - Canary deployments for model updates

Model Serving Scaling

  • Current: Single uvicorn worker, model in memory
  • Next: Multiple workers with shared model cache (Redis)
  • Future: Dedicated model serving (TorchServe, TFServing, Triton)

Data Pipeline Scaling

  • Current: JSON files on disk
  • Next: Apache Kafka for real-time log streaming
  • Future: AWS Kinesis → S3 → SageMaker Pipeline

⚖️ Trade-offs & Design Decisions

Decision Chosen Alternative Why
ML Model Isolation Forest Autoencoder, LSTM Unsupervised, fast, no GPU needed
API Framework FastAPI Flask, Django Async, auto-docs, Pydantic validation
Serialization joblib .pkl ONNX, PMML Native sklearn support, includes scaler
Monitoring Prometheus Datadog, CloudWatch Free, powerful PromQL, industry standard
IaC Terraform CDK, Pulumi Multi-cloud, mature ecosystem
Deployment Docker Compose ECS, K8s Simple to start, easy migration path
CI/CD GitHub Actions Jenkins, GitLab CI Native, free for public repos
Log Format JSONL CSV, Parquet Streaming-friendly, human-readable
Feature Scaling StandardScaler MinMaxScaler Better for Gaussian-like distributions
Anomaly Rate 5% contamination Adaptive Matches known injection rate

Key Architectural Principles

  1. 12-Factor App: Config via env vars, stateless services, port binding
  2. Clean Architecture: Separation of concerns (metrics, model loading, API logic)
  3. Observability First: Metrics exposed from day one, not bolted on later
  4. Infrastructure as Code: All infra is version-controlled and reproducible
  5. Security by Default: Non-root containers, encrypted S3, least-privilege IAM

📚 API Reference

POST /predict

Request:

{
  "latency_ms": 150.0,
  "order_amount": 79.99,
  "quantity": 2,
  "http_status": 200,
  "hour_of_day": 14
}

Response:

{
  "is_anomaly": false,
  "anomaly_score": 0.123456,
  "confidence": "low",
  "model_version": "20240215_143022",
  "prediction_time_ms": 1.234,
  "timestamp": "2024-02-15T14:30:22.123456+00:00"
}

POST /predict/batch

Accepts {"events": [...]} with up to 1000 events.

GET /health

Returns service status, model version, and uptime.

POST /reload

Hot-reloads the model without restarting the service.

GET /metrics

Prometheus-formatted metrics for scraping.

GET /docs

Interactive Swagger UI documentation.


🤝 Contributing

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

Code Standards

  • Python: Black formatter, flake8 linting, isort imports
  • Terraform: terraform fmt, terraform validate
  • Commits: Conventional Commits format

📄 License

This project is licensed under the MIT License.

About

No description, website, or topics provided.

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages