A production-grade AIOps + MLOps platform that generates e-commerce order logs, trains an anomaly detection model using Isolation Forest, serves predictions via FastAPI, and deploys infrastructure using Terraform on AWS.
- Architecture Overview
- Project Structure
- Components
- Quick Start
- Infrastructure Deployment
- CI/CD Pipeline
- Monitoring & Observability
- Scaling Considerations
- Trade-offs & Design Decisions
- API Reference
- Contributing
┌─────────────────────────────────────────────────────────────────────┐
│ AWS Cloud (VPC) │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │
│ │ Data │───▶│ ML Training │───▶│ S3 Model Store │ │
│ │ Generator │ │ Pipeline │ │ (Versioned .pkl) │ │
│ │ (JSON Logs) │ │ (IsoForest) │ └────────┬─────────┘ │
│ └──────────────┘ └──────────────┘ │ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │
│ │ Log Storage │ │ FastAPI │◀───│ Model Loader │ │
│ │ (JSONL files)│ │ Inference │ │ (Local / S3) │ │
│ └──────────────┘ │ Service │ └──────────────────┘ │
│ │ :8000 │ │
│ └──────┬───────┘ │
│ │ │
│ ▼ │
│ ┌──────────────┐ ┌──────────────────┐ │
│ │ Prometheus │───▶│ CloudWatch │ │
│ │ :9090 │ │ Alarms & SNS │ │
│ └──────────────┘ └──────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
- Data Generator produces structured JSON logs simulating e-commerce order events
- Training Pipeline reads logs, extracts features, trains Isolation Forest model
- Trained model is saved as versioned
.pkland uploaded to S3 - Inference Service loads model from S3 or local storage
- Clients send order event features to
/predictendpoint - Prometheus scrapes metrics (latency, anomaly count, errors)
- CloudWatch triggers alarms on CPU spikes, error rates, and anomaly surges
cloud-order-system/
│
├── services/
│ ├── inference-service/ # FastAPI anomaly detection API
│ │ ├── main.py # FastAPI app with /predict, /health, /reload
│ │ ├── metrics.py # Prometheus metric definitions
│ │ ├── model_loader.py # Model I/O (local + S3 fallback)
│ │ ├── requirements.txt
│ │ └── Dockerfile
│ │
│ └── data-generator/ # Structured JSON log generator
│ ├── app.py # Event generation with anomaly injection
│ ├── requirements.txt
│ └── Dockerfile
│
├── ml/
│ ├── training-pipeline/ # ML training code
│ │ ├── train.py # Feature extraction + Isolation Forest training
│ │ └── requirements.txt
│ └── models/ # Trained model artifacts (.pkl)
│
├── infrastructure/
│ └── terraform/ # AWS infrastructure as code
│ ├── main.tf # Provider and backend config
│ ├── vpc.tf # VPC, subnets, route tables
│ ├── ec2.tf # Application server instance
│ ├── security_groups.tf # Network access control
│ ├── iam.tf # IAM roles and policies
│ ├── s3.tf # Model storage bucket
│ ├── cloudwatch.tf # Monitoring alarms
│ ├── variables.tf # Input variables
│ ├── outputs.tf # Output values
│ └── user_data.sh # EC2 bootstrap script
│
├── observability/
│ └── prometheus.yml # Prometheus scrape configuration
│
├── .github/
│ └── workflows/
│ └── ci.yml # GitHub Actions CI pipeline
│
├── docker-compose.yml # Multi-service local orchestration
└── README.md # This file
- Generates realistic e-commerce order events (placed, payment, shipping, errors)
- Injects anomalies at configurable rate (default 5%): latency spikes, amount spikes, error bursts
- Outputs structured JSON logs to files (
.jsonl) and stdout - Anomaly types:
latency_spike,amount_spike,error_burst,quantity_spike
- Reads
.jsonllog files and extracts numerical features - Features:
latency_ms,order_amount,quantity,http_status,hour_of_day - Trains
sklearn.ensemble.IsolationForestwith configurable hyperparameters - Saves versioned model bundles (model + scaler + metadata) as
.pkl - Uploads models to S3 with encryption
/predict— Single event anomaly detection/predict/batch— Batch prediction (up to 1000 events)/health— Health check with model status/reload— Hot-reload model without restart/metrics— Prometheus metrics exposition/docs— Auto-generated Swagger UI
| Metric | Type | Description |
|---|---|---|
inference_request_latency_seconds |
Histogram | Request processing time |
inference_anomaly_count_total |
Counter | Total anomalies detected |
inference_error_count_total |
Counter | Total errors by type |
inference_predictions_total |
Counter | Total predictions (normal/anomaly) |
inference_model_load_seconds |
Gauge | Model load duration |
inference_active_requests |
Gauge | Currently active requests |
- Python 3.11
- Docker & Docker Compose
- AWS CLI (for S3 operations)
# 1. Start all services
docker-compose up -d
# 2. Check service status
docker-compose ps
# 3. Generate training data (wait ~60 seconds)
# The data-generator is already running and producing logs
# 4. Train the model (run inside the container or locally)
docker exec -it data-generator ls /data/logs/
# Or run locally:
pip install -r ml/training-pipeline/requirements.txt
export DATA_DIR=./data/logs
export MODEL_DIR=./ml/models
python ml/training-pipeline/train.py
# 5. Test the inference service
curl http://localhost:8000/health
curl -X POST http://localhost:8000/predict \
-H "Content-Type: application/json" \
-d '{
"latency_ms": 150.0,
"order_amount": 79.99,
"quantity": 2,
"http_status": 200,
"hour_of_day": 14
}'
# 6. Test with anomalous data
curl -X POST http://localhost:8000/predict \
-H "Content-Type: application/json" \
-d '{
"latency_ms": 2500.0,
"order_amount": 50000.0,
"quantity": 300,
"http_status": 503,
"hour_of_day": 3
}'
# 7. View Prometheus metrics
curl http://localhost:8000/metrics
# 8. Open Prometheus UI
# Navigate to http://localhost:9090
# 9. View API documentation
# Navigate to http://localhost:8000/docscd infrastructure/terraform
# Initialize Terraform
terraform init
# Review planned changes
terraform plan -var="environment=dev" -var="alarm_email=your@email.com"
# Apply infrastructure
terraform apply -var="environment=dev" -var="alarm_email=your@email.com"
# Get outputs
terraform output| Resource | Purpose |
|---|---|
| VPC | Isolated network with public/private subnets across 2 AZs |
| EC2 (t3.medium) | Application server running Docker containers |
| Security Groups | Allows ports 8000 (API), 9090 (Prometheus), 22 (SSH) |
| IAM Role | EC2 access to S3 (models) and CloudWatch (metrics) |
| S3 Bucket | Versioned ML model storage with encryption & lifecycle policies |
| CloudWatch Alarms | CPU > 80%, error rate > 50/5min, anomaly spike > 100/5min |
| SNS Topic | Alarm notification delivery (email, Slack, PagerDuty) |
The GitHub Actions pipeline runs on every push to main/develop and all PRs:
┌──────┐ ┌──────┐ ┌──────┐ ┌──────────┐
│ Lint │───▶│ Test │───▶│Build │ │ Security │
│flake8│ │pytest│ │Docker│ │ safety │
│black │ │ │ │ │ │ │
└──────┘ └──────┘ └──────┘ └──────────┘
┌──────────┐
│Terraform │
│ validate │
└──────────┘
Jobs:
- Lint — flake8 style checks, Black formatting, isort import ordering
- Test — pytest for all services
- Build — Docker image compilation with Buildx caching
- Security — Dependency vulnerability scanning
- Terraform —
terraform validateandterraform fmtchecks
- Scrapes inference service every 15 seconds
- Stores 15 days of metric history
- Access UI at
http://<host>:9090
# Request rate (requests per second)
rate(inference_predictions_total[5m])
# 95th percentile latency
histogram_quantile(0.95, rate(inference_request_latency_seconds_bucket[5m]))
# Anomaly rate percentage
rate(inference_anomaly_count_total[5m]) / rate(inference_predictions_total[5m]) * 100
# Error rate
rate(inference_error_count_total[5m])
| Alarm | Threshold | Action |
|---|---|---|
| CPU High | > 80% for 5 min | Scale up / investigate |
| Status Check Failed | Any failure | Instance recovery |
| Error Rate | > 50 errors / 5 min | Check logs, rollback |
| Anomaly Spike | > 100 anomalies / 5 min | Investigate incident |
Current: Single EC2 + Docker Compose
│
▼
Phase 2: EC2 Auto Scaling Group + ALB
│ - Multiple EC2 instances behind a load balancer
│ - Auto-scale based on CPU / request count
│
▼
Phase 3: ECS Fargate
│ - Serverless containers (no EC2 management)
│ - Per-service scaling (inference can scale independently)
│
▼
Phase 4: EKS (Kubernetes)
- Full orchestration with K8s
- Horizontal Pod Autoscaler
- Service mesh (Istio) for traffic management
- Canary deployments for model updates
- Current: Single uvicorn worker, model in memory
- Next: Multiple workers with shared model cache (Redis)
- Future: Dedicated model serving (TorchServe, TFServing, Triton)
- Current: JSON files on disk
- Next: Apache Kafka for real-time log streaming
- Future: AWS Kinesis → S3 → SageMaker Pipeline
| Decision | Chosen | Alternative | Why |
|---|---|---|---|
| ML Model | Isolation Forest | Autoencoder, LSTM | Unsupervised, fast, no GPU needed |
| API Framework | FastAPI | Flask, Django | Async, auto-docs, Pydantic validation |
| Serialization | joblib .pkl |
ONNX, PMML | Native sklearn support, includes scaler |
| Monitoring | Prometheus | Datadog, CloudWatch | Free, powerful PromQL, industry standard |
| IaC | Terraform | CDK, Pulumi | Multi-cloud, mature ecosystem |
| Deployment | Docker Compose | ECS, K8s | Simple to start, easy migration path |
| CI/CD | GitHub Actions | Jenkins, GitLab CI | Native, free for public repos |
| Log Format | JSONL | CSV, Parquet | Streaming-friendly, human-readable |
| Feature Scaling | StandardScaler | MinMaxScaler | Better for Gaussian-like distributions |
| Anomaly Rate | 5% contamination | Adaptive | Matches known injection rate |
- 12-Factor App: Config via env vars, stateless services, port binding
- Clean Architecture: Separation of concerns (metrics, model loading, API logic)
- Observability First: Metrics exposed from day one, not bolted on later
- Infrastructure as Code: All infra is version-controlled and reproducible
- Security by Default: Non-root containers, encrypted S3, least-privilege IAM
Request:
{
"latency_ms": 150.0,
"order_amount": 79.99,
"quantity": 2,
"http_status": 200,
"hour_of_day": 14
}Response:
{
"is_anomaly": false,
"anomaly_score": 0.123456,
"confidence": "low",
"model_version": "20240215_143022",
"prediction_time_ms": 1.234,
"timestamp": "2024-02-15T14:30:22.123456+00:00"
}Accepts {"events": [...]} with up to 1000 events.
Returns service status, model version, and uptime.
Hot-reloads the model without restarting the service.
Prometheus-formatted metrics for scraping.
Interactive Swagger UI documentation.
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
- Python: Black formatter, flake8 linting, isort imports
- Terraform:
terraform fmt,terraform validate - Commits: Conventional Commits format
This project is licensed under the MIT License.