consul-ecs ECS Cluster Setup

This directory contains Terraform and scripts to provision a minimal AWS environment running a Consul-connected ECS cluster using the consul-ecs sidecar pattern.

Architecture

                        ┌─────────────────────────────────┐
                        │            AWS VPC               │
                        │         10.0.0.0/16              │
                        │                                  │
   ┌────────────┐        │  ┌──────────────────────────┐   │
   │  Operator  │───SSH──┼─▶│   Consul Server (EC2)    │   │
   │  (you)     │        │  │   t3.small, public subnet │   │
   └────────────┘        │  │   Consul 1.22.0 + ACLs   │   │
                        │  └──────────┬───────────────┘   │
                        │             │ HTTP :8500          │
                        │             │ gRPC :8502          │
                        │  ┌──────────▼───────────────┐   │
                        │  │    ECS Cluster (EC2)      │   │
                        │  │   t3.medium, private subnet│   │
                        │  │                           │   │
                        │  │  ┌─────────────────────┐  │   │
                        │  │  │    ECS Task         │  │   │
                        │  │  │  ┌───────────────┐  │  │   │
                        │  │  │  │  mesh-init    │  │  │   │
                        │  │  │  │  health-sync  │  │  │   │
                        │  │  │  │  consul-dp    │  │  │   │
                        │  │  │  │  app (nginx)  │  │  │   │
                        │  │  │  └───────────────┘  │  │   │
                        │  │  └─────────────────────┘  │   │
                        │  └───────────────────────────┘   │
                        └─────────────────────────────────┘

Components

Component	Description
Consul server	Single-node Consul 1.22.0 on EC2, ACLs enabled, management token pre-provisioned by Terraform
ECS cluster	EC2-backed ECS cluster (Amazon Linux 2023, t3.medium) in a private subnet
mesh-init	consul-ecs sidecar — registers the service and proxy in Consul, writes consul-dataplane.json to shared volume
consul-dataplane	Envoy proxy sidecar — reads config from shared volume, connects to Consul via gRPC
health-sync	consul-ecs sidecar — syncs ECS container health checks to Consul TTL checks
app	nginx:alpine — the application container (port 80)

ECS Task Layout

All four containers share a /consul volume. The startup order is:

mesh-init (runs once, essential=false)
    └── consul-dataplane (depends on mesh-init SUCCESS)
    └── health-sync      (depends on mesh-init SUCCESS)
    └── app              (no dependency, starts immediately)

mesh-init exits cleanly after registration. The remaining three containers run for the lifetime of the task.

Networking

Subnet	CIDR	Used for
Public 1	10.0.1.0/24	Consul EC2, NAT Gateway
Public 2	10.0.2.0/24	Reserved
Private	10.0.10.0/24	ECS container instances

ECS tasks in the private subnet reach Consul and the internet via NAT Gateway. The Consul HTTP API (port 8500) is accessible from your IP for operator queries.

---

Prerequisites

• Terraform >= 1.0
• AWS CLI configured with credentials
• Go >= 1.21 (to build the consul-ecs binary)
• Docker (to build and push the image)
• An SSH key pair for accessing the Consul EC2 instance

# Generate SSH key if you don't have one
ssh-keygen -t ed25519 -f ~/.ssh/consul-ecs -N ""

---

Setup

All commands are run from the test-setup/ directory.

Step 1 — Create the ECR repository

cd terraform
terraform init
terraform apply -auto-approve -target=aws_ecr_repository.consul_ecs

Step 2 — Build and push the consul-ecs image

cd ..
bash scripts/build_and_push.sh

This builds the consul-ecs binary for linux/amd64, builds the Docker image, and pushes it to ECR.

Step 3 — Provision all infrastructure

cd terraform
terraform apply -auto-approve

This provisions:

• VPC, subnets, IGW, NAT Gateway
• Consul EC2 instance (with management token pre-configured)
• ECR repository
• IAM roles and instance profiles
• ECS cluster, launch template, autoscaling group
• ECS task definition and service
• CloudWatch log group

The Consul management token is automatically generated by Terraform (random_uuid) and embedded in the Consul server config as initial_management. No manual bootstrap step is required.

Step 4 — Verify the cluster is up

Wait for the ECS service to stabilize (typically 2–3 minutes):

REGION=$(terraform output -raw region)
CLUSTER=$(terraform output -raw ecs_cluster_name)
SERVICE=$(terraform output -raw ecs_service_name)

aws ecs wait services-stable --region "$REGION" --cluster "$CLUSTER" --services "$SERVICE"
echo "ECS service is stable"

Check that the service is registered in Consul:

CONSUL_IP=$(terraform output -raw consul_server_ip)
TOKEN=$(terraform output -raw consul_token)

curl -s -H "X-Consul-Token: $TOKEN" \
  "http://$CONSUL_IP:8500/v1/health/service/test-service" | jq .

---

Accessing the cluster

Consul HTTP API

export CONSUL_HTTP_ADDR="http://$(terraform output -raw consul_server_ip):8500"
export CONSUL_HTTP_TOKEN="$(terraform output -raw consul_token)"

consul members
consul catalog services

CloudWatch logs

Each container writes to the /consul-ecs/test log group with a per-container prefix:

REGION=$(terraform output -raw region)
aws logs tail /consul-ecs/test --region "$REGION" --log-stream-name-prefix mesh-init --follow
aws logs tail /consul-ecs/test --region "$REGION" --log-stream-name-prefix health-sync --follow
aws logs tail /consul-ecs/test --region "$REGION" --log-stream-name-prefix consul-dataplane --follow

ECS Exec (shell into a running container)

The ECS service has enableExecuteCommand = true. To open a shell in the app container:

REGION=$(terraform output -raw region)
CLUSTER=$(terraform output -raw ecs_cluster_name)
SERVICE=$(terraform output -raw ecs_service_name)

TASK=$(aws ecs list-tasks --region "$REGION" --cluster "$CLUSTER" \
  --service-name "$SERVICE" --query 'taskArns[0]' --output text)

aws ecs execute-command \
  --region "$REGION" \
  --cluster "$CLUSTER" \
  --task "$TASK" \
  --container app \
  --interactive \
  --command "/bin/sh"

SSH into Consul EC2

ssh -i ~/.ssh/consul-ecs ec2-user@$(terraform output -raw consul_server_ip)

---

Configuration

Terraform variables

All variables have defaults and nothing needs to be set manually. Override in terraform/terraform.tfvars if needed:

Variable	Default	Description
region	ap-south-1	AWS region
consul_version	1.22.0	Consul version to install
instance_type_consul	t3.small	Consul EC2 instance type
instance_type_ecs	t3.medium	ECS container instance type
ssh_public_key_path	~/.ssh/consul-ecs.pub	Path to SSH public key

Your public IP is auto-detected at terraform apply time and used to restrict SSH and Consul API access in the security group.

The Consul management token is auto-generated — retrieve it at any time with:

terraform output consul_token

consul-ecs config

The CONSUL_ECS_CONFIG_JSON passed to mesh-init and health-sync is rendered from terraform/templates/consul_ecs_config.json.tpl:

{
  "bootstrapDir": "/consul",
  "consulServers": {
    "hosts": "<consul-private-ip>",
    "http":  { "port": 8500, "https": false },
    "grpc":  { "port": 8502 },
    "skipServerWatch": true,
    "defaults": { "tls": false }
  },
  "service": { "name": "test-service", "port": 80 },
  "proxy": { "publicListenerPort": 20000, "healthCheckPort": 22000 },
  "healthSyncContainers": ["app"],
  "transparentProxy": { "enabled": false }
}

healthSyncContainers: ["app"] tells health-sync to mirror the ECS health status of the app container into the Consul proxy check.

---

Tear down

cd terraform
terraform destroy -auto-approve