otel-loki-stack

A vendor-neutral, cloud-portable logging stack — OpenTelemetry Collector + Loki + Grafana on S3-compatible object storage. Built so that moving between clouds is a config edit, not a rewrite.

Every cloud wants to own your logs (CloudWatch, Cloud Logging, Azure Monitor) — and each one welds your dashboards, alerts and half your app code to a single provider. This stack keeps the logging layer portable: a standard wire format in (OTLP), a standard query language out (LogQL), and durable state that lives in a bucket instead of a vendor.

📖 Full write-up: OpenTelemetry + Loki: The Logging Stack That Survives a Cloud Migration

Architecture

flowchart LR
    A[Apps & services<br/>OTLP / stdout] --> C
    subgraph C[OpenTelemetry Collector]
      R[receivers] --> P[processors] --> E[exporters]
    end
    C --> L[Loki<br/>index + chunks]
    L --> S[(Object storage<br/>S3 · GCS · MinIO)]
    G[Grafana<br/>LogQL + dashboards] -.reads via LogQL.-> L

Loading

Nothing in that path is provider-specific. Applications talk only to the Collector's receivers; the backend is chosen entirely in the exporters. Loki stores its index and chunks on any S3-compatible bucket. Change cloud and the only thing that moves is where the storage lives.

Quick start

cp .env.example .env      # then edit the values
docker compose up -d

• Grafana → http://localhost:3000 (admin / GRAFANA_ADMIN_PASSWORD)
• Send OTLP from an app:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
export OTEL_RESOURCE_ATTRIBUTES="service.name=checkout,deployment.environment=production"

The bundled MinIO stands in for cloud object storage, so your local stack and your production stack speak the identical S3 API — dev and prod differ only by S3_ENDPOINT and credentials.

What's in here

.
├── docker-compose.yml                     # collector + loki + grafana + minio
├── otel/config.yaml                       # the portability seam (receivers→processors→exporters)
├── loki/loki.yaml                         # storage on S3, retention, compaction, ruler
├── grafana/provisioning/datasources/loki.yaml
├── rules/production.yaml                  # log-based alerting (Loki ruler)
└── .env.example

Moving between clouds

The migration is a bucket copy and an endpoint swap — apps, Collector pipeline and dashboards are untouched.

# 1. move the history between clouds, S3 API to S3 API
rclone sync a-cloud:logs-prod b-cloud:logs-prod --transfers 32 --checksum

# 2. stand up the identical stack on the new cloud (same configs, new S3_ENDPOINT)
# 3. dual-write during cutover: add a second exporter in otel/config.yaml
#      exporters: [ loki, loki/b ]
# 4. flip DNS for Grafana + the OTLP ingress
# 5. drop the old exporter once retention has aged out

The one rule about labels

Index labels, not content. Labels are for bounded sets (service, environment, level). High-cardinality values — user IDs, request IDs, trace IDs, URLs — go in the log line and are reached by filter, never as labels. Break this and Loki creates a stream per unique value, the index balloons and queries crawl.

# GOOD
{service_name="checkout", environment="production", level="error"} | json | user_id="u_8842"

# BAD — user_id as a label explodes the index
{service_name="checkout", user_id="u_8842"}

Alerting

Loki's built-in ruler evaluates LogQL on a schedule (see rules/production.yaml) and posts to Alertmanager or straight to a webhook — e.g. an n8n webhook that classifies and fans out to Telegram and internal apps.

License

MIT