🌗 Half-Moon

An observability stack built from the wire up: a custom exporter for network edge devices, a distributed service instrumented with OpenTelemetry, and an SLO engine that turns both signal sources into alerting and dashboards as code.

Overview

Most observability portfolio projects stop at standing up Prometheus and Grafana with off-the-shelf exporters. Vantage covers the parts that are harder to fake: producing signals that no exporter gives you, correlating them across services, and converting them into decisions about reliability.

The project is built in three stages, each usable on its own and each feeding the next.

Status

• 🟢 Stage 1
• ⚠️ Stage 2
• 🛑 Stage 3

Stage 1

produces telemetry from infrastructure whose source code you do not control. A Python exporter polls a SonicWall or Palo Alto firewall over SNMP and its REST API, and publishes interface throughput, session table utilization, VPN tunnel state, and license and certificate expiry as Prometheus metrics. This is the black box problem: you own the reliability of a device you cannot instrument.

Stage 2

produces telemetry from code you do own. A three-service FastAPI application, deliberately built with realistic failure modes, is instrumented with the OpenTelemetry SDK. Traces go to Tempo, metrics to Prometheus, logs to Loki, and exemplars connect a latency spike on a graph to the exact trace that caused it. A fault injection switch makes the failures reproducible on demand.

Stage 3

turns raw signal into judgment. An SLO engine reads service level objectives defined in YAML, queries Prometheus for historical performance, computes error budget consumption and burn rate, and generates multi-window multi-burn-rate Alertmanager rules and Grafana dashboards as versioned artifacts. Pandas handles the historical analysis. Nothing in Stage 3 is written by hand in the monitoring stack; it is all generated from the objective definitions.

The full stack runs locally with docker compose up, including provisioned dashboards and a seeded metrics history, so the demo starts working rather than starting empty.

Architecture

  ┌─────────────────┐        ┌──────────────────────┐
  │  Edge devices   │        │  Instrumented app    │
  │  (SNMP / REST)  │        │  (FastAPI x3 + PG)   │
  └────────┬────────┘        └──────────┬───────────┘
           │ Stage 1                    │ Stage 2
           ▼                            ▼
  ┌─────────────────┐        ┌──────────────────────┐
  │  edge-exporter  │        │  OTel Collector      │
  └────────┬────────┘        └───┬──────────┬───────┘
           │                     │          │
           ▼                     ▼          ▼
     ┌──────────────────────────────┐  ┌────────┐  ┌──────┐
     │         Prometheus           │  │ Tempo  │  │ Loki │
     └──────────┬───────────────────┘  └────────┘  └──────┘
                │ PromQL API
                ▼  Stage 3
       ┌──────────────────┐
       │   slo-engine     │──▶ generated: alert rules, dashboards
       └──────────────────┘

Stage 1: Edge Exporter

Description

A Prometheus exporter for network edge hardware. It polls a firewall on a configurable interval and exposes device state as metrics on a scrape endpoint. Device access is abstracted behind a collector interface so that SNMP and vendor REST APIs are interchangeable backends, and so additional device families can be added without touching the exporter core.

Metrics of interest: per interface throughput and error counters, session table utilization against the platform maximum, IPsec and SSL VPN tunnel state, high availability failover status, DHCP pool exhaustion, and days remaining on licenses and certificates.

Skills and tools

• Python 3.12, asyncio
• prometheus_client for metric types, registries, and the exposition format
• pysnmp for SNMP v2c/v3 polling and MIB walking
• httpx for vendor REST APIs (SonicOS, PAN-OS)
• pydantic and pydantic-settings for typed configuration
• Docker, multi-stage builds, non-root container user
• pytest with recorded device fixtures so tests run without hardware

Milestones

• Poll a single device over SNMP and print parsed interface counters
• Expose the first metric on /metrics in valid Prometheus exposition format
• Define the collector interface and implement a second backend against a vendor REST API
• Add all target metrics with correct types (counter vs gauge) and stable label sets
• Handle failure without lying: unreachable devices produce a scrape error and an up-style health gauge, not stale values
• Package as a container image and add it as a scrape target in the compose stack
• Write a Grafana dashboard for the device, provisioned from JSON in the repo
• Record fixtures from real devices and get the test suite passing offline
• Document the metric catalog: name, type, labels, meaning, and why it matters operationally

Stage 2: Instrumented Service

Description

A small distributed system that exists to be observed. An API gateway accepts requests, a worker service does the processing, and Postgres holds state. Every hop is instrumented with OpenTelemetry so a single request produces a connected trace, RED metrics, and structured logs carrying the same trace ID.

The application ships with a fault injection controller that can add latency, raise error rates, exhaust the database connection pool, or introduce a slow dependency. This makes the failure scenarios reproducible, which is what turns the project into a demonstration rather than a screenshot.

Skills and tools

• FastAPI, uvicorn, SQLAlchemy, PostgreSQL
• OpenTelemetry Python SDK: automatic and manual instrumentation, span attributes, context propagation across service boundaries
• OpenTelemetry Collector: receivers, processors, exporters, tail sampling
• Grafana Tempo for traces, Loki for logs, Prometheus for metrics
• Exemplars linking metric samples to trace IDs
• structlog for JSON logs with trace correlation
• Docker Compose for the full local stack

Milestones

• Build the three service components with at least one genuinely slow code path
• Add automatic instrumentation and confirm a trace appears in Tempo
• Verify trace context propagates across all service boundaries, including the database calls
• Add manual spans and attributes for the business logic that automatic instrumentation misses
• Emit RED metrics (rate, errors, duration) as histograms with sensible buckets
• Wire structured logging so every log line carries its trace ID, and confirm the Loki to Tempo jump works
• Enable exemplars and confirm the click-through from a latency graph to a trace
• Build the fault injection controller with at least four distinct failure modes
• Configure tail sampling in the Collector and explain the sampling policy in the docs
• Write one incident walkthrough: trigger a failure, then document the path from alert to root cause using only the tooling

Stage 3: SLO Engine

Description

A command line tool and library that treats service level objectives as source code. Objectives are declared in YAML with an SLI query, a target, and a compliance window. The engine validates them, queries Prometheus over its HTTP API for historical performance, calculates error budget consumption and burn rate, and generates the monitoring configuration needed to enforce them.

Generated output includes multi-window multi-burn-rate Alertmanager rules following the Google SRE workbook approach, and a Grafana dashboard per service showing budget remaining and burn rate over the window. Because the output is generated and committed, changes to reliability targets go through code review like any other change.

Skills and tools

• Python packaging with uv, CLI built on typer
• pydantic for the SLO schema and validation errors that are actually readable
• Prometheus HTTP API: instant and range queries, httpx client with retry handling
• Pandas for time series analysis of budget consumption and burn windows
• jinja2 for rule and dashboard generation
• Multi-window multi-burn-rate alerting theory
• GitHub Actions: validate SLO definitions and check that generated artifacts are current on every pull request

Milestones

• Define and validate the SLO schema against a real objective from Stage 1 or 2
• Query Prometheus and compute current compliance for a single objective
• Compute error budget remaining and burn rate over configurable windows
• Generate a recording rule set so budget calculations are cheap at query time
• Generate multi-window multi-burn-rate alert rules and verify them with promtool
• Generate a Grafana dashboard per service and provision it automatically
• Add historical analysis with Pandas: budget consumption by day, worst burn periods, projected exhaustion
• Add the CI check that fails when generated artifacts drift from the definitions
• Seed a realistic metrics history so the dashboards are populated on first run
• Write objectives for the firewall from Stage 1 and the API from Stage 2, closing the loop across all three stages

Repository layout

half-moon/
├── edge-exporter/        Stage 1
├── demo-service/         Stage 2
├── slo-engine/           Stage 3
├── deploy/
│   ├── compose/          Full local stack
│   ├── grafana/          Provisioned dashboards and datasources
│   └── prometheus/       Scrape config, generated rules
├── docs/
│   ├── metric-catalog.md
│   ├── incident-walkthroughs/
│   └── decisions/        Architecture decision records
└── slos/                 SLO definitions in YAML

Running locally

git clone https://github.com/<user>/half-moon.git
cd vantage
docker compose -f deploy/compose/docker-compose.yml up

Grafana is available at http://localhost:3000 with dashboards already provisioned. The demo service is at http://localhost:8000, and its fault injection endpoints are documented at /docs.