🛡️ Production Monitoring & Observability Layer for AI Systems (RAG)

AI Site Reliability Engineering (AI SRE) Architecture Blueprint & Implementation

---

📂 Repository Layout

.
├── .github/
│   ├── ISSUE_TEMPLATE/
│   │   ├── bug_report.md
│   │   └── feature_request.md
│   ├── pull_request_template.md
│   └── workflows/
│       └── ai_regression_gate.yml   # Automated PR Latency & Quality Gate
├── config/
│   ├── prompt_config.yaml           # Version-controlled prompts & inference params
│   └── slo_thresholds.yaml          # AI SRE Service Level Objectives & CI gates
├── src/
│   ├── evaluator.py                 # Groundedness & Citation Coverage evaluators
│   ├── metrics_collector.py         # P50/P95 latency, cost, and failure telemetry
│   └── pipeline.py                  # End-to-end RAG pipeline instrumented with Langfuse
├── ci/
│   ├── baseline_metrics.json        # Production baseline snapshot for regression gating
│   ├── golden_dataset.json          # Benchmark dataset for CI runs
│   └── regression_gate.py           # CI regression gating evaluation engine
├── tests/
│   ├── test_evaluator.py            # Unit tests for citation & faithfulness metrics
│   ├── test_metrics.py              # Unit tests for percentiles & cost attribution
│   └── test_pipeline.py             # Unit tests for RAG pipeline components
├── docker-compose.yml               # Self-hosted Langfuse stack (Server, Worker, DB)
├── pyproject.toml                   # Project metadata and tool configuration
├── Makefile                         # Common developer automation commands
├── requirements.txt                 # Dependencies
├── .env.example                     # Environment configuration template
├── .gitignore                       # Git ignore rules
├── LICENSE                          # Apache 2.0 License
├── CONTRIBUTING.md                  # Development & contribution guidelines
├── SECURITY.md                      # Vulnerability reporting policy
└── README.md                        # Documentation and architecture guide

---

📌 Executive Architecture Overview

In classical SRE, we monitor the Four Golden Signals (Latency, Traffic, Errors, and Saturation). For production Retrieval-Augmented Generation (RAG) systems, non-deterministic model behavior, semantic drift, and variable token costs require an extended paradigm: The Five Golden Signals of AI Systems:

flowchart TD
    subgraph GoldenSignals["The 5 Golden Signals of AI Systems"]
        L["1. Latency (P50, P95, P99)<br/>Retrieval vs. Generation"]
        T["2. Traffic & Saturation<br/>RPS, TPM, RPM, Concurrency"]
        E["3. Failure & Error Rates<br/>5xx, Rate Limits, Fallbacks"]
        C["4. Token Cost Economics<br/>USD/query, Prompt vs. Completion"]
        Q["5. Quality & Faithfulness<br/>Citation Coverage, Hallucination Rate"]
    end

    Client["Client / User Query"] --> Gateway["API Gateway / FastAPI"]
    Gateway --> Trace["Langfuse Tracing Root Span"]
    
    Trace --> Ret["Span 1: Vector/BM25 Retrieval"]
    Ret --> Rerank["Span 2: Cross-Encoder Re-ranker"]
    Rerank --> Gen["Generation: LLM Inference & Token Tracking"]
    Gen --> Eval["Quality Evaluator: Citation Coverage & Faithfulness"]
    Eval --> LangfuseBackend["Self-Hosted Langfuse Server + Worker"]
    
    LangfuseBackend --> Postgres["PostgreSQL (Metadata & State)"]
    LangfuseBackend --> ClickHouse["ClickHouse / Analytics Engine"]
    
    ClickHouse --> Dashboards["SRE Dashboards (Langfuse & Grafana)"]
    ClickHouse --> CIGate["CI/CD Regression Gate (Pull Requests)"]

Loading

---

🏗️ Self-Hosted Langfuse Setup Guide

This deployment utilizes the open-source, self-hosted distribution of Langfuse, decoupled into a web service and background worker backed by PostgreSQL and persistent storage.

1. Stack Topology (docker-compose.yml)

The stack consists of:

• langfuse-server: Next.js web application and ingestion REST API.
• langfuse-worker: Asynchronous execution worker for background evaluations, analytics, and exports.
• db: PostgreSQL 16 transactional store for traces, scores, prompts, and users.

version: "3.9"

services:
  langfuse-server:
    image: langfuse/langfuse:2
    depends_on:
      db:
        condition: service_healthy
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=production
      - DATABASE_URL=postgresql://langfuse:langfuse_secure_pass@db:5432/langfuse
      - NEXTAUTH_SECRET=${NEXTAUTH_SECRET}
      - SALT=${SALT}
      - NEXTAUTH_URL=http://localhost:3000
      - TELEMETRY_ENABLED=false
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "wget", "--spider", "http://localhost:3000/api/public/health"]
      interval: 10s
      timeout: 5s
      retries: 5

  langfuse-worker:
    image: langfuse/langfuse-worker:2
    depends_on:
      db:
        condition: service_healthy
    environment:
      - DATABASE_URL=postgresql://langfuse:langfuse_secure_pass@db:5432/langfuse
      - SALT=${SALT}
      - TELEMETRY_ENABLED=false
    restart: unless-stopped

  db:
    image: postgres:16-alpine
    restart: unless-stopped
    environment:
      - POSTGRES_USER=langfuse
      - POSTGRES_PASSWORD=langfuse_secure_pass
      - POSTGRES_DB=langfuse
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U langfuse -d langfuse"]
      interval: 5s
      timeout: 5s
      retries: 5

volumes:
  pgdata:
    driver: local

2. Initialization Runbook

1. Bootstrap Environment:

   cp .env.example .env
   # Generate high-entropy secrets
   openssl rand -hex 32 # Set as NEXTAUTH_SECRET
   openssl rand -hex 16 # Set as SALT

2. Start Services:

   docker compose up -d

3. Obtain API Keys:

   • Navigate to http://localhost:3000.
   • Create your administrator account and initial Organization.
   • Go to Settings > API Keys and generate a new key pair:
     • pk-lf-... (Public Key)
     • sk-lf-... (Secret Key)
   • Export these keys in your application environment or secrets vault.

---

📊 Conceptual Dashboard Architecture

An effective AI SRE observability platform structures telemetry into three operational tiers:

┌────────────────────────────────────────────────────────────────────────┐
│ TIER 1: Executive / SLO Health Dashboard (Single Pane of Glass)        │
│ • End-to-End P95 Latency SLO Gauge (Target: < 2.8s, Current: 1.8s)    │
│ • Faithfulness SLO Gauge (Target: > 85%, Current: 94.2%)              │
│ • Hourly Error Rate & Unsupported Query Percentage                    │
│ • Daily Cumulative Token Cost Burn Rate & Budget Burnout Forecast     │
└────────────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
┌────────────────────────────────────────────────────────────────────────┐
│ TIER 2: Pipeline Component Diagnostics (Component Isolation)          │
│ • Retrieval Stage: Vector search latency (P50/P95), Cosine Score Dist. │
│ • Re-Ranking Stage: Cross-encoder latency, Rank Position Drift         │
│ • Generation Stage: TTFT (Time to First Token), Token Gen Speed (tok/s)│
│ • Citation Engine: Coverage distribution (% claims citing docs)        │
└────────────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
┌────────────────────────────────────────────────────────────────────────┐
│ TIER 3: Trace Waterfall & Drilldown (Incident Root Cause Analysis)     │
│ • Individual Trace Execution Timeline & Span Hierarchy                │
│ • Exact Prompts, Model System Messages, & Raw Completions             │
│ • Ingested Score Annotations (Groundedness, Citation Validity)         │
│ • Exception Call Stacks, HTTP Status Codes, & Rate Limit Headers       │
└────────────────────────────────────────────────────────────────────────┘

Core SLI / SLO Matrix

Dimension	Service Level Indicator (SLI)	SLO Target	Error Budget (30-day)	Pager Alert Condition
Latency	End-to-End P95 request duration	$\le 2800\text{ ms}$	$5%$ of requests	P95 $> 3500\text{ ms}$ for 10 min
Quality	Faithfulness / Groundedness score	$\ge 85.0%$	$15%$ tolerance	Average $< 80.0%$ for 15 min
Quality	Citation Coverage (valid claims)	$\ge 80.0%$	$20%$ tolerance	Average $< 75.0%$ for 15 min
Reliability	API 5xx & Provider Exception Rate	$\le 2.0%$	$2%$ error budget	$> 5.0%$ errors over 5 min
Economics	Average LLM Cost per Query	$\le $0.015$	Soft budget cap	Spike $> $0.035$/req for 1 hr

---

🔍 Phase 1: Pipeline Tracing Architecture

The RAG pipeline is instrumented using Langfuse's @observe context system. Every invocation establishes a hierarchical execution DAG:

sequenceDiagram
    autonumber
    actor Client
    participant Pipeline as Root: rag_query_pipeline
    participant VectorDB as Span: retrieval_step
    participant Reranker as Span: reranking_step
    participant LLM as Generation: llm_generation
    participant Evaluator as Evaluator: RAGEvaluator
    participant Langfuse as Langfuse Server

    Client->>Pipeline: Query("What is canary rollback criteria?")
    Pipeline->>Langfuse: Open Root Trace (user_id, session_id, tags)
    
    Pipeline->>VectorDB: Query Vector Store (top_k=4)
    VectorDB-->>Pipeline: Candidate Docs + Cosine Scores
    VectorDB->>Langfuse: Close Span (latency, doc_count, score_dist)

    Pipeline->>Reranker: Re-rank Candidates (top_n=2)
    Reranker-->>Pipeline: Reordered Chunks + Cross-Encoder Scores
    Reranker->>Langfuse: Close Span (score_delta, rerank_latency)

    Pipeline->>LLM: Formatted Prompt (System + Context + User)
    LLM-->>Pipeline: Completion Output + Token Counts
    LLM->>Langfuse: Close Generation (prompt_tokens, completion_tokens, cost)

    Pipeline->>Evaluator: Compute Groundedness & Citation Coverage
    Evaluator-->>Pipeline: Scores (citation_coverage=1.0, faithfulness=1.0)
    Pipeline->>Langfuse: score_current_trace(citation_coverage, faithfulness)
    
    Pipeline-->>Client: Final Verified Response

Loading

Key Instrumentation Code: src/pipeline.py

Every single step is traced with dedicated span semantics:

• @observe(name="retrieval_step"): Enriches span metadata with document similarity distributions and vector latency.
• @observe(name="reranking_step"): Records candidate reordering, cross-encoder scores, and rank displacement.
• @observe(as_type="generation", name="llm_generation"): Captures prompt messages, completion output, exact token usage breakdown (prompt_tokens, completion_tokens), and model parameters (temperature, seed, max_tokens).
• langfuse_context.score_current_trace(...): Real-time ingestion of calculated reliability scores directly into the root trace.

---

📈 Phase 2: Metric Tracking & AI SRE Telemetry

The metrics engine (src/metrics_collector.py) queries telemetry to calculate:

1. Latency Percentiles: Accurate percentile interpolation ($P_{50}$, $P_{95}$, $P_{99}$) for both end-to-end and individual pipeline stages.
2. LLM Cost Tracking: Dynamic model pricing engine tracking cost per request and projecting 100k query burn rates.
3. Citation Coverage & Faithfulness: Automated validation verifying that statements reference valid document IDs and match context semantics.
4. Reliability & Failure Rates: Tracking provider errors, rate-limiting HTTP 429s, and unhandled exceptions.

Production SQL Queries for ClickHouse / PostgreSQL

For high-throughput environments scraping metrics into Prometheus or Grafana:

-- 1. Latency Percentiles (ClickHouse)
SELECT
    quantile(0.50)(latency) * 1000 AS p50_latency_ms,
    quantile(0.95)(latency) * 1000 AS p95_latency_ms,
    quantile(0.99)(latency) * 1000 AS p99_latency_ms
FROM traces
WHERE project_id = 'PROJECT_ID'
  AND timestamp >= now() - INTERVAL 1 HOUR;

-- 2. LLM Cost & Token Breakdown
SELECT
    model,
    count(*) AS total_queries,
    sum(prompt_tokens) AS prompt_tokens,
    sum(completion_tokens) AS completion_tokens,
    sum(calculated_total_cost) AS total_cost_usd,
    avg(calculated_total_cost) AS avg_cost_per_query
FROM observations
WHERE type = 'GENERATION'
  AND timestamp >= now() - INTERVAL 24 HOUR
GROUP BY model;

-- 3. Citation Coverage & Groundedness
SELECT
    name AS metric,
    round(avg(value), 4) AS mean_score,
    round(quantile(0.10)(value), 4) AS p10_tail_score
FROM scores
WHERE name IN ('citation_coverage', 'faithfulness')
  AND timestamp >= now() - INTERVAL 24 HOUR
GROUP BY name;

---

🚦 Phase 3: CI/CD Regression Gating

To prevent code or prompt regressions from reaching production, we version control all SLO targets and prompts alongside the application code:

• config/slo_thresholds.yaml: Defines maximum acceptable regression deltas and absolute floors.
• config/prompt_config.yaml: Versioned system prompts and inference hyperparameters.
• ci/baseline_metrics.json: Main-branch production baseline snapshot.

CI Regression Evaluation Script: ci/regression_gate.py

The gate executes candidate code across the golden benchmark suite (ci/golden_dataset.json), extracts telemetry, compares results against baseline, and enforces the rules:

$$\Delta P_{95} = \frac{P_{95}^{\text{candidate}} - P_{95}^{\text{baseline}}}{P_{95}^{\text{baseline}}} \times 100 \le +15%$$

$$\Delta \text{Faithfulness} = \frac{\text{Faithfulness}^{\text{baseline}} - \text{Faithfulness}^{\text{candidate}}}{\text{Faithfulness}^{\text{baseline}}} \times 100 \le +5%$$

$$\text{Faithfulness}^{\text{candidate}} \ge 0.85 \quad (\text{Strict Floor})$$

+-----------------------------------------------------------------------------------------+
|                  AI SRE OBSERVABILITY & QUALITY REGRESSION GATE                         |
+-----------------------------------------------------------------------------------------+
| Metric Check               | Baseline    | Candidate   | Delta    | SLO Gate   | Result |
+----------------------------+-------------+-------------+----------+------------+--------+
| P95 Latency Regression     | 2150.0 ms   | 186.0 ms    | -91.35%  | <= +15.0%  |   ✅   |
| P95 Absolute Ceiling       | N/A         | 186.0 ms    | -        | <= 3200 ms |   ✅   |
| Faithfulness Regression    | 0.920       | 1.000       | +8.70%   | Drop <= 5% |   ✅   |
| Faithfulness Floor         | N/A         | 1.000       | -        | >= 0.85    |   ✅   |
| Cost per Request Spike     | $0.000180   | $0.000018   | -85.26%  | <= +20.0%  |   ✅   |
| Unhandled Error Rate       | 0.00%       | 0.00%       | -        | 0 errors   |   ✅   |
+----------------------------+-------------+-------------+----------+------------+--------+
| OVERALL STATUS: PASSED (EXIT CODE 0)                                                    |
+-----------------------------------------------------------------------------------------+

If any check fails, ci/regression_gate.py exits with code 1, halting the GitHub Actions workflow and blocking the Pull Request.

---

🏃 Verification & Quickstart

To run the complete suite locally:

# 1. Run the instrumented RAG pipeline
python3 src/pipeline.py

# 2. Compute AI SRE Metrics & SLO Compliance
python3 src/metrics_collector.py

# 3. Execute CI Regression Gate Benchmark
python3 ci/regression_gate.py