SentinelAI

SentinelAI — Enterprise AI Reliability & Governance Platform

• Detect model drift
• Monitor inference anomalies
• Track LLM hallucination risk
• Provide real-time observability
• Automate AI governance workflows

It combines statistical ML monitoring with LLM-powered incident intelligence.

Production Readiness Guide

This section is the portfolio audit entry point for SentinelAI. It describes an engineering promotion path; it is not a claim that the repository is already production-authorized.

Architecture flowchart

flowchart LR
    Source --> Build[Release binary] --> Tests[Unit + sanitizer tests] --> Artifact[Versioned artifact]

Loading

Quickstart and local validation

The supported local path should be reproducible from a clean checkout. The inferred stack for this repository is C++.

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build
ctest --test-dir build --output-on-failure

If the project uses external services, model artifacts, cloud credentials, or private data, start them through documented local fixtures or mocks. Never place secrets or identifiable records in the repository.

Research-style metrics and benchmarks

Evidence	Required record
Correctness	Test command, commit SHA, runtime, and pass/fail result
Performance	Warm-up, sample count, concurrency, median, p95, p99, throughput, and memory
Data/model quality	Dataset version, split strategy, leakage controls, calibration, subgroup results, and uncertainty
Runtime	Image digest, health-check latency, resource limits, and rollback target
Security	Dependency, secret, SAST, container, and SBOM results

A benchmark number belongs in a versioned artifact tied to a commit and hardware/runtime description. Engineering benchmarks must not be presented as clinical, financial, safety, or model-quality validation without the appropriate domain evidence.

Extended Q&A

What is production-ready for this repository?
A reproducible build, tested public contract, controlled configuration, observable runtime, documented security boundary, versioned artifacts, and a tested rollback path.

What must remain explicit?
The intended use, excluded use, data/credential handling, model or algorithm limitations, and which metrics are measured versus aspirational.

What should be completed next?
Use the linked production-readiness issue for this repository as the checklist. Resolve missing tests, deployment instructions, observability, supply-chain controls, and release evidence before attaching a production claim.

🏛️ Advanced Platform Architecture & Telemetry Decoupling

To guarantee enterprise-grade performance, SentinelAI enforces strict architectural separation between primary inference loops and the intelligent evaluation layers. [ Incoming User Query ] ───► [ Async Proxy Gateway ] ───► [ Downstream Application ] │ (Non-Blocking Telemetry Mirror) ▼ ┌──────────────────────────────────────┐ │ SentinelAI Asynchronous Engine │ ├──────────────────────────────────────┤ │ • Parallelized Guardrail Evaluation │ │ • GPT-4 Intelligent SRE Diagnostics │ │ • Token Cost & Allocation Trackers │ └──────────────────┬───────────────────┘ ▼ [ Streamlit Observability Control Plane ]

🚀 Quickstart — Docker Compose (recommended)

Prerequisites: Docker 24+ with Compose v2 (docker compose version).

# 1. Copy environment defaults
cp .env.example .env

# 2. Start the full local stack
docker compose up --build

Once running, open:

Service	URL
Streamlit Dashboard	http://localhost:8501
Prometheus	http://localhost:9090
Grafana (admin / admin)	http://localhost:3000
Ingestion API	http://localhost:8080
Drift Engine API	http://localhost:7070
LLM Guard API	http://localhost:8000

Send a sample inference log

curl -X POST http://localhost:8080/log \
  -H "Content-Type: application/json" \
  -d '{"model_id":"demo","model_version":"v1","latency_ms":120,"tokens_in":32,"tokens_out":64,"status":"ok"}'

Run a drift computation

curl -X POST http://localhost:7070/drift \
  -H "Content-Type: application/json" \
  -d '{"model_id":"demo","feature_name":"latency","expected":[0.2,0.3,0.25,0.25],"actual":[0.1,0.35,0.30,0.25]}'

Summarize an incident (Ollama optional — falls back to rule-based stub)

curl -X POST http://localhost:8000/summarize \
  -H "Content-Type: application/json" \
  -d '{"log_data":"PSI 0.35 on latency feature, model demo v1","persist":false}'

---

⚙️ Configuration

All configuration is via environment variables. Copy .env.example to .env and adjust.

Variable	Default	Description
WAREHOUSE_MODE	postgres	postgres (local) or snowflake
DATABASE_URL	Postgres DSN	Full Postgres connection string
POSTGRES_USER	sentinel	Postgres user
POSTGRES_PASSWORD	sentinel	Postgres password
POSTGRES_DB	sentinel	Postgres database
OLLAMA_HOST	http://ollama:11434	Ollama endpoint (optional)
LLM_MODEL	llama2	LLM model name
API_BEARER_TOKEN	(unset)	Required shared bearer token for POST /infer; the endpoint returns 503 until configured

Snowflake (optional): set WAREHOUSE_MODE=snowflake and fill in SNOWFLAKE_ACCOUNT, SNOWFLAKE_USER, SNOWFLAKE_PASSWORD, SNOWFLAKE_DATABASE, SNOWFLAKE_SCHEMA, SNOWFLAKE_WAREHOUSE.

---

🏗️ Architecture

User → Go Ingestion API (8080) → Postgres (local) / Snowflake (optional)
                                        ↓
                              Drift Engine C++ (7070)
                                        ↓
                               LLM Guard Python (8000)
                                        ↓
                           Streamlit Dashboard (8501)
                                        ↓
                         Prometheus (9090) + Grafana (3000)

Services

Service	Language	Port	Description
ingestion-service	Go	8080	Receives inference logs, writes to warehouse
drift-engine	C++ + Python	7070	PSI/KS drift detection
llm-guard	Python	8000	LLM-powered incident summarization
streamlit-dashboard	Python	8501	Control plane UI
postgres	—	5432	Local warehouse (default)
prometheus	—	9090	Metrics scraping
grafana	—	3000	Dashboards

---

📊 Research Metrics & Benchmarks

The committed baseline is generated by a seeded, dependency-free harness that mirrors the PSI/KS decision rule in the C++ drift engine. These numbers measure the Python reference implementation—not native C++ or end-to-end HTTP latency. See the full methodology, interpretation, and limitations and raw JSON evidence.

Latest reproducible baseline

Metric	Value	Protocol	Source
Timed evaluations	20,000	100 warm-ups, 32 bins, seed 20260718	python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json; benchmarks/run_benchmark.py → benchmarks/latest.json
Mean latency	41.706 µs	Per-decision reference latency	python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json; benchmarks/run_benchmark.py → benchmarks/latest.json
Median latency	39.700 µs	Per-decision reference latency	python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json; benchmarks/run_benchmark.py → benchmarks/latest.json
P95 / P99 latency	54.700 / 76.200 µs	Linear percentile interpolation	python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json; benchmarks/run_benchmark.py → benchmarks/latest.json
Minimum / maximum	23.700 / 319.200 µs	Observed range	python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json; benchmarks/run_benchmark.py → benchmarks/latest.json
Throughput	23,031.13 operations/s	Single-process CPython reference	python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json; benchmarks/run_benchmark.py → benchmarks/latest.json
Peak traced memory	0.623 MiB	Python tracemalloc	python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json; benchmarks/run_benchmark.py → benchmarks/latest.json
Precision / recall / F1	1.000 / 1.000 / 1.000	2,000 balanced synthetic cases	python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json; benchmarks/run_benchmark.py → benchmarks/latest.json
Confusion matrix	TP 1000 · TN 1000 · FP 0 · FN 0	Controlled seeded classes	python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json; benchmarks/run_benchmark.py → benchmarks/latest.json
Environment	CPython 3.12.13 · Windows 11	Recorded 2026-07-18	python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json; benchmarks/run_benchmark.py → benchmarks/latest.json

Benchmark scope and reproducibility

python benchmarks/run_benchmark.py --output benchmarks/latest.json

CI reruns the benchmark on every pull request, validates its schema and F1 regression floor, and uploads raw evidence for 30 days. For comparable hosts, median or P95 increases above 15% require investigation and a documented baseline update.

The perfect synthetic classification result is a regression signal for deliberately separated perturbation classes; it is not a production accuracy claim. Native C++, service concurrency, network, warehouse, GPU, and real-world labeled drift benchmarks remain future evaluation layers.

Test and evidence status

Evidence	Current state	Source
Benchmark raw data	Versioned JSON	benchmarks/latest.json
Benchmark methodology	Versioned report	benchmarks/benchmark_report.md
Benchmark CI	Required execution + artifact	.github/workflows/benchmarks.yml
Drift thresholds	PSI 0.20 · KS 0.10	g++ -std=c++17 drift-engine/drift_engine.cpp -o drift-engine/drift_engine; source: drift-engine/drift_engine.cpp

Observability Metrics

Metric Name	Type	Emitted By	Purpose
sentinel_requests_total	Counter	monitoring/metrics.py	API request volume
sentinel_request_latency_seconds	Histogram	monitoring/metrics.py	API request latency
inference_requests_total	Counter	monitoring/prometheus.py	Inference request volume
ingestion_logs_total{status}	Counter	ingestion-service/main.go	Ingestion outcome counts
ingestion_handler_seconds	Histogram	ingestion-service/main.go	Go ingestion handler latency
drift_detected_total	Counter	drift-engine/server.py	Drift event count
drift_compute_seconds	Histogram	drift-engine/server.py	Drift calculation latency
llm_guard_summaries_total{method}	Counter	llm-guard/app.py	Ollama vs fallback summary count
llm_guard_summary_seconds	Histogram	llm-guard/app.py	Summary generation latency
requests_total	Counter	backend/app/main.py	Backend request volume

Design Targets (Not Measured)

The following historical values are retained for planning and comparison, but no committed generator or CI artifact establishes them as current measurements at this commit. They must not be treated as benchmark results or release evidence.

Historical validation claims

Evidence	Current state	Evidence status
Focused API tests	4 passed	No committed command/output establishes this historical audit value
Focused API coverage	24%	No committed command/output establishes this historical audit value; the former static badge was removed

Historical project inventory

Area	Metric	Current Value	Source
Codebase	Tracked files	98	git ls-files
Codebase	Python files	32	*.py files
Codebase	Go files	1	ingestion-service/main.go
Codebase	C++ files	4	Drift and ingestion engine sources
Codebase	TypeScript files	7	frontend/
Codebase	Source NCLOC	1,201	Non-empty, non-comment Python/Go/C++/TS lines
Tests	Python test files	6	tests/
Tests	Test declarations	5	def test_* scan
Tests	Focused API validation	4 passed	pytest tests/test_*.py focused API scope
Tests	Focused api coverage	24%	Local coverage run
CI/CD	GitHub Actions workflows	7	.github/workflows/*.yml
Dependencies	Python runtime dependencies	11	requirements.txt
Delivery	Dockerfiles	5	Root/services/dashboard Docker assets
Delivery	Kubernetes manifests	9	k8s/*.yaml
Delivery	Helm chart files	1	helm/sentinel/templates/deployment.yaml
Infrastructure	Terraform files	1	terraform/main,TF
Monitoring	Monitoring config files	5	monitoring/
Services	Docker Compose service URLs	6	Dashboard, Prometheus, Grafana, ingestion, drift, LLM guard
Validation limits	Native Go/C++ compile checks	Not run locally	Go/g++/MSVC unavailable in workspace

---

🧠 Extended Q&A

Why use C++ for drift detection?

To achieve sub-millisecond statistical scoring at scale.

Why Go for ingestion?

Go provides efficient concurrency and low-latency HTTP handling.

Why Postgres locally (not Snowflake)?

Postgres is free, runs in Docker, and supports the same SQL schema. Switch to WAREHOUSE_MODE=snowflake when you're ready to push to production.

Why MLflow?

Experiment tracking, reproducibility, and version control.

Why LangChain + Ollama?

LLM-powered root cause summarization and RAG over historical incidents.

Why Kubernetes?

Horizontal scaling and production-grade orchestration.

Why Terraform?

Reproducible infrastructure as code.

---

🏢 Enterprise Value

SentinelAI demonstrates:

• AI system lifecycle management
• Drift monitoring
• MLOps integration
• Distributed systems engineering
• Cloud-native architecture
• LLM augmentation
• Observability & metrics-driven design

---

🔧 Recent Code Improvements

See CHANGELOG.md for the dated fix history.

Running tests locally

pip install -r requirements.txt
pytest tests/ -v

Environment variables added

Variable	Default	Description
API_USERNAME	admin	Login username for the API auth endpoint
API_PASSWORD	(unset — auth disabled until set)	Login password; must be set to enable auth
LLM_MODEL_NAME	meta-llama/Meta-Llama-3-8B	HuggingFace model used by the inference route

---

• Add automated retraining pipeline
• Add Shadow Model Deployment
• Add Cost Optimization Engine
• Add Hallucination Classifier Model