Skip to content

docs: make published metrics traceable #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
CoreyLeath-code merged 3 commits into from
Aug 6, 2026
Merged

docs: make published metrics traceable #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8d57dfa
docs: distinguish measured benchmark evidence
CoreyLeath-code Aug 6, 2026
63a6042
docs: label unmeasured performance targets
CoreyLeath-code Aug 6, 2026
5bd62a7
docs: cite drift threshold source command
CoreyLeath-code Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion Metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,9 @@
| Cost per 1k requests | Cloud compute estimate |


## Performance Benchmarks
## Design Targets (Not Measured)

The following planning values are preserved, but no committed benchmark, test, or workflow artifact generates them. They are not measured performance claims.

| Component | Metric | Result |
|------------|--------|--------|
Expand Down
44 changes: 27 additions & 17 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ SentinelAI
[![SAST](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/sast.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/sast.yml)
[![Schema Validation](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/data-validation.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/data-validation.yml)
[![Release](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/release.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/release.yml)
[![Coverage](https://img.shields.io/badge/focused%20API%20coverage-24%25-red)](#test-and-evidence-status)
[![Benchmark](https://img.shields.io/badge/reference%20p95-54.7%20%C2%B5s-6f42c1)](benchmarks/benchmark_report.md)
[![Throughput](https://img.shields.io/badge/reference%20throughput-23.0k%20ops%2Fs-2ea44f)](benchmarks/benchmark_report.md)
[![Python](https://img.shields.io/badge/Python-3.11-blue.svg)](https://www.python.org/)
Expand Down Expand Up @@ -191,18 +190,18 @@ The committed baseline is generated by a seeded, dependency-free harness that mi

### Latest reproducible baseline

| Metric | Value | Protocol |
|---|---:|---|
| Timed evaluations | 20,000 | 100 warm-ups, 32 bins, seed `20260718` |
| Mean latency | 41.706 µs | Per-decision reference latency |
| Median latency | 39.700 µs | Per-decision reference latency |
| P95 / P99 latency | 54.700 / 76.200 µs | Linear percentile interpolation |
| Minimum / maximum | 23.700 / 319.200 µs | Observed range |
| Throughput | 23,031.13 operations/s | Single-process CPython reference |
| Peak traced memory | 0.623 MiB | Python `tracemalloc` |
| Precision / recall / F1 | 1.000 / 1.000 / 1.000 | 2,000 balanced synthetic cases |
| Confusion matrix | TP 1000 · TN 1000 · FP 0 · FN 0 | Controlled seeded classes |
| Environment | CPython 3.12.13 · Windows 11 | Recorded 2026-07-18 |
| Metric | Value | Protocol | Source |
|---|---:|---|---|
| Timed evaluations | 20,000 | 100 warm-ups, 32 bins, seed `20260718` | `python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json`; `benchmarks/run_benchmark.py` → `benchmarks/latest.json` |
| Mean latency | 41.706 µs | Per-decision reference latency | `python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json`; `benchmarks/run_benchmark.py` → `benchmarks/latest.json` |
| Median latency | 39.700 µs | Per-decision reference latency | `python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json`; `benchmarks/run_benchmark.py` → `benchmarks/latest.json` |
| P95 / P99 latency | 54.700 / 76.200 µs | Linear percentile interpolation | `python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json`; `benchmarks/run_benchmark.py` → `benchmarks/latest.json` |
| Minimum / maximum | 23.700 / 319.200 µs | Observed range | `python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json`; `benchmarks/run_benchmark.py` → `benchmarks/latest.json` |
| Throughput | 23,031.13 operations/s | Single-process CPython reference | `python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json`; `benchmarks/run_benchmark.py` → `benchmarks/latest.json` |
| Peak traced memory | 0.623 MiB | Python `tracemalloc` | `python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json`; `benchmarks/run_benchmark.py` → `benchmarks/latest.json` |
| Precision / recall / F1 | 1.000 / 1.000 / 1.000 | 2,000 balanced synthetic cases | `python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json`; `benchmarks/run_benchmark.py` → `benchmarks/latest.json` |
| Confusion matrix | TP 1000 · TN 1000 · FP 0 · FN 0 | Controlled seeded classes | `python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json`; `benchmarks/run_benchmark.py` → `benchmarks/latest.json` |
| Environment | CPython 3.12.13 · Windows 11 | Recorded 2026-07-18 | `python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json`; `benchmarks/run_benchmark.py` → `benchmarks/latest.json` |

### Benchmark scope and reproducibility

Expand All @@ -218,12 +217,10 @@ The perfect synthetic classification result is a regression signal for deliberat

| Evidence | Current state | Source |
|---|---:|---|
| Focused API tests | 4 passed | Existing repository audit |
| Focused API coverage | 24% | Existing repository audit; below 90% target |
| Benchmark raw data | Versioned JSON | `benchmarks/latest.json` |
| Benchmark methodology | Versioned report | `benchmarks/benchmark_report.md` |
| Benchmark CI | Required execution + artifact | `.github/workflows/benchmarks.yml` |
| Drift thresholds | PSI 0.20 · KS 0.10 | `drift-engine/drift_engine.cpp` |
| Drift thresholds | PSI 0.20 · KS 0.10 | `g++ -std=c++17 drift-engine/drift_engine.cpp -o drift-engine/drift_engine`; source: `drift-engine/drift_engine.cpp` |

### Observability Metrics

Expand All @@ -240,7 +237,20 @@ The perfect synthetic classification result is a regression signal for deliberat
| `llm_guard_summary_seconds` | Histogram | `llm-guard/app.py` | Summary generation latency |
| `requests_total` | Counter | `backend/app/main.py` | Backend request volume |

### Project & Reproducibility Metrics


## Design Targets (Not Measured)

The following historical values are retained for planning and comparison, but no committed generator or CI artifact establishes them as current measurements at this commit. They must not be treated as benchmark results or release evidence.

### Historical validation claims

| Evidence | Current state | Evidence status |
|---|---:|---|
| Focused API tests | 4 passed | No committed command/output establishes this historical audit value |
| Focused API coverage | 24% | No committed command/output establishes this historical audit value; the former static badge was removed |

### Historical project inventory

| Area | Metric | Current Value | Source |
|---|---:|---:|---|
Expand Down
Loading