Skip to content

docs: make published metrics traceable - #17

Merged
CoreyLeath-code merged 3 commits into
mainfrom
codex/credible-metrics-evidence
Aug 6, 2026
Merged

docs: make published metrics traceable#17
CoreyLeath-code merged 3 commits into
mainfrom
codex/credible-metrics-evidence

Conversation

@CoreyLeath-code

Copy link
Copy Markdown
Owner

Summary

Makes published SentinelAI metrics traceable. The reproducible benchmark baseline now names its committed generator, exact command, and output file. Values with no committed generator or CI artifact remain visible but are explicitly classified as Design Targets (Not Measured).

No benchmark value was invented, regenerated, or changed.

Files changed

File Why
README.md Adds a source column for committed benchmark evidence, removes the unsupported static coverage badge, preserves unsupported historical audit/inventory values under Design Targets (Not Measured), and gives the PSI/KS threshold a compile/source command.
Metrics.md Reclassifies the ungenerated C++/Go/LLM/container performance figures as design targets instead of measured benchmarks.

Metric evidence audit

Metric Measured? Evidence Command File
Timed evaluations Yes Seeded raw JSON python benchmarks/run_benchmark.py --iterations 20000 --evaluation-samples 2000 --output benchmarks/latest.json benchmarks/run_benchmark.py, benchmarks/latest.json
Mean latency Yes Seeded raw JSON Same benchmark command Same files
Median latency Yes Seeded raw JSON Same benchmark command Same files
P95 / P99 latency Yes Seeded raw JSON Same benchmark command Same files
Minimum / maximum latency Yes Seeded raw JSON Same benchmark command Same files
Throughput Yes Seeded raw JSON Same benchmark command Same files
Peak traced memory Yes Seeded raw JSON Same benchmark command Same files
Precision / recall / F1 Yes Seeded raw JSON; synthetic labels Same benchmark command Same files
Confusion matrix Yes Seeded raw JSON; synthetic labels Same benchmark command Same files
Benchmark environment Yes Recorded in raw JSON Same benchmark command Same files
PSI / KS threshold Yes—configuration, not a benchmark result C++ decision rule g++ -std=c++17 drift-engine/drift_engine.cpp -o drift-engine/drift_engine drift-engine/drift_engine.cpp
Focused API tests (4 passed) No Historical audit only None committed README.md
Focused API coverage (24%) No Historical audit only None committed README.md
Tracked files No Historical inventory only None committed README.md
Python / Go / C++ / TypeScript file counts No Historical inventory only None committed README.md
Source NCLOC No Historical inventory only None committed README.md
Test file / declaration counts No Historical inventory only None committed README.md
Workflow / dependency / Dockerfile counts No Historical inventory only None committed README.md
Kubernetes / Helm / Terraform / monitoring counts No Historical inventory only None committed README.md
Docker Compose service URL count No Historical inventory only None committed README.md
C++ PSI time (< 2ms) No No committed generator or artifact None Metrics.md
Go P95 latency (180ms) No No committed generator or artifact None Metrics.md
LLM summary time (1.2s) No No committed generator or artifact None Metrics.md
Throughput (150 RPS) No No committed generator or artifact None Metrics.md
Container cold start (< 3s) No No committed generator or artifact None Metrics.md

A row named Compute Time does not exist in the current README or Metrics.md; no value was added.

Validation performed

  • Verified benchmarks/run_benchmark.py produces the schema and fields stored in benchmarks/latest.json.
  • Verified .github/workflows/benchmarks.yml executes the benchmark with 20,000 iterations and 2,000 synthetic evaluation samples, validates its schema/F1 floor, and retains the artifact for 30 days.
  • Verified the PSI/KS limits are coded in drift-engine/drift_engine.cpp.
  • CI is triggered on this PR. This change does not alter runtime or benchmark logic.

Risks

  • Historical values move from apparent measurements to clearly labelled, non-release evidence. Consumers relying on those figures as current performance data must use the versioned benchmark instead.

Follow-up recommendations

  • Add committed, CI-executed native C++, Go, LLM, container-startup, and end-to-end benchmark harnesses before promoting those design targets to measured claims.
  • Add a committed inventory-generation script if codebase counts are intended to be release evidence.

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@CoreyLeath-code, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 59 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e84f7437-cb2a-4e89-bf2a-349379894ab3

📥 Commits

Reviewing files that changed from the base of the PR and between 3e1a234 and 5bd62a7.

📒 Files selected for processing (2)
  • Metrics.md
  • README.md

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@CoreyLeath-code
CoreyLeath-code marked this pull request as ready for review August 6, 2026 14:32
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@CoreyLeath-code
CoreyLeath-code merged commit ff6aa8c into main Aug 6, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant