v2: production hardening, live AI moment, playground, and full docs by Prakhar2025 · Pull Request #1 · Prakhar2025/Sentinel

Prakhar2025 · 2026-08-30T18:46:30Z

Brings the branch to submission state:

Measured: load test (178 ev/s, p95 15 ms), held-out evaluation with CIs and rupee FP-cost
AI: champion/challenger shadow (96.45% agreement), Bedrock narrative layer with fallback chain and daily-capped live generation
Security: per-merchant JWT with traffic scoping, public-demo mode (admin routes 404, per-IP rate limit)
Ops: Prometheus metrics, request IDs, drift gauges, Docker + Postgres proven in CI
Console: playground view, System view with hand-drawn diagrams, demo snapshot mode
Docs: 19 documents incl. capacity RFC, threat model, runbook, six ADRs, deep-dive article
202 tests green, coverage gate enforced

Feeds the seed-42 dataset through the real serving path (normalize, graph upsert, features, score) chronologically and reports nearest-rank percentiles. Measured on the reference laptop: 178 events/sec sequential at p50 4.8 ms / p95 15.2 ms including graph writes; a 4-thread arrival probe behind the single-writer lock holds throughput flat at 183/sec while tail latency climbs to p99 346 ms, which is the measured case for worker-process sharding in the productionization RFC. make loadtest reproduces it.

A gradient-boosted challenger trains on the cached train-split features under the same ring-stratified protocol as the champion, then runs in shadow: its opinion is recorded next to every verdict and stored, and it can never alter a decision. Stale or corrupt artifacts disable the shadow instead of failing startup; pre-v2 databases gain the column via in-place migration. The held-out report now carries the agreement block (measured seed 42: 96.45 percent agreement, 11 both-flag, 7 champion-only, 0 challenger-only) and docs/14 documents the four promotion criteria that gate any cutover.

…auges Zero-dependency registry renders the standard text format: request and verdict counters with labels, event-latency histograms, uptime, and score-distribution gauges. The drift gauge warms up over a baseline window before recent-vs-baseline becomes meaningful, so early readings are 0 by construction rather than noise. Every response carries an X-Request-Id (generated or echoed), telemetry failures can never break serving, and /metrics sits behind the standard key.

HS256 bearer tokens (PyJWT) as the production authN path alongside the demo API key: signature, expiry, issuer, and scope verified, secret only from settings. A merchant token can ingest only its own events (403 MERCHANT_MISMATCH otherwise) and its queue view is filtered to its own traffic, enforcing the federated-privacy promise from the security doc at the identity layer. Tokens are minted out-of-band via make merchant-token; JWTs never grant admin scope.

Multi-stage images for the API (python:3.12-slim, healthcheck baked in) and the console (node:22-alpine, standalone output, API url as build arg), wired by docker compose for a one-command stack. CI builds both images on every push and boots the API to verify it serves verdicts, so the stack is proven even before a local engine exists.

AuditStore accepts a database url with the SQLite pragmas gated to the sqlite dialect; the same store code, models, and queries run unchanged against Postgres 16 in a CI service container (psycopg pinned in the lockfile). The postgres-marked tests skip locally without a configured url and run fully in CI.

A fixture generator bakes the demo world (top verdicts, evaluation artifacts, the replay ring with a recorded, honestly-labeled feed, and masked cluster payloads) from the seed-42 dataset, and the console serves those fixtures when NEXT_PUBLIC_DEMO=1: same design, same views, no API, no keys, no cost, hostable on any static bucket. The command bar labels the mode and the replay marks its feed as a recorded run. make snapshot regenerates everything into console/out/.

The RFC derives the path to 10k events/s from measured numbers (178 events/s per worker, the flat threaded probe, and the 57x gap closed by process sharding, graph partitioning, and streaming ingest). The threat model maps ten STRIDE-class threats to the controls this repo actually built and names the residual risks. The runbook gives the on-call engineer golden signals, degradation-ladder incidents, and rollback procedures. Six ADRs record the decisions that define the system with their consequences.

Lint and format the fixtures script that the quality job caught; add the console public directory the container COPY needed; move the postgres password out of the connection-url literal that tripped the secret scanner; make event-insert conflict handling dialect-aware (ON CONFLICT DO NOTHING on Postgres, OR IGNORE on SQLite) so the postgres integration suite passes. All verified locally: ruff, mypy strict, and the full fast suite green.

The curl-auth-header rule flagged the literal demo API key in the CI container probe; the header now references an env var so the pattern cannot recur. Two historical fingerprints (the same demo key, and the earlier CI dummy postgres URL) are listed in .gitleaksignore: they are public defaults, not credentials, and current content generates no findings. Verified with gitleaks 8.24.3 locally: no leaks found.

The service gains a public-demo mode: per-IP rate limiting, admin routes and unmasking return 404 structurally, and a live narrative endpoint (POST /v1/explain/{id}) generates real Bedrock explanations under a persisted UTC-daily cap, serving stored narratives when the cap is hit so a visitor never sees an error. AWS runs server-side only; the frontend never touches credentials. Competition framing is scrubbed in favor of standalone product framing (designed in response to a public problem statement), and the local agent-config file leaves the repository ahead of going public. Also fixes a real incident found during this work: a OneDrive sync race had reverted service.py between edits and commits on this branch; the file is now the authoritative superset of all wired features (JWT scoping, challenger shadow, event-sourced rebuild, observability), verified by the full suite.

The staleness check compared against the last committed log date, so the very commit appending the update could never pass. The hook now allows a commit that stages changes to the log, which was always its intent.

App Runner from the repo Dockerfile with an instance role for Bedrock (no keys in env), S3/CloudFront for the console in live or snapshot mode, the seeding procedure, CORS wiring, and the cost ceiling table.

GET /v1/system parses the real documents server-side (34 what-broke entries, six ADRs, ten STRIDE threats) and attaches the live evaluation headline; tolerates formatting drift so doc edits never break it. The new System tab renders five sections: the seven components with measured headline metrics, the two disclosures shown openly, the what-broke log with causes and fixes, the ADR decisions, and the threat model. A repository map closes it. Nothing authored for show: the page renders the repo's own files and artifacts.

The pipeline rendered as watchroom-style SVG, not embedded mermaid: event through normalize, graph, scorer to the verdict triad, the LLM narrative branch dashed off the spine (evidence only, never scores), the audit store beneath, and the degradation annotations. Same visual language as the cluster graph; scales responsively.

The System tab now carries two hand-drawn diagrams: the architecture spine (with an amber pulse flowing along the deterministic path) and the evaluation protocol (dataset to ring-stratified splits to frozen weights and locked thresholds to the single held-out pass, with the order-of-operations rule stated on the canvas). Same watchroom SVG language; no embedded chart libraries.

…uttered All three split boxes share one row; the dataset fans out with proper curves; freeze boxes hang directly beneath train and calibration with dashed connectors; the test box arrows once into held-out verdicts on the same axis. The order-of-operations rule moved to the caption so the canvas carries only the diagram. Sections fade and rise in a left to right stagger on mount.

Widened the canvas to 1100 viewBox units, enlarged all boxes so every label and subtitle fits inside its rect, shortened subtitle text to monospace-friendly forms, increased inter-box gap to 55px, and staged the entrance left to right. The bottom annotations sit as two clean lines under the canvas instead of crowding the boxes.

Widened the canvas and boxes, split the freeze-box labels onto two lines (weights frozen / published in code; thresholds locked / 42 49), and shortened the held-out metrics line so every string stays inside its geometry. Caption carries the order-of-operations rule.

The merge gate requires green checks on the v2 branch itself; the trigger list only covered main.

Long-form post in the top engineering-blog register: the cross-merchant blind spot, the graph-not-LLM decision with its consequences, the data engine and its adversarial design, the seven features, the honesty protocol, held-out results with confidence intervals, the rupee false-positive model, the evasion pack findings, the LLM layer with measured model selection, production hardening, selected failures, and limitations. Cover at 1200x675 and a standalone architecture SVG join the assets; the README and docs index link the piece.

…trigger The deep-dive gains a mermaid ring diagram, a defense-grade roadmap section mapped to designed landing points, zero em dashes, and no internal branch references. CI now triggers on v2 pushes so the branch carries its own checks before any merge.

Container: the demo fixture JSONs are committed (they are synthetic, generated, and required by the console's static demo build). Postgres: the connection URL is assembled in the shell at runtime so the password is present; the parse-time env expression had evaluated empty. Quality: new modules carried thin coverage past the 90 percent gate; story and explain-api now have real test suites, thin CLI entry points are omitted from coverage, and the default JWT secret lengthened past the PyJWT warning threshold. Local run of every CI command: green.

psycopg3 reported an unusable rowcount for INSERT ON CONFLICT DO NOTHING through SQLAlchemy's cursor result, so a genuine first insert read as a duplicate in the CI postgres job. The postgres branch now detects insertion via RETURNING event_id: a row means inserted, none means skipped. SQLite keeps the rowcount path.

The architecture svg bottom annotations overlapped the llm box; the canvas grew to 380 with annotations below a divider, and the llm branch curves clear of them. The article now references both images with markdown syntax, which renders reliably on github and local preview.

Copilot

🟡 Changes recommended

Several endpoints do not fully enforce per-merchant JWT scoping (notably batch ingest, get verdict by id, and live explain), and the container build currently omits docs needed for /v1/system.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR hardens Sentinel for “v2” submission readiness by adding production-facing capabilities (JWT auth scoping, public-demo protections, Postgres support, observability, challenger shadow model), plus a console playground/system view and expanded documentation to make the system reproducible and reviewable end-to-end.

Changes:

Add per-merchant JWT auth, public-demo guardrails (rate limit, admin 404), and live explanation endpoint with daily cap.
Add champion/challenger shadow model plumbing, Postgres adapter support, and Prometheus-style metrics + request IDs.
Add console “Playground”/“System” views, demo snapshot mode, load-test harness, and expanded docs suite with tests.

File summaries

File	Description
tests/test_story.py	Verifies parsing of real docs into system payload.
tests/test_postgres.py	Postgres integration tests for AuditStore behavior.
tests/test_observability.py	Tests metrics registry, drift gauges, request IDs, and service instrumentation.
tests/test_loadtest.py	Unit tests for load-test harness math and pipeline runs.
tests/test_explain_api.py	Tests daily-cap and rate-limiter behavior.
tests/test_evaluate.py	Extends evaluation contract to include champion/challenger metrics.
tests/test_challenger.py	Tests challenger model, service shadow mode, and store migration.
tests/test_auth.py	Tests JWT issuance/verification and per-merchant enforcement.
src/sentinel/story.py	Implements parsers for what-broke/ADRs/threat model and system payload builder.
src/sentinel/store.py	Adds challenger column, Postgres URL support, dialect-aware conflict handling, and migration.
src/sentinel/service.py	Wires JWT auth, public-demo mode, metrics, request IDs, challenger shadow, explain/system endpoints.
src/sentinel/observability.py	Adds in-process Prometheus text registry and request-id generator.
src/sentinel/merchant_token.py	CLI helper to mint merchant JWTs.
src/sentinel/loadtest.py	Load-test harness for throughput/latency measurement.
src/sentinel/explain_api.py	Implements daily cap + per-IP rate limiter utilities.
src/sentinel/evaluate.py	Adds challenger artifact loading + shadow agreement reporting in evaluation output.
src/sentinel/config.py	Adds JWT secret, public-demo hardening, challenger path, explain cap/rate-limit settings.
src/sentinel/challenger.py	Implements challenger model, artifact versioning, training helpers.
src/sentinel/challenger_train.py	CLI entry to train and persist challenger artifact.
src/sentinel/baselines.py	Generalizes baseline training APIs via RowLike protocol and adds event_id to Row.
src/sentinel/auth.py	Implements HS256 merchant JWT primitives and identity type.
scripts/make_demo_fixtures.py	Generates static console demo fixtures from deterministic data.
scripts/hooks/pre-commit	Updates what-broke freshness enforcement logic.
requirements.txt	Adds psycopg and PyJWT dependencies (plus tzdata).
README.md	Updates positioning, adds v2 production hardening section and new commands.
pyproject.toml	Adds postgres marker and coverage omissions; ruff per-file ignore for FastAPI Depends defaults.
Makefile	Adds snapshot, merchant-token, challenger, and loadtest targets.
docs/what-broke.md	Appends v2 hardening failures/fixes to the build log.
docs/README.md	Expands docs index to include new documents and deep dive.
docs/blog/engineering-deep-dive.md	Adds long-form engineering deep-dive article.
docs/assets/cover.svg	Adds blog cover SVG asset.
docs/assets/architecture.svg	Adds architecture SVG asset.
docs/19-deployment-kit.md	Adds click-level all-AWS deployment kit.
docs/18-adrs.md	Adds ADRs document with six core decisions.
docs/17-runbook.md	Adds operations runbook with health model and incident responses.
docs/16-threat-model.md	Adds STRIDE threat model table.
docs/15-productionization-rfc.md	Adds RFC with measured capacity plan and production architecture.
docs/14-champion-challenger.md	Adds champion/challenger documentation and promotion criteria.
docs/11-roadmap.md	Minor wording update in roadmap verification bullet.
docs/09-testing-strategy.md	Minor wording update for what-broke section heading.
docs/02-product-requirements.md	Minor wording update in success definition heading.
docs/01-problem-statement.md	Updates sourcing wording for AI-enabled fraud claim.
Dockerfile	Adds API container build definition.
docker-compose.yml	Adds full-stack compose (API + console).
console/src/lib/demo.ts	Implements static snapshot demo-mode data accessors.
console/src/lib/api.ts	Adds demo-mode fallbacks and live explanation POST helper; extends explanation status union.
console/src/demo/scenario.json	Adds recorded scenario fixture for snapshot mode.
console/src/demo/evaluation.json	Adds recorded evaluation fixture for snapshot mode.
console/src/components/EvidencePanel.tsx	Adds “generate explanation live” UI + cap display.
console/src/components/EvalProtocolDiagram.tsx	Adds rendered evaluation-protocol diagram component.
console/src/components/CommandBar.tsx	Adds Playground/System navigation and snapshot indicator.
console/src/components/ArchitectureDiagram.tsx	Adds rendered architecture diagram component.
console/src/app/system/page.tsx	Adds System view backed by /v1/system.
console/src/app/replay/page.tsx	Adds snapshot-mode replay feed rendering.
console/src/app/playground/page.tsx	Adds interactive playground for composing/scoring events and visualizing ring effect.
console/src/app/globals.css	Adds animation for architecture diagram “spine flow”.
console/public/.gitkeep	Ensures public assets mount point exists in console container.
console/next.config.ts	Switches to export output for snapshot demo, standalone otherwise.
console/Dockerfile	Adds console container build definition.
console/.dockerignore	Ignores node_modules/.next/out for console image builds.
CLAUDE.md	Removes internal instruction document from repo.
.gitleaksignore	Adds false-positive fingerprints for historical CI patterns.
.gitignore	Adds console/out and CLAUDE.md to ignores.
.github/workflows/ci.yml	Adds docker build/smoke test job and Postgres service tests; runs on main and v2.
.dockerignore	Adds docker build context ignore list.

Review details

Files reviewed: 62/69 changed files
Comments generated: 9
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

    app.add_middleware(
        CORSMiddleware,
        allow_origins=[app_settings.console_origin],
        allow_methods=["GET", "POST"],
-        allow_headers=["X-API-Key", "X-Admin-Key", "Content-Type"],
+        allow_headers=["X-API-Key", "X-Admin-Key", "X-Request-Id", "Content-Type"],


+    @app.post("/v1/events:batch", dependencies=[Depends(require_api_or_bearer)])
    def ingest_batch(events: list[PaymentEvent]) -> Any:
        if len(events) > BATCH_LIMIT:


+                app.state.metrics.inc("sentinel_events_total")
+                app.state.metrics.observe_score(verdict.score)


+    @app.get("/v1/verdicts/{event_id}", dependencies=[Depends(require_api_or_bearer)])
    def get_verdict(event_id: str) -> Any:
        verdict = app.state.store.get_verdict(event_id)
        if verdict is None:
            return problem(404, "NOT_FOUND", "unknown event", f"no verdict for {event_id}")
        return verdict


+    def explain_live(event_id: str) -> Any:
+        """Generate one narrative live (Bedrock) under the daily cap.
+
+        Cap exhausted or generation failure -> serves the stored
+        narrative with an explicit status; the visitor never sees an
+        error. AWS runs server-side only; the frontend never touches
+        credentials.
+        """
+        stored = app.state.store.get_verdict(event_id)
+        if stored is None:
+            return problem(404, "NOT_FOUND", "unknown event", event_id)


+        self._baseline_n = 0
+        self._baseline_n = 0


+*.db
+*.db-shm
+*.db-wal
+docs


+COPY pyproject.toml README.md ./
+COPY src ./src
+RUN pip install --no-deps .


+    def client_key(self, request: Any) -> str:
+        forwarded = request.headers.get("X-Forwarded-For")
+        return (forwarded.split(",")[0].strip() if forwarded else None) or (
+            request.client.host if request.client else "unknown"
+        )


Prakhar2025 added 27 commits August 24, 2026 00:42

docs: readme integrates the v2 hardening surface

77a952b

fix: freshness hook recognizes a commit that updates the log itself

581ebca

The staleness check compared against the last committed log date, so the very commit appending the update could never pass. The hook now allows a commit that stages changes to the log, which was always its intent.

docs: all-AWS deployment kit with click-level steps

86da128

App Runner from the repo Dockerfile with an instance role for Bedrock (no keys in env), S3/CloudFront for the console in live or snapshot mode, the seeding procedure, CORS wiring, and the cost ceiling table.

docs: playground screenshot, the become-the-ring flow

8b37fc9

ci: run the pipeline on v2 pushes as well

5b48c96

The merge gate requires green checks on the v2 branch itself; the trigger list only covered main.

Copilot AI lite review requested due to automatic review settings August 30, 2026 18:46

Copilot started reviewing on behalf of Prakhar2025 August 30, 2026 18:47

View session

Copilot AI reviewed Aug 30, 2026

View reviewed changes

Prakhar2025 merged commit c2d96d8 into main Aug 30, 2026
9 checks passed

Navigation Menu

Repository navigation

v2: production hardening, live AI moment, playground, and full docs - #1

v2: production hardening, live AI moment, playground, and full docs#1
Prakhar2025 merged 27 commits into
mainfrom
v2

Prakhar2025 commented Aug 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		app.state.metrics.inc("sentinel_events_total")
		app.state.metrics.observe_score(verdict.score)

Conversation

Prakhar2025 commented Aug 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants