diff --git a/README.md b/README.md index 171234b..f61c18b 100644 --- a/README.md +++ b/README.md @@ -1,191 +1,36 @@ # AI Test Intelligence Platform -A reference-grade engineering platform that applies LLM-backed analysis to a codebase -and its test suite, with the governance and CI/CD scaffolding a real production system -around AI-generated output would need — not just the analysis itself. - -## Problem Statement - -Engineering teams routinely under-invest in three things because they're tedious to do -by hand and hard to automate with static analysis alone: - -1. **Where is testing risk concentrated in a change?** Line/branch coverage tells you - what's *tested*, not what's *risky* — a one-line change to an auth check and a - hundred-line change to a logging format string look identical to a coverage tool but - are not remotely equivalent in review priority. -2. **What tests should exist but don't?** Spotting an undertested code path requires - understanding what the code *does*, not just how many lines are covered. -3. **Why did this test fail, and does it matter?** Regression, flake, or environment - issue — CI failure triage is manual pattern-matching that scales badly with test - suite size and team turnover. - -An LLM can meaningfully help with all three, but "an LLM says so" is not the same thing -as "this is safe to merge." The harder, less-demoed problem is wiring that output into -an engineering workflow *responsibly*: never as a full generated-test-source dump into a -PR thread, never as a silent auto-approval, always with an audit trail and a human in -the loop for anything risky. That's what most of this platform's actual engineering -effort went into, and it's the part a from-scratch AI coding demo usually skips. - -## Architecture -![architecture](/docs/screenshots/architecture.png) - -## Core Capabilities - -| Capability | What it does | Status | -|---|---|---| -| Risk analysis | Scores a diff's release risk, categorizes it (auth/authz, API contract, schema, security-sensitive, etc.), and gives a release recommendation | Implemented | -| Test intelligence | Suggests specific tests for undertested/high-risk code, with rationale and confidence | Implemented | -| Failure intelligence | Classifies a CI failure as regression/flaky/environment from logs and stack traces | Implemented | -| GitHub PR integration | Webhook-triggered analysis, commit status + PR comment publishing | Implemented | -| Governance & human review | Policy-gated review queue, immutable audit trail, sensitive-data redaction | Implemented | -| CI webhook ingestion (non-PR test-run results) | Feeding raw CI test-run results in independently of a PR | Not implemented — see Production Gaps | -| Multi-provider LLM support | Anthropic Claude (real) + Mock (deterministic, free) implemented; the `LLMProvider` interface is provider-agnostic | OpenAI/other providers not implemented — interface supports it | - -## AI vs Deterministic Responsibilities - -The platform deliberately separates probabilistic analysis from deterministic -engineering controls. - -### LLM-backed - -- Semantic analysis of code changes -- Identification of potential release risk -- Test suggestions for undertested or high-risk code -- Failure classification and likely root-cause hypotheses -- Explanatory rationale for findings and recommendations - -### Deterministic - -- GitHub webhook signature verification -- Repository registration and event normalization -- Request and schema validation -- Rules that determine when Test Intelligence runs -- Governance policy thresholds and review requirements -- Sensitive-data redaction -- Audit-event persistence -- Human approve/reject decisions -- GitHub status and PR-comment publishing -- CI/CD execution - -The LLM produces engineering intelligence; it does not independently authorize a -release. High-risk or policy-sensitive findings are evaluated by deterministic -governance rules and can require explicit human review before a final GitHub status is -published. - -## Design Decisions & Trade-offs - -Written down here so a reviewer doesn't have to reverse-engineer *why* from the code — -each of these was a real choice with a real alternative, not an oversight. - -- **In-process `TaskQueue` (threads), not Celery/Temporal.** Simpler to run and reason - about for a single-process reference deployment; the interface - (`orchestration/queue.py`) is the seam a distributed queue would slot behind without - touching any caller. Trade-off: no horizontal scaling of analysis workers, and a - process restart loses in-flight (not yet-persisted) work. -- **SQLite for unit/component tests, PostgreSQL only for integration tests and - production.** Keeps the fast test suite (~10s for 420 tests) free of a database - dependency; `tests/persistence/postgres/` is what actually proves Postgres-specific - behavior (JSON columns, migrations, FK enforcement). Trade-off: a bug that only - manifests on Postgres (a stricter type, a locking behavior) won't be caught by the - fast suite — `integration.yml` is what catches that class of gap, not `ci.yml`. -- **Statuses API, not the Checks API, for the GitHub commit status.** Checks requires a - GitHub App installation (a different auth/identity model); Statuses works with a plain - token, matching the rest of the integration's auth story. Trade-off: no rich Checks UI - (annotations, requested re-runs) — see Future Evolution. -- **Governance policy is process-wide, not per-repository.** One set of thresholds - (`GOVERNANCE_*` env vars) for every registered repository. Simpler to configure and - reason about; the real trade-off is that a monorepo with wildly different risk - tolerances per subproject can't express that yet. -- **PR comments never include full model output** (full rationale text, full generated - test source) **by design**, not as a token-budget optimization — a concise, capped - summary plus a link to the full detail in the dashboard. This is a deliberate scope - boundary on what belongs in PR history versus what belongs in a reviewed UI, decided - before it became a token-cost problem. -- **No authentication/authorization model.** Every API endpoint is open; CORS is - wildcard. This is the single largest gap between "reference implementation" and - "production-ready" — see Production Gaps. It was deferred deliberately rather than - built partially, since a half-built auth model creates a false sense of security worse - than no auth model plus an honest README section about it. -- **Redaction is pattern-based over secret *shapes*, not a secrets-manager - integration.** Catches AWS/GitHub/Slack tokens, PEM blocks, and generic - password/secret assignments by regex before they reach an LLM call or the audit log. - Trade-off: it's a safety net, not a guarantee — a secret in an unrecognized format - will not be caught. See `backend/app/governance/redaction.py`'s module docstring for - why it targets values, not the security-related keywords the risk/test-intelligence - heuristics themselves depend on. -- **Repository removal is soft-delete (an `is_active` flag), not a `DELETE` endpoint.** - A registered repository accumulates `AnalysisRun`/`RiskFinding`/`ReviewRequest`/ - `AuditEvent` rows that reference it by foreign key; hard-deleting the row would either - cascade-destroy that history — directly contradicting the immutable-audit-trail - guarantee `AuditEvent` exists for — or leave orphaned rows behind. Archiving - (`POST /repositories/{id}/archive`) removes a repository from the default list view - and stops it accepting new analysis (manual trigger or webhook) while leaving every - row that already references it fully intact. This is a distinct concern from - *retention* at scale — see Future Evolution. - -## Example Workflow - -1. A developer opens a PR that touches `app/auth/login.py`. -2. GitHub sends a signed `pull_request` webhook to `POST /api/v1/webhooks/github`. -3. The platform verifies the signature, fetches the PR's diff, and triggers Risk - analysis (always) and Test Intelligence analysis (since the diff touches non-test - source). -4. The Risk Engine scores the change, detects the `authentication_authorization` - category, and produces a risk assessment. -5. Governance policy evaluates that assessment: the auth-change category trips a rule. - Instead of an automatic commit status, the platform publishes a `pending` status and - a short PR comment noting review is required, and creates a review request — visible - at `/review-queue` in the dashboard, with an immutable `policy_evaluated` + - `review_required` audit trail already recorded. -6. A reviewer opens the dashboard, sees the flagged request with its risk summary and - reasons, and calls `POST /api/v1/review-queue/{id}/approve` with their identity and - an optional reason. -7. The platform records the decision (another immutable audit event), then publishes - the final `success` commit status and a decision comment back to the PR. -8. The developer sees a green check and a clear paper trail of who approved it and why — - never an AI-generated status that resolved itself. - -## Screenshots - -The dashboard running locally against `MockProvider` — no real LLM calls, no external -accounts. See Try It Yourself Locally below to reproduce any of these. - -**Repository Overview** — registered repositories, with the archive/soft-delete toggle -(deactivates a repo without destroying its analysis history — see Design Decisions). +A reference implementation for analyzing pull-request risk, suggesting targeted tests, and investigating CI failures. LLMs generate findings and explanations; deterministic policy and human review control release-related actions. -![Repository Overview](docs/screenshots/01-repository-overview.png) - -**Risk Analysis** — a triggered assessment: deterministic heuristic score, category -detection, and the provider's assessment side by side. +![Architecture](docs/screenshots/architecture.png) -![Risk Analysis](docs/screenshots/02-risk-analysis.png) +## What it does -**Test Suggestions** — generated suggestions with confidence, supporting evidence, and -an accept/reject decision per suggestion. +| Capability | Behavior | +| --- | --- | +| Risk analysis | Reviews diffs, identifies sensitive changes, and recommends a risk level | +| Test intelligence | Suggests tests for changed or undertested code, with rationale and confidence | +| Failure intelligence | Classifies test failures and proposes debugging hypotheses | +| GitHub integration | Accepts signed PR webhooks and publishes commit statuses and concise comments | +| Governance | Applies policy thresholds, routes sensitive findings for review, and records audit events | -![Test Suggestions](docs/screenshots/03-test-suggestions.png) - -**Failure Intelligence** — a classified CI failure (regression, in this case) with -factual evidence, root-cause hypotheses, and concrete debugging recommendations. - -![Failure Intelligence](docs/screenshots/04-failure-intelligence.png) +The application supports Anthropic Claude and a deterministic mock provider. OpenAI is not implemented. CI test-run webhook ingestion is also not implemented; see [Limitations](#limitations). -**Pending Approvals** — a risk result governance flagged (low confidence, in this case) -before it can be treated as approved, with the reviewer identity + reason form that -records the decision. +## Request flow -![Pending Approvals](docs/screenshots/05-pending-approvals.png) +1. GitHub sends a signed PR event; the backend verifies it and retrieves the diff. +2. Background analysis produces risk findings and, for non-test source changes, test suggestions. +3. Deterministic governance policy decides whether human review is required. +4. The platform publishes a concise GitHub status/comment, or a pending status until a reviewer approves or rejects the finding. +5. The dashboard shows analysis details, review decisions, and audit history. -**Analysis Run History** — every run for a repository with per-run LLM invocation -detail (provider, model, tokens, latency, cost) expandable inline. +The model never independently approves a release. See [Architecture](docs/architecture.md) for component boundaries and [System design](docs/system-design.md) for detailed request flows. -![Analysis Run History](docs/screenshots/06-analysis-run-history.png) +## Technology -Not pictured: a GitHub PR showing the published commit status + comment — requires a -real webhook pointed at a publicly reachable URL (see GitHub PR Integration below), -more setup than the rest and skippable for evaluating the platform itself. +Python, FastAPI, Pydantic, React, TypeScript, PostgreSQL, SQLite for fast tests, Anthropic Claude, GitHub webhooks/Statuses API, Docker, GitHub Actions, pytest, and frontend tests. A mock provider supports local development without paid model calls. -## Local Setup +## Local setup Requires Python 3.12+, Node 22+, and a running PostgreSQL instance — the default `DATABASE_URL` (see `backend/app/persistence/config.py`) points at @@ -231,7 +76,7 @@ API — so the platform runs end-to-end locally with zero external accounts requ `GOVERNANCE_ENABLED=false` additionally disables the review-queue gate for local iteration, if you want every risk result to auto-approve while developing. -## Try It Yourself Locally +## Try it locally With the backend and frontend both running (`make backend`, `make frontend`), the fastest way to see every core capability work end to end, with zero external accounts: @@ -287,283 +132,36 @@ exercised by design so the platform is fully clickable without an API key. Confi `PROVIDER_ANTHROPIC_API_KEY` + `PROVIDER_DEFAULT_PROVIDER=anthropic` to see real generated output instead — optional, and it spends real API budget. -## AI Evaluation Strategy - -The automated test suites described below validate that the platform behaves correctly -as software. AI evaluation is a separate concern: it asks whether the intelligence -produced by the platform is accurate, useful, and stable enough to trust in an -engineering workflow. - -The repository already has a foundation for this — `observability/eval_datasets.py`'s -three representative-scenario datasets (risk analysis, test intelligence, failure -intelligence) and `experiments.py`'s replay mechanism, both LangSmith-backed and -optional (see [Architecture §8](docs/architecture.md#8-observability-strategy)). But -that foundation is explicitly not a grading harness: `experiments.py`'s own docstring -calls it out — no automated scoring, no baseline comparison, no CI gate. A versioned -golden-dataset pipeline with regression detection is the next layer to build on top of -that foundation, not a from-scratch build. +## Dashboard + +![Repository Overview](docs/screenshots/01-repository-overview.png) + +![Risk Analysis](docs/screenshots/02-risk-analysis.png) + +![Test Suggestions](docs/screenshots/03-test-suggestions.png) + +![Failure Intelligence](docs/screenshots/04-failure-intelligence.png) + +![Pending Approvals](docs/screenshots/05-pending-approvals.png) + +![Analysis Run History](docs/screenshots/06-analysis-run-history.png) + +## Testing and CI + +Run the backend and frontend test commands documented in the setup sections above. The fast backend suite uses SQLite; PostgreSQL-specific behavior is exercised separately in integration tests. CI also checks formatting, frontend tests, and build paths. Mock-provider testing does not require model API credentials. + +## Design trade-offs + +- **Provider boundary:** model-specific calls stay behind `LLMProvider`; the deterministic mock supports reproducible tests. +- **Background queue:** an in-process thread queue is simple locally, but does not provide durable jobs or horizontal worker scaling. +- **Persistence:** SQLite keeps component tests fast; PostgreSQL integration tests catch database-specific differences. +- **GitHub Statuses API:** avoids requiring a GitHub App, but lacks richer Checks API annotations. +- **Human review:** policy, not model output, determines when review is required; comments deliberately omit full generated test source. + +Additional implementation decisions are in [Architecture](docs/architecture.md). + +## Limitations -Evaluation should be capability-specific: +This is a reference implementation, not a deployed multi-tenant service. It does not provide application-wide authentication/authorization; the API and permissive CORS settings are unsuitable for public exposure. Policy thresholds are process-wide, redaction is pattern-based rather than comprehensive, and the in-process queue loses unfinished jobs on restart. CI-result webhook ingestion and additional model providers remain future work. -### Risk Analysis - -Evaluate: - -- risk-category classification accuracy -- recall on known high-risk changes -- false-positive rate on low-risk changes -- evidence quality -- release-recommendation accuracy -- confidence calibration - -### Test Intelligence - -Evaluate: - -- relevance of suggested tests -- coverage of important edge and negative cases -- duplicate or redundant suggestions -- executability of generated tests where source is produced -- usefulness of rationale - -### Failure Intelligence - -Evaluate: - -- regression/flaky/environment classification accuracy -- likely root-cause accuracy -- confidence calibration -- debugging-recommendation usefulness - -### Golden Dataset - -A production evaluation suite should be versioned and include representative examples -such as: - -- low-risk and high-risk code changes -- authentication/authorization changes -- API and schema changes -- security-sensitive changes -- known historical regressions -- flaky tests -- environment failures -- ambiguous or insufficient-evidence cases - -Deterministic checks should be used wherever the expected property is deterministic, -such as schema validity or required fields. Semantic quality can use task-specific -scoring, carefully calibrated model-based evaluation, and human review where necessary. - -Prompt, model, provider, and evaluation-dataset changes should be evaluated against a -stable baseline before promotion so that quality regressions are visible rather than -hidden behind a successful software test run. - -## Testing Strategy - -- **420 backend tests** (pytest), **96% line coverage**, run in ~10-15s with zero - external dependencies (`MockProvider`, in-memory SQLite). -- **36 frontend component tests** (Vitest + React Testing Library). -- **PostgreSQL integration suite** (`tests/persistence/postgres/`) — migrations, FK - enforcement, and repository behavior against a real Postgres 16 instance. -- **Playwright e2e** (`frontend/e2e/`) — the primary user flows against a real backend - (MockProvider, disposable SQLite) and a real Vite dev server. -- **Provider contract tests** — a shared suite run against every `LLMProvider` - implementation, including a manually-triggered live-Anthropic variant that never runs - in automatic CI. -- **Webhook/governance integration tests** — the full webhook → policy evaluation → - review queue → approve/reject → GitHub-publish loop, driven with a fake `GitHubClient` - and no live repository or network access. - -Full breakdown, including what each suite deliberately does *not* cover, in -[Architecture §9](docs/architecture.md#9-testing-strategy). - -## CI/CD - -Everything below runs against `MockProvider` (`PROVIDER_DEFAULT_PROVIDER=mock`, the -default) — no automatic workflow ever calls a real LLM or spends API budget. Four -workflows are required checks on `main`; a fifth is opt-in and manual. - -| Workflow | Trigger | What it checks | -|---|---|---| -| `ci.yml` | every push/PR | Backend: Ruff lint, Ruff format check, mypy, PyTest (incl. API contract tests under `tests/api/`) with coverage. Frontend: oxlint, Vitest component tests, production build (`tsc -b && vite build`). | -| `integration.yml` | every push/PR | Spins up a PostgreSQL 16 service container, runs Alembic migrations (upgrade to head, then a downgrade/upgrade round trip) and the repository-layer integration suite against it — see `tests/persistence/postgres/`. | -| `e2e.yml` | every push/PR | Installs Playwright + Chromium and runs `frontend/e2e/` (`playwright.config.ts`) against a real backend (MockProvider, disposable SQLite) and a real Vite dev server, both started by Playwright itself. | -| `docker.yml` | every push/PR | Builds the backend and frontend Docker images independently, then validates `docker-compose.yml` (`docker compose config` + `docker compose build`). Build-only — nothing is pushed to a registry. | -| `live-smoke.yml` | **manual** (`workflow_dispatch`) only | Runs `tests/providers/test_anthropic_live.py` against the real Anthropic API, using an `ANTHROPIC_API_KEY` repository secret. The only workflow permitted to make paid calls; never runs automatically. | - -Test results, coverage reports, and the Playwright HTML report are uploaded as workflow -artifacts (14-day retention) on every run, pass or fail, so a CI failure can be -diagnosed without re-running locally. - -## GitHub PR Integration - -The platform can trigger AI-backed analysis directly from GitHub pull requests and -publish the results back as a commit status check and a PR comment. - -**Setup:** - -1. Register the repository with the platform first (`POST /api/v1/repositories`) — the - webhook is ignored for any repository whose GitHub URL isn't already registered. -2. Set two environment variables (see `.env.example`): - - `GITHUB_WEBHOOK_SECRET` — required. Requests without a valid - `X-Hub-Signature-256` HMAC-SHA256 signature against this secret are rejected with - 401. The webhook endpoint fails closed if this isn't set at all. - - `GITHUB_API_TOKEN` — required to actually fetch PR diffs and publish status - checks/comments back to GitHub. Without it, webhooks are still received and - verified and analysis still runs, but nothing is published back to GitHub (a - `NullGitHubClient` no-ops the outbound calls, same "no key, no provider" pattern - `PROVIDER_ANTHROPIC_API_KEY` follows). -3. Point a GitHub webhook at `POST /api/v1/webhooks/github` for the `pull_request` event, - using the same secret as `GITHUB_WEBHOOK_SECRET`. - -**What happens on `opened` / `synchronize` / `reopened`:** - -- Risk analysis is always triggered. -- Test Intelligence analysis is triggered only when the diff touches non-test source - (a deterministic heuristic — no LLM call spent deciding this). -- Once analysis completes, governance policy (see Governance and Human Review below) is - checked first. If nothing triggers: a commit status check (`ai-test-intelligence/risk` - context) and a single PR comment are published, covering overall risk, top findings, - recommended tests (when triggered), and a link back to the full analysis in the - platform dashboard. **The comment never includes full model output** — no raw AI - rationale text, no full generated test source — by design. If a policy rule triggers: - a `pending` status and a "human review required" comment are published instead, and - the run waits in the review queue until a human approves or rejects it. - -## Governance and Human Review - -Risk findings are advisory, not self-executing. A completed risk assessment is checked -against a fixed set of policy rules before it's allowed to become a `success`/`failure` -commit status on a PR (or, for manually-triggered runs, before it's treated as resolved -at all): - -- high release risk (a `block` recommendation, or `risk_score` over a threshold) -- low confidence (`confidence_score` under a threshold) -- authentication/authorization changes -- breaking API or schema changes -- security-sensitive findings -- insufficient evidence for an elevated-risk claim - -Any triggered rule creates a **review request** — visible at `/review-queue` in the -dashboard — and, for webhook-originated runs, replaces the automatic status/comment with -a `pending` status and a short "review required" note. Nothing publishes a final -success/failure status for that run except a human decision: `POST -/api/v1/review-queue/{id}/approve` or `/reject`, each requiring a reviewer identity and -accepting an optional reason. Every step — the policy check itself, a triggered -requirement, and the eventual decision — is recorded as an **immutable audit event** -(`GET /api/v1/review-queue/{id}/audit-events`); there is no code path that updates or -deletes one once written. Any sensitive-looking values (API keys, tokens, private key -blocks, password/secret assignments) are redacted before they reach either an LLM -provider or the audit log — see `backend/app/governance/redaction.py`. - -Policy thresholds are configurable via environment variables (see `.env.example`, -`GOVERNANCE_*`); `GOVERNANCE_ENABLED=false` disables the gate entirely (every result -auto-approves), which is useful for local development. See -[architecture.md §12](docs/architecture.md#12-governance-and-human-review) for the full -design. - -## Production Gaps - -This is a reference implementation, not a deployed production system. The following -gaps are **known and deliberate** — flagged here explicitly rather than left for a -reviewer to discover, per the Sprint 14 hardening review this section is a product of: - -- **No authentication/authorization model.** Every API endpoint is open; CORS is - wildcard. This is the largest gap. A production deployment needs this before it's - reachable from anywhere but a trusted internal network. -- **No multi-tenancy.** One shared database, one shared governance policy configuration, - no tenant isolation. -- **No horizontal scaling of analysis workers.** The in-process `TaskQueue` (threads) - doesn't survive a process restart or scale beyond one process — see Design Decisions. -- **Governance policy is process-wide, not per-repository.** A monorepo or - multi-team deployment can't express different risk tolerances per project yet. -- **No rate limiting** on any endpoint, including the GitHub webhook receiver. -- **No secrets-manager integration.** `GITHUB_API_TOKEN`, `PROVIDER_ANTHROPIC_API_KEY`, - and `GITHUB_WEBHOOK_SECRET` are plain environment variables — fine for this reference - deployment, not for a real one holding real repository access tokens. -- **Redaction is a regex safety net, not a guarantee.** It catches recognizable secret - *shapes*; a credential in an unrecognized format passes through unredacted. -- **The Docker healthcheck only proves the process is alive and serving HTTP**, not that - the database is reachable — a real production readiness probe needs to check that too. -- **No OpenTelemetry tracing** across ingestion → orchestration → provider call → - persistence — structured logging and LangSmith tracing exist, distributed tracing - does not. -- **CI webhook ingestion (raw CI test-run results, independent of a PR) is not - implemented** — only the PR-triggered path exists today. -- **No database backup/restore strategy documented or automated.** -- **Every status-column enum (`analysis_runs.status`, `test_suggestions.status`, etc.) - except `review_requests.status` is validated at the application level only, not with a - database CHECK constraint** — Sprint 14 closed the gap for the newest, governance- - critical table; the others remain application-validated-only by scope decision, not - because the gap wasn't noticed. - -## Scaling to 10x Usage - -See Production Gaps above for what's missing today; this section describes how those -gaps would actually get closed under materially higher load, not a separate list. - -The current architecture is intentionally optimized for a single-process reference -deployment. At materially higher usage, the first architectural pressure would be the -in-process `TaskQueue`: analysis work is tied to one process, in-flight work is not -durable, and LLM-provider rate limits become a shared bottleneck. - -At roughly 10x usage, the architecture would evolve without changing the three -intelligence engines themselves: - -1. **Move analysis jobs to a durable distributed queue** behind the existing - `TaskQueue` interface, using a system such as Celery or Temporal, so API ingestion - and analysis execution can scale independently. -2. **Run analysis asynchronously with horizontally-scaled workers**, keeping webhook - handling fast while longer model calls execute outside the request path. -3. **Persist job state before execution** so process or worker restarts do not lose - accepted work, and use idempotency controls to avoid duplicate analysis from repeated - webhook delivery. -4. **Introduce provider-level concurrency and rate-limit controls**, including bounded - retries and backpressure rather than allowing traffic spikes to fan out directly to - the model provider. -5. **Add per-repository or per-team governance configuration** instead of the current - process-wide thresholds. -6. **Add authentication, RBAC, and tenant isolation** before supporting multiple teams - on shared infrastructure. -7. **Add production observability and cost controls**, including distributed tracing, - queue depth, worker saturation, provider latency/error rates, token usage, and - per-team usage budgets. -8. **Scale persistence deliberately**, with PostgreSQL connection pooling, backup and - restore procedures, retention policies, and indexing driven by actual query and - audit-volume patterns. - -Caching would be applied selectively. Stable metadata or reusable repository context may -be cacheable, but risk decisions and governance outcomes should not be blindly reused -across commits because the code, evidence, model version, prompt version, or policy may -have changed. - -## Future Evolution - -Natural next increments, roughly in order of likely value: - -1. **Authentication/authorization** — the prerequisite for deploying this anywhere - multi-user or internet-reachable. -2. **A GitHub App integration** replacing the Statuses-API approach, unlocking the - Checks API's richer UI (annotations, requested re-runs) without changing the - underlying webhook/governance flow. -3. **A distributed task queue** (Celery or Temporal) behind the existing `TaskQueue` - interface, once real concurrency demands outgrow a single process. -4. **Per-repository governance policy configuration**, letting a monorepo or - multi-team deployment express different risk tolerances per project. -5. **CI webhook ingestion** for raw test-run results, completing the Failure - Intelligence Engine's trigger path independent of a PR. -6. **OpenTelemetry tracing** across the full request/analysis lifecycle. -7. **Additional LLM providers** (the `LLMProvider` interface already supports this; - only Anthropic and Mock are implemented today). -8. **Tiered data retention for high-volume tables** (`analysis_runs`, `risk_findings`, - `llm_invocations`) — distinct from repository archiving above, and worth being - explicit about the difference: archiving a repository is small (one row, `{id, name, - url, ...}`) and reversible, aimed at "stop this repo from cluttering my view." At real - scale, the tables that actually grow unbounded are the ones recording *every analysis - run*, not the repository list itself. `audit_events` specifically should never be - time-boxed the way the others could — it's the compliance record Sprint 13's - governance model exists to produce, and a 30/60-day retention window there would - defeat the point (see Governance and Human Review). A sound version of this feature - varies retention by table: aggressive pruning for raw `llm_invocations` latency/cost - logs nobody needs after a few months, longer or indefinite retention for - `risk_findings`/`audit_events`, with cold storage rather than deletion where the data - has long-tail value (a team's risk-trend history) but doesn't need to stay query-fast. +See [Architecture](docs/architecture.md) for the corresponding scaling and security trade-offs. diff --git a/docs/architecture.md b/docs/architecture.md index 1fe3f88..5c435ae 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,37 +1,8 @@ # Architecture -## 1. Product Definition - -The AI Test Intelligence Platform ingests a codebase, its test suite, and its CI test-run -history, then applies LLM-backed analysis to answer three questions engineering teams -routinely under-invest in tooling for: - -1. **Where is testing risk concentrated?** — coverage/risk scoring of code relative to - change frequency, complexity, and existing test coverage. -2. **What tests should exist but don't?** — AI-generated test suggestions for - undertested or high-risk code paths. -3. **Why did this test fail, and does it matter?** — AI-assisted triage of CI failures, - classifying them as regressions, flaky tests, or environment issues, and clustering - recurring flaky patterns over time. - -These three capabilities share one pipeline: ingestion → orchestration → provider-backed -analysis → persistence → API → dashboard. That shared spine is what makes this a -*platform* rather than three unrelated scripts — a new analysis capability is a new -"engine" plugged into the same orchestration and provider layers, not a new system. - -A fourth, cross-cutting concern sits on top of all three: **governance.** AI-generated -findings are advisory by construction, not self-executing — a risk assessment that meets -certain conditions (high release risk, low confidence, authentication/authorization or -breaking-change categories, security-sensitive findings, insufficient evidence) requires -an explicit human decision before it can be treated as an approved signal anywhere -outside the platform's own dashboard. See §12. - -**Primary integration point:** GitHub Actions / pull requests. A PR is the natural unit -of "what changed" and the natural place to surface risk findings and test suggestions -before merge. CI webhook ingestion covers the failure-triage capability independently of -PR analysis. - -## 2. High-Level Architecture +This document describes implementation boundaries and engineering trade-offs. For setup and features, start with the [README](../README.md). For request sequences and data models, see [System design](system-design.md). + +## High-Level Architecture ```mermaid flowchart TB @@ -106,7 +77,7 @@ that feeds analysis and the status-check/PR-comment publish that closes the loop GitHub. CI webhook ingestion (the other half of this diagram's "webhook" arrow) remains future scope — see system-design.md §4's `/webhooks/ci` entry. -## 3. Component Diagram +## Component Diagram ```mermaid flowchart LR @@ -164,48 +135,7 @@ repository pattern — adding a fourth engine later requires no change to the orchestrator, provider layer, or API contracts beyond registering the new engine and its result schema. -## 4. Repository Layout - -``` -ai-test-intelligence-platform/ -├── README.md -├── docs/ -│ ├── architecture.md -│ └── system-design.md -├── backend/ -│ ├── app/ -│ │ ├── api/ # FastAPI routers, request/response schemas, GitHub webhook endpoint -│ │ ├── ingestion/ # Diff parsing, GitHub PR webhook event normalization -│ │ ├── integrations/ -│ │ │ └── github/ # GitHubClient abstraction, HMAC verification, PR comment/status publishing -│ │ ├── governance/ # Policy rules, sensitive-data redaction, review-request/audit-event service -│ │ ├── orchestration/ # TaskQueue interface + in-process implementation -│ │ ├── analysis/ -│ │ │ ├── risk/ # Coverage/risk analyzer engine -│ │ │ ├── test_intelligence/ # Test Intelligence Engine -│ │ │ ├── failure_intelligence/ # Failure Intelligence Engine -│ │ │ └── prompts/ # Versioned prompt templates, per engine -│ │ ├── providers/ # LLM provider abstraction + implementations -│ │ ├── persistence/ # SQLAlchemy models, repositories -│ │ └── observability/ # Logging, metrics, LangSmith tracing setup -│ ├── migrations/ # Alembic migrations -│ └── tests/ -├── frontend/ -│ ├── src/ -│ │ ├── pages/ # Route-level views (Repo overview, Risk, Suggestions, Flaky Tests) -│ │ ├── components/ # Shared presentational components -│ │ ├── api-client/ # Typed API client (generated or hand-written) -│ │ └── state/ # Data-fetching/cache state (TanStack Query) -│ └── e2e/ # Playwright smoke tests -└── .github/ - └── workflows/ # ci.yml, integration.yml, e2e.yml, docker.yml, live-smoke.yml -``` - -This is a monorepo: backend and frontend evolve together, and a single PR can span both -when a feature requires it, which matters for a project of this size and single-team -ownership model. - -## 5. Backend Module Organization +## Backend Module Organization - **`api/`** depends on `persistence/` (via repositories) and `orchestration/` (to enqueue analysis runs). It never imports `providers/` or engine internals directly — @@ -277,7 +207,7 @@ gained a redaction pass over `inputs` this sprint (`governance/redaction.py`), a before any engine (not just risk) ever sees the content, independent of governance's review-gating logic. -## 6. Frontend Organization +## Frontend Organization - **`pages/`** — one page per top-level dashboard view: repository overview, risk findings, test suggestions, flaky test history, and (Sprint 13) the review queue @@ -307,7 +237,7 @@ the provider/model/latency/token-usage detail `LLMInvocation` has captured since 9) — both thin routes over repository methods that already existed. CORS is wide open (`allow_origins=["*"]`) since there's still no auth model to scope it against. -## 7. Provider Abstraction Strategy +## Provider Abstraction Strategy ```mermaid classDiagram @@ -362,7 +292,7 @@ Two scoping decisions worth recording: `AnthropicProvider` and `OpenAIProvider` remain unimplemented, per Sprint 2 scope. -## 8. Observability Strategy +## Observability Strategy - **Structured logging**: JSON logs carrying `repo_id`, `analysis_run_id`, and a correlation ID threaded from ingestion through to persistence, so one run's full @@ -414,80 +344,7 @@ Two scoping decisions worth recording: every analysis run's `inputs` before an engine ever sees it, and from every `AuditEvent.payload` before it's persisted — see §12. -## 9. Testing Strategy - -- **Backend unit tests** (pytest): each module tested in isolation; analysis engines - tested exclusively against `MockProvider` so test runs are deterministic and free. -- **Backend integration tests**: exercise ingestion → orchestration → persistence - end-to-end against a real (dockerized/testcontainers) PostgreSQL instance, still using - `MockProvider` for LLM calls. -- **Provider contract tests**: a shared test suite run against every `LLMProvider` - implementation (including real ones) to guarantee interface compliance; the real - provider variants are gated behind a manual/opt-in CI job so normal PRs never spend - API budget or become flaky due to live network calls. -- **API contract tests**: request/response shape validation per endpoint. -- **Frontend**: component tests (Vitest + React Testing Library) once frontend code - exists; end-to-end (Playwright) deferred until there's a UI worth covering. -- **GitHub webhook tests (Sprint 12)**: `tests/api/test_webhooks.py` drives the full - webhook -> signature verification -> orchestration -> engine -> publish flow through - FastAPI's TestClient, with a `FakeGitHubClient` (implementing the `GitHubClient` - interface) standing in for the real GitHub API — no live repository or network access - required, same "fake the boundary interface" pattern `MockProvider` uses for LLM - calls. `tests/integrations/github/` covers signature verification, comment/status - text building (including an explicit assertion that no full model output — raw - rationale text, generated test source — ever appears in what gets built), the REST - client against an `httpx.MockTransport`, and `PRAnalysisPublisher`'s completion- - coordination logic (both orderings of risk/test-intelligence completion, failure - paths, duplicate-callback safety) directly, without going through HTTP. -- **Governance tests (Sprint 13)**: `tests/governance/` covers `policy.py` (every rule, - each threshold's boundary, the `insufficient_evidence` conditional logic, the global - kill-switch) and `redaction.py` (each secret pattern, plus an explicit "security - keywords survive redaction" check) as pure-function unit tests with no I/O; - `test_review_service.py` covers `ReviewRequest`/`AuditEvent` creation, the decision - workflow, and the "cannot decide twice" invariant against a real in-memory SQLite - session. `tests/api/test_webhooks.py` and `test_review_queue.py` cover the two - trigger paths end-to-end (webhook and manual), including the full - webhook → pending → approve/reject → GitHub-publish loop. - -## 10. CI/CD Strategy - -- **`ci.yml`**: Ruff lint, Ruff format check, mypy, and PyTest with coverage for the - backend; oxlint, Vitest, and a production build for the frontend. Runs on every push - and PR; required for merge. -- **`integration.yml`**: spins up a PostgreSQL 16 service container, runs Alembic - migrations (upgrade to head, then a downgrade/upgrade round trip), and runs the - PostgreSQL integration suite against it. Required for merge. -- **`e2e.yml`**: installs Playwright + Chromium and runs `frontend/e2e/` against a real - backend (MockProvider, disposable SQLite) and a real Vite dev server, both started by - Playwright itself. Required for merge. -- **`docker.yml`**: builds the backend and frontend Docker images independently, then - validates `docker-compose.yml` (`docker compose config` + `docker compose build`). - Build-only — nothing is pushed to a registry yet; that remains deferred until the - platform has an actual deployment target (§11). -- **`live-smoke.yml`**: manually triggered only (`workflow_dispatch`), runs the Anthropic - provider contract tests against the real API using an `ANTHROPIC_API_KEY` repository - secret. Never runs automatically, to keep normal CI free of external API cost and - flakiness — the only workflow in the repo permitted to spend real API budget. -- Branch protection on `main` requires `ci.yml`, `integration.yml`, `e2e.yml`, and - `docker.yml` to pass. - -## 11. Deliberately Deferred - -To keep this document honest about scope, the following are acknowledged as future -decisions, not oversights: authentication/authorization model, multi-tenancy, -deployment target (container platform vs. serverless), a distributed task queue -(Celery/Temporal) to replace the in-process orchestrator once real concurrency demands -it, CI webhook ingestion (the failure-intelligence-triggering half of the ingestion -diagram in §2 — GitHub PR ingestion landed in Sprint 12, CI ingestion has not), a -GitHub App integration (§5's Sprint 12 status note explains why the Statuses API was -used instead for now), and **per-repository** governance policy configuration — -`GovernancePolicySettings` (§12) is process-wide, not scoped to a repository, so every -registered repository shares the same risk-score/confidence thresholds and rule -toggles today. (Sprint 12's original note here — "a per-repository configurable -risk-gating policy... hardcodes 'only `block` fails the check'" — is superseded by -Sprint 13: the gating conditions are now genuinely configurable, just not per-repo yet.) - -## 12. Governance and Human Review +## Governance and Human Review Sprint 13's requirement in one sentence: AI output must never silently become an approved operational engineering action. Concretely, in this platform, the only @@ -568,15 +425,7 @@ to persist (a database error mid-write), the result is treated the same as a fai run — an `error` status, never a silent success — because a governance write failure must not be indistinguishable from "nothing to review." -**Closing the loop (`api/review.py`).** `POST /api/v1/review-queue/{id}/approve` and -`/reject` are the only other code path that can publish a final success/failure commit -status, and only as the direct, synchronous result of `governance/review_service.py`'s -`decide_review()` having already durably recorded the decision — GitHub is a downstream -notification of that decision, not the source of truth for it. A review request can be -decided exactly once (`ReviewAlreadyDecidedError` on a second attempt) — a decision, once -made, is final, the same principle the audit trail's immutability is built on. - -## 13. Sprint 14: Production Hardening Review +## Sprint 14: Production Hardening Review A staff/principal-level audit against the standard checklist — module boundaries, dependency direction, error handling, retry logic, configuration, secret handling, @@ -641,4 +490,3 @@ integration, OpenTelemetry tracing, CI webhook ingestion for raw test-run result database backup/restore strategy, and CHECK constraints on the remaining status-enum columns (`test_runs.status`, `analysis_runs.status`, `test_suggestions.status`, `test_results.status`) beyond the one closed this sprint. - diff --git a/docs/system-design.md b/docs/system-design.md index 743311a..f24b985 100644 --- a/docs/system-design.md +++ b/docs/system-design.md @@ -1,9 +1,8 @@ -# System Design +# System design -Companion to [architecture.md](architecture.md). This document covers data flow, -database schema, and API boundaries in detail. +Detailed request and data contracts for the implemented platform. See [Architecture](architecture.md) for component ownership. -## 1. Data Flow: PR-Triggered Analysis +## Pull-request analysis ```mermaid sequenceDiagram @@ -34,61 +33,13 @@ sequenceDiagram GH-->>Dev: Check result visible on PR ``` -**Implementation status (Sprint 12):** live, with two refinements to what the diagram -shows at a high level: +The webhook handler verifies GitHub's signature and retrieves the diff. Analysis runs asynchronously; results may complete in either order. Policy can hold publication pending human review rather than publishing an automatic success status. -- `API->>Ing: normalize event` and the diff itself are two separate steps in practice — - `app/ingestion/github_webhook.py` normalizes the webhook *payload* into a - `PullRequestWebhookEvent`; `app/api/webhooks.py` then fetches the PR's *diff* via - `GitHubClient.get_pull_request_diff()` (a second GitHub API call, since `pull_request` - webhook payloads don't embed the diff itself) before calling `Orch->>Risk` / - `Orch->>TestIntel`. Test Intelligence is only enqueued when - `diff_touches_non_test_source()` finds non-test source in the diff — not - unconditionally, as the diagram's `AnalysisRun(type=risk, test_intelligence)` might - imply. -- `API->>GH: POST status check + PR comment` is really `Orch->>API`-adjacent, not - `API`-initiated: Risk and Test Intelligence run on independent background threads and - complete in no guaranteed order, so `AnalysisOrchestrator.submit()`'s `on_result` - completion hook (added this sprint) is what actually fires the publish, from - `integrations/github/publisher.py`'s `PRAnalysisPublisher` — once both results this PR - is waiting on have arrived, not synchronously as part of the original webhook request/ - response cycle. +## CI failure analysis -**Implementation status (Sprint 13):** the diagram's final `POST status check + PR -comment` step is no longer unconditional. Governance policy (architecture.md §12) is -evaluated on the risk result first; if it triggers, what gets published is a `pending` -status plus a "review required" comment instead — the diagram's success/failure publish -only happens automatically when nothing triggers. When something does, the corresponding -edges become a *separate* flow entirely, initiated later by a human decision, not by -this sequence: `Human->>API: POST /review-queue/{id}/approve` (or `/reject`) -> -`API->>DB: record decision + audit event` -> `API->>GH: POST final status + decision -comment`. +The failure-intelligence engine analyzes supplied logs and test results. Automatic ingestion of independent CI test-run webhooks is **not implemented**; the original design sequence should not be read as a working integration. -## 2. Data Flow: CI Failure Analysis - -```mermaid -sequenceDiagram - participant CI as CI Test Run - participant API as API Layer - participant Ing as Ingestion Service - participant Orch as Orchestrator - participant FailureIntel as Failure Intelligence Engine - participant Prov as Provider Abstraction - participant DB as PostgreSQL - - CI->>API: POST /webhooks/ci (test results) - API->>Ing: normalize test run + results - Ing->>DB: persist TestRun, TestResults - Ing->>Orch: enqueue AnalysisRun(type=failure_intelligence) - Orch->>FailureIntel: run(context: failed TestResults) - FailureIntel->>DB: read historical TestResults for same TestCase - FailureIntel->>Prov: generate(failure intelligence prompt) - Prov-->>FailureIntel: LLMResponse (root cause hypotheses + rationale) - FailureIntel->>DB: write FlakyTestFindings / FailureFindings - Orch->>DB: mark AnalysisRun complete -``` - -## 3. Database Schema (High-Level) +## Data model ```mermaid erDiagram @@ -221,26 +172,9 @@ erDiagram } ``` -**Notes:** - -- `analysis_runs` is the anchor entity for observability and cost accounting — every - engine invocation belongs to exactly one run, and every finding traces back to the run - (and therefore the provider/model) that produced it. -- `test_suggestions.status` (`pending` / `accepted` / `rejected`) makes suggestion - review an explicit workflow state rather than an implicit one, so acceptance-rate - becomes a measurable signal on generation quality. -- **`review_requests`** (Sprint 13) is mutable current-state — `status`, `reviewer`, - `review_reason`, `decided_at` are updated in place by a decision. `github_*` columns - are nullable and only populated for webhook-originated runs; see architecture.md §12. -- **`audit_events`** (Sprint 13) is append-only by repository API design (no update/ - delete method exists on `AuditEventRepository` at all — see architecture.md §12) — - the immutable history of how a `review_requests` row reached its current status, and - also records `policy_evaluated` events (`review_request_id NULL`) for runs where - governance ran but nothing triggered. -- Schema is intentionally high-level here; exact column types, indexes, and constraints - are defined when persistence code is implemented, not in this design doc. +`analysis_runs` ties findings to provider/model execution. Review requests hold the current decision state; audit events retain the decision history. This diagram is a conceptual view; refer to migrations and models for the authoritative schema. -## 4. API Boundaries +## API boundaries All endpoints are versioned under `/api/v1`. @@ -255,9 +189,7 @@ All endpoints are versioned under `/api/v1`. | `POST` | `/test-suggestions/{id}/accept` | Mark a suggestion accepted | | `POST` | `/test-suggestions/{id}/reject` | Mark a suggestion rejected | | `GET` | `/repositories/{id}/flaky-tests` | List flaky test findings | -| `POST` | `/webhooks/github` | GitHub PR event ingestion — **implemented, Sprint 12** | | `POST` | `/webhooks/ci` | CI test-run result ingestion — not yet implemented | -| `GET` | `/review-queue` | List review requests (defaults to `status=pending`) — **implemented, Sprint 13** | | `GET` | `/review-queue/{id}` | Review request detail (incl. redacted risk_summary) | | `GET` | `/review-queue/{id}/audit-events` | Immutable audit trail for one review request | | `POST` | `/review-queue/{id}/approve` | Record an approval (reviewer, optional reason) | @@ -270,7 +202,6 @@ All endpoints are versioned under `/api/v1`. or a provider implementation directly. - Webhook endpoints are the only unauthenticated-by-default surface (secured instead by provider signature verification — GitHub's HMAC-SHA256 `X-Hub-Signature-256`, verified - against the raw request body by `integrations/github/signature.py`, Sprint 12) since they're called by external systems, not end users; every other endpoint sits behind whatever authentication model is adopted (deferred — see architecture.md §11). - All analysis-triggering endpoints return immediately with an `analysis_run_id` in @@ -280,7 +211,7 @@ All endpoints are versioned under `/api/v1`. OpenAPI spec FastAPI generates automatically — the frontend's typed API client is intended to be generated from that spec rather than hand-synced. -## 5. Module Contracts +## Module contracts - **`AnalysisEngine` interface** (implemented by Risk, Test Intelligence, Failure Intelligence engines): `run(context: AnalysisContext) -> AnalysisResult`.