From 837068aa29008815f6e1d4b385e108194158881a Mon Sep 17 00:00:00 2001 From: Aman koli <2025.amana@isu.ac.in> Date: Sun, 12 Jul 2026 18:21:41 +0530 Subject: [PATCH] docs: Strengthen Documentation Architecture & Developer Guides (#38) --- Docs/README.md | 153 +++++++++++++++ Docs/api-docs/websocket-api.md | 230 +++++++++++++++++++++++ Docs/architecture/agent-overview.md | 152 +++++++++++++++ Docs/architecture/data-flow.md | 203 ++++++++++++++++++++ Docs/architecture/execution-flow.md | 204 ++++++++++++++++++++ Docs/architecture/module-inventory.md | 107 +++++++++++ Docs/architecture/system-overview.md | 149 +++++++++++++++ Docs/ci-cd/discord-notifications.md | 82 ++++++++ Docs/ci-cd/workflows.md | 101 ++++++++++ Docs/contributing/guidelines.md | 142 ++++++++++++++ Docs/contributing/main-agent-issues.md | 221 ++++++++++++++++++++++ Docs/deployment/environment-variables.md | 88 +++++++++ Docs/error-handling/bias-mitigation.md | 155 +++++++++++++++ Docs/error-handling/known-issues.md | 174 +++++++++++++++++ Docs/frontend/ui-overview.md | 125 ++++++++++++ Docs/getting-started/README.md | 80 ++++++++ Docs/getting-started/installation.md | 119 ++++++++++++ Docs/monitoring-logging/observability.md | 122 ++++++++++++ Docs/security/overview.md | 171 +++++++++++++++++ Docs/testing/calibration.md | 197 +++++++++++++++++++ Docs/testing/human-testing-protocol.md | 191 +++++++++++++++++++ Docs/user-flows/analysis-workflow.md | 133 +++++++++++++ Docs/viva-intelligence/session-flow.md | 166 ++++++++++++++++ Docs/viva-intelligence/viva-overview.md | 118 ++++++++++++ 24 files changed, 3583 insertions(+) create mode 100644 Docs/README.md create mode 100644 Docs/api-docs/websocket-api.md create mode 100644 Docs/architecture/agent-overview.md create mode 100644 Docs/architecture/data-flow.md create mode 100644 Docs/architecture/execution-flow.md create mode 100644 Docs/architecture/module-inventory.md create mode 100644 Docs/architecture/system-overview.md create mode 100644 Docs/ci-cd/discord-notifications.md create mode 100644 Docs/ci-cd/workflows.md create mode 100644 Docs/contributing/guidelines.md create mode 100644 Docs/contributing/main-agent-issues.md create mode 100644 Docs/deployment/environment-variables.md create mode 100644 Docs/error-handling/bias-mitigation.md create mode 100644 Docs/error-handling/known-issues.md create mode 100644 Docs/frontend/ui-overview.md create mode 100644 Docs/getting-started/README.md create mode 100644 Docs/getting-started/installation.md create mode 100644 Docs/monitoring-logging/observability.md create mode 100644 Docs/security/overview.md create mode 100644 Docs/testing/calibration.md create mode 100644 Docs/testing/human-testing-protocol.md create mode 100644 Docs/user-flows/analysis-workflow.md create mode 100644 Docs/viva-intelligence/session-flow.md create mode 100644 Docs/viva-intelligence/viva-overview.md diff --git a/Docs/README.md b/Docs/README.md new file mode 100644 index 0000000..d51b35f --- /dev/null +++ b/Docs/README.md @@ -0,0 +1,153 @@ +# ORACLE Documentation + +> **ORACLE** β€” Optimized Repository Analysis & Candidate Learning Engine +> An evidence-grounded Implementation Familiarity Assessment System + +--- + +## Documentation Index + +### πŸš€ Getting Started + +| Document | Description | +|---|---| +| [README](./getting-started/README.md) | What ORACLE does, quick navigation, system requirements | +| [Installation](./getting-started/installation.md) | Local setup, environment, running the server | + +--- + +### πŸ›οΈ Architecture + +| Document | Description | +|---|---| +| [System Overview](./architecture/system-overview.md) | Top-level architecture, agent pipeline, design principles | +| [Module Inventory](./architecture/module-inventory.md) | All modules, LOC, ownership, active/review/archived status | +| [Execution Flow](./architecture/execution-flow.md) | Step-by-step pipeline: repo clone β†’ AST β†’ viva β†’ assessment | +| [Data Flow](./architecture/data-flow.md) | Data structures, transformations, WebSocket payload schemas | +| [Agent Overview](./architecture/agent-overview.md) | Per-agent responsibilities, boundaries, and decision routing | + +--- + +### πŸ”Œ API Reference + +| Document | Description | +|---|---| +| [WebSocket API](./api-docs/websocket-api.md) | Full WebSocket protocol, input/output schemas, streaming format | + +--- + +### 🧠 Viva Intelligence + +| Document | Description | +|---|---| +| [Viva Overview](./viva-intelligence/viva-overview.md) | How the viva system works, indicators, familiarity levels, examples | +| [Session Flow](./viva-intelligence/session-flow.md) | Turn-by-turn session walkthrough with state and examples | + +--- + +### 🎨 Frontend + +| Document | Description | +|---|---| +| [UI Overview](./frontend/ui-overview.md) | Dashboard panels, UI alignment status, known minor issues | + +--- + +### 🚒 Deployment + +| Document | Description | +|---|---| +| [Environment Variables](./deployment/environment-variables.md) | All env vars, required scopes, security rules | + +--- + +### βš™οΈ CI/CD + +| Document | Description | +|---|---| +| [Workflows](./ci-cd/workflows.md) | GitHub Actions workflows: calibration, Discord notify | +| [Discord Notifications](./ci-cd/discord-notifications.md) | Commit-to-Discord bot setup guide | + +--- + +### πŸ§ͺ Testing + +| Document | Description | +|---|---| +| [Calibration Framework](./testing/calibration.md) | Automated P/R metrics, confidence calibration, CI thresholds | +| [Human Testing Protocol](./testing/human-testing-protocol.md) | 4-phase real-human validation protocol | + +--- + +### πŸ” Security + +| Document | Description | +|---|---| +| [Security Overview](./security/overview.md) | Fairness audit, trust audit, overconfidence detection, access control | + +--- + +### ⚠️ Error Handling + +| Document | Description | +|---|---| +| [Known Issues](./error-handling/known-issues.md) | Active known issues, error codes, recovery behaviour | +| [Bias Mitigation](./error-handling/bias-mitigation.md) | All 6 FairnessAuditor patterns, mitigation strategies | + +--- + +### πŸ“Š Monitoring & Logging + +| Document | Description | +|---|---| +| [Observability](./monitoring-logging/observability.md) | Runtime tracing, WebSocket log streaming, calibration dashboard | + +--- + +### 🀝 Contributing + +| Document | Description | +|---|---| +| [Guidelines](./contributing/guidelines.md) | Code standards, PR process, naming conventions, review checklist | +| [MAIN Agent Issues](./contributing/main-agent-issues.md) | 6 pre-scoped contributor issues with file structures | + +--- + +### πŸ—ΊοΈ User Flows + +| Document | Description | +|---|---| +| [Analysis Workflow](./user-flows/analysis-workflow.md) | End-to-end user journey: open UI β†’ analyse β†’ review β†’ assess | + +--- + +## Project Phase Status + +| Phase | Status | +|---|---| +| Phase 1 – Architecture Assessment & Cleanup | βœ… Complete | +| Phase 2 – Evidence-Grounded Intelligence (3 engines) | βœ… Complete | +| Phase 3.5 – Stabilization & Reality Hardening | πŸ”„ In Progress | +| Phase 4 – Real Human Testing (10–15 participants) | ⏳ Planned | + +--- + +## Performance Baseline (Phase 2 Calibration) + +| Component | Precision | Recall | F1 | +|---|---|---|---| +| Observable Signals | 0.847 | 0.823 | 0.835 | +| Failure Propagation | 0.805 | 0.778 | 0.791 | +| Viva Question Validity | 0.856 | β€” | β€” | +| Viva Question Grounding | 0.912 | β€” | β€” | +| Confidence Calibration RMSE | β€” | β€” | 0.062 βœ… | + +--- + +## Quick Links + +- **Run the system:** [Installation Guide](./getting-started/installation.md) +- **Understand the pipeline:** [Execution Flow](./architecture/execution-flow.md) +- **Read the API:** [WebSocket API](./api-docs/websocket-api.md) +- **Run calibration:** [Calibration Framework](./testing/calibration.md) +- **Contribute:** [Contributing Guidelines](./contributing/guidelines.md) diff --git a/Docs/api-docs/websocket-api.md b/Docs/api-docs/websocket-api.md new file mode 100644 index 0000000..de49ae2 --- /dev/null +++ b/Docs/api-docs/websocket-api.md @@ -0,0 +1,230 @@ +# WebSocket API Reference + +ORACLE communicates with the browser UI via a persistent WebSocket connection. + +**Endpoint:** `ws://localhost:8001/ws/analyze` +**Protocol:** JSON messages over WebSocket +**Server:** FastAPI + `python-multipart`, `uvicorn` + +--- + +## Connection Flow + +``` +Browser FastAPI Server + β”‚ β”‚ + │──── ws://localhost:8001/ws/analyze ──→│ + β”‚ β”‚ + │──── { "repo_url": "..." } ────────→│ + β”‚ β”‚ + │←─── { "type": "log", ... } ───────│ (streaming) + │←─── { "type": "log", ... } ───────│ (streaming) + β”‚ ... many log messages ... β”‚ + │←─── { "type": "result", ... } ────│ (final payload) + β”‚ β”‚ + │──── close ────────────────────────│ +``` + +--- + +## Input Message + +Sent once after WebSocket connection is established. + +```json +{ + "repo_url": "https://github.com/org/repo-name" +} +``` + +| Field | Type | Required | Description | +|---|---|---|---| +| `repo_url` | `string` | βœ… Yes | Full HTTPS GitHub URL of the repository to analyse | + +--- + +## Streaming Messages (Backend β†’ UI) + +Sent continuously during analysis. Used to update the terminal log panel. + +```json +{ + "type": "log", + "message": "[Oracle] Building execution graph...", + "log_type": "info" +} +``` + +| Field | Type | Values | Description | +|---|---|---|---| +| `type` | `string` | `"log"` | Always `"log"` for streaming messages | +| `message` | `string` | Any | Human-readable progress message | +| `log_type` | `string` | `info` \| `warn` \| `success` \| `error` | Controls colour in terminal panel | + +--- + +## Final Result Payload (Backend β†’ UI) + +Sent once when analysis is complete. + +```json +{ + "type": "result", + "data": { ... } +} +``` + +### `data` Object Schema + +#### Framework Detection + +```json +"backend_framework": { + "value": "FastAPI", + "confidence": 0.95, + "evidence": ["backend/src/main.py:12 β€” @app.get('/health')"] +} +``` + +| Field | Type | Description | +|---|---|---| +| `value` | `string` | Detected framework name | +| `confidence` | `float` | Detection confidence (0–1) | +| `evidence` | `List[string]` | Code locations that support detection | + +Same schema applies to: +- `architecture_pattern` β€” e.g. "REST + WebSocket" +- `authentication_system` β€” e.g. "JWT Bearer" + +--- + +#### Execution Graph + +```json +"execution_graph": { + "nodes": [ + { + "id": "node_001", + "label": "GET /users", + "type": "ROUTE", + "metadata": { + "file_path": "backend/src/main.py", + "line_number": 42, + "snippet": "@app.get('/users')" + } + } + ], + "edges": [ + { + "source": "node_001", + "target": "node_002", + "relationship": "calls" + } + ] +} +``` + +**Node types:** `ROUTE` | `MIDDLEWARE` | `DB_QUERY` | `STATE_STORE` | `AUTH_HANDLER` + +--- + +#### Viva Intelligence Targets + +```json +"viva_intelligence_targets": [ + { + "topic": "N+1 Query in /users endpoint", + "question_target": "What performance concern does the /users endpoint have?", + "difficulty": "hard", + "importance_score": 0.95, + "category": "Architecture", + "depth_score": 8.5, + "related_node": "node_001", + "confidence": 0.88, + "reasoning_summary": "Endpoint loads 100+ related resources without batching" + } +] +``` + +**Difficulty values:** `"hard"` | `"medium"` | `"foundational"` +**Category values:** `"Architecture"` | `"Tradeoff"` | `"Security"` | `"Scalability"` | `"Failure-Path"` | `"Runtime"` + +--- + +#### Runtime Risks + +```json +"runtime_risks": [ + { + "severity": "HIGH", + "value": "Unhandled async exception in payment service", + "confidence": 0.87, + "evidence": ["backend/src/services/payment.py:88"] + } +] +``` + +--- + +#### Failure Paths + +```json +"failure_paths": [ + { + "value": "DB connection loss causes cascading timeout in /checkout", + "confidence": 0.82, + "evidence": ["backend/src/routes/checkout.py:55"] + } +] +``` + +--- + +#### Evaluation Metrics + +```json +"evaluation_metrics": { + "metrics": { + "stack_accuracy": 0.95, + "auth_detection_accuracy": 0.91 + }, + "mismatches": [], + "expected": { + "expected_stack": "FastAPI + PostgreSQL", + "expected_protected_routes": ["/admin", "/dashboard"], + "expected_architecture": "Layered REST" + } +} +``` + +--- + +## Error Handling + +If analysis fails, the backend sends: + +```json +{ + "type": "log", + "message": "[Oracle] ERROR: Failed to clone repository. Check GITHUB_TOKEN.", + "log_type": "error" +} +``` + +The WebSocket connection is then closed. + +--- + +## Backend Source Reference + +- Handler: `backend/src/main.py` β€” `websocket_analyze()` at line 42 +- OracleAgent: `backend/src/agents/oracle/agent.py` +- StructuredContext model: `backend/src/models/` + +--- + +## Related Docs + +- [Payload Schemas](./payload-schemas.md) β€” Full Pydantic model reference +- [Data Flow](../architecture/data-flow.md) β€” How data is assembled +- [UI Overview](../frontend/ui-overview.md) β€” How the UI consumes this payload diff --git a/Docs/architecture/agent-overview.md b/Docs/architecture/agent-overview.md new file mode 100644 index 0000000..4cf9f74 --- /dev/null +++ b/Docs/architecture/agent-overview.md @@ -0,0 +1,152 @@ +# Agent Overview + +ORACLE uses a four-agent pipeline. Each agent has a **strictly defined** responsibility boundary. No agent duplicates another's intelligence. + +--- + +## Agent Pipeline + +``` +Request (repo_url) + ↓ +MainAgent + β”œβ”€β”€β†’ GatekeeperAgent (identity + access) + β”œβ”€β”€β†’ OracleAgent (analysis + intelligence) + └──→ SentinelAgent (audit + moderation) + ↓ +FinalAssessment +``` + +--- + +## 1. MainAgent + +**File:** `backend/src/agents/main_agent/agent.py` +**Role:** Viva session orchestrator + +### Responsibilities +- Coordinates GatekeeperAgent β†’ OracleAgent β†’ SentinelAgent pipeline +- Manages session lifecycle (init β†’ active β†’ complete β†’ closed) +- Persists viva session state across multiple turns +- Controls question sequencing and pacing +- Routes follow-up decisions to ORACLE outputs β€” does NOT recompute intelligence +- Maintains topic coverage tracker to avoid repetitive questioning +- Persists full transcript for audit and replay + +### MAIN Must NOT +- Perform AST analysis +- Duplicate ORACLE reasoning logic +- Invent speculative questions +- Hide scoring systems +- Behave like a generic chatbot + +### Key Sub-Modules (Planned) + +``` +backend/src/agents/main_agent/ + session/ ← Session state management + orchestration/ ← Flow control, pacing, branching + followups/ ← Evidence-grounded follow-up strategy + coverage/ ← Topic coverage tracker + transcript/ ← Persistence layer + integration/ ← ORACLE adapter (normalises outputs) +``` + +--- + +## 2. GatekeeperAgent + +**File:** `backend/src/agents/gatekeeper/` +**Role:** Identity context + access validation + +### Responsibilities +- Validates the requesting identity +- Establishes access context for the session +- Provides identity metadata to downstream agents +- Returns identity context as input to ORACLE pipeline + +### Gatekeeper Must NOT +- Perform code analysis +- Make session decisions +- Modify viva question generation + +--- + +## 3. OracleAgent + +**File:** `backend/src/agents/oracle/agent.py` +**Role:** Core intelligence and analysis engine + +### Responsibilities +- Clones repository and builds ExecutionGraph via AST +- Detects backend framework, architecture pattern, authentication system +- Runs ObservableSignalsEngine (error handling, resilience, observability) +- Runs ExecutionGraphFailureAnalyzer (failure propagation paths) +- Runs EvidenceGroundedVivaGenerator (grounded viva questions) +- Conducts viva session scoring (per-response specificity, correctness, quality) +- Classifies reasoning patterns and implementation familiarity +- Runs TrustAuditPipeline and FairnessAuditor +- Assembles and returns StructuredContext + +### OracleAgent Pipeline + +``` +OracleAgent.process() + β”œβ”€β”€ Clone + AST Parse β†’ ExecutionGraph + β”œβ”€β”€ ObservableSignalsEngine.extract_signals() β†’ observable_signals + β”œβ”€β”€ ExecutionGraphFailureAnalyzer.analyze() β†’ failure_paths + β”œβ”€β”€ EvidenceGroundedVivaGenerator.generate() β†’ viva_intelligence_targets + β”œβ”€β”€ TechDetector β†’ backend_framework, architecture_pattern, auth_system + β”œβ”€β”€ OracleEvaluator β†’ evaluation_metrics (benchmarked vs ground truth) + └── StructuredContext.assemble() β†’ final JSON payload +``` + +--- + +## 4. SentinelAgent + +**File:** `backend/src/agents/sentinel/` +**Role:** Audit + moderation layer + +### Responsibilities +- Logs audit events for all session decisions +- Enforces moderation policies on outputs +- Appends fairness-related audit annotations to transcripts +- Does NOT own session state β€” reads only + +### Sentinel Must NOT +- Generate questions +- Score candidates +- Override ORACLE assessments without documented policy + +--- + +## Agent Boundaries (Summary) + +| Decision | Owner | +|---|---| +| AST analysis, code intelligence | OracleAgent | +| Session flow, turn progression | MainAgent | +| Identity, access | GatekeeperAgent | +| Audit logs, moderation | SentinelAgent | +| Follow-up strategy | MainAgent (consuming ORACLE outputs) | +| Confidence scoring | OracleAgent | +| Fairness detection | OracleAgent (FairnessAuditor) | +| Transcript persistence | MainAgent | + +--- + +## Decision Routing Rule + +> If a change starts to look like **analysis** β†’ move it to OracleAgent. +> If it looks like **moderation** β†’ move it to SentinelAgent. +> If it looks like **submission validation** β†’ move it to GatekeeperAgent. +> If it's **orchestration** β†’ it belongs in MainAgent. + +--- + +## Related Docs + +- [MAIN Agent Issue Breakdown](../contributing/main-agent-issues.md) +- [Execution Flow](./execution-flow.md) +- [Module Inventory](./module-inventory.md) diff --git a/Docs/architecture/data-flow.md b/Docs/architecture/data-flow.md new file mode 100644 index 0000000..536dddc --- /dev/null +++ b/Docs/architecture/data-flow.md @@ -0,0 +1,203 @@ +# Data Flow + +This document describes how data structures are created, transformed, and passed between modules throughout the ORACLE pipeline. + +--- + +## Request Phase + +``` +Repository Metadata (URL, branch, concern) + ↓ +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Code Parsing β”‚ +β”‚ AST β†’ ExecutionGraph β”‚ +β”‚ nodes: List[GraphNode] β”‚ +β”‚ edges: List[GraphEdge] β”‚ +β”‚ node types: ROUTE | MIDDLEWARE | DB_QUERY | ... β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + ↓ +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Corpus Loading β”‚ +β”‚ EngineeringReviewCorpus β†’ CorpusContext β”‚ +β”‚ reviews: List[EngineeredReviewEntry] β”‚ +β”‚ signals: List[str] β”‚ +β”‚ concern_match_score: float β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + ↓ +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Failure Mapping β”‚ +β”‚ FailureCorpusRepository β†’ FailureSignalMap β”‚ +β”‚ scenarios: List[FailureScenario] β”‚ +β”‚ severity: HIGH | MEDIUM | LOW β”‚ +β”‚ propagation_path: List[GraphNode] β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +--- + +## Session Phase + +``` +VivaTarget (per question) + topic: str + question_target: str + difficulty: "hard" | "medium" | "foundational" + importance_score: float + category: "Architecture" | "Tradeoff" | "Security" | "Scalability" | "Runtime" + depth_score: float (0–10) + related_node: str (graph node ID) + confidence: float (0–1) + reasoning_summary: str + ↓ +CandidateResponse + question_id: str + response_text: str + turn: int + ↓ +ResponseEvaluation + specificity_score: float (0–100) + correctness_score: float (0–100) + quality: EXCELLENT | GOOD | ADEQUATE | WEAK | EVASIVE | CONTRADICTION + red_flags: List[str] +``` + +--- + +## Analysis Phase + +``` +IndicatorSet (per response) + understanding: List[IndicatorType] + memorisation: List[IndicatorType] + ↓ +ReasoningPatternAssessment + pattern: DEEP | PRACTICED | INFORMED | LOW | INSUFFICIENT + implementation_familiarity_score: float (0–1) + familiarity_confidence: HIGH | MEDIUM | LOW | VERY_HIGH + indicator_evidence: List[IndicatorEvidence] + ↓ +AggregateProfile + overall_familiarity: FamiliarityLevel + overall_confidence: ConfidenceLevel + red_flags: List[str] + contradiction_log: List[ContradictionEvent] + ↓ +TrustAuditResult + is_overconfident: bool + evidence_grounded: bool + uncertainty_flags: List[str] + adjusted_confidence: float + ↓ +FairnessAuditReport + issues: List[FairnessAuditIssue] + bias_types: List[BiasType] + manual_review_recommended: bool + confidence_adjusted: bool + ↓ +FinalAssessment + classification: str (DEEP_IMPLEMENTATION_FAMILIARITY | PRACTICED | INFORMED | INSUFFICIENT) + confidence: str (HIGH | MEDIUM | LOW) + evidence_trace: List[EvidenceTraceEntry] + uncertainty_statement: str + transcript: List[TranscriptEntry] +``` + +--- + +## WebSocket Payload (UI β†’ Backend) + +```json +{ + "repo_url": "https://github.com/org/repo" +} +``` + +--- + +## WebSocket Streaming Messages (Backend β†’ UI) + +### Log Message (streaming) +```json +{ + "type": "log", + "message": "[Oracle] Building execution graph...", + "log_type": "info" +} +``` + +`log_type` values: `info` | `warn` | `success` | `error` + +--- + +## WebSocket Result Payload (Backend β†’ UI) + +```json +{ + "type": "result", + "data": { + "backend_framework": { "value": "FastAPI", "confidence": 0.95, "evidence": [] }, + "architecture_pattern": { "value": "REST + WebSocket", "confidence": 0.88, "evidence": [] }, + "authentication_system": { "value": "JWT Bearer", "confidence": 0.91, "evidence": [] }, + "execution_graph": { + "nodes": [...], + "edges": [...] + }, + "viva_intelligence_targets": [...], + "runtime_risks": [...], + "failure_paths": [...], + "evaluation_metrics": { + "metrics": { "stack_accuracy": 0.95, "auth_detection_accuracy": 0.91 }, + "mismatches": [], + "expected": {} + } + } +} +``` + +--- + +## Model Dependency Map + +``` +reasoning_depth_analyzer.py + β”œβ”€ No internal HV imports + └─ Used by: __init__.py, comparative_calibration_runner.py + +viva_session_conductor.py + β”œβ”€ Imports: engineering_review_corpus + └─ Used by: __init__.py, comparative_calibration_runner.py + +trust_audit.py + β”œβ”€ No internal HV imports + └─ Used by: comparative_calibration_runner.py + +engineering_review_corpus.py + β”œβ”€ No internal HV imports + └─ Used by: viva_session_conductor, comparative_reasoning_evaluator + +failure_corpus.py + β”œβ”€ No internal HV imports + └─ Used by: comparative_evaluator, execution_behavior_analysis + +fairness_audit.py + β”œβ”€ No internal HV imports + └─ Used by: comparative_calibration_runner + +comparative_reasoning_evaluator.py + β”œβ”€ Imports: engineering_review_corpus, failure_corpus + └─ Used by: comparative_calibration_runner + +⚠️ NEEDS REVIEW: +evaluator.py β†’ imports human_evaluator_models, models +comparative_evaluator.py β†’ imports human_evaluator_models, execution_behavior_analysis +execution_behavior_analysis.py β†’ imports models, failure_corpus +``` + +--- + +## Related Docs + +- [Execution Flow](./execution-flow.md) β€” Step-by-step pipeline +- [WebSocket API](../api-docs/websocket-api.md) β€” Full API contract +- [Module Inventory](./module-inventory.md) β€” Ownership and status diff --git a/Docs/architecture/execution-flow.md b/Docs/architecture/execution-flow.md new file mode 100644 index 0000000..fb671ac --- /dev/null +++ b/Docs/architecture/execution-flow.md @@ -0,0 +1,204 @@ +# Execution Flow + +This document traces the complete step-by-step pipeline from repository URL input to final assessment output. + +--- + +## Overview + +``` +Input: Repository URL + Engineering Concern + ↓ + [8 Pipeline Steps] + ↓ +Output: Assessment Report + Transcript + Evidence Trace +``` + +--- + +## Step 1: Repository Analysis + +``` +Input: GitHub Repository URL + ↓ +Git Clone (shallow, --depth 1) + ↓ +AST Parsing (Python AST / language-specific parser) + β”œβ”€ Extract route definitions + β”œβ”€ Extract middleware chains + β”œβ”€ Extract DB query patterns + β”œβ”€ Extract auth handlers + └─ Extract state store interactions + ↓ +Output: ExecutionGraph (nodes, edges, types, metadata) +``` + +**Node Types:** `ROUTE`, `MIDDLEWARE`, `DB_QUERY`, `STATE_STORE`, `AUTH_HANDLER` + +Each node carries: `file_path`, `line_number`, `snippet` + +--- + +## Step 2: Engineering Review Corpus Loading + +``` +Input: Engineering Concern (e.g. "auth", "caching", "error handling") + ↓ +Load relevant engineering reviews from corpus + β”œβ”€ Match concern to review topics + └─ Extract implementation signals from reviews + ↓ +Output: CorpusContext (grounding evidence for questions) +``` + +--- + +## Step 3: Failure Pattern Mapping + +``` +Input: ExecutionGraph + ↓ +FailureCorpusRepository.get_patterns() + β”œβ”€ Identify potential failure modes for this code + β”œβ”€ Map to observable signals (what to look for) + └─ Create probing targets (which failure paths to question) + ↓ +Output: FailureSignalMap +``` + +--- + +## Step 4: Viva Session Planning + +``` +Input: CorpusContext + FailureSignalMap + ExecutionGraph + ↓ +EvidenceGroundedVivaGenerator + β”œβ”€ Create 3–4 opening questions + β”‚ └─ Each grounded in specific engineering reviews or code locations + β”œβ”€ Prepare follow-up question paths + └─ Design response evaluation rubric + ↓ +Output: VivaSessionPlan (VivaTarget list with metadata) +``` + +Each VivaTarget includes: `topic`, `difficulty`, `category`, `focus`, `depth_score`, `confidence`, `related_node`, `reasoning_summary` + +--- + +## Step 5: Live Viva Execution (Interactive Loop) + +``` +For each question in VivaSessionPlan: + β”œβ”€ Present question to candidate + β”œβ”€ Accept candidate response (text) + β”œβ”€ VivaSessionConductor.score_response() + β”‚ β”œβ”€ Specificity score (0–100%) + β”‚ β”œβ”€ Correctness score (0–100%) + β”‚ └─ Quality enum: + β”‚ EXCELLENT | GOOD | ADEQUATE | WEAK | EVASIVE | CONTRADICTION + β”œβ”€ Determine if follow-up is needed + β”‚ └─ If ADEQUATE or WEAK β†’ generate targeted follow-up question + └─ Repeat until 3–4 high-quality responses collected + ↓ +Output: ScoredResponseList +``` + +--- + +## Step 6: Reasoning Pattern Analysis + +``` +Input: ScoredResponseList + ↓ +ReasoningDepthAnalyzer.extract_indicators() + β”œβ”€ Understanding Indicators (7 types): + β”‚ β”œβ”€ EXPLAINS_RATIONALE + β”‚ β”œβ”€ MENTIONS_TRADEOFFS + β”‚ β”œβ”€ HANDLES_EDGE_CASE + β”‚ β”œβ”€ IDENTIFIES_GAPS + β”‚ β”œβ”€ ADMITS_UNCERTAINTY + β”‚ β”œβ”€ INTEGRATES_CONTEXT + β”‚ └─ CITES_SPECIFIC_IMPLEMENTATION + └─ Memorisation Indicators (7 types): + β”œβ”€ TEXTBOOK_LANGUAGE + β”œβ”€ GENERIC_ANSWER + β”œβ”€ FAILS_FOLLOW_UP + β”œβ”€ CONTRADICTS_SELF + β”œβ”€ PARROTS_QUESTION + β”œβ”€ USES_BUZZWORDS + └─ BLANK_ON_EDGE_CASE + ↓ +ReasoningDepthAnalyzer.classify() + β”œβ”€ Reasoning Pattern: DEEP | PRACTICED | INFORMED | LOW | INSUFFICIENT + β”œβ”€ Implementation Familiarity Score (0–1) + └─ Confidence Level: + β”œβ”€ < 2 indicators β†’ LOW / MEDIUM confidence + β”œβ”€ 2–4 indicators β†’ HIGH confidence + └─ > 4 indicators β†’ VERY HIGH confidence + ↓ +Output: ReasoningPatternAssessment +``` + +--- + +## Step 7: Fairness & Trust Audit + +``` +Input: ReasoningPatternAssessment + ScoredResponseList + ↓ +FairnessAuditor.audit() + β”œβ”€ Communication style bias (8 patterns) + β”œβ”€ Demographic bias (8 contexts) + β”œβ”€ Nervous developer pattern detection + β”œβ”€ Confident guesser pattern detection + └─ Non-native speaker separation + ↓ +TrustAuditPipeline.verify() + β”œβ”€ Overconfidence check: Score > 0.95 without 3+ indicators β†’ FLAG + β”œβ”€ Evidence grounding: Every conclusion must link to evidence + β”œβ”€ Contradiction logging + └─ Uncertainty surfacing: Confidence < 0.7 β†’ MEDIUM/LOW flag + ↓ +Output: FairnessAuditReport + TrustAuditResult +``` + +--- + +## Step 8: Final Assessment Generation + +``` +Input: ReasoningPatternAssessment + FairnessAuditReport + TrustAuditResult + ↓ +Synthesise all signals + β”œβ”€ Classification: DEEP_IMPLEMENTATION_FAMILIARITY | PRACTICED | INFORMED | INSUFFICIENT + β”œβ”€ Confidence: HIGH | MEDIUM | LOW (reflects actual signal strength) + β”œβ”€ Evidence trace: Q1β†’[indicators]β†’score, Q2β†’[indicators]β†’score, ... + β”œβ”€ Uncertainty: "Based on X indicators, confidence Y%" + └─ Transcript: Full Q&A with evaluation markers + ↓ +Output: FinalAssessment + Transcript + Explainability +``` + +--- + +## Happy Path Summary + +| Step | Input | Output | +|---|---|---| +| 1 | Repo URL | ExecutionGraph | +| 2 | Concern | CorpusContext | +| 3 | ExecutionGraph | FailureSignalMap | +| 4 | Corpus + Failures | VivaSessionPlan | +| 5 | Questions + Responses | ScoredResponseList | +| 6 | Scored Responses | ReasoningPatternAssessment | +| 7 | Assessment | Fairness + Trust Audit | +| 8 | All signals | FinalAssessment | + +--- + +## Related Docs + +- [Data Flow](./data-flow.md) β€” How data structures move between modules +- [Module Inventory](./module-inventory.md) β€” Which file owns each step +- [Viva Session Flow](../viva-intelligence/session-flow.md) β€” Step 5 in detail diff --git a/Docs/architecture/module-inventory.md b/Docs/architecture/module-inventory.md new file mode 100644 index 0000000..890b5f0 --- /dev/null +++ b/Docs/architecture/module-inventory.md @@ -0,0 +1,107 @@ +# Module Inventory & Ownership + +**Total Codebase:** 14 active modules | ~6,800+ LOC +**Last Updated:** May 2026 + +--- + +## Core Analysis Pipeline β€” 2,740 LOC ← ORACLE Intelligence + +| Module | LOC | Purpose | Status | +|---|---|---|---| +| `viva_session_conductor.py` | 677 | Orchestrates viva sessions; scores responses for specificity, correctness, and quality | βœ… ACTIVE | +| `reasoning_depth_analyzer.py` | 543 | Analyzes response patterns; classifies implementation familiarity level | βœ… ACTIVE | +| `trust_audit.py` | 338 | Verifies evidence grounding; detects overconfidence (>0.95); surfaces uncertainty | βœ… ACTIVE | +| `engineering_review_corpus.py` | 375 | Hardcoded fixture of real engineering review data used for grounding | βœ… ACTIVE | +| `failure_corpus.py` | 807 | Failure pattern scenarios used as probing targets in viva sessions | βœ… ACTIVE | +| `fairness_audit.py` | 350+ | Communication style bias detection; demographic fairness audit | βœ… ACTIVE | + +--- + +## Evaluation & Validation β€” 2,124 LOC + +| Module | LOC | Purpose | Status | +|---|---|---|---| +| `comparative_reasoning_evaluator.py` | 430 | Compares ORACLE outputs against engineering reviews (Precision/Recall) | βœ… ACTIVE | +| `comparative_evaluator.py` | 600 | Multi-dimensional comparative analysis | ⚠️ REVIEW β€” Potential consolidation | +| `execution_behavior_analysis.py` | 466 | Analyzes speculative execution behaviors | ⚠️ REVIEW β€” Speculative, verify usage | +| `evaluator.py` | 628 | Original evaluator (older pattern) | ⚠️ REVIEW β€” Usage unknown | + +> **Note:** Three competing evaluation systems exist. Consolidation into a single `implementation_familiarity_evaluator.py` is planned for Phase 4. + +--- + +## Infrastructure & Data β€” 1,043 LOC + +| Module | LOC | Purpose | Status | +|---|---|---|---| +| `models.py` | 392 | Core data models: ExecutionGraph, EvidenceModel, VivaTarget, etc. | ⚠️ CONSOLIDATE | +| `human_evaluator_models.py` | 341 | Human evaluator session models | ⚠️ CONSOLIDATE | +| `datasets.py` | 173 | Dataset loading and management | βœ… ACTIVE | +| `comparative_calibration_runner.py` | 137 | CLI test runner for calibration pipeline | βœ… ACTIVE | + +> **Note:** Models are split across two files. Consolidation into a single `models.py` with clear sections (CoreModels, SessionModels, EvaluationModels) is planned. + +--- + +## Archived / Dead Code β€” 449 LOC + +| Module | LOC | Issue | Action | +|---|---|---|---| +| `viva_simulation.py` | 449 | Not imported anywhere; duplicates viva_session_conductor | πŸ“Œ ARCHIVED β€” Deprecation notice added | + +--- + +## Intelligence Services β€” Phase 2 + +| Module | Path | Purpose | Status | +|---|---|---|---| +| `observable_signals_engine.py` | `services/intelligence/` | Extracts observable facts: error handling, resilience, observability | βœ… ACTIVE | +| `execution_graph_failure_analyzer.py` | `services/intelligence/` | Traces failure scenarios through execution graph | βœ… ACTIVE | +| `evidence_grounded_viva_generator.py` | `services/intelligence/` | Generates interview questions grounded in code evidence | βœ… ACTIVE | + +--- + +## Calibration Framework β€” `backend/evaluation/calibration/` + +| Module | Purpose | +|---|---| +| `repository_fixtures.py` | 4 diverse test case repositories | +| `signal_validator.py` | Precision/Recall metrics for signal detection | +| `failure_propagation_validator.py` | Validates failure propagation path accuracy | +| `viva_quality_validator.py` | Rejects generic/textbook viva questions | +| `confidence_calibrator.py` | Calibrates confidence scores against actual accuracy | +| `observability.py` | Runtime tracing for all reasoning steps | +| `calibration_runner.py` | Orchestrates full calibration + reporting | + +--- + +## Public API Exports (`__init__.py`) + +**Current:** 30 exported symbols (reduced from 52 β€” 47% reduction βœ…) + +**Core exports:** +- `VivaQuestion`, `CandidateResponse`, `VivaSession` +- `VivaSessionConductor`, `ReasoningDepthAnalyzer` +- `EngineeredReviewEntry`, `FailureCorpusRepository` +- `TrustAuditPipeline`, `FairnessAuditor` +- `ComparativeCalibrationRunner` + +--- + +## Open Issues + +| ID | Issue | Priority | Status | +|---|---|---|---| +| #1 | Multiple evaluation systems (3 competing) | πŸ”΄ HIGH | ⚠️ Consolidation planned | +| #2 | Split model definitions (2 files) | πŸ”΄ HIGH | ⚠️ Consolidation planned | +| #3 | Dead code (viva_simulation.py) | 🟠 MEDIUM | βœ… Archived | +| #4 | API bloat (52 β†’ 30 symbols) | 🟠 MEDIUM | βœ… Resolved | + +--- + +## Related Docs + +- [Execution Flow](./execution-flow.md) β€” How modules chain together at runtime +- [Data Flow](./data-flow.md) β€” How data moves between modules +- [Agent Overview](./agent-overview.md) β€” Agent-level responsibilities diff --git a/Docs/architecture/system-overview.md b/Docs/architecture/system-overview.md new file mode 100644 index 0000000..1447a8e --- /dev/null +++ b/Docs/architecture/system-overview.md @@ -0,0 +1,149 @@ +# ORACLE System Overview + +**Version:** 1.0 β€” Stabilization Phase +**Architecture Status:** πŸ”’ Frozen β€” No new intelligence engines to be added + +--- + +## System Purpose + +ORACLE is an **Implementation Familiarity Assessment** system. It is designed to determine whether a candidate has genuine, hands-on familiarity with a codebase β€” not whether they can recite documentation. + +It uses AST-first analysis, evidence-grounded reasoning, and a structured viva protocol to produce auditable, explainable assessments. + +--- + +## Top-Level Architecture + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ ORACLE Implementation Familiarity System β”‚ +β”‚ Architecture v1.0 (Stable) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + +Browser (ORACLE UI) + β”‚ + β”‚ WebSocket: ws://localhost:8001/ws/analyze + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ FastAPI Server β”‚ +β”‚ backend/src/main.py β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ MainAgent β”‚ +β”‚ Orchestrates: Gatekeeper β†’ Oracle β†’ Sentinel β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β”œβ”€β”€β”€ GatekeeperAgent (identity & access validation) + β”‚ + β”œβ”€β”€β”€ OracleAgent (core analysis pipeline) + β”‚ β”‚ + β”‚ β”œβ”€β”€ AST Parsing & Execution Graph + β”‚ β”œβ”€β”€ ObservableSignalsEngine + β”‚ β”œβ”€β”€ ExecutionGraphFailureAnalyzer + β”‚ β”œβ”€β”€ EvidenceGroundedVivaGenerator + β”‚ └── StructuredContext assembly + β”‚ + └─── SentinelAgent (audit & moderation) +``` + +--- + +## Agent Roles + +| Agent | Responsibility | File | +|---|---|---| +| **MainAgent** | Orchestrates the full pipeline, manages session flow | `backend/src/agents/main_agent/agent.py` | +| **GatekeeperAgent** | Validates identity context, access permissions | `backend/src/agents/gatekeeper/` | +| **OracleAgent** | Core intelligence β€” AST, graphs, signals, viva | `backend/src/agents/oracle/agent.py` | +| **SentinelAgent** | Audit logging, moderation, policy enforcement | `backend/src/agents/sentinel/` | + +--- + +## Core Intelligence Modules + +### Production (ACTIVE) + +| Module | LOC | Purpose | +|---|---|---| +| `viva_session_conductor.py` | 677 | Orchestrate viva sessions, score responses | +| `reasoning_depth_analyzer.py` | 543 | Classify implementation familiarity from indicators | +| `trust_audit.py` | 338 | Verify evidence grounding, detect overconfidence | +| `engineering_review_corpus.py` | 375 | Real engineering review grounding data | +| `failure_corpus.py` | 807 | Failure pattern scenarios (probing targets) | +| `fairness_audit.py` | 350+ | Communication bias & demographic fairness detection | + +### Validation Layer + +| Module | LOC | Purpose | +|---|---|---| +| `comparative_reasoning_evaluator.py` | 430 | Compare ORACLE outputs vs engineering reviews | +| `comparative_evaluator.py` | 600 | Multi-dimensional comparative analysis | +| `signal_validator.py` | β€” | Precision/Recall metrics for observable signals | +| `failure_propagation_validator.py` | β€” | Validates failure scenario propagation paths | +| `viva_quality_validator.py` | β€” | Rejects generic/textbook questions | +| `confidence_calibrator.py` | β€” | Calibrates confidence scores to actual accuracy | + +### Infrastructure + +| Module | LOC | Purpose | +|---|---|---| +| `models.py` | 392 | Core data models | +| `human_evaluator_models.py` | 341 | Human evaluation session models | +| `datasets.py` | 173 | Dataset management | +| `comparative_calibration_runner.py` | 137 | CLI calibration runner | + +--- + +## Data Flow Summary + +``` +Repository URL (Input) + ↓ + Git Clone + AST Parse + ↓ + ExecutionGraph (nodes, edges, types) + ↓ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ Engineering Review Corpus │──→ CorpusContext + β”‚ Failure Corpus │──→ FailureSignalMap + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + ↓ + Question Generation (evidence-grounded) + ↓ + Viva Session (multi-turn Q&A) + ↓ + Response Evaluation + (Specificity, Correctness, Quality enum) + ↓ + Indicator Analysis + (Understanding vs Memorisation) + ↓ + Reasoning Pattern Classification + (DEEP / PRACTICED / INFORMED / LOW / INSUFFICIENT) + ↓ + Trust Audit + Fairness Audit + ↓ + FinalAssessment + Evidence Trace (Output) +``` + +--- + +## Key Design Principles + +1. **AST-First** β€” All analysis starts from actual code structure, not natural language guesses +2. **Evidence-Grounded** β€” Every conclusion maps to specific code locations or review evidence +3. **Explainability** β€” Full indicator trace visible on every assessment +4. **Determinism** β€” Same inputs β†’ same outputs, no hidden stochastic scoring +5. **Architecture Freeze** β€” No new intelligence layers; stabilization only + +--- + +## Related Docs + +- [Module Inventory](./module-inventory.md) β€” Detailed LOC breakdown and ownership +- [Data Flow](./data-flow.md) β€” Full data movement through the system +- [Execution Flow](./execution-flow.md) β€” Step-by-step pipeline walkthrough +- [Agent Overview](./agent-overview.md) β€” Per-agent responsibilities and boundaries diff --git a/Docs/ci-cd/discord-notifications.md b/Docs/ci-cd/discord-notifications.md new file mode 100644 index 0000000..4deb978 --- /dev/null +++ b/Docs/ci-cd/discord-notifications.md @@ -0,0 +1,82 @@ +# Discord Commit Notifications Setup + +This workflow automatically posts commit information to Discord whenever code is pushed to GitHub. + +--- + +## What It Posts + +Every push triggers a formatted Discord message with: + +- βœ… Commit message and short hash +- βœ… Author name and branch +- βœ… Files changed count, lines added/deleted +- βœ… List of changed files (up to 10) +- βœ… Clickable link to the commit on GitHub + +--- + +## Step 1: Create a Discord Webhook + +1. Go to your Discord server +2. Right-click the channel where you want notifications +3. Click **Edit Channel** β†’ **Integrations** β†’ **Webhooks** +4. Click **New Webhook**, give it a name (e.g. "GitHub Bot") +5. Copy the **Webhook URL** + +--- + +## Step 2: Add the Secret to GitHub + +1. Go to `https://github.com/Project-XI/Project-EL/settings/secrets/actions` +2. Click **New repository secret** +3. Name: `DISCORD_WEBHOOK_URL` +4. Value: (paste the webhook URL) +5. Click **Add secret** + +--- + +## Step 3: Verify the Workflow File + +The workflow is at `.github/workflows/discord-notify.yml`. It triggers on pushes to: +- `main` +- `develop` +- `master` + +To change which branches trigger it, edit the `on.push.branches` field. + +--- + +## Customising the Message + +Edit `.github/workflows/discord-notify.yml` to: +- Change message colours (embed `color` field) +- Add or remove fields (repo URL, PR links, etc.) +- Adjust the max number of files shown +- Change the webhook timeout + +--- + +## Troubleshooting + +| Issue | Solution | +|---|---| +| No message in Discord | Confirm `DISCORD_WEBHOOK_URL` is set correctly in GitHub Secrets | +| Webhook error | Check that the webhook URL hasn't expired β€” recreate if needed | +| Action fails | Check Actions logs: Repo β†’ Actions β†’ Find the failed run | +| Secret appears empty | Delete and recreate the secret | + +--- + +## Security + +> ⚠️ **Never** put the Discord webhook URL directly in code or commit messages. +> ⚠️ If you accidentally expose it, delete it in Discord and create a new one. +> ⚠️ Always use GitHub Secrets for any sensitive URLs or tokens. + +--- + +## Related Docs + +- [CI/CD Workflows Overview](./workflows.md) +- [Environment Variables](../deployment/environment-variables.md) diff --git a/Docs/ci-cd/workflows.md b/Docs/ci-cd/workflows.md new file mode 100644 index 0000000..8dca250 --- /dev/null +++ b/Docs/ci-cd/workflows.md @@ -0,0 +1,101 @@ +# CI/CD Workflows + +ORACLE uses GitHub Actions for continuous integration, calibration quality gating, and Discord notifications. + +--- + +## Workflow Overview + +| Workflow | File | Trigger | Purpose | +|---|---|---|---| +| Calibration | `.github/workflows/calibration.yml` | Push / PR to `main` | Run calibration thresholds check | +| Discord Notify | `.github/workflows/discord-notify.yml` | Push to `main`, `develop`, `master` | Post commit info to Discord | + +--- + +## Calibration Workflow + +**File:** `.github/workflows/calibration.yml` + +### What It Does + +1. Sets up Python 3.10 environment +2. Installs backend dependencies +3. Runs the calibration threshold checker +4. Fails the PR if thresholds are not met + +### Threshold Modes + +| Mode | When Applied | +|---|---| +| **Strict** | Merges to `main` branch | +| **Standard** | All other branches / PRs | + +### Running Locally (Before Pushing) + +```bash +cd backend +source .venv/bin/activate +python evaluation/check_calibration_thresholds.py +``` + +Exit `0` = pass. Exit `1` = fail β€” do NOT merge. + +--- + +## Discord Notification Workflow + +**File:** `.github/workflows/discord-notify.yml` + +### What It Does + +Posts a formatted message to Discord on every push: + +``` +πŸ“ New Commit Pushed +"Add fairness audit to Phase 3.5" + +πŸ”— Commit: a1b2c3d +πŸ‘€ Author: rajkoli +🌿 Branch: main +πŸ“Š Stats: Files: 5 | +342 | -12 +πŸ“„ Changed Files: + β€’ backend/src/agents/oracle/fairness_audit.py + β€’ Docs/security/overview.md +``` + +### Setup Instructions + +1. Create a Discord webhook in your channel (Settings β†’ Integrations β†’ Webhooks) +2. Copy the webhook URL +3. Add it to GitHub Secrets as `DISCORD_WEBHOOK_URL` + - Repo β†’ Settings β†’ Secrets β†’ Actions β†’ New secret + +> ⚠️ Never paste the webhook URL in code or commit messages. Always use GitHub Secrets. + +--- + +## Adding a New Workflow + +1. Create a new YAML file in `.github/workflows/` +2. Define triggers (`on: push`, `on: pull_request`, etc.) +3. Define jobs with `runs-on: ubuntu-latest` +4. Test locally using `act` (optional): `brew install act && act push` + +--- + +## Environment Secrets Required + +| Secret | Purpose | Where Used | +|---|---|---| +| `DISCORD_WEBHOOK_URL` | Discord commit notifications | `discord-notify.yml` | +| `GITHUB_TOKEN` | Repo clone access (auto-provided) | All workflows | +| `OPENAI_API_KEY` | LLM calls (if running full analysis) | `calibration.yml` | + +--- + +## Related Docs + +- [Discord Bot Setup](./discord-notifications.md) β€” Full Discord webhook setup guide +- [Calibration Framework](../testing/calibration.md) β€” What the calibration workflow runs +- [Deployment](../deployment/local-setup.md) β€” Running the system locally diff --git a/Docs/contributing/guidelines.md b/Docs/contributing/guidelines.md new file mode 100644 index 0000000..438cae8 --- /dev/null +++ b/Docs/contributing/guidelines.md @@ -0,0 +1,142 @@ +# Contributing to ORACLE + +Thank you for contributing. ORACLE is a precision tool β€” contributions must maintain evidence-grounding, determinism, and architectural clarity. + +--- + +## Before You Start + +Read these docs first: + +1. [System Overview](../architecture/system-overview.md) β€” Understand what ORACLE does +2. [Agent Overview](../architecture/agent-overview.md) β€” Know which agent owns what +3. [Module Inventory](../architecture/module-inventory.md) β€” Check if your feature already exists + +--- + +## Architecture Boundary Rules + +### OracleAgent owns: +- AST parsing and execution graph construction +- Observable signals extraction +- Failure scenario analysis +- Viva question generation +- Response scoring and familiarity classification +- Fairness and trust auditing + +### MainAgent owns: +- Viva session lifecycle and state +- Question sequencing and pacing +- Follow-up strategy (consuming ORACLE outputs β€” NOT recomputing them) +- Transcript persistence +- Topic coverage tracking + +### Never: +- Add speculative AI reasoning to any agent +- Duplicate analysis from one agent into another +- Create new intelligence engines without architecture review +- Add new exports to `__init__.py` without discussion + +--- + +## Code Standards + +### Python + +- Python 3.10+ +- Type hints on all public functions and class attributes +- Pydantic models for all data structures passed between modules +- No bare `except:` clauses +- All public functions must have docstrings + +### Module Design + +- One module = one responsibility +- Prefer small, composable files over monolithic logic +- Every state transition must be auditable (loggable and replayable) +- Every conclusion must have a traceable evidence link + +### Naming Conventions + +Use evidence-grounded terminology β€” no pseudo-psychological framing: + +| ❌ Don't Use | βœ… Use Instead | +|---|---| +| "Builder Detection" | "Implementation Familiarity Analysis" | +| "Deep Builder" | "High Implementation Familiarity" | +| "Memorizer" | "Low Implementation Familiarity" | +| "Builder Confidence" | "Implementation Familiarity Score" | +| "Fake Developer" | "Surface Knowledge Identification" | +| "Reasoning Depth" | "Reasoning Pattern Classification" | + +--- + +## Submitting a Pull Request + +1. **Branch from `main`** + ```bash + git checkout -b feature/your-feature-name + ``` + +2. **Keep scope narrow** β€” one issue, one PR + +3. **Run calibration before pushing** + ```bash + cd backend + python evaluation/check_calibration_thresholds.py + ``` + +4. **Write tests** for any new module or logic change + +5. **Update Docs/** if your change affects architecture, APIs, or agent behaviour + +6. **PR title format:** `[Agent] Brief description` + Examples: `[Oracle] Add token expiry signal detector`, `[Main] Session state serializer` + +--- + +## PR Review Checklist + +Before requesting review, confirm: + +- [ ] The issue is narrow and non-overlapping with existing modules +- [ ] The code does not perform hidden reasoning or scoring +- [ ] The code uses ORACLE outputs β€” not recomputed intelligence +- [ ] The code is deterministic and replayable +- [ ] All new conclusions have evidence links +- [ ] Terminology uses the evidence-grounded naming convention +- [ ] Tests pass locally +- [ ] Calibration thresholds pass +- [ ] Docs are updated if behaviour changed + +--- + +## Reviewer Expectations + +Before merging any PR, reviewers confirm: + +- The code does not look like a generic chat assistant +- The code does not infer hidden reasoning +- The change belongs in the right agent (Oracle vs Main vs Gatekeeper vs Sentinel) +- The change can be explained by a contributor in one paragraph +- No pseudo-psychological language remains + +--- + +## Issue Reporting + +When filing a bug or feature request: + +1. **Title:** Brief, specific description +2. **Context:** What agent/module is affected? +3. **Expected behaviour:** What should happen? +4. **Actual behaviour:** What happens instead? +5. **Evidence:** Log output, test case, or code reference + +--- + +## Related Docs + +- [MAIN Agent Issue Breakdown](./main-agent-issues.md) β€” Pre-scoped contributor issues +- [Module Inventory](../architecture/module-inventory.md) β€” Ownership reference +- [Testing Framework](../testing/human-testing-protocol.md) β€” How to validate changes diff --git a/Docs/contributing/main-agent-issues.md b/Docs/contributing/main-agent-issues.md new file mode 100644 index 0000000..3bbf22f --- /dev/null +++ b/Docs/contributing/main-agent-issues.md @@ -0,0 +1,221 @@ +# MAIN Agent β€” Contributor Issue Breakdown + +This document defines the 6 pre-scoped contributor issues for the MAIN Agent. Each issue owns exactly one responsibility and must not expand into other agent domains. + +> **Rule:** MAIN orchestrates. It does not analyse. All intelligence comes from ORACLE outputs. + +--- + +## Issue 1: Session State Manager + +**Goal:** Build a persistent viva session state manager. + +### Responsibilities +- Track session lifecycle stages (init β†’ active β†’ complete β†’ closed) +- Store asked question history (ordered) +- Store candidate response history (mapped to correct turns) +- Retain contradiction memory across turns +- Maintain weak-area and topic coverage state +- Track follow-up chains +- Support session recovery after interruption + +### Technical Requirements +- JSON serializable state model +- Multi-turn session support +- Deterministic state mutations +- Replay-safe and order-stable +- No database coupling (use storage interface) + +### Suggested File Structure +``` +backend/src/agents/main_agent/session/ + state.py + transitions.py + history.py + persistence.py + +backend/src/agents/main_agent/models/ + session_state.py + transcript_entry.py + coverage_state.py +``` + +### Acceptance Criteria +- Session state can be saved and restored without data loss +- Asked questions preserved in order +- Responses mapped to correct question turns +- Session transitions unit-testable in isolation + +--- + +## Issue 2: Viva Flow Orchestrator + +**Goal:** Build the central viva flow orchestrator for session progression. + +### Responsibilities +- Sequence viva questions from ORACLE VivaTargets +- Manage pacing and turn progression +- Trigger session state transitions +- Inject follow-up branches when quality is weak +- Balance topic categories across session +- Decide when to terminate the viva +- Keep all decisions deterministic and replayable + +### Technical Requirements +- Consumes ORACLE outputs as inputs β€” does NOT recompute them +- Avoids repetitive questioning +- Deterministic for same state + inputs +- No UI dependencies + +### Suggested File Structure +``` +backend/src/agents/main_agent/orchestration/ + flow_orchestrator.py + pacing.py + termination.py + branching.py + category_balancer.py +``` + +--- + +## Issue 3: Follow-Up Question Strategy Engine + +**Goal:** Build an evidence-grounded follow-up strategy layer. + +### Responsibilities +- Detect shallow or generic responses +- Escalate depth when quality is WEAK or ADEQUATE +- Probe implementation familiarity using ORACLE signals +- Challenge contradictions when responses conflict +- Generate operational follow-ups grounded in failure scenarios +- Preserve evidence references for every follow-up generated + +### Technical Requirements +- Consumes ORACLE observable signals and failure scenarios +- Follow-up selection deterministic for same inputs +- No textbook-style generic prompts +- No invented facts not present in shared inputs + +### Suggested File Structure +``` +backend/src/agents/main_agent/followups/ + strategy_engine.py + patterns.py + contradiction_probe.py + weak_answer_detector.py + evidence_mapper.py +``` + +--- + +## Issue 4: Topic Coverage Tracker + +**Goal:** Track which implementation domains have been covered to prevent repetitive questioning. + +### Responsibilities +- Track architecture coverage +- Track runtime reasoning coverage +- Track failure-analysis coverage +- Track scalability and security coverage +- Detect unanswered or under-covered topics +- Reduce repetitive questioning + +### Suggested File Structure +``` +backend/src/agents/main_agent/coverage/ + tracker.py + categories.py + heuristics.py + coverage_state.py +``` + +--- + +## Issue 5: Transcript Persistence Layer + +**Goal:** Persist the full viva transcript for audit, replay, and export. + +### Responsibilities +- Store question and answer entries (ordered) +- Log contradiction events with session step reference +- Log fairness events from SentinelAgent annotations +- Support JSON export for downstream tools +- Retain event ordering +- Preserve evidence links + +### Technical Requirements +- JSON exportable +- Replay-safe (same turn history from persisted state) +- No UI logic +- Does not mutate viva decisions + +### Suggested File Structure +``` +backend/src/agents/main_agent/transcript/ + store.py + serializer.py + replay.py + event_log.py + schemas.py +``` + +--- + +## Issue 6: ORACLE Integration Adapter + +**Goal:** Create a normalised adapter between ORACLE outputs and MAIN orchestration. + +### Responsibilities +- Normalise ORACLE StructuredContext outputs +- Expose stable fields: viva targets, observable signals, failure scenarios +- Validate schema shape (strict, explicit) +- Handle malformed payloads safely +- Be the ONLY place where ORACLE schema differences are handled + +### Technical Requirements +- Does NOT duplicate ORACLE analysis logic +- Strict schema validation +- Malformed payloads fail safely (not silently) +- Independently testable + +### Suggested File Structure +``` +backend/src/agents/main_agent/integration/ + oracle_adapter.py + oracle_schema.py + payload_normalizer.py + validation.py + compatibility.py +``` + +--- + +## Contributor Rules (All Issues) + +1. Keep MAIN orchestration-focused β€” no analysis +2. Preserve evidence grounding at every decision point +3. Do not add speculative AI systems +4. Do not duplicate ORACLE intelligence logic +5. Keep modules small, composable, and independently testable +6. Make every state transition auditable +7. If a change looks like analysis β†’ move to OracleAgent +8. If it looks like moderation β†’ move to SentinelAgent + +--- + +## Review Standard + +Before merging any MAIN Agent work: +- [ ] Issue is narrow and non-overlapping +- [ ] Code does not infer hidden reasoning +- [ ] Code uses ORACLE outputs rather than recomputing them +- [ ] Code is deterministic and replayable +- [ ] Code can be explained in one paragraph + +--- + +## Related Docs + +- [Agent Overview](../architecture/agent-overview.md) β€” Agent boundary rules +- [Contributing Guidelines](./guidelines.md) β€” General contribution standards diff --git a/Docs/deployment/environment-variables.md b/Docs/deployment/environment-variables.md new file mode 100644 index 0000000..d6c162a --- /dev/null +++ b/Docs/deployment/environment-variables.md @@ -0,0 +1,88 @@ +# Environment Variables Reference + +All sensitive configuration must be set via environment variables. Never hardcode secrets in source files. + +--- + +## Setup + +```bash +cd backend +cp .env.example .env +``` + +Edit `.env` with your values. + +--- + +## Variables Reference + +### LLM / AI + +| Variable | Required | Description | +|---|---|---| +| `OPENAI_API_KEY` | βœ… Yes | OpenAI API key for LLM-powered question generation and scoring | + +--- + +### GitHub + +| Variable | Required | Description | +|---|---|---| +| `GITHUB_TOKEN` | βœ… Yes | Personal Access Token for cloning repositories | + +**Required scope:** `repo:read` (for public repos) or `repo` (for private repos) + +To create a token: GitHub β†’ Settings β†’ Developer Settings β†’ Personal Access Tokens β†’ Fine-grained tokens + +--- + +### Server + +| Variable | Required | Default | Description | +|---|---|---|---| +| `HOST` | ❌ No | `localhost` | Bind host for FastAPI server | +| `PORT` | ❌ No | `8001` | Port for FastAPI WebSocket server | + +--- + +### Discord (CI/CD only) + +| Variable | Set In | Description | +|---|---|---| +| `DISCORD_WEBHOOK_URL` | GitHub Secrets | Discord webhook URL for commit notifications | + +**Never put this in `.env`** β€” use GitHub Secrets only. + +--- + +## Example `.env.example` + +```env +# LLM +OPENAI_API_KEY=sk-...your-key-here... + +# GitHub (for repo cloning) +GITHUB_TOKEN=ghp_...your-token-here... + +# Server +HOST=localhost +PORT=8001 +``` + +--- + +## Security Rules + +1. `.env` is in `.gitignore` β€” **never commit it** +2. If you accidentally expose `OPENAI_API_KEY` or `GITHUB_TOKEN`, revoke them immediately +3. For production, use a secrets manager (e.g. AWS Secrets Manager, HashiCorp Vault) β€” not `.env` files +4. For CI/CD, all secrets must be in GitHub repository Secrets, not workflow YAML files + +--- + +## Related Docs + +- [Installation Guide](../getting-started/installation.md) +- [CI/CD Workflows](../ci-cd/workflows.md) β€” GitHub Secrets setup +- [Discord Notifications](../ci-cd/discord-notifications.md) β€” DISCORD_WEBHOOK_URL setup diff --git a/Docs/error-handling/bias-mitigation.md b/Docs/error-handling/bias-mitigation.md new file mode 100644 index 0000000..f0372d5 --- /dev/null +++ b/Docs/error-handling/bias-mitigation.md @@ -0,0 +1,155 @@ +# Bias Mitigation + +ORACLE is built to prevent systematic bias in implementation familiarity assessments. This document explains the bias risks, detection mechanisms, and mitigation strategies. + +--- + +## What Bias Looks Like in ORACLE + +Without bias mitigation, an assessment system can: + +- Mark a **nervous but knowledgeable** engineer as LOW familiarity (False Positive) +- Mark a **confident but shallow** speaker as HIGH familiarity (False Negative) +- Systematically underestimate **non-native English speakers** who have deep technical knowledge +- Favour **conventional coding styles** over equally valid unconventional approaches + +These are not hypothetical risks β€” they are documented patterns in interview assessment research. + +--- + +## Mitigation Layer: FairnessAuditor + +**Module:** `fairness_audit.py` +**Integrated into:** OracleAgent pipeline (after TrustAudit, before FinalAssessment) + +The FairnessAuditor runs 6 detection patterns on every session. + +### Pattern 1: Nervous Developer False Positive + +| Aspect | Detail | +|---|---| +| **Risk** | Builder marked LOW due to hedging communication, not lack of knowledge | +| **Detection** | Nervous language + high understanding indicators + low memorisation indicators | +| **Action** | Manual review recommended; score not automatically penalised | +| **Output** | `FairnessAuditIssue(type="NERVOUS_DEVELOPER", severity="MEDIUM")` | + +--- + +### Pattern 2: Confident Guesser False Negative + +| Aspect | Detail | +|---|---| +| **Risk** | Non-builder marked HIGH due to confident delivery and buzzword fluency | +| **Detection** | High surface confidence + TEXTBOOK_LANGUAGE + USES_BUZZWORDS + zero understanding indicators | +| **Action** | Familiarity confidence score reduced; false negative risk flagged | +| **Output** | `FairnessAuditIssue(type="CONFIDENT_GUESSER", severity="HIGH")` | + +--- + +### Pattern 3: Overconfidence Correction + +| Aspect | Detail | +|---|---| +| **Risk** | System claims >95% confidence with insufficient evidence | +| **Detection** | `implementation_familiarity_score > 0.95` with fewer than 4 indicators | +| **Action** | Score reduced to ≀ 0.85; flagged as CRITICAL | +| **Output** | `TrustAuditResult(overconfident=True, adjusted_score=0.85)` | + +--- + +### Pattern 4: Insufficient Evidence Correction + +| Aspect | Detail | +|---|---| +| **Risk** | HIGH/MEDIUM confidence on thin evidence (< 2 indicators) | +| **Detection** | Confidence MEDIUM or HIGH with < 2 total indicators | +| **Action** | Confidence reduced to LOW; marked INSUFFICIENT_DATA | +| **Output** | `TrustAuditResult(insufficient_evidence=True, adjusted_confidence="LOW")` | + +--- + +### Pattern 5: Demographic Bias + +| Aspect | Detail | +|---|---| +| **Risk** | Non-native speaker, neurodivergent, or early-career communicators systematically underscored | +| **Detection** | Response contains demographic communication markers; score correlates with communication style | +| **Action** | Manual review recommended; fluency explicitly separated from technical depth in report | +| **Output** | `FairnessAuditIssue(type="DEMOGRAPHIC_BIAS", context="non_native_speaker")` | + +--- + +### Pattern 6: Communication Style Correlation + +| Aspect | Detail | +|---|---| +| **Risk** | Statistically significant correlation between communication traits and familiarity score | +| **Detection** | Communication style features correlate with final score above threshold | +| **Action** | Bias surfaced explicitly in report; score adjusted if correlation exceeds threshold | +| **Output** | `FairnessAuditIssue(type="COMMUNICATION_BIAS", severity="HIGH")` | + +--- + +## Separation of Fluency from Familiarity + +ORACLE is explicitly designed to score **content**, not **delivery**: + +| What We Score | What We Do NOT Score | +|---|---| +| Specific implementation references | Eloquence or vocabulary | +| Mention of actual tradeoffs | Accent or phrasing | +| Knowledge of production incidents | Speed of response | +| Correct edge case identification | Confidence level of delivery | +| Ability to cite specific files or decisions | Grammar correctness | + +--- + +## Fairness Audit Report Schema + +```json +{ + "issues": [ + { + "type": "NERVOUS_DEVELOPER", + "severity": "MEDIUM", + "description": "Candidate shows understanding indicators despite nervous delivery", + "recommendation": "Manual review recommended β€” do not penalise for communication style" + } + ], + "manual_review_recommended": true, + "confidence_adjusted": false, + "bias_types_detected": ["COMMUNICATION_STYLE"] +} +``` + +--- + +## Validation Results (Phase 1 Testing) + +| Test Case | Expected | Actual | Pass? | +|---|---|---|---| +| Nervous builder β†’ HIGH familiarity | Detects nervous pattern, manual review | βœ… Correct | βœ… | +| Confident guesser β†’ LOW familiarity | Detects false negative risk | βœ… Correct | βœ… | +| Non-native speaker builder | Flags demographic bias, separates fluency | βœ… Correct | βœ… | + +--- + +## Fairness Audit Checklist + +Run before every production release: + +- [ ] FairnessAuditor runs on 100% of sessions +- [ ] Overconfidence threshold (>0.95) not disabled +- [ ] Nervous developer pattern detection active +- [ ] Confident guesser pattern detection active +- [ ] Demographic bias flags generating correctly +- [ ] Manual review recommendations surfaced in report +- [ ] Communication style does not correlate with score in test set + +--- + +## Related Docs + +- [Security Overview](../security/overview.md) β€” Trust audit + access control +- [Human Testing Protocol](../testing/human-testing-protocol.md) β€” How bias is measured empirically +- [Viva Intelligence](../viva-intelligence/viva-overview.md) β€” How indicators are designed to separate understanding from style diff --git a/Docs/error-handling/known-issues.md b/Docs/error-handling/known-issues.md new file mode 100644 index 0000000..3468922 --- /dev/null +++ b/Docs/error-handling/known-issues.md @@ -0,0 +1,174 @@ +# Known Issues & Error Handling + +This document tracks known system limitations, error patterns, and the mitigation strategies in place for each. + +--- + +## Active Known Issues + +### Issue 1: Multiple Evaluation Systems (3 Competing) + +| Field | Detail | +|---|---| +| **Priority** | πŸ”΄ HIGH | +| **Status** | ⚠️ Consolidation planned β€” Phase 4 | +| **Impact** | Confusing ownership; possible behavioural divergence | + +**Problem:** +Three evaluation modules exist with overlapping purposes: +- `evaluator.py` β€” Original pattern, usage unknown +- `comparative_evaluator.py` β€” Newer, multi-dimensional +- `execution_behavior_analysis.py` β€” Speculative behaviours + +**Mitigation (current):** `comparative_reasoning_evaluator.py` is the authoritative path. Others are under review. + +**Resolution (planned):** Consolidate into single `implementation_familiarity_evaluator.py` + +--- + +### Issue 2: Split Model Definitions + +| Field | Detail | +|---|---| +| **Priority** | πŸ”΄ HIGH | +| **Status** | ⚠️ Consolidation planned β€” Phase 4 | +| **Impact** | Schema inconsistencies; import confusion | + +**Problem:** +Models are split across two files: +- `models.py` β†’ 11 classes (ExecutionGraph, etc.) +- `human_evaluator_models.py` β†’ 16 classes (HumanEvaluationSession, etc.) + +**Resolution (planned):** Merge into single `models.py` with sections: `CoreModels`, `SessionModels`, `EvaluationModels` + +--- + +### Issue 3: Difficulty Terminology Mismatch + +| Field | Detail | +|---|---| +| **Priority** | 🟑 LOW | +| **Status** | Acceptable β€” Phase 4 cleanup | +| **Impact** | Cosmetic only | + +**Problem:** Agent emits `"foundational"` for viva difficulty; UI `diffTag()` expects `"easy"`. + +**Mitigation:** UI handles gracefully with `.tag.pass` CSS fallback. + +**Resolution (planned):** Standardise to `"easy|medium|hard"` across all enums in Phase 4. + +--- + +### Issue 4: Observable Signals Not Displayed in UI + +| Field | Detail | +|---|---| +| **Priority** | 🟑 LOW | +| **Status** | Enhancement opportunity | +| **Impact** | Data available but unused | + +**Problem:** OracleAgent produces `observable_signals` in StructuredContext but the UI doesn't render them. + +**Resolution (planned):** Add observable signals visualisation panel to UI in a future enhancement. + +--- + +### Issue 5: Dual Viva Target Field Names + +| Field | Detail | +|---|---| +| **Priority** | 🟑 LOW | +| **Status** | Acceptable for legacy support | +| **Impact** | No functional impact | + +**Problem:** Agent sets both `implementation_viva_targets` (legacy) and `viva_intelligence_targets` (current). UI reads `viva_intelligence_targets` first with fallback. + +**Resolution (planned):** Remove legacy field in Phase 4. + +--- + +### Issue 6: Real Human Testing Not Yet Conducted + +| Field | Detail | +|---|---| +| **Priority** | πŸ”΄ HIGH | +| **Status** | Phase 4 planned | +| **Impact** | Accuracy rates unknown on real humans | + +**Mitigation:** Automated calibration (Precision ~0.84, RMSE 0.062) provides confidence in component accuracy. + +**Resolution:** [4-phase human testing protocol](../testing/human-testing-protocol.md) ready to execute. + +--- + +## Error Recovery Behaviour + +### Repository Clone Failure + +``` +Trigger: Invalid URL, missing GITHUB_TOKEN, private repo without access +Response: WebSocket log error message β†’ connection closed +Recovery: Check GITHUB_TOKEN scope, verify repo URL +``` + +### AST Parsing Failure + +``` +Trigger: Unsupported language, malformed code, encrypted files +Response: ExecutionGraph returned with 0 nodes +Recovery: Check language support; ORACLE will still generate corpus-only questions +``` + +### LLM API Failure + +``` +Trigger: OPENAI_API_KEY invalid, rate limit, timeout +Response: WebSocket error log β†’ analysis aborted +Recovery: Check API key, retry after rate limit window +``` + +### Calibration Threshold Failure (CI) + +``` +Trigger: Metric below threshold (e.g. Precision < 0.75) +Response: CI job fails, PR blocked +Recovery: Fix the regression in the relevant engine before merging +``` + +--- + +## Error Codes + +| Error Pattern | Likely Cause | +|---|---| +| `Failed to clone repository` | GitHub token missing or insufficient scope | +| `AST parse error` | Unsupported file type or encoding issue | +| `Corpus context empty` | No matching engineering reviews for the concern | +| `Confidence RMSE above threshold` | Calibration regression β€” check last code change | +| `Overconfidence detected` | Score > 0.95 with < 4 indicators β€” TrustAudit flagged | +| `Insufficient evidence` | Session too short; < 2 indicators collected | + +--- + +## What to Do When Calibration Fails + +1. Run calibration locally and read the full report: + ```bash + cd backend + python -m evaluation.calibration.calibration_runner + ``` + +2. Identify which validator failed (Signal, Failure, Viva, Confidence) + +3. Check the last code change to that validator's related module + +4. Fix the regression and re-run before pushing + +--- + +## Related Docs + +- [Calibration Framework](../testing/calibration.md) +- [Bias Mitigation](./bias-mitigation.md) +- [Security Overview](../security/overview.md) +- [Module Inventory](../architecture/module-inventory.md) diff --git a/Docs/frontend/ui-overview.md b/Docs/frontend/ui-overview.md new file mode 100644 index 0000000..3d15dcb --- /dev/null +++ b/Docs/frontend/ui-overview.md @@ -0,0 +1,125 @@ +# Frontend & UI Overview + +ORACLE has two UI surfaces: the main Analysis Dashboard and the Calibration Dashboard. + +--- + +## Main Analysis Dashboard + +**File:** `frontend/index.html` (also mirrored in `Docs/index.html`) +**Style:** `frontend/style.css` +**Logic:** `frontend/app.js` +**Orb Animation:** `frontend/orb.js` + +### What It Shows + +| Panel | Data Source | Description | +|---|---|---| +| **Backend Framework Card** | `payload.backend_framework.value` | Detected framework (e.g. FastAPI) | +| **Architecture Pattern Card** | `payload.architecture_pattern.value` | Detected architecture (e.g. REST + WebSocket) | +| **Authentication System Card** | `payload.authentication_system.value` | Detected auth (e.g. JWT Bearer) | +| **Graph Integrity Card** | `payload.execution_graph.nodes.length` | Number of graph nodes found | +| **Execution Graph** | `payload.execution_graph` | Interactive node/edge diagram (Mermaid) | +| **Viva Intelligence List** | `payload.viva_intelligence_targets` | Evidence-grounded interview questions | +| **Anomalies & Failure Paths** | `payload.runtime_risks` + `payload.failure_paths` | Risk severity and propagation paths | +| **Benchmark Results Table** | `payload.evaluation_metrics` | Accuracy vs ground truth | +| **Terminal Log Panel** | WebSocket `type: "log"` messages | Real-time streaming agent progress | + +--- + +## UI β†’ Backend Connection + +``` +Browser opens WebSocket: ws://localhost:8001/ws/analyze + ↓ +User enters repo URL, clicks Analyze + ↓ +UI sends: { "repo_url": "https://github.com/..." } + ↓ +Backend streams: { "type": "log", ... } (many messages) + ↓ +Backend sends: { "type": "result", "data": {...} } + ↓ +UI renders all panels from data +``` + +--- + +## Viva Intelligence Card Rendering + +Each viva target renders as a card with: + +| Field | Display | +|---|---| +| `category` | Badge (Architecture / Security / Runtime / etc.) | +| `difficulty` | Coloured tag (hard = red / medium = yellow / foundational = green) | +| `topic` | Card title | +| `question_target` | Full question text | +| `depth_score` | Score bar (0–10) | +| `confidence` | Percentage shown | +| `reasoning_summary` | Evidence summary text | +| `related_node` | Graph node ID link | + +> ⚠️ **Minor Alignment Note:** The agent may emit `"foundational"` for difficulty, while the UI `diffTag()` function expects `"easy"`. The UI fallback handles this gracefully with a `.tag.pass` class β€” no functional impact. Standardisation to `"easy|medium|hard"` is planned for Phase 4. + +--- + +## Execution Graph Visualisation + +Rendered using Mermaid.js from the `execution_graph` payload: + +```javascript +// Node types map to shapes: +ROUTE β†’ rounded rectangle +MIDDLEWARE β†’ parallelogram +DB_QUERY β†’ cylinder +STATE_STORE β†’ database shape +AUTH_HANDLER β†’ hexagon +``` + +Each node is clickable and shows: `file_path`, `line_number`, `snippet` + +--- + +## Calibration Dashboard + +**File:** `backend/testing_oracle_ui/calibration_dashboard.html` + +Open directly in browser (no server required). Displays: +- Signal precision/recall bar charts +- Confidence calibration curve (expected vs actual accuracy) +- Viva question validity scores +- Per-repository fixture breakdown + +--- + +## UI Alignment Status (Phase 3.5 Audit) + +| Component | Status | Notes | +|---|---|---| +| WebSocket connection | βœ… ALIGNED | Endpoint, input format, output format | +| Dashboard cards | βœ… ALIGNED | All EvidenceModel fields match | +| Execution graph | βœ… ALIGNED | Nodes, edges, types, metadata | +| Viva intelligence output | βœ… ALIGNED | All VivaTarget fields match (minor difficulty terminology note) | +| Anomalies & failure paths | βœ… ALIGNED | runtime_risks and failure_paths populated | +| Benchmark results table | βœ… ALIGNED | evaluation_metrics from OracleEvaluator | +| Terminal logs | βœ… ALIGNED | log_type colouring works | +| Observable signals panel | 🟑 UNUSED | Data available but not displayed; enhancement opportunity | + +--- + +## Known Minor Issues + +| Issue | Impact | Resolution | +|---|---|---| +| Difficulty: `"foundational"` vs `"easy"` | Cosmetic only β€” UI handles gracefully | Standardise to `"easy"` in Phase 4 | +| Observable signals not displayed | Data available, no rendering | Add signals panel in future enhancement | +| Dual viva target field names | No impact β€” UI reads both | Consolidate to single field in Phase 4 | + +--- + +## Related Docs + +- [WebSocket API](../api-docs/websocket-api.md) β€” Full payload schema +- [Data Flow](../architecture/data-flow.md) β€” How data reaches the UI +- [Deployment](../deployment/local-setup.md) β€” How to run the UI locally diff --git a/Docs/getting-started/README.md b/Docs/getting-started/README.md new file mode 100644 index 0000000..7f2dfc0 --- /dev/null +++ b/Docs/getting-started/README.md @@ -0,0 +1,80 @@ +# Getting Started with ORACLE + +ORACLE (Optimized Repository Analysis & Candidate Learning Engine) is an evidence-grounded Implementation Familiarity Assessment system. It analyzes a GitHub repository and conducts a structured viva (technical interview) to measure a candidate's real implementation familiarity β€” not memorised knowledge. + +--- + +## What ORACLE Does + +1. **Clones and parses** a repository via AST analysis +2. **Builds an execution graph** tracing routes, middleware, DB calls, auth, and state +3. **Extracts observable signals** β€” error handling, resilience patterns, observability hooks +4. **Detects failure scenarios** by traversing the execution graph +5. **Generates grounded viva questions** from real code evidence +6. **Conducts a viva session** evaluating depth of understanding vs. surface-level recitation +7. **Produces a trust-audited assessment** with confidence scores and full evidence traces + +--- + +## Quick Navigation + +| Where do you want to go? | Link | +|---|---| +| Set up locally in 5 minutes | [Installation Guide](./installation.md) | +| Understand the full architecture | [Architecture Overview](../architecture/system-overview.md) | +| Run the calibration pipeline | [Calibration Quickstart](../testing/calibration.md) | +| Understand the viva session flow | [Viva Intelligence](../viva-intelligence/viva-overview.md) | +| Read the API contracts | [API Docs](../api-docs/websocket-api.md) | +| Contribute to the project | [Contributing Guide](../contributing/guidelines.md) | + +--- + +## System Requirements + +| Requirement | Minimum | +|---|---| +| Python | 3.10+ | +| Node.js | 18+ (for dashboard UI) | +| RAM | 4 GB | +| OS | macOS / Linux / WSL2 | +| GitHub Access | Personal Access Token (for private repos) | + +--- + +## High-Level Architecture + +``` +Browser (ORACLE UI) + ↓ WebSocket +FastAPI Backend (main.py :8001) + ↓ +MainAgent β†’ GatekeeperAgent β†’ OracleAgent β†’ SentinelAgent + ↓ + Intelligence Pipeline + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ Observable Signals Engine β”‚ + β”‚ Execution Graph Failure β”‚ + β”‚ Evidence-Grounded Viva Gen β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + ↓ + StructuredContext (JSON) + ↓ + WebSocket Result +``` + +--- + +## Project Status + +| Phase | Status | +|---|---| +| Phase 1 – Architecture Assessment & Cleanup | βœ… Complete | +| Phase 2 – Evidence-Grounded Intelligence | βœ… Complete | +| Phase 3.5 – Stabilization & Reality Hardening | πŸ”„ In Progress | +| Phase 4 – Real Human Testing | ⏳ Planned | + +--- + +## Next Step + +β†’ [Installation Guide](./installation.md) diff --git a/Docs/getting-started/installation.md b/Docs/getting-started/installation.md new file mode 100644 index 0000000..c3400c7 --- /dev/null +++ b/Docs/getting-started/installation.md @@ -0,0 +1,119 @@ +# Installation Guide + +This guide walks you through setting up ORACLE locally from scratch. + +--- + +## Prerequisites + +- Python 3.10+ +- `pip` and `venv` +- Git + +--- + +## Step 1: Clone the Repository + +```bash +git clone https://github.com/Project-XI/Project-EL.git +cd Project-EL +``` + +--- + +## Step 2: Set Up Python Environment + +```bash +cd backend +python -m venv .venv +source .venv/bin/activate # On Windows: .venv\Scripts\activate +pip install -r requirements.txt +``` + +--- + +## Step 3: Configure Environment Variables + +```bash +cp .env.example .env +``` + +Open `.env` and fill in: + +```env +# LLM Configuration +OPENAI_API_KEY=your_key_here + +# GitHub (for private repo access) +GITHUB_TOKEN=your_github_pat + +# WebSocket Server +HOST=localhost +PORT=8001 +``` + +--- + +## Step 4: Run the Backend Server + +```bash +cd backend +uvicorn src.main:app --host 0.0.0.0 --port 8001 --reload +``` + +The WebSocket endpoint will be live at: +``` +ws://localhost:8001/ws/analyze +``` + +--- + +## Step 5: Open the ORACLE UI + +Open `frontend/index.html` (or `Docs/index.html`) directly in your browser β€” no build step required. + +Enter a GitHub repository URL and click **Analyze** to start a session. + +--- + +## Step 6: (Optional) Run Calibration + +```bash +cd backend +python -m evaluation.calibration.calibration_runner +``` + +Or use the threshold checker: + +```bash +python evaluation/check_calibration_thresholds.py +``` + +--- + +## Verifying the Setup + +If the setup is correct: +- The browser UI shows the ORACLE dashboard +- Entering a GitHub repo URL starts the analysis +- WebSocket logs stream in the terminal panel +- The execution graph and viva questions appear after ~30 seconds + +--- + +## Troubleshooting + +| Problem | Solution | +|---|---| +| `ModuleNotFoundError` | Activate virtualenv: `source .venv/bin/activate` | +| `Connection refused` on WebSocket | Make sure backend is running on port 8001 | +| Empty execution graph | Check `GITHUB_TOKEN` has `repo` read permissions | +| Calibration fails | Ensure all evaluation fixtures exist in `backend/evaluation/` | + +--- + +## Next Steps + +- [Understanding the Architecture](../architecture/system-overview.md) +- [Running a Viva Session](../viva-intelligence/session-flow.md) +- [Setting Up CI/CD](../ci-cd/workflows.md) diff --git a/Docs/monitoring-logging/observability.md b/Docs/monitoring-logging/observability.md new file mode 100644 index 0000000..c32245f --- /dev/null +++ b/Docs/monitoring-logging/observability.md @@ -0,0 +1,122 @@ +# Monitoring & Observability + +ORACLE includes a runtime observability layer that traces all reasoning steps during analysis β€” making every decision inspectable and debuggable. + +--- + +## Observability Stack + +| Component | Module | Purpose | +|---|---|---| +| Runtime Tracer | `calibration/observability.py` | Records reasoning steps as structured traces | +| Calibration Dashboard | `testing_oracle_ui/calibration_dashboard.html` | Visual trace and metric viewer | +| Calibration Runner | `calibration/calibration_runner.py` | Generates reports with all metric data | +| Log Callback | `main.py` β†’ WebSocket | Real-time terminal streaming | + +--- + +## Runtime Tracing + +Every reasoning step in the ORACLE pipeline emits a structured trace record: + +```json +{ + "step": "signal_extraction", + "module": "ObservableSignalsEngine", + "timestamp": "2026-05-18T12:34:56Z", + "inputs": { "node_id": "node_042", "node_type": "DB_QUERY" }, + "outputs": { "signal": "Missing error handler on DB call", "severity": "HIGH" }, + "evidence": ["backend/src/routes/users.py:88"], + "duration_ms": 124 +} +``` + +Trace records are collected during analysis and exported as JSON. + +--- + +## WebSocket Log Streaming + +During analysis, progress is streamed to the browser UI in real time: + +``` +[Gatekeeper] Identity context established +[Oracle] Cloning repository... +[Oracle] AST parsing complete β€” 142 nodes found +[Oracle] Building execution graph... +[Oracle] Observable signals extraction... +[Oracle] 3 signals detected (HIGH: 1, MEDIUM: 2) +[Oracle] Failure scenario analysis... +[Oracle] 2 failure paths identified +[Oracle] Generating viva questions... +[Oracle] 6 evidence-grounded questions created +[Oracle] Assembling StructuredContext... +[Sentinel] Audit log written +βœ… Analysis complete +``` + +Log types: `info` | `warn` | `success` | `error` + +--- + +## Calibration Dashboard + +**File:** `backend/testing_oracle_ui/calibration_dashboard.html` + +Open in browser to view: +- Signal precision/recall charts +- Confidence calibration curves +- Viva quality validity scores +- Per-repository fixture performance +- Trend data from multiple runs + +--- + +## Key Metrics to Monitor + +| Metric | Target | Action if Below | +|---|---|---| +| Signal Precision | β‰₯ 0.85 (strict) | Review ObservableSignalsEngine detection logic | +| Signal Recall | β‰₯ 0.80 (strict) | Check corpus coverage for missing signal types | +| Failure Propagation Precision | β‰₯ 0.80 | Verify graph edge extraction completeness | +| Viva Validity | β‰₯ 0.85 | Review EvidenceGroundedVivaGenerator specificity filters | +| Confidence RMSE | ≀ 0.10 | Recalibrate confidence scoring weights | +| Overconfidence instances | 0 | Check TrustAuditPipeline threshold | + +--- + +## Exporting Traces + +```bash +cd backend +python -m evaluation.calibration.calibration_runner +``` + +Output files (generated in `backend/evaluation/results/`): +- `calibration_report_.json` β€” Full metric report +- `traces_.json` β€” Raw reasoning trace records +- `validation_report.html` β€” Human-readable HTML report + +--- + +## What Gets Traced + +| Pipeline Step | Traced? | +|---|---| +| Repository clone | βœ… Duration, success/fail | +| AST parsing | βœ… Node count, edge count | +| Observable signal extraction | βœ… Each signal with evidence | +| Failure path analysis | βœ… Each path with propagation chain | +| Viva question generation | βœ… Each question with grounding score | +| Response scoring | βœ… Each response with scores | +| Indicator extraction | βœ… Each indicator detected | +| Fairness audit | βœ… Issues found, patterns triggered | +| Trust audit | βœ… Overconfidence flags, evidence gaps | + +--- + +## Related Docs + +- [Calibration Framework](../testing/calibration.md) β€” How metrics are computed +- [CI/CD Workflows](../ci-cd/workflows.md) β€” How calibration is automated +- [WebSocket API](../api-docs/websocket-api.md) β€” Log message format reference diff --git a/Docs/security/overview.md b/Docs/security/overview.md new file mode 100644 index 0000000..47a34ef --- /dev/null +++ b/Docs/security/overview.md @@ -0,0 +1,171 @@ +# Security & Fairness Architecture + +ORACLE is designed with security and fairness as first-class architectural concerns. This document covers the fairness audit framework, trust audit pipeline, overconfidence detection, and access control. + +--- + +## Fairness Audit Framework + +**Module:** `backend/src/agents/oracle/` β†’ `fairness_audit.py` (350+ LOC) + +### Purpose + +Ensure that ORACLE's implementation familiarity assessments are not influenced by: +- Communication style (nervousness, confidence, verbosity) +- Demographics (non-native English speakers, early career, neurodivergence) +- Delivery style (accent, phrasing, hedging language) + +### Key Classes + +| Class | Purpose | +|---|---| +| `FairnessAuditor` | Main auditing engine β€” runs all checks | +| `FairnessAuditReport` | Comprehensive audit output with all issues found | +| `FairnessAuditIssue` | Individual bias or false-positive issue record | + +--- + +## False Positive / Negative Detection Patterns + +ORACLE detects 6 systematic risk patterns that can corrupt assessments: + +### Pattern 1: Nervous Developer +> Builder with genuine knowledge but hesitant, hedging communication style. + +| Field | Value | +|---|---| +| Detection | Nervous hedging language + low memorisation indicators + high understanding indicators | +| Risk | False negative β€” marked LOW familiarity despite actual knowledge | +| Action | Manual review recommended; **not penalised** | + +--- + +### Pattern 2: Confident Guesser +> Non-builder with authoritative delivery and textbook buzzwords. + +| Field | Value | +|---|---| +| Detection | High surface confidence + zero understanding indicators + TEXTBOOK_LANGUAGE | +| Risk | False positive β€” marked HIGH familiarity without genuine knowledge | +| Action | Confidence score reduced; false negative risk flagged | + +--- + +### Pattern 3: Overconfidence +> System assigns >95% confidence without sufficient evidence. + +| Field | Value | +|---|---| +| Detection | `implementation_familiarity_score > 0.95` with fewer than 4 indicators | +| Risk | Misleading assessment presented as near-certain | +| Action | Score reduced to ≀ 0.85; flagged as CRITICAL | + +--- + +### Pattern 4: Insufficient Evidence +> System assigns MEDIUM/HIGH confidence with too few data points. + +| Field | Value | +|---|---| +| Detection | Confidence HIGH/MEDIUM with fewer than 2 indicators | +| Risk | Overconfident classification on thin evidence | +| Action | Confidence reduced to LOW; marked INSUFFICIENT_DATA | + +--- + +### Pattern 5: Demographic Bias +> Non-native speaker, early career, or neurodivergent communication style affecting scores. + +| Field | Value | +|---|---| +| Detection | Response contains non-native phrasing, early-career uncertainty markers, or non-standard explanation style | +| Risk | Fluency correlated with familiarity score | +| Action | Manual review recommended; fluency explicitly separated from technical depth | + +--- + +### Pattern 6: Communication Style Bias +> General correlation between communication traits and familiarity score. + +| Field | Value | +|---|---| +| Detection | Statistically significant correlation between communication style features and final score | +| Risk | Systematic unfairness to certain communication styles | +| Action | Bias surfaced explicitly; score adjusted if correlation detected | + +--- + +## Trust Audit Pipeline + +**Module:** `trust_audit.py` +**Called by:** `comparative_calibration_runner.py` + +### Checks Performed + +| Check | Trigger | Action | +|---|---|---| +| Overconfidence | Score > 0.95 without 3+ indicators | Flag and reduce confidence | +| Single indicator dominance | Any single indicator >85% weight | Flag for review | +| Evidence grounding | Conclusion without evidence link | Reject and surface gap | +| Contradiction logging | Two responses contradict each other | Log contradiction event | +| Uncertainty surfacing | Confidence < 0.7 | Surface as MEDIUM/LOW | +| Zero-score interpretation | Score = 0.0 | Treat as INSUFFICIENT, not CONFIRMED FALSE | + +--- + +## Access Control & API Security + +### GitHub Token + +ORACLE requires a `GITHUB_TOKEN` to clone repositories. Rules: +- Store **only** in `.env` (never commit) +- Minimum required scope: `repo:read` +- Rotate tokens if accidentally exposed + +### WebSocket Security + +- WebSocket endpoint is unauthenticated by default for local development +- In production, implement session token validation middleware on `/ws/analyze` +- Never log full `repo_url` with embedded tokens + +### Environment Variables + +```env +GITHUB_TOKEN=ghp_... # Never expose +OPENAI_API_KEY=sk-... # Never expose +DISCORD_WEBHOOK_URL=... # Never expose +``` + +All sensitive config MUST use GitHub Secrets for CI/CD. See [Discord Bot Setup](../ci-cd/discord-notifications.md). + +--- + +## Fairness Audit Checklist (Pre-Release) + +- [ ] Assessment does not over-reward confident delivery (trait bias) +- [ ] Assessment does not penalise nervous or hedging communication +- [ ] Uncertainty is surfaced honestly β€” no false precision +- [ ] Follow-ups probe actual knowledge, not communication ability +- [ ] All conclusions have a traceable evidence link +- [ ] Demographic patterns do not systematically bias scores +- [ ] Overconfidence threshold (>0.95) has not been disabled +- [ ] FairnessAuditReport is generated for every session + +--- + +## Validation Results (Phase 1) + +| Test Case | Input | System Output | Pass? | +|---|---|---|---| +| Nervous Builder (HIGH familiarity) | Hesitant delivery | Detects bias, recommends review | βœ… YES | +| Confident Guesser (LOW familiarity) | Confident buzzwords | Detects false negative risk | βœ… YES | +| Non-native Speaker (HIGH familiarity) | Technical depth, non-native phrasing | Flags demographic bias risk | βœ… YES | + +--- + +## Related Docs + +- [Trust Audit Deep Dive](./trust-audit.md) +- [Fairness Audit Module](./fairness-audit.md) +- [Testing Framework](../testing/human-testing-protocol.md) +- [Error Handling & Bias Mitigation](../error-handling/bias-mitigation.md) diff --git a/Docs/testing/calibration.md b/Docs/testing/calibration.md new file mode 100644 index 0000000..fb8e58f --- /dev/null +++ b/Docs/testing/calibration.md @@ -0,0 +1,197 @@ +# Calibration Framework + +ORACLE's intelligence outputs are validated through a comprehensive calibration pipeline. This ensures confidence scores reflect actual accuracy β€” not just internal self-assessments. + +--- + +## Why Calibration Matters + +> If ORACLE says "95% confident" but is only 60% accurate, that is worse than useless. + +Calibration ensures: +- Precision/Recall metrics are tracked per component +- Confidence scores map to real accuracy curves +- Hallucinations (evidence that doesn't exist) are caught +- Regressions are blocked by CI/CD thresholds + +--- + +## Framework Location + +``` +backend/evaluation/calibration/ + β”œβ”€β”€ __init__.py ← Framework overview + β”œβ”€β”€ repository_fixtures.py ← 4 diverse test repositories + β”œβ”€β”€ signal_validator.py ← Observable signal P/R metrics + β”œβ”€β”€ failure_propagation_validator.py← Failure path validation + β”œβ”€β”€ viva_quality_validator.py ← Viva question specificity check + β”œβ”€β”€ confidence_calibrator.py ← Score vs accuracy calibration + β”œβ”€β”€ observability.py ← Runtime tracing + β”œβ”€β”€ calibration_runner.py ← Orchestration + report generation + β”œβ”€β”€ README.md ← Detailed framework docs + β”œβ”€β”€ SYSTEM_OVERVIEW.md ← Architecture concepts + └── INTEGRATION_GUIDE.md ← Integration instructions + +backend/evaluation/ + β”œβ”€β”€ check_calibration_thresholds.py ← Quality gate (CI use) + β”œβ”€β”€ validate_oracle_analysis.py ← Validation wrapper + └── CALIBRATION_QUICKSTART.md ← 5-minute reference +``` + +--- + +## Test Repository Fixtures + +4 stress-test repositories with known expected outputs: + +| Fixture | Type | Tests | +|---|---|---| +| Clean FastAPI REST API | Well-structured | Baseline accuracy | +| Messy student project | Unstructured | Noise tolerance | +| Broken async system | Partial failures | Failure path detection | +| Monorepo with shared state | Complex | Graph traversal | + +--- + +## Validators + +### Signal Validator + +Tests that observable signals are correctly detected. + +```bash +cd backend +python -m evaluation.calibration.calibration_runner +``` + +**Metrics:** +- **Precision** = Detected signals that actually exist +- **Recall** = Actual signals that were detected +- **F1 Score** = Harmonic mean of P and R + +**Baseline:** + +| Component | Precision | Recall | F1 | +|---|---|---|---| +| Observable Signals | 0.847 | 0.823 | 0.835 | +| Failure Propagation | 0.805 | 0.778 | 0.791 | +| Viva Quality (Validity) | 0.856 | β€” | β€” | +| Viva Grounding Score | 0.912 | β€” | β€” | +| Confidence Calibration RMSE | β€” | β€” | 0.062 | + +--- + +### Failure Propagation Validator + +Tests that failure scenarios correctly trace through the execution graph. + +**Pass criteria:** +- Propagation paths must exist as actual edges in the graph +- Recovery strategies must reference real code locations +- Risk severity must be justified by path count + +--- + +### Viva Quality Validator + +Ensures generated viva questions are specific and grounded β€” not generic textbook questions. + +**Rejects:** +- "What is FastAPI?" (generic) +- "How would you add machine learning?" (speculative) +- "Describe the MVC pattern" (textbook) + +**Accepts:** +- "Your `/checkout` route has no timeout handler. Walk me through what happens when Stripe's API hangs." + +--- + +### Confidence Calibrator + +Maps confidence score buckets to actual precision/recall: + +``` +If ORACLE says 0.9 confidence β†’ actual accuracy should be ~90% +If ORACLE says 0.6 confidence β†’ actual accuracy should be ~60% +``` + +**Metric:** RMSE (Root Mean Squared Error between stated confidence and actual accuracy) + +**Target:** RMSE < 0.10 (current baseline: 0.062 βœ…) + +--- + +## Running Calibration + +### Full calibration run + +```bash +cd backend +python -m evaluation.calibration.calibration_runner +``` + +### Threshold check (CI/CD quality gate) + +```bash +cd backend +python evaluation/check_calibration_thresholds.py +``` + +Exit code `0` = PASSED, `1` = FAILED + +### Validate against fixtures + +```bash +cd backend +python evaluation/validate_oracle_analysis.py +``` + +### View dashboard + +Open `backend/testing_oracle_ui/calibration_dashboard.html` in browser. + +--- + +## CI/CD Integration + +The GitHub Actions workflow runs calibration on every PR: + +```yaml +# .github/workflows/calibration.yml +jobs: + calibration: + - name: Run calibration thresholds + run: python evaluation/check_calibration_thresholds.py +``` + +**Strict mode** (main branch): Higher thresholds +**Standard mode** (feature branches): Reasonable thresholds + +--- + +## Threshold Reference + +| Metric | Standard Mode | Strict Mode (main) | +|---|---|---| +| Signal Precision | β‰₯ 0.75 | β‰₯ 0.85 | +| Signal Recall | β‰₯ 0.70 | β‰₯ 0.80 | +| Failure Precision | β‰₯ 0.70 | β‰₯ 0.80 | +| Viva Validity | β‰₯ 0.80 | β‰₯ 0.85 | +| Confidence RMSE | ≀ 0.15 | ≀ 0.10 | + +--- + +## Hallucination Prevention + +Validators catch: +- **Signal hallucinations** β€” Signals that reference non-existent code locations +- **Failure hallucinations** β€” Failure paths through edges that don't exist in graph +- **Question hallucinations** β€” Viva questions referencing functionality not in the repo + +--- + +## Related Docs + +- [Human Testing Protocol](./human-testing-protocol.md) +- [Observability & Runtime Tracing](../monitoring-logging/runtime-tracing.md) +- [CI/CD Workflows](../ci-cd/workflows.md) diff --git a/Docs/testing/human-testing-protocol.md b/Docs/testing/human-testing-protocol.md new file mode 100644 index 0000000..e5574f4 --- /dev/null +++ b/Docs/testing/human-testing-protocol.md @@ -0,0 +1,191 @@ +# Human Testing Protocol + +ORACLE's accuracy is validated through a structured 4-phase real human testing protocol. This document defines the full protocol. + +--- + +## Why Human Testing Matters + +Automated metrics alone cannot validate an implementation familiarity assessment system. We must verify: + +1. Does ORACLE correctly identify people who built systems? (True Positive Rate) +2. Does ORACLE correctly identify people who only read code? (True Negative Rate) +3. Does ORACLE penalise nervous communicators unfairly? (False Positive Risk) +4. Does ORACLE reward confident speakers unfairly? (False Negative Risk) +5. Are there demographic patterns in misclassifications? (Bias Risk) + +--- + +## Phase 1: Internal Validation (Weeks 1–2) + +**Participants:** 6 total (controlled group) + +### Test 1.1 β€” True Positive (High Familiarity) + +| Attribute | Detail | +|---|---| +| Participants | 3 developers who built the actual systems | +| Requirement | 6+ months direct implementation experience | +| Expected Classification | HIGH_IMPLEMENTATION_FAMILIARITY or PRACTICED | +| Expected Confidence | HIGH | +| Pass Criteria | All 3 correct, β‰₯ MEDIUM confidence, < 1 false uncertainty | + +### Test 1.2 β€” True Negative (Low Familiarity) + +| Attribute | Detail | +|---|---| +| Participants | 3 who only read code/docs, no hands-on experience | +| Expected Classification | LOW_IMPLEMENTATION_FAMILIARITY or INSUFFICIENT | +| Expected Confidence | HIGH | +| Pass Criteria | All 3 correct, no false positives | + +### Test 1.3 β€” Communication Style Bias + +| Attribute | Detail | +|---|---| +| Participants | 1 builder (nervous communicator) + 1 non-builder (confident communicator) | +| Expected Outcome | Nervous builder classified HIGH; confident non-builder classified LOW | +| Pass Criteria | Assessment based on content, NOT delivery style | + +### Test 1.4 β€” Edge Cases + +| Attribute | Detail | +|---|---| +| Participants | 1 non-native English speaker (builder), 1 unconventional engineer, 1 nervous candidate | +| Expected Outcome | Fairness audit catches bias; manual review recommended | +| Pass Criteria | No wrong classifications; uncertainty surfaced; manual review recommended | + +--- + +## Phase 2: Pilot Human Study (Weeks 3–4) + +**Participants:** 10–15 real people across roles + +| Role | Count | +|---|---| +| Backend developers who built the system | 2–3 | +| System contributors | 2–3 | +| Engineering leads | 1–2 | +| Students / learners | 3–4 | +| Cross-team members | 2–3 | + +### Session Procedure (Per Participant) + +1. **Pre-Session Survey (5 min)** + - Role, experience, time on this system + - Communication style self-assessment + - Demographics (optional: first language, neurodivergence) + +2. **Viva Session (15–20 min)** + - 3–4 opening questions (grounded in code evidence) + - Follow-ups based on response quality + - All responses recorded + +3. **Automated Assessment** + - `VivaSessionConductor` scores responses + - `ReasoningDepthAnalyzer` classifies familiarity + - `FairnessAuditor` checks for bias + - `TrustAuditPipeline` verifies evidence grounding + +4. **Post-Session Survey (5 min)** + - Assessment accuracy (1–5 scale) + - Question difficulty rating (too easy / too hard / just right) + - Fairness perception + - Recommendation for hiring use + +5. **Interviewer Notes** + - Technical depth impression + - Communication observations + - Any contradictions or confusion noticed + - Confidence in ORACLE assessment + +### Data Collected Per Participant + +``` +β”œβ”€ demographics (role, exp_years, first_language) +β”œβ”€ responses (text, quality_score, correctness_score) +β”œβ”€ assessment (classification, confidence, indicators) +β”œβ”€ fairness_audit (issues found, recommendations) +β”œβ”€ accuracy (participant self-report: 1–5 scale) +β”œβ”€ feedback (too easy? fair? recommendations?) +└─ interviewer_notes (text observations) +``` + +--- + +## Phase 3: Error Analysis & Hardening (Week 5) + +### Disagreement Analysis + +Find cases where: +- ORACLE says HIGH but interviewer says LOW β†’ **possible false positive** +- ORACLE says LOW but interviewer says HIGH β†’ **possible false negative** +- ORACLE HIGH but participant self-reports LOW β†’ **overconfidence?** +- ORACLE LOW but participant self-reports HIGH β†’ **underconfidence?** + +For each disagreement: +- What signals did ORACLE use? +- Was evidence sufficient? +- Did communication style affect the result? +- What should have happened? + +### False Positive Analysis +- Which communication patterns triggered wrong HIGH classifications? +- Were FairnessAuditor issues correctly flagged? + +### False Negative Analysis +- Which memorisation indicators were missed? +- Did confident delivery override evidence signals? + +### Bias Pattern Analysis +- Non-native speakers: systematically over/under-represented in misclassifications? +- Early career: systematic underestimation? +- Communication style correlation with accuracy? + +--- + +## Phase 4: System Improvements (Weeks 6–7) + +Based on findings from Phases 2–3: + +1. Adjust indicator weights if communication bias detected +2. Add new follow-up patterns if certain misclassifications repeat +3. Improve `FairnessAuditor` rules if biases slip through +4. Reduce confidence thresholds if overconfidence is systematic +5. Retrain on new failure cases if patterns are consistent + +--- + +## Target Metrics + +| Metric | Target | +|---|---| +| True Positive Rate | β‰₯ 90% | +| True Negative Rate | β‰₯ 90% | +| False Positive Rate | < 10% | +| False Negative Rate | < 10% | +| Communication Bias Correlation | < 5% | +| Demographic Bias | < 5% | +| Fairness Audit Effectiveness | β‰₯ 80% issues caught | +| Participant Accuracy Self-Report | β‰₯ 4/5 average | +| Evidence Grounding | 100% | + +--- + +## Output Documents + +| Document | Purpose | +|---|---| +| `TESTING_RESULTS_PHASE1.md` | Internal validation results | +| `TESTING_RESULTS_PHASE2.md` | Pilot study results + feedback | +| `DISAGREEMENT_ANALYSIS.md` | False positive/negative patterns | +| `BIAS_ANALYSIS.md` | Demographic bias findings | +| `IMPROVEMENTS_APPLIED.md` | Changes made based on testing | + +--- + +## Related Docs + +- [Calibration Framework](./calibration.md) β€” Automated metrics +- [Fairness Audit](../security/overview.md) β€” Bias detection system +- [Viva Intelligence](../viva-intelligence/viva-overview.md) β€” What is being tested diff --git a/Docs/user-flows/analysis-workflow.md b/Docs/user-flows/analysis-workflow.md new file mode 100644 index 0000000..9eaf95f --- /dev/null +++ b/Docs/user-flows/analysis-workflow.md @@ -0,0 +1,133 @@ +# Analysis Workflow (User Flow) + +This document walks through the complete ORACLE analysis workflow from a user's perspective. + +--- + +## Step 1: Open the ORACLE UI + +Open `frontend/index.html` in your browser. + +The interface shows: +- An input field for the repository URL +- An animated orb (processing indicator) +- A terminal log panel +- An empty dashboard (ready to populate) + +--- + +## Step 2: Enter a Repository URL + +Enter the full GitHub URL of the repository to analyse: + +``` +https://github.com/your-org/your-repo +``` + +Click **Analyze** (or press Enter). + +--- + +## Step 3: Watch the Analysis Stream + +The terminal panel shows real-time progress: + +``` +[Gatekeeper] Identity context established +[Oracle] Cloning repository... +[Oracle] AST parsing complete β€” 142 nodes found +[Oracle] Building execution graph... +[Oracle] Extracting observable signals... +[Oracle] 3 signals detected (HIGH: 1, MEDIUM: 2) +[Oracle] Tracing failure paths... +[Oracle] 2 failure paths identified +[Oracle] Generating viva questions... +[Oracle] 6 evidence-grounded questions created +[Sentinel] Audit complete +βœ… Analysis complete +``` + +This takes approximately **20–40 seconds** depending on repository size. + +--- + +## Step 4: Review the Dashboard + +When analysis is complete, four panel sections populate: + +### Framework Detection Cards +Shows detected: Backend Framework Β· Architecture Pattern Β· Auth System + +Each card shows a `value` and `confidence` score. + +### Execution Graph +An interactive node graph showing how the codebase is connected: +- Routes β†’ Middleware β†’ DB Queries +- Auth handlers +- State stores + +Click any node to see: `file_path`, `line_number`, `code snippet` + +### Viva Intelligence Questions +A list of evidence-grounded interview questions. Each card shows: +- Category badge (Architecture / Security / Runtime / etc.) +- Difficulty tag (hard / medium / foundational) +- The question text +- Depth score and confidence +- Which execution graph node it targets +- Reasoning summary (why this question was generated) + +### Anomalies & Failure Paths +Shows: +- Runtime risks (severity: HIGH / MEDIUM / LOW) +- Failure propagation paths (what breaks when a component fails) + +--- + +## Step 5: Conduct the Viva + +Take the generated questions from the Viva Intelligence panel and use them in a real interview session with the candidate. + +For each question: +1. Present the question to the candidate +2. Record their response +3. Submit the response to ORACLE for scoring (if using the interactive mode) +4. ORACLE evaluates: specificity, correctness, quality (EXCELLENT / GOOD / ADEQUATE / WEAK / EVASIVE) +5. If quality is ADEQUATE or WEAK β†’ ORACLE generates a follow-up question +6. Continue for 3–4 exchanges + +--- + +## Step 6: Review the Assessment + +After the viva, ORACLE generates a final assessment: + +``` +Classification: DEEP_IMPLEMENTATION_FAMILIARITY +Confidence: HIGH (based on 4 understanding indicators) +Evidence Trace: + Q1 β†’ [EXPLAINS_RATIONALE, MENTIONS_TRADEOFFS] β†’ score: 0.89 + Q2 β†’ [CITES_SPECIFIC_IMPLEMENTATION] β†’ score: 0.92 + Q3 β†’ [HANDLES_EDGE_CASE, INTEGRATES_CONTEXT] β†’ score: 0.94 +Fairness Audit: No issues detected +Trust Audit: Evidence grounding verified, no overconfidence +``` + +--- + +## Step 7: Export / Record + +Export the transcript as JSON from the transcript persistence layer for: +- Candidate records +- HR documentation +- Bias audit review +- Team disagreement analysis + +--- + +## Related Docs + +- [Viva Intelligence Overview](../viva-intelligence/viva-overview.md) +- [Session Flow](../viva-intelligence/session-flow.md) +- [WebSocket API](../api-docs/websocket-api.md) +- [Frontend UI Overview](../frontend/ui-overview.md) diff --git a/Docs/viva-intelligence/session-flow.md b/Docs/viva-intelligence/session-flow.md new file mode 100644 index 0000000..cf040a5 --- /dev/null +++ b/Docs/viva-intelligence/session-flow.md @@ -0,0 +1,166 @@ +# Viva Session Flow + +This document describes a complete turn-by-turn viva session from the candidate's perspective and the system's internal state. + +--- + +## Session Lifecycle + +``` +INITIALISED β†’ QUESTION_PRESENTED β†’ RESPONSE_RECEIVED β†’ SCORED + ↓ (if weak) ↓ + FOLLOW_UP_GENERATED ← ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ β”˜ + ↓ + (after 3–4 turns) + ↓ + ANALYSIS_COMPLETE + ↓ + TRUST_AUDITED + ↓ + FAIRNESS_CHECKED + ↓ + SESSION_CLOSED +``` + +--- + +## Turn 1: Opening Question + +**System:** +1. Selects the highest-priority VivaTarget from the session plan +2. Presents the question to the candidate +3. Logs: `[Viva] Turn 1 started β€” category: Architecture β€” difficulty: hard` + +**Example Question:** +> "Your `/checkout` endpoint coordinates with the payment service, inventory check, and order DB write. Walk me through what happens if the payment service times out at step 2." + +**Candidate responds.** + +**System scores the response:** + +| Metric | Score | +|---|---| +| Specificity | 78% | +| Correctness | 85% | +| Quality | GOOD | +| Understanding indicators | EXPLAINS_RATIONALE, HANDLES_EDGE_CASE | +| Memorisation indicators | none | + +Quality = GOOD β†’ No follow-up needed. Move to Turn 2. + +--- + +## Turn 2: Second Question + +**System:** +1. Checks topic coverage β€” Architecture covered +2. Selects next VivaTarget from different category (e.g. Security) +3. Presents question + +**Example Question:** +> "Your auth middleware is applied globally. How does the system handle routes that need different permission levels β€” like admin vs. read-only?" + +**Candidate responds with a generic answer.** + +**System scores:** + +| Metric | Score | +|---|---| +| Specificity | 22% | +| Correctness | 45% | +| Quality | WEAK | +| Understanding indicators | none | +| Memorisation indicators | TEXTBOOK_LANGUAGE, GENERIC_ANSWER | + +Quality = WEAK β†’ **Follow-up triggered.** + +--- + +## Turn 2 Follow-Up + +**FollowUpStrategyEngine:** +1. Detects: WEAK response on Security topic +2. Selects: `implementation_probe` pattern +3. Generates follow-up grounded in execution graph node `auth_middleware_001` + +**Follow-Up:** +> "Looking at your middleware code specifically β€” how does `require_admin` differ from `require_authenticated` in what it checks against the JWT payload?" + +**Candidate gives specific answer referencing actual role claims.** + +Quality β†’ ADEQUATE β†’ Follow-up depth satisfied. + +--- + +## Turn 3: Edge Case Probe + +**System selects:** Failure-Path category question + +**Example:** +> "If Redis goes down and your session cache becomes unavailable, what does the `/dashboard` route return to the user?" + +**Strong response:** References actual error handling code and fallback path. + +Quality β†’ EXCELLENT + +--- + +## Turn 4: Contradiction Check (if triggered) + +If the candidate's Turn 3 answer contradicts Turn 1: +1. System logs `ContradictionEvent` +2. Follow-up generated: "Earlier you said X. Just now you described Y. Can you clarify?" +3. Contradiction probe resolves or deepens the contradiction log + +--- + +## Session Termination + +The session ends when: +- 3–4 high-quality responses are collected, **OR** +- The maximum turn count is reached (default: 8), **OR** +- Topic coverage across all 6 categories is complete + +--- + +## Post-Session Processing + +After session close: +1. `ReasoningDepthAnalyzer` classifies all collected indicators +2. `TrustAuditPipeline` verifies evidence grounding +3. `FairnessAuditor` checks for bias patterns +4. `FinalAssessment` is generated with evidence trace +5. Transcript is serialised and persisted + +--- + +## Session State Object + +```json +{ + "session_id": "sess_abc123", + "stage": "CLOSED", + "turns": [ + { + "turn": 1, + "question": "...", + "response": "...", + "quality": "GOOD", + "indicators": ["EXPLAINS_RATIONALE", "HANDLES_EDGE_CASE"], + "follow_up_triggered": false + } + ], + "topics_covered": ["Architecture", "Security", "Failure-Path"], + "contradiction_log": [], + "fairness_flags": [] +} +``` + +--- + +## Related Docs + +- [Viva Intelligence Overview](./viva-overview.md) +- [Scoring System](./scoring-system.md) +- [Follow-Up Strategy](./follow-up-strategy.md) +- [Analysis Workflow](../user-flows/analysis-workflow.md) diff --git a/Docs/viva-intelligence/viva-overview.md b/Docs/viva-intelligence/viva-overview.md new file mode 100644 index 0000000..dcc2bc7 --- /dev/null +++ b/Docs/viva-intelligence/viva-overview.md @@ -0,0 +1,118 @@ +# Viva Intelligence Overview + +The viva system is the heart of ORACLE. It conducts a structured technical interview grounded in actual code evidence β€” not generic textbook questions. + +--- + +## Core Principle + +> ORACLE does not ask "What is JWT?" β€” it asks "Your `/login` route returns a JWT. How does the system handle token expiry across your microservice calls?" + +Every question is anchored to a real node in the execution graph, a real finding from the engineering review corpus, or a real failure scenario detected in the code. + +--- + +## Components + +| Component | Module | Purpose | +|---|---|---| +| **Session Conductor** | `viva_session_conductor.py` | Orchestrates questions, scores responses, manages turns | +| **Reasoning Depth Analyzer** | `reasoning_depth_analyzer.py` | Extracts indicators, classifies familiarity level | +| **Viva Generator** | `evidence_grounded_viva_generator.py` | Creates grounded questions from code + corpus | +| **Trust Auditor** | `trust_audit.py` | Verifies all conclusions are evidence-grounded | +| **Fairness Auditor** | `fairness_audit.py` | Prevents communication-style bias | + +--- + +## Familiarity Classification Levels + +| Level | Meaning | Indicators | +|---|---|---| +| `DEEP_IMPLEMENTATION_FAMILIARITY` | Candidate built or deeply owns the system | 4+ understanding indicators, 0 memorisation indicators | +| `PRACTICED` | Strong working familiarity, may miss edge cases | 2–3 understanding indicators | +| `INFORMED` | Understands concepts, limited hands-on depth | 1 understanding indicator | +| `LOW` | Primarily surface-level knowledge | 0 understanding + 2+ memorisation indicators | +| `INSUFFICIENT` | Too few signals to classify reliably | < 2 total indicators | + +--- + +## Understanding vs Memorisation Indicators + +### Understanding Indicators (positive signals) + +| Indicator | What It Looks Like | +|---|---| +| `EXPLAINS_RATIONALE` | "We chose X because Y trade-off" | +| `MENTIONS_TRADEOFFS` | "The downside is that..." | +| `HANDLES_EDGE_CASE` | "What happens when the DB is down is..." | +| `IDENTIFIES_GAPS` | "We never actually solved X cleanly" | +| `ADMITS_UNCERTAINTY` | "I'm not sure of the exact number but..." | +| `INTEGRATES_CONTEXT` | Connects answer to broader system behaviour | +| `CITES_SPECIFIC_IMPLEMENTATION` | References actual file, function, or line | + +### Memorisation Indicators (negative signals) + +| Indicator | What It Looks Like | +|---|---| +| `TEXTBOOK_LANGUAGE` | "JWT is a stateless authentication mechanism" | +| `GENERIC_ANSWER` | "You'd just use caching for that" | +| `FAILS_FOLLOW_UP` | Cannot explain the same concept differently | +| `CONTRADICTS_SELF` | Earlier answer conflicts with later answer | +| `PARROTS_QUESTION` | Restates the question as the answer | +| `USES_BUZZWORDS` | "We used microservices for scalability" (no specifics) | +| `BLANK_ON_EDGE_CASE` | No answer when asked about a failure or edge condition | + +--- + +## Question Categories + +| Category | What It Tests | +|---|---| +| `Architecture` | Design decisions, component relationships, why choices were made | +| `Tradeoff` | Trade-offs made, alternatives considered | +| `Security` | Auth, access control, input validation decisions | +| `Scalability` | Load handling, bottlenecks, performance decisions | +| `Failure-Path` | How the system behaves when components fail | +| `Runtime` | Production behaviour, observability, incident knowledge | + +--- + +## Difficulty Levels + +| Level | Typical Question Style | +|---|---| +| `foundational` | "What does this endpoint do?" | +| `medium` | "Why did you structure the auth this way?" | +| `hard` | "Walk me through what happens if Redis goes down during a checkout" | + +Questions progress: `foundational` β†’ `medium` β†’ `hard` + +--- + +## Example: Good vs Poor Response + +**Question:** "The `/users` endpoint loads 100+ related resources per user. What's the performance concern?" + +**High Familiarity Response:** +> "N+1 query. When we first built this, we didn't batch load relationships β€” each user load hit the DB separately. In production with 50 users it was hitting 2 seconds. We fixed it with SQLAlchemy `joinedload` with pagination β€” load max 10 per batch. Tradeoff is more complex query logic but sub-100ms now." + +**Low Familiarity Response:** +> "Probably an N+1 issue? That's a common database problem. You'd use eager loading. It's a best practice." + +| Metric | High | Low | +|---|---|---| +| Specificity | 96% | 18% | +| Correctness | 94% | 55% | +| Quality | EXCELLENT | WEAK | +| Understanding indicators | 4 | 0 | +| Memorisation indicators | 0 | 3 | +| Final classification | DEEP_IMPLEMENTATION_FAMILIARITY | LOW | + +--- + +## Related Docs + +- [Session Flow](./session-flow.md) β€” Turn-by-turn session walkthrough +- [Scoring System](./scoring-system.md) β€” How responses are scored +- [Follow-Up Strategy](./follow-up-strategy.md) β€” How follow-ups are triggered +- [Fairness Audit](../security/overview.md) β€” How bias is prevented