fix: remove private-method boundary leak, dedupe ingest retry loop

.env.example

@@ -5,8 +5,15 @@ OLLAMA_LLM_MODEL=gemma3:4b
 CHROMA_PERSIST_PATH=./data/chroma
 CHROMA_COLLECTION_NAME=localrag
 
+# Optional tenant tag written to every chunk's metadata and usable as a
+# QueryRequest.metadata_filter key ({"tenant_id": "..."}). Empty = untagged
+# (single-tenant deployments, the common case, pay zero extra cost). NOTE:
+# this is an equality-filter convenience, not full multi-tenant isolation —
+# see docs/rag-retrieval.md.
+TENANT_ID=
+
 CHUNK_CHARS=512
-CHUNK_OVERLAP_CHARS=50
+CHUNK_OVERLAP_CHARS=150
 CHUNKING_MODE=structural
 CHUNK_MAX_CHARS=1200
 CHUNK_MIN_CHARS=200
@@ -20,19 +27,37 @@ INGEST_ROOTS=[]
 UPLOAD_DIR=./data/uploads
 UPLOAD_MAX_BYTES=100000000
 
+# Append-only local audit trail (one JSON line per query: question, sources, answer, model, latency). Empty disables.
+AUDIT_LOG_PATH=
+
 # OCR fallback for scanned/image-only PDF pages (requires the `tesseract` binary
 # on the host; see docs/ocr.md). Set OCR_ENABLED=false to disable.
 OCR_ENABLED=true
 OCR_LANGUAGE=eng
 OCR_MIN_CHARS_PER_PAGE=20
 
 RAG_TOP_K=5
+# Below this top-context score, refuse instead of generating (0 disables).
+RAG_MIN_CONTEXT_SCORE=0.0
 RETRIEVAL_MODE=hybrid
 BM25_WEIGHT=0.5
 RRF_K=60
 FRESHNESS_HALF_LIFE_DAYS=30.0
+# Expand top retrieval hits to their full markdown section (via heading_path) before prompting. Set false to disable.
+PARENT_EXPANSION_ENABLED=true
+# LLM rewrites the query for retrieval only (adds one extra LLM round-trip per query); the original question is still used for the final answer prompt.
+QUERY_REWRITE_ENABLED=false
 RAG_SYSTEM_PROMPT=You are a helpful assistant. Answer only based on the provided context.
 
+# Optional cross-encoder reranking (requires `uv sync --extra rerank`).
+RERANK_ENABLED=false
+RERANK_MODEL=cross-encoder/ms-marco-MiniLM-L-6-v2
+RERANK_FETCH_K=20
+
+# In-process TTL cache for repeated/near-identical queries (0 disables; no external cache).
+QUERY_CACHE_TTL_SECONDS=0
+QUERY_CACHE_MAXSIZE=256
+
 API_HOST=0.0.0.0
 API_PORT=8000
 
@@ -43,6 +68,13 @@ API_KEY=
 # LLM backend: ollama | openai | anthropic
 LLM_BACKEND=ollama
 
+# Optional automatic failover backend when the primary trips its circuit breaker
+# (ollama | openai | anthropic; empty disables fallback).
+LLM_FALLBACK_BACKEND=
+LLM_RETRY_MAX_ATTEMPTS=3
+LLM_CIRCUIT_FAIL_MAX=5
+LLM_CIRCUIT_RESET_TIMEOUT_SECONDS=30.0
+
 # Canonical embedding model alias (overrides OLLAMA_EMBED_MODEL when set).
 EMBEDDING_MODEL=
 

.github/workflows/evals-pr.yml

@@ -0,0 +1,47 @@
+name: RAGAS Evals (PR gate)
+
+on:
+  pull_request:
+    paths:
+      - "localrag/rag/**"
+      - "localrag/ingestion/**"
+      - "evals/**"
+      - "pyproject.toml"
+
+permissions:
+  contents: read
+
+jobs:
+  evals:
+    name: RAGAS evaluation (offline, blocking)
+    runs-on: ubuntu-latest
+    services:
+      ollama:
+        image: ollama/ollama:latest
+        ports:
+          - 11434:11434
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: astral-sh/setup-uv@v6
+        with:
+          version: "latest"
+
+      - name: Install dependencies
+        run: uv sync --all-extras
+
+      - name: Pull judge models
+        run: |
+          curl -s http://localhost:11434/api/pull -d '{"name": "gemma3:4b"}'
+          curl -s http://localhost:11434/api/pull -d '{"name": "nomic-embed-text"}'
+
+      - name: Run RAGAS evals (offline — judge runs against local Ollama, no external API key needed)
+        run: uv run python evals/run_evals.py --offline
+
+      - name: Upload eval results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: eval-results-pr-${{ github.event.pull_request.number }}
+          path: evals/results/
+          if-no-files-found: warn

AGENTS.md → CLAUDE.md

@@ -22,6 +22,26 @@ The FastAPI layer follows a **light domain-driven** split:
 
 Domain packages (`localrag/ingestion/`, `localrag/rag/`, `localrag/storage/`) keep their own services and types; the API service calls into them (e.g. `IngestionService`, `RAGEngine`).
 
+## Claude Code skills for this repo
+
+If you're running as Claude Code, these installed skills/subagents map directly onto LocalRAG's stack—reach for them instead of re-deriving generic advice:
+
+| Area | Skill / agent | When it applies here |
+| --- | --- | --- |
+| Git workflow | `superpowers:using-git-worktrees`, `git-pr-workflows:git-workflow` | Isolating longer-lived `feat/…`/`fix/…` work, or orchestrating review → PR beyond the basic steps in `.cursor/skills/trunk-feature-workflow/SKILL.md` |
+| API/DDD layering | `backend-development:architecture-patterns`, `backend-development:api-design-principles` | Reshaping the schemas/service/repository/router split above, or designing new routes |
+| Python style & types | `python-development:python-code-style`, `python-development:python-type-safety` | Anything ruff/mypy already gate in `.pre-commit-config.yaml`—use before relying on defaults |
+| Config & settings | `python-development:python-configuration` | Changes to `localrag/settings.py` / `.env.example` (pydantic-settings) |
+| Error handling & resilience | `python-development:python-error-handling`, `python-development:python-resilience` | `HttpMappedError` subclasses, ingestion retry/batch logic |
+| Observability | `python-development:python-observability` | structlog / Prometheus metrics work (`localrag/logging_config.py`, `api/routers/metrics.py`) |
+| Testing | `python-development:python-testing-patterns`, `superpowers:test-driven-development` | Adding/extending `tests/` (pytest, pytest-asyncio, respx) |
+| Debugging | `superpowers:systematic-debugging`, `diagnosing-bugs` | Any bug/regression—before proposing a fix |
+| LLM/provider code | `claude-api` | Anthropic/OpenAI/Ollama provider work in `localrag/llm/` |
+| Security | `security-review`, `backend-api-security:backend-security-coder` | Auth (`API_KEY`), path validation (`is_path_allowed`), upload handling, anything bandit flags |
+| Verification | `verify` | Before claiming a change works—exercise the flow, don't just trust lint/tests |
+
+This is a pointer, not a guarantee of installation—confirm availability in your environment before relying on one.
+
 ## Documentation maintenance for agents
 
 When you change anything that affects **how agents find or reason about the codebase**, update the relevant docs **in the same change** (same PR). At minimum:

docs/adr/007-resilient-llm-provider-routing.md

@@ -0,0 +1,42 @@
+# ADR 007: Resilient, uniformly-routed LLM provider abstraction
+
+## Context
+
+`LLM_BACKEND` (ollama/openai/anthropic) existed as a setting, but `RAGEngine`
+hand-rolled its own httpx calls to Ollama directly for prompt generation and
+streaming, bypassing `localrag/llm/factory.py::build_provider` entirely.
+Changing `LLM_BACKEND` had no effect on `/query` — it only ever hit Ollama.
+Separately, no provider had retry, timeout backoff, or fallback behavior: a
+transient Ollama hiccup or OpenAI/Anthropic rate limit surfaced immediately
+as a user-facing failure.
+
+## Decision
+
+- Give `BaseLLMProvider` two abstract methods, `generate_from_prompt` and
+  `stream_from_prompt`, implemented per backend (`ollama.py`,
+  `openai_provider.py`, `anthropic_provider.py`).
+- Wrap every provider `build_provider` returns in `ResilientProvider`
+  (`localrag/llm/resilience.py`): tenacity retry with backoff, a pybreaker
+  circuit breaker, and an optional configured fallback backend
+  (`llm_fallback_backend`) that takes over once the breaker trips.
+- Give `RAGEngine` a required `provider: BaseLLMProvider` field; route
+  `_stream_chat_tokens` through `self.provider.stream_from_prompt(...)`
+  instead of its own httpx client. `get_engine()` wires
+  `provider=build_provider(settings)`.
+
+## Consequences
+
+- `LLM_BACKEND` now genuinely governs `/query` — verified live by setting
+  `LLM_BACKEND=openai` with a bogus key and observing a real
+  `401 AuthenticationError` from the OpenAI SDK instead of a silent
+  fall-through to Ollama.
+- All three backends get retry/circuit-breaker/fallback uniformly, instead of
+  resilience logic living only where someone happened to add it.
+- Every LLM-calling code path (RAG answers, ragas eval judge, query
+  rewriting) now goes through one seam (`build_provider`), so future
+  resilience or observability changes apply everywhere at once.
+
+## Related
+
+`[[../architecture.md]]`, `docs/superpowers/plans/2026-07-07-production-rag-hardening.md`
+(Tasks 13–14).

docs/adr/008-cross-encoder-reranking.md

@@ -0,0 +1,44 @@
+# ADR 008: Optional cross-encoder reranking as a final relevance step
+
+## Context
+
+Hybrid vector+BM25 retrieval (ADR 005) with RRF fusion ranks well on lexical
+and semantic signal separately, but neither embedding similarity nor BM25
+score directly optimizes for "is this chunk actually relevant to this exact
+question" the way a cross-encoder (which scores the pair jointly) does.
+Running a cross-encoder over the full corpus per query would be too slow;
+running it only on the fused top-k throws away recall it could have fixed.
+
+## Decision
+
+Make reranking an optional, off-by-default final step:
+
+- When `RERANK_ENABLED=true`, `Retriever.retrieve` over-fetches
+  `RERANK_FETCH_K` candidates (instead of the usual `top_k * 2`) from the
+  vector/hybrid path.
+- `CrossEncoderReranker` (`localrag/rag/reranker.py`), backed by a local
+  `sentence-transformers` cross-encoder model (`RERANK_MODEL`, default
+  `cross-encoder/ms-marco-MiniLM-L-6-v2`), scores each `(question, chunk)`
+  pair and trims back down to `top_k`.
+- Reranking runs on the raw candidate list, strictly before freshness decay
+  (ADR 006) and parent-section expansion — it is the last relevance-ordering
+  step; freshness/expansion apply to the reranked, already-trimmed result.
+- `sentence-transformers` is an optional dependency (`uv sync --extra
+  rerank`); nothing imports it unless the feature is enabled, keeping the
+  default install lightweight.
+
+## Consequences
+
+- Retrieval quality can improve for ambiguous queries without changing the
+  default (disabled) behavior or the default dependency footprint.
+- Extra latency and a model download when enabled — appropriate to enable
+  per-deployment, not universally.
+- `_fuse_results`'s dead RRF weight branch (only ever taken when
+  `bm25_weight == 0.5`, mathematically identical to the other branch) was
+  removed as part of this change since it was mechanically adjacent cleanup,
+  not a new decision in itself.
+
+## Related
+
+`[[005-hybrid-retrieval]]`, `[[006-freshness-decay]]`,
+`docs/superpowers/plans/2026-07-07-production-rag-hardening.md` (Task 6).

docs/adr/009-offline-ragas-judge.md

@@ -0,0 +1,65 @@
+# ADR 009: RAGAS eval judge runs on local Ollama, not OpenAI
+
+## Context
+
+`evals/run_evals.py` gates the RAG pipeline's quality on faithfulness,
+answer relevancy, context precision, and context recall (ragas metrics).
+Ragas's `evaluate()` defaults its judge LLM to `OpenAI()`, which requires
+`OPENAI_API_KEY` — not present anywhere in this project's `.env`, and at
+odds with LocalRAG's offline-first positioning ("Your documents, your
+models, your machine."). Installed `ragas==0.4.3` also deprecated the
+`ragas.metrics.*` singleton import style this script originally used, in
+favor of `ragas.metrics.collections.*` classes scored per-sample via async
+`.ascore()`.
+
+## Decision
+
+Point the judge LLM and embeddings at the same local Ollama instance
+LocalRAG itself uses, via Ollama's OpenAI-compatible `/v1` endpoint:
+
+```python
+ollama_client = AsyncOpenAI(base_url=f"{ollama_url}/v1", api_key="ollama")
+judge_llm = llm_factory(judge_model, client=ollama_client, adapter="instructor")
+judge_embeddings = OpenAIEmbeddings(client=ollama_client, model="nomic-embed-text")
+```
+
+`openai.AsyncOpenAI`/`ragas.embeddings.OpenAIEmbeddings` are used here purely
+as clients for the OpenAI-shaped wire protocol Ollama speaks — `base_url`
+points at localhost, the API key is a required-but-ignored dummy string, and
+no request ever reaches OpenAI's actual service. This was chosen over:
+
+1. **OpenAI as judge via CI secret** — rejected, can't be verified locally,
+   contradicts offline-first positioning.
+2. **LangChain wrapper** (`langchain-ollama` +
+   `ragas.llms.LangchainLLMWrapper`) — works, but pulls in a dependency this
+   project's owner prefers to avoid for what's fundamentally an HTTP call to
+   a local model.
+3. **Swap the whole eval framework to DeepEval** — rejected as a
+   disproportionate framework migration for a compatibility fix; DeepEval
+   isn't obviously lighter than ragas, just a different footprint.
+4. **LiteLLM adapter** — avoids LangChain, but pulls in `litellm` +
+   `instructor` for the same job the `openai` client (already a core
+   dependency, since it backs the real `LLM_BACKEND=openai` provider) does
+   natively.
+
+Judge model defaults to `gemma3:4b`, embeddings to `nomic-embed-text` — the
+project's own default LLM/embedder, already pulled by the existing Docker
+Ollama container. Both are overridable via `--judge-model`/`--ollama-url`.
+
+## Consequences
+
+- Evals are fully offline-capable and reproducible on any machine running
+  the project's own Docker stack — no external API key needed anywhere,
+  including CI (`.github/workflows/evals-pr.yml` runs an `ollama/ollama`
+  service container and pulls the judge/embedding models before scoring).
+- Judge quality is bounded by a small local model (`gemma3:4b`, 4B params)
+  rather than a GPT-4-class judge — `PASS_THRESHOLDS` may need recalibration
+  if scores drift far from expectations on a real corpus. First full run:
+  faithfulness 0.942, answer_relevancy 0.671, context_precision 0.891,
+  context_recall 1.000 — all above threshold.
+- Zero new dependencies added for this fix.
+
+## Related
+
+`docs/superpowers/plans/2026-07-07-production-rag-hardening.md` (Task 1),
+session narrative in `log_reasonings/2026-07-07-ragas-offline-judge.md`.

docs/agent-navigation.md

@@ -31,6 +31,7 @@ Agents (and humans) move faster when they:
 | API app factory (lifespan, middleware, error handlers) | `localrag/api/main.py` |
 | HTTP ingest path validation (`INGEST_ROOTS`, URL decode) | `localrag/api/service.py`, `localrag/settings.py` (`is_path_allowed`), `localrag/api/exceptions.py` + `main.py` handler |
 | HTTP multipart file upload ingest (`POST /ingest/upload`) | `localrag/api/routers/ingest.py` (`ingest_upload`, Swagger limitations in `_UPLOAD_DESCRIPTION`), `localrag/api/service.py` (`ingest_upload`, `_stream_upload_to_disk`), `UPLOAD_DIR` / `UPLOAD_MAX_BYTES` in `localrag/settings.py` |
+| Background ingest jobs | `localrag/api/jobs.py`, `localrag/api/routers/ingest.py` (async routes), `localrag/api/service.py` (`ingest_directory_async`, `get_ingest_job`) |
 | DI / shared service instances | `localrag/api/dependencies.py` |
 | Log format, levels, request ID | `localrag/logging_config.py`, `localrag/api/middleware.py`, `LOG_LEVEL` in `localrag/settings.py` |
 | API key auth | `localrag/api/dependencies.py` (`require_api_key`), `API_KEY` in `localrag/settings.py` |