title	Chat with PDF
emoji	💬
colorFrom	indigo
colorTo	violet
sdk	gradio
sdk_version	5.35.0
app_file	app.py
pinned	false

Chat with PDF: Enterprise-Grade RAG System

A production-grade PDF question-answering system built on Retrieval-Augmented Generation (RAG). Started as a primitive chatbot and systematically upgraded across 6 layers to enterprise level with observability, error handling, quality evaluation, advanced retrieval, security guardrails, and semantic caching.

---

What This Is

Most RAG tutorials show you how to get something working. This project shows you how to make it production-ready. Every layer adds a capability that enterprise systems require - and the reasoning behind each decision is documented in code comments.

Before: 4 files, no error handling, no logging, no security, flan-t5-large (3GB local model), k=2 retrieval, naive chunking.

After: Full observability pipeline, classified error handling, RAGAS evaluation, hybrid BM25+FAISS+RRF retrieval, cross-encoder reranking, HyDE query expansion, jailbreak/injection/PII protection, semantic caching, and index persistence.

---

Architecture

PDF Upload
    |
    v
[Layer 6] Index Cache Check
    | hit: load from disk (0.5s)
    | miss: build indexes (7s)
    v
pymupdf text extraction
    |
    v
RecursiveCharacterTextSplitter (1500 chars, 200 overlap)
    |
    v
HuggingFace Embeddings (all-MiniLM-L6-v2) + FAISS index
BM25Okapi keyword index
    |
    v
[Saved to disk for next load]

User Query
    |
    v
[Layer 5] Input Security Guard
    | jailbreak patterns -> blocked instantly
    | injection patterns -> blocked instantly
    v
[Layer 6] Semantic Cache Lookup
    | HIT (similarity > 0.92) -> return cached answer (0.014s)
    | MISS -> continue
    v
[Layer 4] HyDE: LLM generates hypothetical answer
    |
    v
[Layer 4] Hybrid Retrieval
    | BM25 (keyword) on original query -> top 8 chunks
    | FAISS (semantic) on hypothesis   -> top 8 chunks
    | RRF combination (k=60)           -> ~13 unique chunks
    v
[Layer 2] Retrieval Score Threshold Check
    | score > 1.8 -> off-topic, blocked
    v
[Layer 5] Chunk Security Guard
    | injection patterns in document -> blocked
    | PII detected -> warning logged
    v
[Layer 4] CrossEncoder Reranking (ms-marco-MiniLM-L-2-v2)
    | scores 13 candidates against query
    | keeps top 5
    v
LLM (Groq llama-3.1-8b-instant)
    |
    v
[Layer 5] Output Security Guard
    | PII in answer -> blocked/redacted
    | harmful content -> blocked
    v
[Layer 6] Cache Store
    |
    v
[Layer 1] Trace Written (JSONL + SQLite)
    |
    v
Answer returned to user

---

The 6 Layers

Layer 0 - Foundation

• Replaced flan-t5-large (3GB local) with Groq API (llama-3.1-8b-instant)
• Rewrote chain using modern LCEL (LangChain Expression Language)
• Switched PDF loader from PyPDFLoader to pymupdf for multi-column academic papers
• Fixed import bugs and dependency conflicts in original repo

Layer 1 - Observability

Every request gets a Trace with Spans measuring each step:

[TRACE a3f9c1b2]  2026-08-09T04:52:36
  Query     : what is the abstract?
  Retrieval : 1.74s  | chunks=5 | top_score=1.55
  LLM       : 1.68s  | ~243 tokens
  Total     : 3.46s

Results persisted to logs/traces.jsonl and logs/traces.db (SQLite). Query your own traces:

import sqlite3
con = sqlite3.connect("logs/traces.db")
con.execute("SELECT query, latency_total, latency_llm FROM traces ORDER BY latency_total DESC LIMIT 10").fetchall()

Layer 2 - Error Handling

Classified failure types with specific responses:

• RETRIEVAL_FAILURE - no relevant chunks found, or score above threshold
• LLM_FAILURE - model timeout or empty response
• EMPTY_DOCUMENT - PDF has no extractable text (scanned image)
• API_ERROR - Groq or external service unavailable

Retrieval score threshold (1.8) blocks off-topic questions before calling the LLM:

"what is the capital of France?" -> score=1.86 > 1.8 -> blocked in 0.026s

Layer 3 - RAG Evaluation

RAGAS evaluation pipeline with golden test set. Baseline scores:

Faithfulness      : 0.844  (hallucination rate)
Context Precision : 0.688  (retrieval quality)

Run evals after any change to measure improvement:

python run_eval.py path/to/your.pdf

Layer 4 - Advanced Retrieval

Three stacked improvements, each measured against baseline:

Hybrid Search (BM25 + FAISS + RRF)

• BM25 finds exact keyword matches ("abstract", "BERTScore")
• FAISS finds semantically similar content
• Reciprocal Rank Fusion (k=60) combines both ranked lists
• Result: Faithfulness 0.844 -> 0.925, Precision 0.688 -> 0.723

CrossEncoder Reranking

• ms-marco-MiniLM-L-2-v2 scores each query-chunk pair together
• Reduces 13 candidates to 5 high-quality chunks
• Bi-encoder retrieves fast, cross-encoder selects accurately

HyDE (Hypothetical Document Embeddings)

• LLM generates a hypothetical answer to the question
• FAISS searches using the hypothesis (closes question-answer semantic gap)
• BM25 still uses original query for exact keyword matching
• Final scores: Faithfulness 0.903, Context Precision 0.775

Layer 5 - Security & Guardrails

Three checkpoints, all in security.py:

Input Guard (before retrieval)

• Detects 12+ jailbreak patterns ("ignore all previous instructions", "DAN mode")
• Detects prompt injection in the query itself
• Blocked queries return in <0.001s - no retrieval, no LLM call

Chunk Guard (after retrieval, before LLM)

• Scans retrieved document chunks for injected instructions
• Detects PII (SSN, credit cards, emails, phone numbers)
• RAG-specific attack: malicious text embedded in uploaded PDF

Output Guard (after LLM, before user)

• Detects PII that leaked into the answer
• Detects harmful content patterns
• Redacts or blocks before returning

Layer 6 - Production Hardening

Index Persistence

• Document hash identifies each PDF uniquely
• FAISS and BM25 indexes saved to disk after first build
• Same PDF uploaded again: loads in 0.5s instead of 7s

Semantic Cache

• Caches question-answer pairs per document (document-aware)
• Uses embedding cosine similarity (threshold 0.92) for fuzzy matching
• "what is the abstract?" and "What's the abstract?" both hit the same cache entry
• Cache hit latency: 0.014s vs 3.5s full pipeline (247x faster)

---

Evaluation Results

Metric	Baseline	After Layer 4	Improvement
Faithfulness	0.844	0.903	+7.0%
Context Precision	0.688	0.775	+12.6%

---

Project Structure

chat-with-pdf/
├── app.py           - entry point
├── chain.py         - RAG pipeline (LCEL, hybrid retrieval, HyDE, reranking)
├── interface.py     - Gradio UI
├── utils.py         - shared state
├── tracer.py        - Layer 1: observability (traces, spans, SQLite)
├── security.py      - Layer 5: input/chunk/output guards
├── cache.py         - Layer 6: semantic cache + index persistence
├── evaluator.py     - Layer 3: RAGAS evaluation pipeline
├── eval_set.py      - Layer 3: golden test set
├── run_eval.py      - Layer 3: eval runner script
├── requirements.txt - pinned dependencies
└── logs/
    ├── traces.jsonl      - request logs
    ├── traces.db         - SQLite for querying
    ├── cache/            - semantic cache per document
    └── indexes/          - persisted FAISS+BM25 indexes

---

Setup

Prerequisites

• Python 3.10+
• Groq API key (free at console.groq.com)

Installation

git clone https://github.com/serasr/Chat-with-PDF.git
cd Chat-with-PDF

python -m venv venv
# Windows:
venv\Scripts\activate
# Mac/Linux:
source venv/bin/activate

pip install -r requirements.txt

Environment

Create a .env file in the project root:

GROQ_API_KEY=your_key_here

Run

python app.py

Open http://127.0.0.1:7860 in your browser.

On first run, the embedding model (~90MB) downloads automatically. Set cache location to avoid filling your system drive:

# Windows PowerShell
$env:HF_HOME = "D:\hf-cache"
python app.py

Run Evaluation

python run_eval.py path/to/your.pdf

---

Key Design Decisions

Why Groq instead of a local model? flan-t5-large (original) required a 3GB download, used 4GB RAM, took 8 seconds per response, and produced poor answers. Groq's llama-3.1-8b-instant responds in ~1 second with dramatically better quality. For a learning project, the free tier is sufficient.

Why pymupdf instead of PyPDFLoader? PyPDFLoader reads PDFs left-to-right across the page, mixing columns in two-column academic papers. pymupdf (fitz) understands page structure and preserves column separation.

Why BM25 alongside FAISS? Semantic search misses exact term matches. Asking "what is the abstract?" in a RAG paper returned copyright text and references which are semantically similar to academic content but not the abstract. BM25 finds the chunk with the literal word "Abstract" as a section header. Both are needed.

Why RRF over score normalization? BM25 scores and FAISS L2 distances are on completely different scales. Normalizing them requires assumptions about score distributions. RRF uses only rank positions, which are scale-independent. A chunk ranked 2nd by both systems beats one ranked 1st by only one system.

Why document-aware caching? "What is the abstract?" means completely different things for different PDFs. The cache uses a MD5 hash of document content as a namespace. Cache entries from one document never contaminate lookups for another.

---

Security Notes

• Never commit your .env file (already in .gitignore)
• The security layer uses pattern matching, not an LLM judge - fast but not exhaustive
• PII detection covers SSN, credit cards, emails, US phone numbers, passports
• For production deployment, add rate limiting per user/IP

---

Tech Stack

Component	Technology
LLM	Groq (llama-3.1-8b-instant)
Embeddings	sentence-transformers/all-MiniLM-L6-v2
Reranker	cross-encoder/ms-marco-MiniLM-L-2-v2
Vector store	FAISS (local)
Keyword search	BM25Okapi (rank-bm25)
PDF parsing	pymupdf (fitz)
RAG framework	LangChain 0.3.25 (LCEL)
UI	Gradio
Evaluation	RAGAS
Tracing	Custom (SQLite + JSONL)

---

License

MIT License. See LICENSE.

Author

Sera Singha Roy

---