🧠 Local Agentic RAG System

A Retrieval-Augmented Generation system that routes between a local FAISS knowledge base and live web search based on confidence scoring, built with LangGraph, Qwen 2.5, and Tavily - deployed free on HuggingFace Spaces using ZeroGPU.

---

Table of Contents

• 🧠 Local Agentic RAG System
  • Table of Contents
  • Overview
  • Architecture
  • The Engineering Journey
  • Key Features
  • Tech Stack
  • Live Demo
  • Getting Started
    • Run the reference implementation (Colab)
    • Run the deployed version locally
  • Configuration
  • Engineering Decisions & Lessons Learned
  • Known Limitations
  • Roadmap
  • License

---

Overview

This system answers questions by first checking a local FAISS vector store for relevant, previously-indexed knowledge. If the best match falls below a confidence threshold, it falls back to a live Tavily web search and indexes those results into FAISS so the next related question is answered from cache instead of hitting the API again.

Every response includes conversation-aware follow-up handling (a bare "who scored?" after a football question resolves correctly) and deterministically-extracted source citations regardless of whether the underlying LLM remembers to cite inline.

"What was the Brazil vs Haiti World Cup 2026 result?"
    -> FAISS empty -> Tavily search -> indexed -> answered, cited

"Who scored the goals?"
    -> follow-up detected -> enriched query -> FAISS hit (cached) -> answered

Architecture

                        User Query
                            │
                            ▼
                 ┌─────────────────────┐
                 │   retrieve_node     │   FAISS similarity search
                 │  (confidence score) │   sim = 1 / (1 + L2_distance)
                 └──────────┬──────────┘
                            │
                  sim ≥ 0.60?
                 ┌──────────┴──────────┐
                 │                     │
                YES                   NO
                 │                     │
                 │                     ▼
                 │          ┌─────────────────────┐
                 │          │      web_node       │ Tavily search + FAISS indexing
                 │          │  (index + answer)   │   
                 │          └──────────┬──────────┘
                 │                     │
                 └──────────┬──────────┘
                            ▼
                 ┌─────────────────────┐
                 │     answer_node     │ Qwen 2.5, greedy decoding + deterministic citations
                 │  (history-aware)    │   
                 └──────────┬──────────┘
                            ▼
                     Answer + Sources

A clean three-node LangGraph StateGraph - no ReAct prompting, no regex parsing of model output. Routing is a single confidence-score comparison, and generation is deterministic (greedy decoding) for reproducible, citation-grounded answers.

The Engineering Journey

This repository intentionally keeps three notebooks, each representing a real stage of the build — not a cleaned-up final draft. The middle notebook documents an approach that didn't ship; it's kept because the reasoning behind abandoning it is as informative as the approach that worked.

Notebook	Environment	Outcome
01_agentic_rag_notebook_version	Google Colab, free T4 GPU	✅ Working - Qwen 2.5-7B, 4-bit quantized, full pipeline
02_agentic_rag_hf_cpu_deployment	HuggingFace Spaces, CPU Basic	⚠️ Built, then blocked - see below
03_agentic_rag_hf_zerogpu_deployment	HuggingFace Spaces, ZeroGPU	✅ Deployed - Qwen 2.5-7B restored, @spaces.GPU

Why three attempts, not one:

1. Notebook 01 was developed and debugged on Colab's free T4 GPU: Qwen 2.5-7B-Instruct with 4-bit NF4 quantization (bitsandbytes), a Gradio UI with real token streaming via TextIteratorStreamer, and FAISS-backed session memory. This is the reference implementation.

2. Notebook 02 targeted HuggingFace Spaces' CPU Basic tier for free hosting. This required downgrading to Qwen 2.5-3B (no GPU means no 4-bit quantization), removing torch.compile, and switching the streaming approach entirely - from model.generate() + a background thread to a simpler "generate complete response, then simulate streaming word-by-word" pattern, after the threaded approach caused multi-minute hangs from torch.compile recompiling on every call. Mid-deployment, HuggingFace changed its free-tier policy: CPU Basic now requires a PRO subscription for Gradio/Docker Spaces. The create_repo() call returned 402 Payment Required.

3. Notebook 03 pivoted to ZeroGPU — HuggingFace's free shared-GPU tier, which remained available without a subscription. This meant restoring the 7B model and 4-bit quantization, but required learning ZeroGPU's specific execution model: real GPU access only exists inside functions decorated with @spaces.GPU, model weights must be placed on cuda at module scope (not lazily), and torch.compile isn't supported on this tier either. The final architecture isolates GPU access to a single small function - everything else (retrieval, prompt building, history handling) runs on CPU and never competes for the shared GPU slot.

Each pivot was driven by a concrete constraint, not a style preference - which is why all three are documented here rather than only the one that shipped.

Key Features

• Confidence-based routing - FAISS similarity scores decide whether a question is answered from cache or triggers a live web search
• Session memory — Tavily results are indexed into FAISS immediately, so repeated or related questions don't re-spend API credits
• Multi-turn conversation — a lightweight three-signal heuristic (short message + reference word + question starter) detects genuine follow-ups and enriches the retrieval query without polluting unrelated new questions
• Deterministic citations — source URLs are extracted directly from retrieved context and appended after every answer, independent of whether the model remembered to cite inline
• Live timing metrics — retrieval time, web search time, generation time, and tokens/sec are surfaced per query in the UI
• Graph-native pipeline — no ReAct scratchpad, no regex parsing of model output; routing is a single numeric comparison

Tech Stack

Component	Technology
Orchestration	LangGraph
Vector store	FAISS + sentence-transformers/all-MiniLM-L6-v2
Web search	Tavily Search API
LLM	Qwen/Qwen2.5-7B-Instruct (4-bit NF4, bitsandbytes)
GPU access (deployed)	HuggingFace ZeroGPU (@spaces.GPU)
UI	Gradio (ChatInterface, streaming via sync generators)

Live Demo

👉 At my HuggingFace Space

Getting Started

Run the reference implementation (Colab)

1. Open notebooks/01_agentic_rag_notebook_version.ipynb in Google Colab
2. Set the runtime to a T4 GPU (Runtime -> Change runtime type)
3. Add your TAVILY_API_KEY to Colab Secrets (key icon, left sidebar)
4. Run all cells - the last cell launches a Gradio UI with a public link

Run the deployed version locally

Same with the notebook version, since all the modules are created and available in the notebook, all that needs to be set is the HuggingFace Hub and the Tavily API key. Also, the HuggingFace Space need to be set manually for the ZeroGPU tier, and to be loaded programmatically from the notebook setting using huggingface_hub.upload_file function. All are demonstrated and implemented in the notebook.

Note: space/requirements.txt includes bitsandbytes and spaces, both of which expect a CUDA-capable environment. Running this locally without a GPU will not work as-is - see 02_agentic_rag_hf_cpu_deployment for a CPU-only reference (Qwen 2.5-3B, no quantization).

Configuration

Variable	Where	Purpose
TAVILY_API_KEY	Colab Secrets / HF Space secrets / env var	Web search
CONFIDENCE_THRESHOLD	config.py	FAISS similarity cutoff for routing (default 0.60)
MAX_PROMPT_TOKENS	config.py	Prompt token budget before left-trimming
ZEROGPU_DURATION	config.py	Max seconds per @spaces.GPU call (ZeroGPU only)

Engineering Decisions & Lessons Learned

• FAISS confidence scoring: similarity_search_with_score() returns L2 distance, not similarity. Converting via sim = 1 / (1 + L2) gives a bounded (0, 1] score that's stable enough to threshold on directly.

• text_gen_pipeline() over direct model.generate(): an earlier attempt called model.generate() from a background thread with TextIteratorStreamer for true token streaming. On a torch.compile-d model this triggered full JIT recompilation on the first direct call, hanging for several minutes. Routing through the HF pipeline() object avoided this entirely; streaming became "generate fully, then reveal word-by-word" instead of true token streaming — a reasonable trade for reliability.

• _safe_content() defensive helper: Gradio 6.x can store chat message content as either a plain string or a list of typed content blocks ([{"type": "text", "text": "..."}]), even in text-only conversations. Code that assumed .strip() always worked crashed on the second turn of every conversation until this was normalized in one place.

• Follow-up detection needs all three signals, not one: initially, any short message got the previous question prepended to its retrieval query. This polluted genuinely new questions ("What is machine learning?" right after a football question). Requiring a short message and a reference word and a question starter together turned out to reliably separate real follow-ups from new topics.

• Module-scope GPU placement on ZeroGPU: counter-intuitively, model weights must be moved to cuda at import time, not lazily inside the @spaces.GPU-decorated function - import spaces patches torch to intercept that placement correctly even before a real GPU is attached.

Known Limitations

• FAISS index is in-memory and resets on Space restart - no persistent vector store across sessions (by design, for a free-tier deployment)
• ZeroGPU is a shared resource; occasional queueing under load is expected
• Follow-up detection is a heuristic, not a learned classifier - it will occasionally misclassify an ambiguous short question
• Citations are extracted from retrieved context, not verified against the generated claim sentence-by-sentence

Roadmap

• Persistent vector store option (e.g., a hosted FAISS/pgvector backend)
• Evaluate swapping the follow-up heuristic for a lightweight classifier
• Add a docs/ write-up with benchmark numbers (retrieval time, tok/s, end-to-end latency) across all three deployment environments

License

This project is licensed under the Apache license 2.0.

see LICENSE for details.

---

Built by @Saint Jeane.