Code Compass

Code Compass lets you paste in a public GitHub repository and ask questions about the codebase in plain English.

It clones and indexes the repository, retrieves the most relevant parts of the code for each question, and generates an answer grounded in those sources. Answers include clickable citations back to the files and line ranges used.

I built this project to explore RAG on real codebases instead of documents. Code retrieval has different challenges from normal document search. Exact symbols and filenames matter, implementations are often spread across multiple files, and retrieving code that is only loosely related to the question can lead to a convincing but incorrect answer.

Screenshots

How it works

1. The user submits a public GitHub repository.
2. The backend clones the repository and filters supported files.
3. Source code is parsed with tree-sitter and split around functions, classes, symbols, and module boundaries. Unsupported files fall back to plain-text chunking.
4. Chunks are embedded and stored in Chroma.
5. Each question runs through semantic vector search and BM25 lexical search.
6. Results are combined using Reciprocal Rank Fusion.
7. A cross-encoder reranks the strongest candidates.
8. The final context is sent to the LLM.
9. The LLM generates an answer with inline citations such as [1] and [2].
10. Citations are validated against the retrieved sources before the response is returned.

Architecture

┌──────────────────────┐
│      React UI        │
│  repo submit + chat  │
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│    FastAPI Server    │
│   API + sessions     │
└──────────┬───────────┘
           │
           ▼
┌──────────────────────────────┐
│      CodebaseRAGSystem       │
│ indexing + query pipeline    │
└───────┬──────────────┬───────┘
        │              │
        ▼              ▼
┌──────────────┐ ┌──────────────┐
│ RepoFetcher  │ │  CodeParser  │
│ clone/filter │ │  tree-sitter │
└──────┬───────┘ └──────┬───────┘
       │                 │
       └────────┬────────┘
                ▼
       ┌──────────────────┐
       │ Hybrid Retrieval │
       │ vectors + BM25   │
       │ RRF + reranking  │
       └────────┬─────────┘
                ▼
       ┌──────────────────┐
       │   LLM Answerer   │
       │ answer + sources │
       └──────────────────┘

Retrieval

The retrieval pipeline combines semantic and lexical search because code questions often contain a mix of natural language and exact identifiers.

Code-aware chunking

tree-sitter splits supported source files around functions, classes, methods, and declarations instead of arbitrary fixed-length chunks.

Each source file also gets a lightweight module overview containing information such as its path, imports, exports, and symbols. This helps with architectural questions and files that mainly connect other parts of the codebase.

Hybrid search

Semantic vector search and BM25 run in parallel.

Semantic search helps with questions such as:

How does an incoming request reach a view?

BM25 is useful when the question contains exact code terminology such as:

QuerySet
APIRouter
OAuth2AuthorizationCodeBearer

The two rankings are combined using Reciprocal Rank Fusion.

Reranking

The strongest candidates are passed through a cross-encoder before the final context is selected.

Question
   │
   ├── Semantic search
   │
   └── BM25
          │
          ▼
         RRF
          │
          ▼
   Cross-encoder
          │
          ▼
    Final context
          │
          ▼
         LLM

Tech stack

Frontend

• React
• Tailwind CSS
• Axios
• Vercel

Backend

• Python
• FastAPI
• Pydantic
• Hugging Face Spaces

RAG

• tree-sitter
• Chroma
• BM25
• Reciprocal Rank Fusion
• Cross-encoder reranking
• Cohere Embed v3
• Qwen3 Coder Next through Amazon Bedrock
• Groq for the hosted version

Evaluation

I added an evaluation harness so retrieval changes can be tested without manually running the frontend and repeating the same questions.

python evals/run_eval.py

The benchmark contains 24 hand-written questions across three repositories:

• Documenso, a large TypeScript monorepo
• FastAPI, a Python API framework
• Django, a large Python web framework

There are eight questions per repository covering architecture, implementation lookup, cross-file flows, APIs, configuration, tests, security and error handling, and conversational follow-ups.

The evaluation checks whether expected sources are retrieved, how highly the first expected source is ranked, and whether the generated answer is supported by the retrieved context.

The evaluation setup uses Qwen3 Coder Next through Amazon Bedrock with Cohere Embed v3 embeddings.

Results

Metric	Result
Retrieval Hit Rate @ 5	83.3%
Top-1 Hit Rate	62.5%
Mean Reciprocal Rank	0.72
Faithfulness	0.92

The retrieval results show that relevant source files are usually present in the final context, while the difference between Top-5 and Top-1 reflects cases where a related documentation file, test, helper, or consumer ranks above the main implementation.

Faithfulness measures whether claims in the generated answer are supported by the retrieved context. It is evaluated separately from retrieval so that a relevant retrieval result does not automatically count as a grounded answer.

The full evaluation set is available at:

server/evals/sample_eval_set.json

Running locally

Backend

cd server

python -m venv .venv
source .venv/bin/activate

pip install -r requirements.txt

export LLM_PROVIDER=bedrock
export EMBEDDING_PROVIDER=bedrock
export AWS_REGION=us-east-1

export BEDROCK_LLM_MODEL=qwen.qwen3-coder-next
export BEDROCK_EMBEDDING_MODEL=cohere.embed-english-v3

export CHROMA_PATH=./data/chroma

python server_app.py

The API runs at http://localhost:8000.

Frontend

cd ui
npm install
npm start

Create ui/.env:

REACT_APP_API_URL=http://localhost:8000

The frontend runs at http://localhost:3000.

Deployment

The frontend is deployed on Vercel.

The FastAPI backend runs as a Docker Space on Hugging Face Spaces and is deployed through GitHub Actions.

The hosted version uses Groq for generation and local all-MiniLM-L6-v2 embeddings to keep the deployment lightweight.

export LLM_PROVIDER=groq
export EMBEDDING_PROVIDER=local
export GROQ_API_KEY=<your-groq-api-key>
export CHROMA_PATH=./data/chroma

Tradeoffs

• Repository and session state is mostly kept in memory, so backend restarts require re-indexing.
• Cloned repositories are deleted after indexing.
• Large repositories can take time to index.
• Hybrid retrieval and cross-encoder reranking improve retrieval quality but add latency.
• Related documentation, tests, or helper code can still rank above the canonical implementation.
• Retrieval works on chunks independently and does not currently use a dependency or call graph.

Project structure

server/
  server_app.py
  evals/
    run_eval.py
    sample_eval_set.json
  src/
    code_parser.py
    embeddings.py
    hybrid_search.py
    rag_system.py
    repo_fetcher.py
    vector_store.py

ui/
  src/

README.md