English · 简体中文
Embedding-free multi-hop evidence retrieval.
The companion benchmark repository for the SAG2 paper. SQLite FTS5, BM25 and JEV connect source evidence at query time, with no model calls for indexing.
Question → Passages → Terms → New passages
SAG2 application: SAG-Wiki · Original SAG reproduction
SAG2-English.mp4
Contents
- SAG2
- Extensions
- Code and Reproduction
SAG2 was evaluated on 1,000 questions each from MuSiQue, 2WikiMultiHopQA and HotpotQA. R@10 is 86.62%, 97.43% and 99.00%, respectively. EnterpriseRAG-Bench covers 511,962 source documents, with document R@10 of 81.10% over 470 labeled questions.
The paper's main table is below; see the evaluation settings for its configuration. Values are percentages; bold denotes the column maximum. R@k is the mean fraction of labeled evidence recovered in the first k deduplicated results; AE@k is the fraction of questions with all evidence recovered. R@k reports evidence recall.
| Method | MuSiQue | 2WikiMultiHopQA | HotpotQA | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R@1 | R@2 | R@5 | R@10 | R@1 | R@2 | R@5 | R@10 | R@1 | R@2 | R@5 | R@10 | |
| Lexical and dense retrieval | ||||||||||||
| BM25 | 28.53 | 37.32 | 47.47 | 54.41 | 40.10 | 57.93 | 67.05 | 71.50 | 39.75 | 57.95 | 74.40 | 86.90 |
| Contriever | 24.50 | 33.54 | 45.32 | 53.58 | 34.40 | 48.60 | 59.25 | 65.20 | 39.30 | 57.45 | 74.85 | 82.75 |
| Large embedding models | ||||||||||||
| BGE-Large-EN-v1.5 | 29.42 | 41.68 | 56.32 | 64.08 | 41.48 | 62.28 | 69.83 | 73.75 | 45.45 | 76.20 | 89.20 | 93.35 |
| GTE-Qwen2-7B | 35.19 | 48.06 | 61.92 | 70.80 | 43.45 | 65.58 | 74.18 | 78.63 | 46.40 | 73.95 | 88.45 | 93.55 |
| GritLM-7B | 34.86 | 50.70 | 66.22 | 75.19 | 41.85 | 66.23 | 75.62 | 79.40 | 46.40 | 81.15 | 93.40 | 97.20 |
| NV-Embed-v2 | 34.02 | 51.08 | 68.84 | 77.27 | 43.23 | 68.93 | 77.00 | 80.75 | 46.90 | 85.55 | 95.25 | 97.70 |
| Structure-enhanced retrieval | ||||||||||||
| HippoRAG 2 | 30.65 | 49.52 | 65.13 | 73.76 | 42.38 | 76.55 | 90.35 | 93.40 | 44.40 | 78.35 | 94.35 | 97.15 |
| SAG (Ours) | 36.82 | 63.62 | 80.36 | 83.37 | 43.81 | 83.93 | 93.34 | 93.57 | 47.80 | 91.55 | 96.50 | 97.70 |
| SAG2 (Ours) | 34.30 | 62.18 | 80.68 | 86.62 | 43.55 | 85.50 | 97.35 | 97.43 | 48.30 | 93.60 | 98.45 | 99.00 |
BM25 uses the paper's fixed lexical reference; cited baselines and provenance are in main table data. These are fixed development subsets; MuSiQue has 518 two-hop, 316 three-hop and 166 four-hop questions and was used for method development. Enterprise results are shown below. Ablations and Agentic results are reported in later sections.
EnterpriseRAG-Bench contains 511,962 source documents. The table reports document R@10 (%) on 470 labeled questions.
| Question type | n | BM25 | Vector | Bash Agent† | SAG2 (Ours) |
|---|---|---|---|---|---|
| Basic retrieval | 175 | 77.7 | 48.6 | 61.7 | 86.86 |
| Semantic retrieval | 125 | 43.2 | 24.8 | 29.6 | 58.40 |
| Within-document reasoning | 40 | 90.0 | 57.5 | 75.0 | 100.00 |
| Project-related | 40 | 65.5 | 49.2 | 43.2 | 83.58 |
| Constrained | 30 | 85.0 | 83.3 | 78.3 | 96.67 |
| Conflicting information | 20 | 82.5 | 55.0 | 72.5 | 92.50 |
| Completeness | 20 | 46.5 | 33.4 | 59.0 | 76.24 |
| Miscellaneous | 20 | 90.0 | 75.0 | 100.0 | 100.00 |
| Overall | 470 | 68.4 | 46.0 | 55.8 | 81.10 |
Enterprise baselines are cited from the dataset paper. Bash Agent has variable output size; other methods in this table return ten documents, so computation and output budgets differ.
- Index source text. Tokenize titles and text into SQLite FTS5; retain source identity, headings and term statistics. No embeddings or offline semantic extraction are required.
- Retrieve passages from the question. BM25 returns up to 100 candidates. JEV
noulscores whether each passage supplies necessary evidence for the original question and available context. - Select terms from passages. Up to 10 expansion seeds contribute up to 20 terms, using frequency, rarity, query context and JEV scores.
- Retrieve more evidence. Search each term, interleave and deduplicate results, retain earlier evidence and score again. The default adds two expansion rounds.
- Return sources. Locally diversify the top ten results without another model request.
JSON corpus text is treated as plain text by default; a supplied path selects Markdown or code parsing by its file extension. No generated annotations are added to source text. Gold answers and evidence labels are used only for evaluation. The algorithm, prompts and query optimizations are in scripts/sag2.py. The original event/entity approach is documented in sag/.
Run the following commands from SAG-Benchmark/sag2/. Original SAG readers should instead follow sag/README.md, which has its own environment and dependencies.
Requirements: Python 3.11+, uv and a C compiler. The SQLite scoring extension compiles on first use: Xcode Command Line Tools on macOS, GCC or Clang on Linux, or MSVC Build Tools on Windows. Default SAG2 needs no Docker, MySQL, Elasticsearch or GPU. Shell examples use Bash (Linux, macOS or WSL); in PowerShell, use Copy-Item for the configuration copy. Paper-control scripts currently require a Unix-like environment.
git clone https://github.com/Zleap-AI/SAG-Benchmark.git
cd SAG-Benchmark/sag2
uv sync --frozenFor an existing checkout, run cd sag2 from the repository root. Run subsequent SAG2 commands here, using sag2/.venv/.
Create the configuration once, preserving an existing file:
test -f .env || cp .env.example .envSet these values in the current directory’s .env (repository path: sag2/.env):
OPENROUTER_API_KEY=your_api_key
JEV_BASE_URL=https://openrouter.ai/api/alpha/decisions
JEV_MODEL=typesafe/jev-1.13The default answer type is noul. The historical paper run returned typesafe/jev-1.13-20260917; provider versions may change, so record the actual returned model. Keys are read from environment variables or --env-file, never passed as command-line arguments.
# Download missing pinned data and index all three datasets; no model requests
uv run python scripts/run_sag2_benchmark.py --download --index-onlyAn index with the same corpus and chunking configuration is reused. The terminal prints source count, chunk count and question count; the suite summary reports indexed. Indexes default to .cache/sag2/. Add --dataset musique to prepare only one dataset.
uv run python scripts/run_sag2_benchmark.py \
--dataset musique --limit 10 --concurrency 20 \
--output-dir output/sag2/musique-smoke--limit changes the question count, not the corpus size. This step uses the Jev API. The terminal prints recall metrics; per-question results are in output/sag2/musique-smoke/musique/details.jsonl, with metrics.json in the same directory.
# 1,000 questions per dataset
uv run python scripts/run_sag2_benchmark.py --datasets all --concurrency 20
# Or choose specific datasets
uv run python scripts/run_sag2_benchmark.py \
--datasets hotpotqa 2wikimultihopqa --concurrency 20Add --output-dir output/sag2/my-run for an explicit destination; it must not already exist for a new run. Check dependencies, data and the small run before starting full evaluation.
| Dataset argument | Questions | Source passages / documents | Evaluation unit |
|---|---|---|---|
musique |
1,000 | 11,656 | Supporting passage |
2wikimultihopqa |
1,000 | 6,119 | Supporting passage |
hotpotqa |
1,000 | 9,811 | Supporting passage |
| EnterpriseRAG-Bench, separate entry point | 470 labeled | 511,962 | Source document |
The three multi-hop datasets come from HippoRAG 2 at revision 5ec05b38deecc3318bb432c69865959c56058990. --datasets all means the first three rows. Enterprise uses the separate entry point below. See dataset/README.md for format and license information.
Prepare the official directory with documents/, questions.jsonl and download_manifest.json. Check for an existing matching index before building another one.
# Only for the first build of a current-format index; preserve older indexes
uv run python scripts/build_enterprise_index.py \
--dataset-dir ../../datasets/EnterpriseRAG-Bench \
--index-path .cache/sag2/enterprise.sqlite \
--output-dir output/enterprise/index
# Reuse that index and evaluate all 470 labeled questions
uv run python scripts/run_enterprise.py \
--dataset-dir ../../datasets/EnterpriseRAG-Bench \
--index-summary output/enterprise/index/summary.json \
--output-dir output/enterprise/evaluation --concurrency 20The evaluator opens the index read-only and writes progress.json, summary.json, per-question qst_*.json files and requests/. Repeating the same command reuses saved results; use a new output directory when changing models, data or settings. A schema4 index cannot be passed directly to the current schema5 reader. Keep the old file and use its matching compatibility implementation; changing a version marker does not convert an index.
| Argument | Default | Purpose |
|---|---|---|
--mode |
sag2 |
sag2, agentic or local |
--dataset / --datasets |
all |
One or more multi-hop datasets |
--dataset-dir |
dataset/ |
Dataset files; defaults to sag2/dataset/ |
--limit / --offset |
All / 0 |
Question range; corpus stays complete |
--concurrency |
20 |
Concurrent questions |
--index-workers |
8 |
Index-building processes, separate from query concurrency |
--candidate-k |
100 |
Lexical candidates per round |
--seed-k |
10 |
Expansion seeds per round |
--expansions / --max-hops |
2 for SAG2; 1 for Agentic |
Additional rounds after the initial retrieval; 0, 1 or 2 |
--expansion-term-limit |
20 |
Expansion terms per round |
--top-k / --k-list |
10 / 1 2 5 10 |
Output size and evaluation cutoffs |
--chunk-chars |
2200 |
Chunk target in UTF-16 units |
--cache-dir |
.cache/sag2 |
SQLite index directory |
--output-dir |
Timestamped directory | Destination for a new run |
--resume |
Unset | Continue an existing run with its original settings |
--env-file |
sag2/.env (repository path) |
Configuration file; SAG_ENV_FILE can override the default |
# One expansion round, still using 100 candidates and 10 seeds
uv run python scripts/run_sag2_benchmark.py \
--dataset musique --limit 10 --expansions 1 --candidate-k 100 --seed-k 10
uv run python scripts/run_sag2_benchmark.py --helpChanging models, chunking or retrieval parameters changes the experiment. Match the model version, corpus, questions, chunking policy and metric units when comparing with published scores.
The following paths are relative to SAG-Benchmark/sag2/. Default output layout:
output/sag2/<timestamp>/
├── summary.json
└── <dataset>/
├── config.json
├── details.jsonl
├── metrics.json
├── search_results.json
├── bm25_results.json
└── requests/ # Agentic / local reranking modes
| File | Content and completion checks |
|---|---|
summary.json |
Final per-dataset summary; index-only runs report indexed |
config.json |
Model, settings, data and code hashes |
details.jsonl |
One completed question per line, with retrieval, metrics and usage |
metrics.json |
Final dataset metrics, all-evidence recall and usage |
search_results.json |
Final retrieved source text per question, for QA or analysis |
bm25_results.json |
Initial BM25 results for the same questions |
requests/ |
Agentic and local request/response records for recovery |
# Reuse the original datasets, range, model and settings
uv run python scripts/run_sag2_benchmark.py \
--dataset musique --limit 10 --concurrency 20 \
--resume output/sag2/musique-smokePlain SAG2 resumes completed questions. Agentic and local modes also reuse successful identical requests. Plain SAG2 does not persist each Jev request separately for recovery, so an interrupted question may send its requests again. Keep different model/configuration runs in separate output directories.
| Script | Purpose |
|---|---|
scripts/run_sag2_benchmark.py |
Multi-hop indexing, SAG2, Agentic and local evaluation |
scripts/build_enterprise_index.py |
Full Enterprise index construction |
scripts/run_enterprise.py |
Enterprise evaluation over an existing read-only index |
scripts/run_paper_controls.py |
MuSiQue paper controls, including model calls |
scripts/verify_paper_results.py |
Offline verification of retained paper records |
scripts/verify_agentic_results.py |
Verify 3,000 retained best Agentic results |
scripts/verify_rerank_results.py |
Verify retained reranking results |
scripts/search_local.py |
Query a custom JSON corpus with local reranking |
The current SAG2 entry points return evidence. The original SAG upload, answer-generation and LLM Judge workflows are preserved separately in sag/.
Optional extension. An LLM retains the original question and retrieval history, calls SAG2 as a tool, decides when to search or stop, and ranks the final ten passages. It adds no generated text at index time.
Best historical results over 1,000 questions per dataset are below. MuSiQue reaches 92.20% R@5, a gain of 11.53 percentage points over a single SAG2 run using unrounded values, and 95.39% R@10. R@5 is 99.88% on 2WikiMultiHopQA and 99.30% on HotpotQA.
| Method | MuSiQue | 2WikiMultiHopQA | HotpotQA | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R@1 | R@2 | R@5 | R@10 | R@1 | R@2 | R@5 | R@10 | R@1 | R@2 | R@5 | R@10 | |
| SAG2 | 34.30 | 62.18 | 80.68 | 86.62 | 43.55 | 85.50 | 97.35 | 97.43 | 48.30 | 93.60 | 98.45 | 99.00 |
| Agentic SAG2 | 38.21 | 71.99 | 92.20 | 95.39 | 44.08 | 88.00 | 99.88 | 99.88 | 49.10 | 96.45 | 99.30 | 99.65 |
Evaluation settings: gpt-6.1-sol, high reasoning, one expansion per SAG2 call, at most three tool calls per question, 20 passages returned per call and 10 final passages. The general API entry point requires no Codex account or CLI. Historical scores came from native MCP runs.
LLM_API_KEY=your_llm_key
LLM_BASE_URL=https://api.openai.com/v1
LLM_MODEL=gpt-6.1-sol
LLM_API=responses
LLM_REASONING_EFFORT=highuv run python scripts/run_sag2_benchmark.py --mode agentic \
--dataset musique --limit 10 --expansions 1 --max-tool-calls 3 --return-k 20Configure --llm-model, --llm-base-url, --llm-api responses|chat, --llm-api-key-env and --reasoning-effort. The provider must support the selected tool-calling format. See the Agentic guide and best results for configuration, tokens and costs.
Optional extension. Replace each Jev scoring round with a local model while keeping BM25, term expansion and final diversity ordering. After preparing weights and data, queries can run offline.
The following results were evaluated through 302 APIs, with 1,000 questions per dataset:
| Method | MuSiQue | 2WikiMultiHopQA | HotpotQA | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R@1 | R@2 | R@5 | R@10 | R@1 | R@2 | R@5 | R@10 | R@1 | R@2 | R@5 | R@10 | |
| BM25 | 28.71 | 37.50 | 47.66 | 54.50 | 40.25 | 58.18 | 67.08 | 71.58 | 39.75 | 58.05 | 74.45 | 86.95 |
| SAG2 + Jev(paired control) | 34.24 | 62.34 | 80.59 | 86.61 | 43.55 | 85.50 | 97.35 | 97.43 | 48.30 | 93.60 | 98.45 | 99.00 |
| SAG2 + Jina(two expansions) | 31.03 | 50.38 | 65.61 | 72.13 | 43.10 | 70.15 | 80.80 | 86.95 | 46.75 | 85.55 | 94.35 | 97.00 |
MuSiQue model and pipeline ablations
| MuSiQue ablation | R@1 | R@2 | R@5 | R@10 | AE@10 |
|---|---|---|---|---|---|
| BM25 | 28.71 | 37.50 | 47.66 | 54.50 | 17.50 |
| BM25+Qwen single scoring round | 31.66 | 45.44 | 60.17 | 66.88 | 32.50 |
| BM25+Jina single scoring round | 35.01 | 52.18 | 63.04 | 67.43 | 34.70 |
| SAG2+Qwen two expansions | 22.64 | 34.75 | 55.55 | 71.61 | 44.60 |
| SAG2+Qwen 0.6B two expansions | 25.67 | 40.81 | 62.54 | 74.18 | 48.40 |
| SAG2+Jina two expansions | 31.03 | 50.38 | 65.61 | 72.13 | 43.80 |
| SAG2+Jev baseline | 34.24 | 62.34 | 80.59 | 86.61 | 69.00 |
Single-round runs use the original query and standard relevance instructions. Two-expansion runs use SAG2 evidence instructions, context and final DPP, so these comparisons change both pipeline and prompts. qwen3-rerank and Qwen 0.6B are distinct models.
# Prepare dependencies and pinned weights while online
uv sync --frozen --group local
uv run --group local hf download Qwen/Qwen3-Reranker-0.6B \
--revision e61197ed45024b0ed8a2d74b80b4d909f1255473 --local-dir models/qwen3-reranker-0.6b
# Run ten questions offline once data and weights are ready
uv run --offline --group local python scripts/run_sag2_benchmark.py \
--mode local --reranker qwen --model-path models/qwen3-reranker-0.6b \
--dataset musique --limit 10 --expansions 2 --concurrency 20Available backends include Qwen Transformers, Jina Transformers and Jina MLX on Apple Silicon. Qwen weights use Apache-2.0; Jina uses CC BY-NC 4.0 and requires additional permission for commercial use. See the local reranking guide for custom corpora, memory, batch sizes and full ablations.
Five external methods live under sag/external/. Each has an independent environment and reads sag/.env. See the external-method guide for full steps, outputs and shared Judge evaluation.
| Method | Reproduction guide |
|---|---|
| Microsoft GraphRAG | graphrag |
| HippoRAG 2 | hipporag2 |
| HyperGraphRAG | hypergraphrag |
| Hyper-RAG | hyperrag |
| LightRAG | lightrag |
For HippoRAG 2, first copy sag/.env.example to sag/.env and configure the LLM and embedding endpoints; preserve an existing configuration. Run from the SAG-Benchmark repository root, reusing the MuSiQue files downloaded above:
cd sag/external/hipporag2
uv sync --frozen
uv run --frozen --env-file ../../.env python reproduce/Step_0_load_dataset.py \
--data_name musique --dataset-root ../../../sag2/dataset
uv run --frozen --env-file ../../.env python reproduce/Step_1_build_index.py --data_name musique
uv run --frozen --env-file ../../.env python reproduce/Step_3_response.py --data_name musiqueThe stages are data preparation, indexing, then retrieval and QA; the last two use model endpoints. GraphRAG separates retrieval and QA, while Hyper-RAG also requires Hypergraph-DB. Follow each method's guide for its parameters and data paths; run shared Judge evaluation from sag/.
SAG-Benchmark/
├── README.md / README-CN.md # SAG2 introduction, results and reproduction / SAG2介绍、成绩与复现
├── sag/ # Original SAG / 原SAG
│ ├── README.md / README-CN.md
│ ├── pipeline/ / scripts/ / prompts/
│ ├── external/ # GraphRAG, HippoRAG 2, HyperGraphRAG, Hyper-RAG, LightRAG
│ └── pyproject.toml / uv.lock / docker-compose.yml
└── sag2/ # SAG2 and its optional extensions
├── README.md / README-CN.md
├── pyproject.toml / uv.lock / .env.example
├── scripts/
│ ├── run_sag2_benchmark.py
│ ├── sag2.py / agentic_sag2.py / local_reranker.py
│ └── build_enterprise_index.py / run_enterprise.py
├── prompts/ / docs/ / dataset/ / results/ / tests/ / assets/
├── .cache/sag2/ # SQLite indexes / SQLite索引(运行时生成)
└── output/ # Evaluation results / 评测结果(运行时生成)
Each method has separate .env, .venv, dataset/ and output/ paths. Run SAG2 from sag2/ and original SAG from sag/. Original SAG uses --strategy sag and the SAGGraphSearcher event/entity graph implementation; SQLite SAG2 is implemented separately in sag2/.
| MuSiQue index and cost | Paper record |
|---|---|
| Source passages / indexed units | 11,656 / 11,656 |
| Index construction | 2.36 seconds |
| SQLite file | 25.87 MiB |
| Offline model calls / cost | 0 / USD 0 |
| Model cost / 100 questions | USD 0.433769 |
| Model cost / 1,000 questions | USD 4.326379 |
Construction time is the median of three builds on Apple M4 Pro, 64 GiB, eight processes. It includes tokenization, index writes, statistics and query initialization; it excludes imports, corpus loading and validation, and does not clear the OS file cache. Costs come from historical responses and exclude compute rental and answer generation. See paper data.
Full MuSiQue controlled experiments, 1,000 questions each
| Method | R@1 | R@2 | R@5 | R@10 | AE@5 | AE@10 |
|---|---|---|---|---|---|---|
| BM25 | 28.53 | 37.32 | 47.47 | 54.41 | 12.20 | 17.40 |
| BM25+JEV (H=0) | 31.13 | 46.79 | 60.78 | 67.93 | 26.60 | 36.00 |
| SAG2 (Ours), H=1 | 34.35 | 61.44 | 77.89 | 83.36 | 54.60 | 63.10 |
| SAG2 (Ours), H=2 | 34.30 | 62.18 | 80.68 | 86.62 | 59.00 | 69.50 |
| BM25+JEV,matched budget | 30.64 | 46.58 | 61.57 | 69.82 | 29.10 | 39.60 |
| SAG2 (Ours),without context | 31.20 | 48.04 | 66.32 | 76.25 | 37.20 | 52.20 |
| Lexical expansion, final JEV only | 31.23 | 45.43 | 57.17 | 61.34 | 25.50 | 29.60 |
| RM3 | 21.79 | 28.50 | 39.69 | 47.83 | 9.80 | 15.70 |
| RM3+final JEV | 30.88 | 45.76 | 60.19 | 66.93 | 29.80 | 37.20 |
| GAR-BM25 | 28.71 | 37.50 | 47.66 | 54.50 | 12.10 | 17.50 |
| GAR-BM25+final JEV | 30.91 | 48.61 | 64.63 | 73.11 | 34.70 | 46.10 |
H counts expansion rounds after initial retrieval; H0/H1/H2 come from the same run. The matched-budget control matches passage judgments, not tokens or cost. Control data · Analysis
Current chunking follows SAG-Desktop: a target of 2,200 UTF-16 units, complete sentences, heading paths and roughly 80 characters of whole-sentence overlap within a heading. Indexes store source ranges and no LLM annotations. Query optimizations include scoring-statistics caching, bounded candidate selection, parallel expansion and persisted corpus statistics.
The index signature includes schema and chunking policy. Matching configurations reuse indexes; changing chunking changes candidate boundaries and scores. Older Enterprise formats require their matching reader; do not overwrite an old index to handle compatibility.
Run these commands from SAG-Benchmark/sag2/.
uv run python scripts/verify_paper_results.py
uv run python scripts/verify_agentic_results.py
uv run python scripts/verify_rerank_results.py
uv run pytest -q
uv run ruff check scripts testsHTTP 499 retries continue within the current request; temporary failures such as 500 and timeouts have bounded retries. See each mode's guide and implementation. Check the host, endpoint, concurrency and output directory before running.
