Skip to content

About

Official benchmark reproduction code for the SAG paper

Topics

Resources

Stars

164 stars

Watchers

1 watching

Forks

Latest commit

 

History

4 Commits

Folders and files

Repository files navigation

Zleap AI

SAG2

English · 简体中文

Paper: arXiv:2610.22588 License: MIT

Embedding-free multi-hop evidence retrieval.

The companion benchmark repository for the SAG2 paper. SQLite FTS5, BM25 and JEV connect source evidence at query time, with no model calls for indexing.

Question → Passages → Terms → New passages

SAG2 application: SAG-Wiki · Original SAG reproduction

SAG2-English.mp4

Contents

Benchmark Score Reproduction

SAG2 was evaluated on 1,000 questions each from MuSiQue, 2WikiMultiHopQA and HotpotQA. R@10 is 86.62%, 97.43% and 99.00%, respectively. EnterpriseRAG-Bench covers 511,962 source documents, with document R@10 of 81.10% over 470 labeled questions.

The paper's main table is below; see the evaluation settings for its configuration. Values are percentages; bold denotes the column maximum. R@k is the mean fraction of labeled evidence recovered in the first k deduplicated results; AE@k is the fraction of questions with all evidence recovered. R@k reports evidence recall.

MethodMuSiQue2WikiMultiHopQAHotpotQA
R@1R@2R@5R@10R@1R@2R@5R@10R@1R@2R@5R@10
Lexical and dense retrieval
BM2528.5337.3247.4754.4140.1057.9367.0571.5039.7557.9574.4086.90
Contriever24.5033.5445.3253.5834.4048.6059.2565.2039.3057.4574.8582.75
Large embedding models
BGE-Large-EN-v1.529.4241.6856.3264.0841.4862.2869.8373.7545.4576.2089.2093.35
GTE-Qwen2-7B35.1948.0661.9270.8043.4565.5874.1878.6346.4073.9588.4593.55
GritLM-7B34.8650.7066.2275.1941.8566.2375.6279.4046.4081.1593.4097.20
NV-Embed-v234.0251.0868.8477.2743.2368.9377.0080.7546.9085.5595.2597.70
Structure-enhanced retrieval
HippoRAG 230.6549.5265.1373.7642.3876.5590.3593.4044.4078.3594.3597.15
SAG (Ours)36.8263.6280.3683.3743.8183.9393.3493.5747.8091.5596.5097.70
SAG2 (Ours)34.3062.1880.6886.6243.5585.5097.3597.4348.3093.6098.4599.00

BM25 uses the paper's fixed lexical reference; cited baselines and provenance are in main table data. These are fixed development subsets; MuSiQue has 518 two-hop, 316 three-hop and 166 four-hop questions and was used for method development. Enterprise results are shown below. Ablations and Agentic results are reported in later sections.

EnterpriseRAG-Bench contains 511,962 source documents. The table reports document R@10 (%) on 470 labeled questions.

Question type n BM25 Vector Bash Agent† SAG2 (Ours)
Basic retrieval 175 77.7 48.6 61.7 86.86
Semantic retrieval 125 43.2 24.8 29.6 58.40
Within-document reasoning 40 90.0 57.5 75.0 100.00
Project-related 40 65.5 49.2 43.2 83.58
Constrained 30 85.0 83.3 78.3 96.67
Conflicting information 20 82.5 55.0 72.5 92.50
Completeness 20 46.5 33.4 59.0 76.24
Miscellaneous 20 90.0 75.0 100.0 100.00
Overall 470 68.4 46.0 55.8 81.10

Enterprise baselines are cited from the dataset paper. Bash Agent has variable output size; other methods in this table return ten documents, so computation and output budgets differ.

Method Figures

SAG2 indexing and retrieval

  1. Index source text. Tokenize titles and text into SQLite FTS5; retain source identity, headings and term statistics. No embeddings or offline semantic extraction are required.
  2. Retrieve passages from the question. BM25 returns up to 100 candidates. JEV noul scores whether each passage supplies necessary evidence for the original question and available context.
  3. Select terms from passages. Up to 10 expansion seeds contribute up to 20 terms, using frequency, rarity, query context and JEV scores.
  4. Retrieve more evidence. Search each term, interleave and deduplicate results, retain earlier evidence and score again. The default adds two expansion rounds.
  5. Return sources. Locally diversify the top ten results without another model request.

JSON corpus text is treated as plain text by default; a supplied path selects Markdown or code parsing by its file extension. No generated annotations are added to source text. Gold answers and evidence labels are used only for evaluation. The algorithm, prompts and query optimizations are in scripts/sag2.py. The original event/entity approach is documented in sag/.

Quick Start

Run the following commands from SAG-Benchmark/sag2/. Original SAG readers should instead follow sag/README.md, which has its own environment and dependencies.

1. Install

Requirements: Python 3.11+, uv and a C compiler. The SQLite scoring extension compiles on first use: Xcode Command Line Tools on macOS, GCC or Clang on Linux, or MSVC Build Tools on Windows. Default SAG2 needs no Docker, MySQL, Elasticsearch or GPU. Shell examples use Bash (Linux, macOS or WSL); in PowerShell, use Copy-Item for the configuration copy. Paper-control scripts currently require a Unix-like environment.

git clone https://github.com/Zleap-AI/SAG-Benchmark.git
cd SAG-Benchmark/sag2
uv sync --frozen

For an existing checkout, run cd sag2 from the repository root. Run subsequent SAG2 commands here, using sag2/.venv/.

2. Configure JEV

Create the configuration once, preserving an existing file:

test -f .env || cp .env.example .env

Set these values in the current directory’s .env (repository path: sag2/.env):

OPENROUTER_API_KEY=your_api_key
JEV_BASE_URL=https://openrouter.ai/api/alpha/decisions
JEV_MODEL=typesafe/jev-1.13

The default answer type is noul. The historical paper run returned typesafe/jev-1.13-20260917; provider versions may change, so record the actual returned model. Keys are read from environment variables or --env-file, never passed as command-line arguments.

3. Prepare Data and Indexes

# Download missing pinned data and index all three datasets; no model requests
uv run python scripts/run_sag2_benchmark.py --download --index-only

An index with the same corpus and chunking configuration is reused. The terminal prints source count, chunk count and question count; the suite summary reports indexed. Indexes default to .cache/sag2/. Add --dataset musique to prepare only one dataset.

4. Run Ten Questions First

uv run python scripts/run_sag2_benchmark.py \
  --dataset musique --limit 10 --concurrency 20 \
  --output-dir output/sag2/musique-smoke

--limit changes the question count, not the corpus size. This step uses the Jev API. The terminal prints recall metrics; per-question results are in output/sag2/musique-smoke/musique/details.jsonl, with metrics.json in the same directory.

5. Run the Full Benchmarks

# 1,000 questions per dataset
uv run python scripts/run_sag2_benchmark.py --datasets all --concurrency 20

# Or choose specific datasets
uv run python scripts/run_sag2_benchmark.py \
  --datasets hotpotqa 2wikimultihopqa --concurrency 20

Add --output-dir output/sag2/my-run for an explicit destination; it must not already exist for a new run. Check dependencies, data and the small run before starting full evaluation.

Datasets

Dataset argument Questions Source passages / documents Evaluation unit
musique 1,000 11,656 Supporting passage
2wikimultihopqa 1,000 6,119 Supporting passage
hotpotqa 1,000 9,811 Supporting passage
EnterpriseRAG-Bench, separate entry point 470 labeled 511,962 Source document

The three multi-hop datasets come from HippoRAG 2 at revision 5ec05b38deecc3318bb432c69865959c56058990. --datasets all means the first three rows. Enterprise uses the separate entry point below. See dataset/README.md for format and license information.

EnterpriseRAG-Bench

Prepare the official directory with documents/, questions.jsonl and download_manifest.json. Check for an existing matching index before building another one.

# Only for the first build of a current-format index; preserve older indexes
uv run python scripts/build_enterprise_index.py \
  --dataset-dir ../../datasets/EnterpriseRAG-Bench \
  --index-path .cache/sag2/enterprise.sqlite \
  --output-dir output/enterprise/index

# Reuse that index and evaluate all 470 labeled questions
uv run python scripts/run_enterprise.py \
  --dataset-dir ../../datasets/EnterpriseRAG-Bench \
  --index-summary output/enterprise/index/summary.json \
  --output-dir output/enterprise/evaluation --concurrency 20

The evaluator opens the index read-only and writes progress.json, summary.json, per-question qst_*.json files and requests/. Repeating the same command reuses saved results; use a new output directory when changing models, data or settings. A schema4 index cannot be passed directly to the current schema5 reader. Keep the old file and use its matching compatibility implementation; changing a version marker does not convert an index.

Retrieval Configuration

Argument Default Purpose
--mode sag2 sag2, agentic or local
--dataset / --datasets all One or more multi-hop datasets
--dataset-dir dataset/ Dataset files; defaults to sag2/dataset/
--limit / --offset All / 0 Question range; corpus stays complete
--concurrency 20 Concurrent questions
--index-workers 8 Index-building processes, separate from query concurrency
--candidate-k 100 Lexical candidates per round
--seed-k 10 Expansion seeds per round
--expansions / --max-hops 2 for SAG2; 1 for Agentic Additional rounds after the initial retrieval; 0, 1 or 2
--expansion-term-limit 20 Expansion terms per round
--top-k / --k-list 10 / 1 2 5 10 Output size and evaluation cutoffs
--chunk-chars 2200 Chunk target in UTF-16 units
--cache-dir .cache/sag2 SQLite index directory
--output-dir Timestamped directory Destination for a new run
--resume Unset Continue an existing run with its original settings
--env-file sag2/.env (repository path) Configuration file; SAG_ENV_FILE can override the default
# One expansion round, still using 100 candidates and 10 seeds
uv run python scripts/run_sag2_benchmark.py \
  --dataset musique --limit 10 --expansions 1 --candidate-k 100 --seed-k 10

uv run python scripts/run_sag2_benchmark.py --help

Changing models, chunking or retrieval parameters changes the experiment. Match the model version, corpus, questions, chunking policy and metric units when comparing with published scores.

Outputs and Resume

The following paths are relative to SAG-Benchmark/sag2/. Default output layout:

output/sag2/<timestamp>/
├── summary.json
└── <dataset>/
    ├── config.json
    ├── details.jsonl
    ├── metrics.json
    ├── search_results.json
    ├── bm25_results.json
    └── requests/          # Agentic / local reranking modes
File Content and completion checks
summary.json Final per-dataset summary; index-only runs report indexed
config.json Model, settings, data and code hashes
details.jsonl One completed question per line, with retrieval, metrics and usage
metrics.json Final dataset metrics, all-evidence recall and usage
search_results.json Final retrieved source text per question, for QA or analysis
bm25_results.json Initial BM25 results for the same questions
requests/ Agentic and local request/response records for recovery
# Reuse the original datasets, range, model and settings
uv run python scripts/run_sag2_benchmark.py \
  --dataset musique --limit 10 --concurrency 20 \
  --resume output/sag2/musique-smoke

Plain SAG2 resumes completed questions. Agentic and local modes also reuse successful identical requests. Plain SAG2 does not persist each Jev request separately for recovery, so an interrupted question may send its requests again. Keep different model/configuration runs in separate output directories.

Common Scripts

Script Purpose
scripts/run_sag2_benchmark.py Multi-hop indexing, SAG2, Agentic and local evaluation
scripts/build_enterprise_index.py Full Enterprise index construction
scripts/run_enterprise.py Enterprise evaluation over an existing read-only index
scripts/run_paper_controls.py MuSiQue paper controls, including model calls
scripts/verify_paper_results.py Offline verification of retained paper records
scripts/verify_agentic_results.py Verify 3,000 retained best Agentic results
scripts/verify_rerank_results.py Verify retained reranking results
scripts/search_local.py Query a custom JSON corpus with local reranking

The current SAG2 entry points return evidence. The original SAG upload, answer-generation and LLM Judge workflows are preserved separately in sag/.

Agentic SAG2

Optional extension. An LLM retains the original question and retrieval history, calls SAG2 as a tool, decides when to search or stop, and ranks the final ten passages. It adds no generated text at index time.

Best historical results over 1,000 questions per dataset are below. MuSiQue reaches 92.20% R@5, a gain of 11.53 percentage points over a single SAG2 run using unrounded values, and 95.39% R@10. R@5 is 99.88% on 2WikiMultiHopQA and 99.30% on HotpotQA.

MethodMuSiQue2WikiMultiHopQAHotpotQA
R@1R@2R@5R@10R@1R@2R@5R@10R@1R@2R@5R@10
SAG234.3062.1880.6886.6243.5585.5097.3597.4348.3093.6098.4599.00
Agentic SAG238.2171.9992.2095.3944.0888.0099.8899.8849.1096.4599.3099.65

Evaluation settings: gpt-6.1-sol, high reasoning, one expansion per SAG2 call, at most three tool calls per question, 20 passages returned per call and 10 final passages. The general API entry point requires no Codex account or CLI. Historical scores came from native MCP runs.

LLM_API_KEY=your_llm_key
LLM_BASE_URL=https://api.openai.com/v1
LLM_MODEL=gpt-6.1-sol
LLM_API=responses
LLM_REASONING_EFFORT=high
uv run python scripts/run_sag2_benchmark.py --mode agentic \
  --dataset musique --limit 10 --expansions 1 --max-tool-calls 3 --return-k 20

Configure --llm-model, --llm-base-url, --llm-api responses|chat, --llm-api-key-env and --reasoning-effort. The provider must support the selected tool-calling format. See the Agentic guide and best results for configuration, tokens and costs.

Local Reranking

Optional extension. Replace each Jev scoring round with a local model while keeping BM25, term expansion and final diversity ordering. After preparing weights and data, queries can run offline.

The following results were evaluated through 302 APIs, with 1,000 questions per dataset:

MethodMuSiQue2WikiMultiHopQAHotpotQA
R@1R@2R@5R@10R@1R@2R@5R@10R@1R@2R@5R@10
BM2528.7137.5047.6654.5040.2558.1867.0871.5839.7558.0574.4586.95
SAG2 + Jev(paired control)34.2462.3480.5986.6143.5585.5097.3597.4348.3093.6098.4599.00
SAG2 + Jina(two expansions)31.0350.3865.6172.1343.1070.1580.8086.9546.7585.5594.3597.00
MuSiQue model and pipeline ablations
MuSiQue ablation R@1 R@2 R@5 R@10 AE@10
BM25 28.71 37.50 47.66 54.50 17.50
BM25+Qwen single scoring round 31.66 45.44 60.17 66.88 32.50
BM25+Jina single scoring round 35.01 52.18 63.04 67.43 34.70
SAG2+Qwen two expansions 22.64 34.75 55.55 71.61 44.60
SAG2+Qwen 0.6B two expansions 25.67 40.81 62.54 74.18 48.40
SAG2+Jina two expansions 31.03 50.38 65.61 72.13 43.80
SAG2+Jev baseline 34.24 62.34 80.59 86.61 69.00

Single-round runs use the original query and standard relevance instructions. Two-expansion runs use SAG2 evidence instructions, context and final DPP, so these comparisons change both pipeline and prompts. qwen3-rerank and Qwen 0.6B are distinct models.

# Prepare dependencies and pinned weights while online
uv sync --frozen --group local
uv run --group local hf download Qwen/Qwen3-Reranker-0.6B \
  --revision e61197ed45024b0ed8a2d74b80b4d909f1255473 --local-dir models/qwen3-reranker-0.6b

# Run ten questions offline once data and weights are ready
uv run --offline --group local python scripts/run_sag2_benchmark.py \
  --mode local --reranker qwen --model-path models/qwen3-reranker-0.6b \
  --dataset musique --limit 10 --expansions 2 --concurrency 20

Available backends include Qwen Transformers, Jina Transformers and Jina MLX on Apple Silicon. Qwen weights use Apache-2.0; Jina uses CC BY-NC 4.0 and requires additional permission for commercial use. See the local reranking guide for custom corpora, memory, batch sizes and full ablations.

Run External Methods

Five external methods live under sag/external/. Each has an independent environment and reads sag/.env. See the external-method guide for full steps, outputs and shared Judge evaluation.

Method Reproduction guide
Microsoft GraphRAG graphrag
HippoRAG 2 hipporag2
HyperGraphRAG hypergraphrag
Hyper-RAG hyperrag
LightRAG lightrag

For HippoRAG 2, first copy sag/.env.example to sag/.env and configure the LLM and embedding endpoints; preserve an existing configuration. Run from the SAG-Benchmark repository root, reusing the MuSiQue files downloaded above:

cd sag/external/hipporag2
uv sync --frozen
uv run --frozen --env-file ../../.env python reproduce/Step_0_load_dataset.py \
  --data_name musique --dataset-root ../../../sag2/dataset
uv run --frozen --env-file ../../.env python reproduce/Step_1_build_index.py --data_name musique
uv run --frozen --env-file ../../.env python reproduce/Step_3_response.py --data_name musique

The stages are data preparation, indexing, then retrieval and QA; the last two use model endpoints. GraphRAG separates retrieval and QA, while Hyper-RAG also requires Hypergraph-DB. Follow each method's guide for its parameters and data paths; run shared Judge evaluation from sag/.

Repository Layout

SAG-Benchmark/
├── README.md / README-CN.md   # SAG2 introduction, results and reproduction / SAG2介绍、成绩与复现
├── sag/                      # Original SAG / 原SAG
│   ├── README.md / README-CN.md
│   ├── pipeline/ / scripts/ / prompts/
│   ├── external/             # GraphRAG, HippoRAG 2, HyperGraphRAG, Hyper-RAG, LightRAG
│   └── pyproject.toml / uv.lock / docker-compose.yml
└── sag2/                     # SAG2 and its optional extensions
    ├── README.md / README-CN.md
    ├── pyproject.toml / uv.lock / .env.example
    ├── scripts/
    │   ├── run_sag2_benchmark.py
    │   ├── sag2.py / agentic_sag2.py / local_reranker.py
    │   └── build_enterprise_index.py / run_enterprise.py
    ├── prompts/ / docs/ / dataset/ / results/ / tests/ / assets/
    ├── .cache/sag2/          # SQLite indexes / SQLite索引(运行时生成)
    └── output/              # Evaluation results / 评测结果(运行时生成)

Each method has separate .env, .venv, dataset/ and output/ paths. Run SAG2 from sag2/ and original SAG from sag/. Original SAG uses --strategy sag and the SAGGraphSearcher event/entity graph implementation; SQLite SAG2 is implemented separately in sag2/.

Reproduction Notes and Ablations

Indexing and Cost

MuSiQue index and cost Paper record
Source passages / indexed units 11,656 / 11,656
Index construction 2.36 seconds
SQLite file 25.87 MiB
Offline model calls / cost 0 / USD 0
Model cost / 100 questions USD 0.433769
Model cost / 1,000 questions USD 4.326379

Construction time is the median of three builds on Apple M4 Pro, 64 GiB, eight processes. It includes tokenization, index writes, statistics and query initialization; it excludes imports, corpus loading and validation, and does not clear the OS file cache. Costs come from historical responses and exclude compute rental and answer generation. See paper data.

Full MuSiQue controlled experiments, 1,000 questions each
Method R@1 R@2 R@5 R@10 AE@5 AE@10
BM25 28.53 37.32 47.47 54.41 12.20 17.40
BM25+JEV (H=0) 31.13 46.79 60.78 67.93 26.60 36.00
SAG2 (Ours), H=1 34.35 61.44 77.89 83.36 54.60 63.10
SAG2 (Ours), H=2 34.30 62.18 80.68 86.62 59.00 69.50
BM25+JEV,matched budget 30.64 46.58 61.57 69.82 29.10 39.60
SAG2 (Ours),without context 31.20 48.04 66.32 76.25 37.20 52.20
Lexical expansion, final JEV only 31.23 45.43 57.17 61.34 25.50 29.60
RM3 21.79 28.50 39.69 47.83 9.80 15.70
RM3+final JEV 30.88 45.76 60.19 66.93 29.80 37.20
GAR-BM25 28.71 37.50 47.66 54.50 12.10 17.50
GAR-BM25+final JEV 30.91 48.61 64.63 73.11 34.70 46.10

H counts expansion rounds after initial retrieval; H0/H1/H2 come from the same run. The matched-budget control matches passage judgments, not tokens or cost. Control data · Analysis

Indexes and Evaluation Settings

Current chunking follows SAG-Desktop: a target of 2,200 UTF-16 units, complete sentences, heading paths and roughly 80 characters of whole-sentence overlap within a heading. Indexes store source ranges and no LLM annotations. Query optimizations include scoring-statistics caching, bounded candidate selection, parallel expansion and persisted corpus statistics.

The index signature includes schema and chunking policy. Matching configurations reuse indexes; changing chunking changes candidate boundaries and scores. Older Enterprise formats require their matching reader; do not overwrite an old index to handle compatibility.

Offline Verification

Run these commands from SAG-Benchmark/sag2/.

uv run python scripts/verify_paper_results.py
uv run python scripts/verify_agentic_results.py
uv run python scripts/verify_rerank_results.py
uv run pytest -q
uv run ruff check scripts tests

HTTP 499 retries continue within the current request; temporary failures such as 500 and timeouts have bounded retries. See each mode's guide and implementation. Check the host, endpoint, concurrency and output directory before running.

About

Official benchmark reproduction code for the SAG paper

Topics

Resources

Stars

164 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages