Unified Retrieval Agent-Based System
for Ambiguous and Unanswerable Question Answering

Paper · Overview · Quickstart · Terminal · Datasets · Evaluation · Results · Findings · Adapt & Extend · Citation

Long S. T. Nguyen · Quynh T. N. Vo · Hung C. Luu · Tho T. Quan
_{URA Research Group · Ho Chi Minh City University of Technology (HCMUT) · Vietnam National University, Ho Chi Minh City}

Overview

URASys brings retrieval, reasoning, and clarification into one question-answering workflow. A Manager Agent coordinates FAQ and document agents to gather complementary evidence, resolve complex questions, and respond according to what the sources support. Under the Just Enough principle, the system asks for missing context when a question is ambiguous and signals when an answer is unavailable.

_{URASys framework: question decomposition, parallel retrieval, and evidence-grounded responses.}

Component	Role
Manager Agent	Decomposes the question, combines evidence, and decides how to respond.
FAQ Search Agent	Retrieves from generated, expanded, and paraphrased question–answer pairs.
Document Search Agent	Searches contextualized document chunks using dense and lexical retrieval.
Two-phase indexing	Builds complementary document and FAQ representations from a shared corpus.

Indexing

The two-phase pipeline creates a document index and an FAQ index from the same source material:

• Chunk-and-Title: segment documents semantically, rewrite chunks with context, and assign descriptive titles.
• Ask-and-Augment: generate and expand FAQs, then paraphrase questions to cover different ways users express an information need.

_{Complementary document and FAQ indexes support the two retrieval agents.}

Quickstart

Use Python 3.13 and uv. Run commands from the repository root. The local walkthrough uses Milvus Lite on Linux.

1. Install

git clone https://github.com/longstnguyen/URASys.git
cd URASys
uv sync --locked
source .venv/bin/activate

test -f environments/.env || cp environments/.template.env environments/.env

Set GEMINI_API_KEY and OPENAI_API_KEY in environments/.env for the default model configuration. Model IDs, embedding dimensions, retrieval parameters, and server settings are configured in urasys.yaml.

2. Index a small sample

Start with two rows and a dedicated index:

MILVUS_LOCAL_DB_PATH=.data/quickstart.db urasys-index \
  --strategy urasys --storage-mode local \
  --dataset datasets/SQuAD2_1000.csv --n 2 \
  --doc-collection quickstart_docs --faq-collection quickstart_faqs \
  --bm25-dir .data/quickstart-bm25

Indexing calls the configured models. The default --mode index rebuilds the selected collections; use separate collection names for separate experiments.

3. Start the server

After indexing finishes:

MILVUS_LOCAL_DB_PATH=.data/quickstart.db urasys-query-server \
  --storage-mode local \
  --doc-collection quickstart_docs --faq-collection quickstart_faqs \
  --bm25-dir .data/quickstart-bm25

4. Open the terminal

In another terminal at the repository root:

source .venv/bin/activate
python scripts/eval_batch.py interactive --storage-mode local

Try a question from the sample: “Beyonce supported which campaign that encourages leadership in girls?” Use :clear to start a fresh conversation and quit to exit.

Terminal

Follow the question through analysis, FAQ and document retrieval, and answer synthesis. The terminal displays agent search rounds, retrieved evidence, scores, and the final response.

_{Interactive retrieval and clarification on a Vietnamese UniQA question.}

Task	Command	Useful options
Build indexes	urasys-index	--strategy, --dataset, --n, --storage-mode
Run the query server	urasys-query-server	--storage-mode, --doc-collection, --faq-collection
Chat	python scripts/eval_batch.py interactive	--storage-mode, --backend-url, --top-k
Evaluate	python scripts/eval_batch.py eval	--dataset, --strategy, --n, --output
Check dataset samples	python scripts/smoke_datasets.py	--mode, --n, --datasets

Every command supports --help.

Storage modes

Choose where retrieval indexes live. LLM and embedding configuration is independent of storage mode.

	Local	Cloud
Dense vectors	Milvus Lite .db file	Zilliz Cloud
Lexical retrieval	Local BM25 state and sparse vectors	Native Zilliz BM25
Configuration	MILVUS_LOCAL_DB_PATH	MILVUS_CLOUD_URI, MILVUS_CLOUD_TOKEN
CLI flag	--storage-mode local	--storage-mode cloud

Set the default with STORAGE_MODE in .env, or select it per command. Use matching collection names and, in local mode, matching BM25 paths for indexing and serving.

Cloud setup

Set the following in environments/.env, alongside your model credentials:

STORAGE_MODE=cloud
MILVUS_CLOUD_URI=https://your-cluster-endpoint
MILVUS_CLOUD_TOKEN=your-cluster-token

With the cluster running, build an index:

urasys-index --strategy urasys --storage-mode cloud \
  --dataset datasets/SQuAD2_1000.csv --n 2 \
  --doc-collection quickstart_cloud_docs \
  --faq-collection quickstart_cloud_faqs

Start the query server:

urasys-query-server --storage-mode cloud \
  --doc-collection quickstart_cloud_docs \
  --faq-collection quickstart_cloud_faqs

In another terminal at the repository root:

source .venv/bin/activate
python scripts/eval_batch.py interactive --storage-mode cloud

Cloud mode stores dense and BM25 indexes in Zilliz. Run commands without --bm25-dir; the client reads input datasets and configuration and writes evaluation outputs on the host machine.

Model configuration and runtime settings

Setting	Location
Model credentials and storage	environments/.env
Indexing LLM	models.llm.indexing_llm in urasys.yaml
Retrieval and terminal Manager	models.llm.retrieval_llm
Embeddings	models.embedding
Baseline generation	baseline.generation
Evaluation judge	evaluation and baseline.evaluation

The model clients support Gemini for generation and OpenAI or OpenAI-compatible endpoints for embeddings. Set the embedding model ID, endpoint, and dimensions together; rebuild indexes when the embedding configuration changes.

Process environment variables take precedence over .env. Use ENVIRONMENT_FILE to select another environment file. The query API defaults to port 8006; GET /health reports readiness and active collections.

Milvus Lite uses one process per database file. Complete indexing before starting the query server, and stop that server before rebuilding the same index. The sample runner manages isolated evaluation indexes automatically.

Datasets

The collection covers answerability with SQuAD 2.0 [1] and UIT-ViQuAD 2.0 [2], and multi-hop reasoning with HotpotQA [3] and VIMQA [4]. UniQA adds Vietnamese academic advising, while the derived ambiguous sets support clarification evaluation.

Dataset	Language	Standard QA rows	Ambiguous rows	Focus
SQuAD 2.0	English	1,000	418	Answerability
UIT-ViQuAD 2.0	Vietnamese	1,000	731	Answerability
HotpotQA	English	1,000	—	Multi-hop reasoning
VIMQA	Vietnamese	1,000	—	Multi-hop reasoning
UniQA	Vietnamese	996	314	Academic advising

For UniQA, index datasets/UniQA_corpus.csv and evaluate datasets/UniQA_1000.csv. The corpus contains 41 document rows; the standard question file contains question and answer columns.

For other datasets, the indexing loader reads paragraph or paragraphs, with optional question–answer pairs as FAQ seeds. Keep held-out evaluation answers separate from the indexing input. See the dataset reference for schemas, sources, and usage terms.

Evaluation

The evaluation CLI supports BM25, dense, hybrid, and URASys. It saves predictions, reference answers, timing, and judge scores to JSON. Datasets with an info column use the clarification evaluation rubric.

With the quickstart server running:

python scripts/eval_batch.py eval \
  --storage-mode local --strategy urasys \
  --dataset datasets/SQuAD2_1000.csv --n 2 \
  --output results/quickstart.json

To exercise all four strategies on two samples per dataset, use the sample runner. It creates isolated indexes, manages the query server, and cleans up its temporary storage:

python scripts/smoke_datasets.py --mode local --n 2
python scripts/smoke_datasets.py --mode cloud --n 2

Select a subset with --datasets squad2 viquad2 uniqa. The sample runner checks indexing, retrieval, generation, and scoring through the configured model APIs. For benchmark evaluation, use the evaluation CLI with your corpus and held-out question split.

Run the retrieval baselines

Baseline indexing uses fixed-size chunks and a dedicated document collection:

MILVUS_LOCAL_DB_PATH=.data/baseline.db urasys-index \
  --strategy baseline --storage-mode local \
  --dataset datasets/SQuAD2_1000.csv --n 2 \
  --doc-collection baseline_docs --bm25-dir .data/baseline-bm25

MILVUS_LOCAL_DB_PATH=.data/baseline.db python scripts/eval_batch.py eval \
  --storage-mode local --strategy baselines \
  --dataset datasets/SQuAD2_1000.csv --n 2 \
  --baseline-doc-collection baseline_docs --bm25-dir .data/baseline-bm25 \
  --output results/baselines.json

Choose --strategy bm25, dense, or hybrid to run one baseline. baselines runs all three. Use --em-f1 for additional Exact Match and token F1 on standard answerable questions.

For local evaluation across baselines and URASys, use sequential runs or the sample runner so the processes do not share an open Milvus Lite database. --strategy all is available for cloud runs with both indexes and the query server prepared.

Experiment tracking

Use uv sync --locked to install the dependencies recorded in uv.lock. For each experiment, retain the code revision, urasys.yaml, corpus and question splits, storage mode, and collection names alongside the output JSON.

The evaluator saves the selected strategies, question count, top-k, judge configuration, timing, predictions, and scores. Use a distinct --output path for each configuration and compare runs on the same question set. Keep model IDs, embedding dimensions, and judge prompts fixed when measuring the effect of a retrieval or indexing change.

Results

Answer correctness (%, higher is better) across five QA benchmarks under the LLM-as-a-Judge protocol. See the paper for the experimental setup and full results.

Method	SQuAD 2.0	UIT-ViQuAD 2.0	HotpotQA	VIMQA	UniQA
Naive RAG · Dense	30.3	18.2	11.3	41.2	40.1
Naive RAG · BM25	30.1	18.7	11.8	40.0	39.8
Naive RAG · Hybrid	30.3	18.3	11.6	42.4	41.3
IRCoT	45.3	46.9	43.2	20.7	56.7
LightRAG	43.4	44.3	59.0	76.0	51.8
MiniRAG	49.0	55.0	21.7	54.5	52.6
NodeRAG	44.1	48.1	56.7	45.3	67.7
HippoRAG 2	67.3	78.8	60.3	75.0	50.7
URASys	75.0	80.0	90.0	83.2	85.0

Unanswerable and ambiguous questions

Method	SQuAD unanswerable	SQuAD ambiguous	ViQuAD unanswerable	ViQuAD ambiguous	UniQA unanswerable	UniQA ambiguous
Naive RAG · Dense	7.2	14.8	2.9	11.6	9.2	13.8
Naive RAG · BM25	13.6	12.9	3.4	31.3	9.3	8.6
Naive RAG · Hybrid	8.3	10.6	3.7	21.3	9.03	3.2
IRCoT	33.4	13.7	28.0	31.7	9.4	67.5
LightRAG	36.2	18.2	49.0	45.6	51.0	68.2
MiniRAG	10.6	17.5	11.3	70.5	49.0	36.9
NodeRAG	23.0	69.3	8.1	68.3	14.0	32.5
HippoRAG 2	56.2	67.4	54.0	62.7	13.0	49.7
URASys	83.7	71.9	86.5	73.9	82.6	81.2

Findings

1. Strong performance across languages and reasoning tasks. URASys leads the evaluated systems on all five benchmarks, reaching 90.0% on HotpotQA and 85.0% on UniQA. On HotpotQA, it exceeds the next-best result in the comparison by 29.7 percentage points.

2. Answerability and clarification are central to reliable QA. URASys achieves 83.7%, 86.5%, and 82.6% correctness on the unanswerable subsets of SQuAD 2.0, UIT-ViQuAD 2.0, and UniQA. On their ambiguous subsets, it reaches 71.9%, 73.9%, and 81.2%, respectively.

3. FAQ and document retrieval contribute complementary evidence. Removing the FAQ agent reduces HotpotQA correctness from 0.90 to 0.43. Removing the document agent reduces UIT-ViQuAD 2.0 correctness from 0.80 to 0.16, highlighting the value of retaining both retrieval paths.

4. Asking for context improves academic advising. On UniQA, removing clarification reduces correctness from 0.85 to 0.50. The full system combines retrieval with interaction to address underspecified student questions.

5. Users value explanation quality and trust. In a human evaluation with 20 participants, the two groups rate explanation quality at 4.51/5 and 4.76/5, and trust at 4.26/5 and 4.37/5.

Ablation study

Judge correctness on a 0–1 scale.

System variant	SQuAD 2.0	UIT-ViQuAD 2.0	HotpotQA	VIMQA	UniQA
URASys	0.75	0.80	0.90	0.83	0.85
Without FAQ Search Agent	0.73	0.26	0.43	0.14	0.48
Without Document Search Agent	0.59	0.16	0.86	0.15	0.62
Without Manager decomposition	0.72	0.22	0.87	0.21	0.63
Without proposed indexing	0.71	0.29	0.85	0.29	0.64
Without clarification	0.64	0.37	0.76	0.44	0.50

Adapting URASys

To use URASys in another domain, provide your corpus, configure the domain vocabulary and response policy, then build dedicated document and FAQ indexes. The same workflow supports local and cloud storage.

1. Prepare your documents and questions

Store source passages in a CSV with a paragraph column. Prepare evaluation questions in a separate CSV with question and answer columns. For example, create the following files under .data/domain/:

corpus.csv

paragraph
"The library opens at 08:00 and closes at 17:00, Monday through Friday."
"Students may borrow up to five books for fourteen days."

questions.csv

question,answer
"When does the library close on weekdays?","17:00"
"How many books may a student borrow?","Five books"

Use one passage per row and UTF-8 encoding. Export text from PDFs, web pages, or other sources into this schema before indexing. Include question and answer in the indexing CSV only when you intend to supply seed FAQs. For clarification evaluation, add an info column describing the missing details the response should request.

2. Configure the domain

Adaptation	Where to configure it
Domain scope, terminology, and clarification policy	src/urasys/prompts/query/manager_agent.py
FAQ and document search instructions	src/urasys/prompts/query/faq_agent.py and document_agent.py
Chunk rewriting, titles, and FAQ coverage	src/urasys/prompts/indexing/
Chunk sizes and FAQ expansion limits	indexing.semantic_chunk and indexing.faq in urasys.yaml
Generation and embedding models	models in urasys.yaml; see model configuration
Agent search limits	retrieval.sub_agent_max_retries and retrieval.agent_timeout_seconds
Domain-specific evaluation criteria	src/urasys/prompts/evaluation/

Adapt the educational examples and domain references in the prompts to your application. Retain the structured output fields and tool names expected by the callers, and keep answers grounded in retrieved evidence. Restart the query server and terminal client after changing their prompts or configuration. Rebuild indexes after changing indexing prompts, chunking, or embeddings.

3. Index and serve your corpus

Run from the repository root with the environment configured. If the quickstart server is still running, stop it before starting the domain server on the same port.

MILVUS_LOCAL_DB_PATH=.data/domain/urasys.db urasys-index \
  --strategy urasys --storage-mode local \
  --dataset .data/domain/corpus.csv \
  --doc-collection domain_docs --faq-collection domain_faqs \
  --bm25-dir .data/domain/bm25

# Start after indexing completes.
MILVUS_LOCAL_DB_PATH=.data/domain/urasys.db urasys-query-server \
  --storage-mode local \
  --doc-collection domain_docs --faq-collection domain_faqs \
  --bm25-dir .data/domain/bm25

For cloud storage, use --storage-mode cloud, omit the local database and BM25 paths, and configure Zilliz as described in Storage modes.

4. Chat and evaluate

In another terminal, with the virtual environment activated:

python scripts/eval_batch.py interactive --storage-mode local

# After leaving the interactive session:
python scripts/eval_batch.py eval \
  --storage-mode local --strategy urasys \
  --dataset .data/domain/questions.csv \
  --output results/domain.json

Start with a small corpus and questions covering direct answers, missing information, ambiguous references, and multi-hop reasoning. Review retrieved evidence alongside the final answers, then expand the corpus and evaluation set.

Extending URASys

URASys separates indexing, retrieval, agent orchestration, and evaluation into modules. Use the following entry points when adding capabilities.

Extension	Entry points	Integration steps
New input format	load_dataset() in src/urasys/workflow/__main__.py	Convert the input to documents and optional FAQ pairs, or export it to the CSV schema above.
New indexing stage	src/urasys/indexing/ and DataIndex in src/urasys/workflow/build_index.py	Add the transformation, connect it to the indexing flow, and expose its settings in urasys.yaml.
Custom retriever or reranker	src/urasys/core/retriever/	Extend a retriever, preserve its result objects, and wire it into _reconfigure_retrievers() in the query server.
Hybrid fusion	src/urasys/utils/database_clients/milvus/database.py	Configure the ranker used by hybrid_search_vectors() and evaluate the resulting evidence ranking.
Embedding backend	BaseEmbedder in src/urasys/core/model_clients/embedder/	Implement its embedding methods, connect it through create_embedder() in config/utils.py, and align the index dimension.
Agent or retrieval tool	src/urasys/server/query_server/server_app.py and src/urasys/prompts/query/	Add the tool and agent loop, include its output in parallel retrieval, and pass its evidence to the Manager.
Evaluation strategy or metric	scripts/eval_batch.py and src/urasys/prompts/evaluation/	Connect execution, scoring, terminal reporting, and JSON export so the new result is available throughout evaluation.

Retriever and agent integration

DocumentRetriever.retrieve_documents() returns a DocumentRetrievalResult containing the query and scored document nodes. FAQRetriever.retrieve_faqs() returns a FAQRetrievalResult containing scored question–answer pairs. These contracts are defined in src/urasys/core/retriever/base_class.py and consumed by the query server's _sync_retrieve_docs() and _sync_retrieve_faqs() helpers.

For a reranker, retrieve candidate nodes, rerank them, and return the selected nodes through the same result type. Preserve source content and scores for terminal inspection. When adding stored metadata, update utils/vectordb_schema.py, indexing writes, and retriever output fields together.

For an additional agent, follow _run_faq_sub_agent() or _run_doc_sub_agent(), then connect the agent in rest_search_parallel(). Extend _run_urasys_pipeline() in scripts/eval_batch.py to carry the new evidence into Manager synthesis, and update the terminal trace and exported results.

For a new evaluation strategy, update STRATEGIES, _run_one(), any required retriever initialization and answer generation, and the reporting/export branches. Keep its corpus and evaluation questions aligned with the strategies being compared.

Validate extensions with the development checks and a small end-to-end run in the storage modes you support.

Development

uv sync --locked --extra dev
uv run --no-sync python -m unittest discover -s tests -v

Directory	Contents
src/urasys/config/	Environment settings and model configuration
src/urasys/indexing/	Document preparation and FAQ augmentation
src/urasys/core/	Model clients and retrieval
src/urasys/prompts/	Indexing, agent, baseline, and evaluation prompts
src/urasys/server/	Query and indexing services
src/urasys/workflow/	URASys and baseline indexing workflows
scripts/	Interactive CLI, evaluation, and sample checks
tests/	Runtime and storage regression tests

Contribution guidelines are in CONTRIBUTING.md.

Citation

If you use URASys in your research, please cite our IJCNLP–AACL 2025 paper:

@inproceedings{nguyen-etal-2025-doubt,
    title = "When in Doubt, Ask First: {A} Unified Retrieval Agent-Based System for
      Ambiguous and Unanswerable Question Answering",
    author = "Nguyen, Long S. T.  and
      Vo, Quynh T. N.  and
      Luu, Hung C.  and
      Quan, Tho T.",
    editor = "Inui, Kentaro  and
      Sakti, Sakriani  and
      Wang, Haofen  and
      Wong, Derek F.  and
      Bhattacharyya, Pushpak  and
      Banerjee, Biplab  and
      Ekbal, Asif  and
      Chakraborty, Tanmoy  and
      Singh, Dhirendra Pratap",
    booktitle = "Proceedings of the 14th International Joint Conference on Natural
      Language Processing and the 4th Conference of the {A}sia-Pacific Chapter of the
      {A}ssociation for {C}omputational {L}inguistics",
    month = dec,
    year = "2025",
    address = "Mumbai, India",
    publisher = "The Asian Federation of Natural Language Processing and The
      Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-ijcnlp.27/",
    doi = "10.18653/v1/2025.findings-ijcnlp.27",
    pages = "452--472",
    ISBN = "979-8-89176-303-6"
}

License

Code: MIT. Datasets retain their respective licenses and usage conditions, documented in the dataset reference.