Which Works Best for Vietnamese?
A Practical Study of Information Retrieval Methods across Domains

Overview · Quickstart · Methods · Datasets · Results · Findings · Evaluate & Extend · Citation

Long S. T. Nguyen^1,2,3 · Tho T. Quan<sup>1,2

1</sup> URA Research Group, Ho Chi Minh City University of Technology (HCMUT), Vietnam
² Vietnam National University Ho Chi Minh City, Vietnam
³ Center for AI Research (CAIR), VinUniversity, Vietnam

Overview

ViRE is a toolkit for evaluating Vietnamese information retrieval across domains. It brings lexical, dense, learned-sparse, late-interaction, and hybrid retrieval into one CLI, with shared data preparation, ranking metrics, and per-query error analysis.

The accompanying study compares retrieval methods on 10 datasets across 6 domains and introduces EduCoQA and CSConDa, drawn from educational counseling and customer-service interactions. The toolkit makes it straightforward to compare a new encoder, inspect retrieval failures, or evaluate your own question–document pairs.

Choose a starting point for your task:

I want to…	Start here	Output
Run a first evaluation	CPU quickstart	Ranking metrics and missed-query examples
Compare retrieval methods	Benchmark results · Full report	Scores grouped by dataset and backbone
Evaluate my own data	Input format · Usage guide	An evaluation on your question–passage pairs
Add a model or method	Extension guide	A retriever connected to the shared evaluation pipeline

Quickstart

Use Python 3.11+ for the dependency versions declared in this repository. The lexical example runs on CPU without API credentials or a model download.

git clone https://github.com/longstnguyen/ViRE.git
cd ViRE
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -e .

vi-retrieval-eval --csv data/CSConDa.csv --method bm25 \
  --max-samples 100 --sample-seed 42 --prefer-unique --dedup \
  --ks 1,10,20 --output-dir outputs/lexical

_{BM25 retrieval and evaluation on a 100-pair CSConDa sample.}

Read metrics.json, ranks.json, and errors/fail@20.csv in outputs/lexical/CSConDa/bm25-s100-uniq-dedup/. The failure file pairs missed queries with retrieved passages for error analysis. This example samples both the queries and their candidate contexts.

Compare TF-IDF on the same sample and export a report

vi-retrieval-eval --csv data/CSConDa.csv --method tfidf \
  --max-samples 100 --sample-seed 42 --prefer-unique --dedup \
  --ks 1,10,20 --output-dir outputs/lexical

python scripts/summarize_report.py --outputs-root outputs \
  --datasets CSConDa --metrics P@1,R@10,MRR@10,nDCG@10,R@20 \
  --percent --save outputs/report.md

_{Comparing BM25 and TF-IDF, then exporting a Markdown report.}

Try a dense–lexical hybrid

Install the optional backends, then combine a local Vietnamese encoder with BM25. Neural runs download the selected model and can use a GPU.

python -m pip install -e ".[all]"

vi-retrieval-eval --csv data/CSConDa.csv \
  --method dense+bm25 --dense-backend sbert \
  --sbert-model AITeamVN/Vietnamese_Embedding_v2 \
  --fusion alpha --alpha 0.7 --batch-size 16 \
  --max-samples 100 --sample-seed 42 --prefer-unique --dedup \
  --ks 1,10,20 --output-dir outputs/vietnamese-v2

See the usage guide for API backends, local LLM embedders, SPLADE, ColBERT, multi-gold relevance judgments, and the full CLI reference.

Methods

ViRE follows a shared workflow: load → normalize and sample → build the corpus → retrieve and fuse → evaluate → inspect errors. Each retrieval family uses the same input data and evaluation interface.

Retrieval family	--method	What it captures
Lexical	tfidf, bm25	Exact terms and vocabulary overlap
Dense	dense	Semantic similarity with local or API embedding models
Learned sparse	splade	Neural term expansion and sparse matching
Late interaction	colbert	Query–document token interactions
Hybrid	dense+tfidf, dense+bm25, splade+tfidf, splade+bm25, splade+dense	Complementary retrieval signals with score or rank fusion

Score fusion uses --fusion alpha --alpha 0.7: dense similarity receives weight α after per-query min–max normalization in dense–lexical hybrids. For dense+bm25, this gives 0.7 to dense similarity and 0.3 to BM25. Rank fusion uses --fusion rrf --rrf-k 60 to combine reciprocal ranks.

Retrieval in a RAG system

_{Retrieved passages provide evidence for answer generation; ViRE evaluates the retrieval stage.}

Datasets

The benchmark covers six domains. The study uses a sampling cap of 1,000 question–context pairs per dataset, retaining all pairs for smaller datasets. Use --max-samples to select a subset from the available files.

Domain	Datasets	Retrieval setting
Education	EduCoQA, ViRHE4QA [1]	Student counseling and institutional regulations
Customer support	CSConDa	Conversational service requests
Legal	ALQAC, ZaloLegalQA	Legal questions and statutory passages
Healthcare	ViNewsQA [2], ViMedAQA [3]	Health news and medical questions
Lifestyle & reviews	VlogQA [4], ViRe4MRC [5]	Spoken narratives and customer reviews
Open knowledge	UIT-ViQuAD 2.0 [6]	Encyclopedic question answering

Bonus datasets from our related work

We also share ViHERMES and VixSTORY, two datasets introduced in our related papers, alongside the ten-dataset ViRE benchmark. If you use either dataset, please cite its corresponding paper; full citations and BibTeX are available in each dataset folder.

Dataset	Focus	Citation
ViHERMES [7]	Multihop question answering over Vietnamese healthcare regulations	Cite ViHERMES
VixSTORY [8]	Structured multihop reasoning for cross-institution university admission advising	Cite VixSTORY

Two datasets from real conversations

EduCoQA captures student questions about admissions and university life; CSConDa contains customer-service conversations. Their frequent terms illustrate the vocabulary of these two domains.

_{EduCoQA: frequent topics in educational counseling.}

_{CSConDa: frequent topics in customer-service conversations.}

Results

MRR@10 across the ten datasets, reported on a 0–100 scale. “Best dense” and “Best hybrid” select the highest recorded MRR@10 in each family for each dataset; the winning backbone may differ by row. Bold marks the best value within a row, and underlining marks the second distinct value, including ties.

Dataset	Domain	BM25	Best dense	Best hybrid
EduCoQA	Education	23.09	34.22	35.24
ViRHE4QA	Education	76.05	72.04	81.42
CSConDa	Customer support	22.99	41.06	43.60
ALQAC	Legal	92.20	94.14	96.66
ZaloLegalQA	Legal	79.39	91.04	91.67
ViNewsQA	Healthcare	66.13	64.91	74.96
ViMedAQA	Healthcare	71.36	87.94	88.82
VlogQA	Lifestyle & reviews	23.90	32.49	39.53
ViRe4MRC	Lifestyle & reviews	10.25	17.25	18.21
UIT-ViQuAD 2.0	Open knowledge	78.09	89.29	93.62

The complete results provide every recorded configuration with P@1, R@10, MRR@10, nDCG@10, and R@20. See the paper for the experimental protocol and domain-level analysis.

Findings

The benchmark comparisons suggest five practical lessons for choosing encoders, tuning fusion, and evaluating Vietnamese retrieval.

• Hybrid retrieval is the strongest overall starting point in this study. Combining lexical and semantic evidence improves the best recorded MRR@10 over standalone dense retrieval across all 10 datasets. BM25-based linear fusion is particularly effective across domains.
• Calibrate the lexical–semantic balance. The full-corpus UIT-ViQuAD experiment peaks around α = 0.6–0.8; the main benchmark uses α = 0.7. Exact terms and semantic similarity both contribute to ranking quality.
• Language specialization and domain robustness are different strengths. Vietnamese-specialized encoders perform strongly on formal institutional text, while multilingual encoders such as multilingual-e5-large and bge-m3 offer strong performance across domains.
• Parameter count does not determine retrieval quality. The study finds substantial variation among LLM-derived embedders; larger models do not consistently outperform smaller encoders across these domains.
• Conversational retrieval remains the harder test. The best recorded MRR@10 is 96.66 on ALQAC, compared with 35.24 on EduCoQA and 43.60 on CSConDa. Abbreviations, register mismatches, and ambiguous intent make realistic user queries valuable evaluation targets.

_{Lexical–semantic balance on full-corpus UIT-ViQuAD with BM25 + multilingual-e5-large.}

Evaluate & Extend

Use the same evaluation workflow with your own data, then customize the retriever or metrics as needed. Start with the input schema below and use the configuration table to keep comparisons consistent.

Bring your own data

Create a UTF-8 CSV with question and context columns. Each row pairs a query with its relevant passage; all retained contexts form the candidate corpus.

question,context
"Hồ sơ nhập học gồm những gì?","Hồ sơ nhập học gồm giấy báo trúng tuyển và giấy tờ cá nhân."
"Khi nào được đổi sản phẩm?","Khách hàng có thể yêu cầu đổi sản phẩm trong bảy ngày."

vi-retrieval-eval --csv your_dataset.csv --method bm25 \
  --normalize-all --dedup --output-dir outputs/your-baseline

For multiple relevant passages per query, use a separate qid,doc_id,rel file and explicit ID-column flags; see multi-gold evaluation.

Configure an evaluation

Choose the data preparation, retrieval settings, and ranking cutoffs for your experiment. Keep the sample and preprocessing fixed when comparing methods.

Setting	Flags	Purpose
Sampling	--max-samples, --sample-frac, --sample-seed, --prefer-unique	Control the sampled query–context pairs
Text preparation	--normalize-all, --dedup	Normalize Vietnamese text and merge identical contexts
Model and batching	--dense-backend, --sbert-model, --batch-size, --max-len	Select a backend and control embedding inputs
Retrieval	--bm25-k1, --bm25-b, --fusion, --alpha, --rrf-k	Configure lexical scoring and fusion
Analysis	--ks, --error-k, --save-intersection	Choose metric cutoffs and compare retrieval failures
Output and cache	--output-dir, --force	Separate experiments and rebuild cached embeddings/indexes

Use distinct --output-dir values for separate experiment configurations. For report aggregation, follow outputs/<model-or-group>/<dataset>/<method-tag>/, as in the quickstart.

The batch evaluation script lists the study's datasets, backbones, and retrieval combinations. Review its model list and resource settings before launching the full suite; the usage guide explains the workflow.

Extend the toolkit

The source separates embedding backends, retrieval, fusion, and evaluation. The table below identifies where to implement each extension and how to expose it through the CLI.

Add or change	Start here	Connect it to the pipeline
Embedding backend	embeddings/base.py and embeddings/	Implement embed, register the backend, import it in embeddings/__init__.py, and expose its CLI configuration
Retrieval algorithm	runner.py, lexical.py, dense_index.py	Add the method branch and CLI choice
Fusion rule	fusion.py	Connect the rule in runner.py and --fusion choices
Metric	metrics.py	Add the per-query calculation and aggregate output; expose its name in reporting
Input format	io_utils.py, qrels.py	Map source data into questions, contexts, and relevance judgments

Citation

If you use the ViRE toolkit or the EduCoQA and CSConDa datasets, please cite our ViRE paper. If you use our bonus datasets, please cite the corresponding ViHERMES or VixSTORY paper.

@inproceedings{nguyen-quan-2026-works,
	title        = {{Which Works Best for Vietnamese?
	                 A Practical Study of Information Retrieval Methods across Domains}},
	author       = {Nguyen, Long S. T. and Quan, Tho T.},
	year         = 2026,
	month        = mar,
	booktitle    = {Findings of the Association for Computational Linguistics: EACL 2026},
	publisher    = {Association for Computational Linguistics},
	address      = {Rabat, Morocco},
	pages        = {2098--2119},
	doi          = {10.18653/v1/2026.findings-eacl.110},
	isbn         = {979-8-89176-386-9},
	url          = {https://aclanthology.org/2026.findings-eacl.110/},
}

License

ViRE's source code is released under the MIT License. The EduCoQA and CSConDa datasets are available under Creative Commons Attribution–NonCommercial 4.0 International. Third-party datasets retain their original licenses and terms of use.

References

[1] Tinh Pham Phuc Do, Ngoc Dinh Duy Cao, Khanh Quoc Tran, and Kiet Van Nguyen, “R2GQA: retriever-reader-generator question answering system to support students understanding legal regulations in higher education”, Artificial Intelligence and Law, May 2025.

[2] Kiet Van Nguyen, Tin Van Huynh, Duc-Vu Nguyen, Anh Gia-Tuan Nguyen, and Ngan Luu-Thuy Nguyen, “New Vietnamese Corpus for Machine Reading Comprehension of Health News Articles”, ACM Transactions on Asian and Low-Resource Language Information Processing, volume 21, number 5, article 105, September 2022.

[3] Minh-Nam Tran, Phu-Vinh Nguyen, Long Nguyen, and Dien Dinh, “ViMedAQA: A Vietnamese Medical Abstractive Question-Answering Dataset and Findings of Large Language Model”, in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), Bangkok, Thailand, pages 252–260, August 2024.

[4] Thinh Ngo, Khoa Dang, Son Luu, Kiet Nguyen, and Ngan Nguyen, “VlogQA: Task, Dataset, and Baseline Models for Vietnamese Spoken-Based Machine Reading Comprehension”, in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), St. Julian's, Malta, pages 1310–1324, March 2024.

[5] Tinh Pham Phuc Do, Ngoc Dinh Duy Cao, Nhan Thanh Nguyen, Tin Van Huynh, and Kiet Van Nguyen, “Machine Reading Comprehension for Vietnamese Customer Reviews: Task, Corpus and Baseline Models”, in Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation, Hong Kong, China, pages 24–35, December 2023.

[6] Kiet Van Nguyen, Duc-Vu Nguyen, Anh Gia-Tuan Nguyen, and Ngan Luu-Thuy Nguyen, “A Vietnamese Dataset for Evaluating Machine Reading Comprehension”, in Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (Online), pages 2595–2605, December 2020.

[7] Long S. T. Nguyen, Quan M. Bui, Tin T. Ngo, Quynh T. N. Vo, Dung N. H. Le, and Tho T. Quan, “ViHERMES: A Graph-Grounded Multihop Question Answering Benchmark and System for Vietnamese Healthcare Regulations”, in Intelligent Information and Database Systems, Springer Nature Singapore, pages 321–336, 2026.

[8] Long S. T. Nguyen, Tin T. Ngo, Dung N. H. Le, Quynh T. N. Vo, Dung H. Nguyen, Khang N. Le, and Tho T. Quan, “A Benchmark for Structured Multihop Reasoning in Cross-Institution University Admission Advisory Question Answering”, in Trends and Applications in Knowledge Discovery and Data Mining, Springer Nature Singapore, pages 235–247, 2027.