The paper states that the RabbitQ algorithm achieves a QPS of 2329 on the OpenAI 1536 dataset. How can this be tested?
The official RabbitQ algorithm documentation supports single-threaded testing. My single-threaded tests show that RabbitQ's QPS is superior.
SAQ:
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260401 17:51:01.745781 304818 test_qps.cpp:252] ivf4096_b5_caq_adj_seg_pca
File ./data/openai1536/openai1536_base_pca.fvecs loaded (Rows 999000 Cols 1536)
File ./data/openai1536/openai1536_query_pca.fvecs loaded (Rows 1000 Cols 1536)
File ./data/openai1536/openai1536_groundtruth.ivecs loaded (Rows 1000 Cols 1000)
I20260401 17:51:04.362443 304818 ivf.hpp:233] Loading IVF...
I20260401 17:51:04.362475 304818 ivf.hpp:238] Loading meta data...
I20260401 17:51:04.835563 304818 ivf.hpp:194] Initializing done... num_points: 999000
I20260401 17:51:06.481058 304818 ivf.hpp:263] Index loaded
data loaded
N: 999000
DIM: 1536
query loaded
NQ: 1000
load index from ./data/openai1536/ivf4096_b5_caq_adj_seg_pca.index
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 5.45367ms qps: 183.363 bw_mbps: 965.627MB/s compute_kopps: 12718.5KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 5.39256ms qps: 185.441 bw_mbps: 976.559MB/s compute_kopps: 12862.5KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 4.85406ms qps: 206.013 bw_mbps: 1084.88MB/s compute_kopps: 14289.2KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 4.91416ms qps: 203.494 bw_mbps: 1071.62MB/s compute_kopps: 14114.6KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 4.57125ms qps: 218.759 bw_mbps: 1151.99MB/s compute_kopps: 15173.1KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 4.59051ms qps: 217.841 bw_mbps: 1147.13MB/s compute_kopps: 15109.1KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 4.59842ms qps: 217.466 bw_mbps: 1145.17MB/s compute_kopps: 15083.4KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 4.74816ms qps: 210.608 bw_mbps: 1109.07MB/s compute_kopps: 14607.9KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 4.87268ms qps: 205.226 bw_mbps: 1080.72MB/s compute_kopps: 14234.5KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 4.53798ms qps: 220.363 bw_mbps: 1160.43MB/s compute_kopps: 15284.4KOP/s
I20260401 17:51:56.015496 304818 test_qps.cpp:238] result log to file: ./results/saq//qps_openai1536_ivf4096_b5_caq_adj_seg_pca_th1_np256_sm4.csv
nprobe,num_threads,QPS,avg_tm_ms,recall,ratio,bw_mbps,compute_kopps
256,1,206.8572,4.853344,0.95473,1.0012699,965.6272,12718.522
Rabitq (https://github.com/VectorDB-NTU/RaBitQ-Library)
File ./data/openai1536/openai1536_query.fvecs loaded
Rows 1000 Cols 1536
File ./data/openai1536/openai1536_groundtruth.ivecs loaded
Rows 1000 Cols 1000
Loading IVF...
Loading meta data...
FhtKacRotator is selected
Allocating memory for IVF...
Index loaded
0.65004 5
0.92521 128
0.95306 256
nprobe QPS recall
5 741.905 0.65004
128 413.515 0.92521
256 296.466 0.95306
The paper states that the RabbitQ algorithm achieves a QPS of 2329 on the OpenAI 1536 dataset. How can this be tested?
The official RabbitQ algorithm documentation supports single-threaded testing. My single-threaded tests show that RabbitQ's QPS is superior.
SAQ:
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260401 17:51:01.745781 304818 test_qps.cpp:252] ivf4096_b5_caq_adj_seg_pca
File ./data/openai1536/openai1536_base_pca.fvecs loaded (Rows 999000 Cols 1536)
File ./data/openai1536/openai1536_query_pca.fvecs loaded (Rows 1000 Cols 1536)
File ./data/openai1536/openai1536_groundtruth.ivecs loaded (Rows 1000 Cols 1000)
I20260401 17:51:04.362443 304818 ivf.hpp:233] Loading IVF...
I20260401 17:51:04.362475 304818 ivf.hpp:238] Loading meta data...
I20260401 17:51:04.835563 304818 ivf.hpp:194] Initializing done... num_points: 999000
I20260401 17:51:06.481058 304818 ivf.hpp:263] Index loaded
data loaded
N: 999000
DIM: 1536
query loaded
NQ: 1000
load index from ./data/openai1536/ivf4096_b5_caq_adj_seg_pca.index
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 5.45367ms qps: 183.363 bw_mbps: 965.627MB/s compute_kopps: 12718.5KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 5.39256ms qps: 185.441 bw_mbps: 976.559MB/s compute_kopps: 12862.5KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 4.85406ms qps: 206.013 bw_mbps: 1084.88MB/s compute_kopps: 14289.2KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 4.91416ms qps: 203.494 bw_mbps: 1071.62MB/s compute_kopps: 14114.6KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 4.57125ms qps: 218.759 bw_mbps: 1151.99MB/s compute_kopps: 15173.1KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 4.59051ms qps: 217.841 bw_mbps: 1147.13MB/s compute_kopps: 15109.1KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 4.59842ms qps: 217.466 bw_mbps: 1145.17MB/s compute_kopps: 15083.4KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 4.74816ms qps: 210.608 bw_mbps: 1109.07MB/s compute_kopps: 14607.9KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 4.87268ms qps: 205.226 bw_mbps: 1080.72MB/s compute_kopps: 14234.5KOP/s
num_threads: 1 recall: 0.95473 dist_rate: 1.00127 q_avg_tm: 4.53798ms qps: 220.363 bw_mbps: 1160.43MB/s compute_kopps: 15284.4KOP/s
I20260401 17:51:56.015496 304818 test_qps.cpp:238] result log to file: ./results/saq//qps_openai1536_ivf4096_b5_caq_adj_seg_pca_th1_np256_sm4.csv
nprobe,num_threads,QPS,avg_tm_ms,recall,ratio,bw_mbps,compute_kopps
256,1,206.8572,4.853344,0.95473,1.0012699,965.6272,12718.522
Rabitq (https://github.com/VectorDB-NTU/RaBitQ-Library)
File ./data/openai1536/openai1536_query.fvecs loaded
Rows 1000 Cols 1536
File ./data/openai1536/openai1536_groundtruth.ivecs loaded
Rows 1000 Cols 1000
Loading IVF...
Loading meta data...
FhtKacRotator is selected
Allocating memory for IVF...
Index loaded
0.65004 5
0.92521 128
0.95306 256
nprobe QPS recall
5 741.905 0.65004
128 413.515 0.92521
256 296.466 0.95306