[experimenal] SINDI BM25 U4 LUT

[alexanderguzhva]

An experimental code that adds U4 LUT code instead of U8.

Please node that the code is not performance optimized, and just serves the demonstration of the approach.

this PR includes #1863 and #1864

All the following numbers are for the window size of 4096.

The format of the data is term storage format / docid storage format

The numbers for hotpotqa dataset on x86 M7i 8c32g are the following:

SIMD	Top-k	u8/u16	u16/u16	u4/u12	u4/u16
AVX2	10	2,636.3 / 0.999892	2,539.7 / 0.999797	2,210.9 / 0.999770	2,409.8 / 0.999770
AVX2	100	1,282.9 / 0.999864	1,230.4 / 0.999841	1,054.0 / 0.999458	1,188.2 / 0.999458
AVX512	10	2,774.9 / 0.999865	2,679.8 / 0.999770	2,474.7 / 0.999770	2,618.1 / 0.999770
AVX512	100	1,417.7 / 0.999812	1,372.3 / 0.999768	1,269.0 / 0.999458	1,377.8 / 0.999458

The numbers for various datasets on ARM G4 M8g 8c32, SVE code, are the following:

Dataset	Top-k	u8/u16	u16/u16	u4/u12	u4/u16
hotpotqa	10	3,236.0 / 1.000000	3,354.6 / 1.000000	2,565.8 / 0.999905	2,843.5 / 0.999905
hotpotqa	100	1,578.1 / 0.999995	1,631.6 / 0.999995	1,177.9 / 0.999522	1,331.2 / 0.999522
fever	10	4,769.3 / 0.998320	4,917.9 / 0.998320	3,964.6 / 0.995635	4,345.8 / 0.995635
fever	100	2,167.3 / 1.000000	2,255.4 / 1.000000	1,646.9 / 0.998066	1,866.0 / 0.998066
nq	10	9,615.1 / 0.998754	9,730.9 / 0.998754	7,875.6 / 0.998320	8,693.9 / 0.998320
nq	100	3,882.3 / 1.000000	3,911.8 / 1.000000	2,924.7 / 0.999519	3,341.2 / 0.999519
dbpedia-entity	10	8,607.4 / 0.995743	8,628.0 / 0.995743	7,647.4 / 0.995743	8,113.3 / 0.995743
dbpedia-entity	100	4,566.3 / 1.000000	4,736.6 / 1.000000	3,666.1 / 0.999724	4,079.0 / 0.999724

Serialized index file sizes, in bytes:

Dataset	u8/u16	u16/u16	u4/u12	u4/u16
hotpotqa	701,585,888	878,698,480	524,473,352	613,029,648
fever	1,114,945,232	1,407,828,688	822,057,768	968,500,512
nq	535,718,876	686,630,976	384,806,828	460,262,880
dbpedia-entity	676,510,472	847,248,064	505,772,616	591,141,492

/kind improvement

generated using Codex

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alexanderguzhva

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [alexanderguzhva]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment