From 0980938098a80e8cfccc04d4aa1c2f6613a96090 Mon Sep 17 00:00:00 2001 From: Simon Pichugin Date: Wed, 22 Jul 2026 20:13:21 -0700 Subject: [PATCH 1/2] Add OR filter equality lookup design doc --- docs/389ds/design/design.md | 4 + docs/389ds/design/or-filter-lookup-design.md | 251 +++++++++++++++++++ 2 files changed, 255 insertions(+) create mode 100644 docs/389ds/design/or-filter-lookup-design.md diff --git a/docs/389ds/design/design.md b/docs/389ds/design/design.md index 1e5055a..a7bb000 100644 --- a/docs/389ds/design/design.md +++ b/docs/389ds/design/design.md @@ -39,6 +39,10 @@ If you are adding a new design document, use the [template](design-template.html - [Replication Monitoring With Ansible](ansible-replication-monitoring-design.html) +## 389 Directory Server 3.x + +- [OR Filter Equality Lookup](or-filter-lookup-design.html) + ## 389 Directory Server 3.1 - [Session Tracking Control client - replication](session-identifier-clients.html) diff --git a/docs/389ds/design/or-filter-lookup-design.md b/docs/389ds/design/or-filter-lookup-design.md new file mode 100644 index 0000000..56091b2 --- /dev/null +++ b/docs/389ds/design/or-filter-lookup-design.md @@ -0,0 +1,251 @@ +--- +title: "OR Filter Equality Lookup" +--- + +# OR Filter Equality Lookup +--------------------------- + +Overview +-------- + +Applications often send searches that are one big OR of equality tests on +the same attribute: + + (|(uid=user1)(uid=user2)(uid=user3) ... (uid=user1000)) + +This is how a sync job fetches a batch of accounts, how an inventory system +checks a list of hostnames, and how an org chart tool finds everyone who +reports to a set of managers. + +These searches are usually fully indexed, and the index side is quick. The +slow part hides somewhere else. Before returning an entry the server has to +prove the entry really matches the filter, and that filter test walked the +OR from left to right, every branch, for every candidate. Each branch +scanned the entry's attributes and normalized its values all over again. +With 1000 branches and 1000 candidates, that's a million branch evaluations +for a single search. + +The server now builds a sorted lookup table from the OR's assertion values, +once per search. Each candidate entry normalizes its values once and finds +the matching branch with a binary search. On a 100,000 entry test database a +355-branch search over 612 candidates went from 1.7 seconds to 0.18 seconds, +and a 10,000 candidate case went from 23 seconds to 0.4 seconds. Search +results are identical, and anything the table can't handle simply falls back +to the old code. + +Use Cases +--------- + +- The case from issue 6275: an application keeps its user list in sync + with the directory and fetches all the accounts it cares about in one + search, a thousand uids in a single OR: + + (|(uid=jsmith)(uid=mjones)(uid=bwilson)...) + +- An inventory or configuration management system keeps its machines in + LDAP and checks a batch of hostnames in one query: + + (|(cn=web01.example.com)(cn=web02.example.com)(cn=db03.example.com)...) + +- A backup or quota tool walks a filesystem, collects the numeric owners + it found, and maps them back to accounts: + + (|(uidNumber=1004)(uidNumber=1027)(uidNumber=1263)...) + +Clients that send small ORs, like SSSD with its handful of branches, aren't +affected. The lookup doesn't even wake up below 16 branches; a table +wouldn't pay for itself there. + +Design +------ + +The backend creates its own normalized copy of the search filter at the +start of each search. After that copy is made, the server walks it and looks +at every OR node. When an OR contains at least 16 equality branches on the +same attribute, and the attribute qualifies, the server builds a sorted +table of the branch values and attaches it to that OR node. + +During the per-entry filter test, an OR node that has a table does not walk +its branches. The entry's values are normalized once, with the same +normalizer that was applied to the branch values, and looked up in the +table. + +The table only decides which branch to test; it's a shortcut to the right +branch, never a substitute for testing it. When a value is found in the +table, the server still runs the normal access check and the normal matching +call for that branch. A branch the client is not allowed to read still never +counts as a match, which has been the rule since ticket 48275. If no table +branch matches, branches outside the table (other attributes, substrings, +and so on) are evaluated the old way. + +An attribute qualifies when: + +- its syntax is Directory String, IA5 String, Integer, Numeric String, + Telephone Number, or DN. For these, two values are equal exactly when + their normalized forms are the same bytes. +- its equality matching rule is one of the eight standard rules + (caseIgnoreMatch, caseExactMatch, caseIgnoreIA5Match, caseExactIA5Match, + integerMatch, numericStringMatch, telephoneNumberMatch, + distinguishedNameMatch). +- the filter uses the plain attribute, without options. `uid` qualifies, + `uid;lang-en` does not. + +When an OR mixes several attributes, the largest qualifying group gets the +table and the remaining branches are evaluated normally. On a tie, the group +that appears first in the filter wins. + +The old code runs unchanged when: + +- the OR has fewer than 16 qualifying branches +- the attribute is not in the schema, or uses generalizedTime, Boolean, + Bit String, or Name And Optional UID syntax +- the matching rule is a collation rule or comes from a plugin +- the search is a tombstone or RUV search +- the attribute may be supplied by CoS or Roles for the entry being tested +- a DN-valued entry has more values than the table has keys, where the old + code is cheaper +- a stored DN value cannot be parsed +- the evaluation is an access-only pass, which must visit every branch + +One more restriction, and it's the subtle one. A filter branch can evaluate +to true, false, or undefined, where undefined usually means "you're not +allowed to read this attribute". Normal searches treat false and undefined +the same, so when all table lookups miss, the server can mark the entry as a +non-match right away. +NOT and VLV are the two places that can tell false and undefined apart, so +under a NOT, and in VLV searches, this shortcut is disabled and the old walk +decides. + +One side effect: the server no longer asks the ACL plugin about branches +whose values the entry does not have. This is only visible with ACL debug +logging enabled, where large OR searches now produce far fewer lines. +Nothing changes about who can see what. + +Implementation +-------------- + +New file `ldap/servers/slapd/filter_or_lookup.c` holds the eligibility +checks, the table build, the probe, and the free function. The per-entry +lookup path is `vattr_test_filter_or_lookup()` in +`ldap/servers/slapd/filterentry.c`. The build is called from the filter +pre-digest block in `ldap/servers/slapd/back-ldbm/ldbm_search.c`. The +configuration attribute is wired in `libglobs.c` like the other `cn=config` +switches. + +The table hangs off the OR node of the backend's private filter copy and is +freed together with it in `slapi_filter_free()`. `slapi_filter_dup()` does +not copy tables. Filters parsed for other purposes (plugins, persistent +search, ACL evaluation, `cn=config` searches) never get tables. The table is +read-only after the build, so later pages of a paged search can read it +safely. + +Memory cost is about 40 bytes per branch for the lifetime of the search. A +10,000-branch OR allocates around 400 KB. There is no upper limit on the +table size beyond the existing limits on incoming requests; in the +measurements below, even a 1000-branch table built for a search with no +candidates cost nothing visible. + +Configuration +------------- + + dn: cn=config + nsslapd-enable-or-filter-lookup: on + +The default is on. The setting is dynamic and needs no restart: + + dsconf config replace nsslapd-enable-or-filter-lookup=off + +The server reads the setting when it prepares a search's filter, so a change +applies to new searches. A paged search that's already running keeps its +tables until the client finishes paging; don't expect a search in flight to +change course. + +Troubleshooting +--------------- + +One line is logged when tables are built, at a debug level that is off by +default. Add 524288 (SLAPI_LOG_BACKLDBM) to `nsslapd-errorlog-level` and +look for: + + DEBUG - ldbm_back_search - OR filter equality lookup engaged: 1 node(s), largest 355 branches + +The line means tables were built for the search, nothing more. It doesn't +say how many entries used them. There's no counter in `cn=monitor` and no +`notes=` keyword in the access log. + +To confirm the lookup is really deciding, also enable filter logging (add +32) and check that the per-branch `test_ava_filter - => AVA:` lines +disappear for the attribute. Do that on a test system; at these log levels a +busy server spends more time logging than searching. + +If the feature does not engage, check that the setting is on, that the OR +has at least 16 equality branches on one attribute, that the attribute's +syntax and equality matching rule are in the supported lists above, and that +the search is not a tombstone or RUV search. + +Performance - To Be Updated With More Data +----------- + +Measured on Fedora with a 100,000 entry database. Every number is the median +of 15 searches. Both runs used the same installed package and the same +indexes; only `nsslapd-enable-or-filter-lookup` was changed. + +| Search | off | on | +|--------|----:|---:| +| 355 DN branches, 612 candidate entries, no matches | 1.73 s | 0.18 s | +| the same filter over 10,000 candidates | 23.2 s | 0.39 s | +| 1000 branches, more than half values present | 2.92 s | 0.29 s | +| the only match on the last of 355 branches | 3.84 s | 0.20 s | +| 128 branches, all values present | 0.26 s | 0.17 s | + +The saving grows with the number of candidates times the number of +branches. Below about 64 branches the difference is within measurement +noise. Searches with no candidates measure the same even with a 1000-branch +table, so building a table that goes unused costs nothing visible. Nothing +got slower beyond noise, and that's the part we watched most closely. + +For context, packaged OpenLDAP on the same data ran the 10,000 candidate +case in 0.85 seconds; 389 DS with the lookup ran it in 0.39 seconds. +Different servers do different work, so read that as context, not as a +scoreboard. + +Testing +------- + +- `dirsrvtests/tests/suites/filter/filter_or_lookup_test.py`: the same + searches with the feature on and off must return the same results. + Covers all supported matching rules, non-ASCII values, escaped DNs, + thresholds 15 and 16, mixed attribute groups, subtypes, CoS, referrals, + paging, VLV with server-side sorting, and proxy authorization. +- `filter_or_lookup_feature_test.py`: reads the debug log to prove the + lookup engaged, fell back, or was disabled where expected. +- `filter_or_union_test.py` and + `filter_large_filter_interaction_test.py`: OR result sets across + `idl_set` consumers, and large DN filters combined with substring and + approximate branches. +- `memory_leaks/filter_not_first_lifecycle_test.py`: filter ownership + under ASan across startup, VLV, paged, and cancelled operations. Needs a + dedicated ASan build and is not part of the default CI run. + +Replication +----------- + +No impact. The change is confined to search evaluation. + +Updates and Upgrades +-------------------- + +Nothing to migrate. The attribute has a default, needs no schema or +`dse.ldif` changes, and upgraded servers get the feature enabled. Setting it +to off restores the previous behavior exactly. + + +Origin +------ + +https://github.com/389ds/389-ds-base/issues/7664 + +Author +------ + + From b87dcfe244391515bd8069c6d3f633ab3f4bd648 Mon Sep 17 00:00:00 2001 From: Simon Pichugin Date: Mon, 27 Jul 2026 16:31:14 -0700 Subject: [PATCH 2/2] Complete performance results and design clarifications/rework --- docs/389ds/design/or-filter-lookup-design.md | 158 ++++++++++++------- 1 file changed, 98 insertions(+), 60 deletions(-) diff --git a/docs/389ds/design/or-filter-lookup-design.md b/docs/389ds/design/or-filter-lookup-design.md index 56091b2..80973f4 100644 --- a/docs/389ds/design/or-filter-lookup-design.md +++ b/docs/389ds/design/or-filter-lookup-design.md @@ -19,19 +19,22 @@ reports to a set of managers. These searches are usually fully indexed, and the index side is quick. The slow part hides somewhere else. Before returning an entry the server has to -prove the entry really matches the filter, and that filter test walked the +prove the entry really matches the filter, and that filter test walks the OR from left to right, every branch, for every candidate. Each branch -scanned the entry's attributes and normalized its values all over again. +scans the entry's attributes and normalizes its values all over again. With 1000 branches and 1000 candidates, that's a million branch evaluations for a single search. -The server now builds a sorted lookup table from the OR's assertion values, -once per search. Each candidate entry normalizes its values once and finds -the matching branch with a binary search. On a 100,000 entry test database a -355-branch search over 612 candidates went from 1.7 seconds to 0.18 seconds, -and a 10,000 candidate case went from 23 seconds to 0.4 seconds. Search -results are identical, and anything the table can't handle simply falls back -to the old code. +The lookup replaces the walk: the server builds a sorted table from the +OR's assertion values, once per search, and each candidate entry +normalizes its values once and finds the matching branch with a binary +search. A 355-branch search over 612 candidates drops from 0.24 seconds +to 0.09 seconds, and a 10,000 candidate case from 2.6 seconds to 0.19 +seconds; measured against current main, which carries neither this +feature nor the companion Issue 7655 fix beneath it, the same +10,000-candidate case starts at 9.0 seconds (details in the Performance +section). Search results are identical, and anything the table can't +handle simply falls back to the classic walk. Use Cases --------- @@ -54,16 +57,19 @@ Use Cases Clients that send small ORs, like SSSD with its handful of branches, aren't affected. The lookup doesn't even wake up below 16 branches; a table -wouldn't pay for itself there. +wouldn't pay for itself there. The one shape that pays above the +threshold is a large OR returning almost nothing - a few milliseconds +for the table (details in the Performance section). Design ------ -The backend creates its own normalized copy of the search filter at the -start of each search. After that copy is made, the server walks it and looks -at every OR node. When an OR contains at least 16 equality branches on the -same attribute, and the attribute qualifies, the server builds a sorted -table of the branch values and attaches it to that OR node. +When a search needs the per-entry filter test, the backend creates its own +normalized copy of the filter, walks it, and looks at every OR node. When +an OR contains at least 16 equality branches on the same attribute, and +the attribute qualifies, the server builds a sorted table of the branch +values and attaches it to that OR node. Duplicate assertion values +collapse to a single key. During the per-entry filter test, an OR node that has a table does not walk its branches. The entry's values are normalized once, with the same @@ -76,17 +82,17 @@ table, the server still runs the normal access check and the normal matching call for that branch. A branch the client is not allowed to read still never counts as a match, which has been the rule since ticket 48275. If no table branch matches, branches outside the table (other attributes, substrings, -and so on) are evaluated the old way. +and so on) are evaluated by the classic walk. An attribute qualifies when: - its syntax is Directory String, IA5 String, Integer, Numeric String, Telephone Number, or DN. For these, two values are equal exactly when their normalized forms are the same bytes. -- its equality matching rule is one of the eight standard rules - (caseIgnoreMatch, caseExactMatch, caseIgnoreIA5Match, caseExactIA5Match, - integerMatch, numericStringMatch, telephoneNumberMatch, - distinguishedNameMatch). +- its equality matching rule, when one is registered, is one of the + eight standard rules (caseIgnoreMatch, caseExactMatch, + caseIgnoreIA5Match, caseExactIA5Match, integerMatch, + numericStringMatch, telephoneNumberMatch, distinguishedNameMatch). - the filter uses the plain attribute, without options. `uid` qualifies, `uid;lang-en` does not. @@ -94,16 +100,19 @@ When an OR mixes several attributes, the largest qualifying group gets the table and the remaining branches are evaluated normally. On a tie, the group that appears first in the filter wins. -The old code runs unchanged when: +The classic walk runs unchanged when: - the OR has fewer than 16 qualifying branches - the attribute is not in the schema, or uses generalizedTime, Boolean, Bit String, or Name And Optional UID syntax -- the matching rule is a collation rule or comes from a plugin +- the matching rule is a collation rule or a plugin rule outside the + eight above - the search is a tombstone or RUV search +- the search skips the per-entry filter test entirely (a fully indexed + filter that needs no test builds no normalized copy and no table) - the attribute may be supplied by CoS or Roles for the entry being tested -- a DN-valued entry has more values than the table has keys, where the old - code is cheaper +- a DN-valued entry has more values than the table has distinct keys, + where the classic walk is cheaper - a stored DN value cannot be parsed - the evaluation is an access-only pass, which must visit every branch @@ -113,13 +122,13 @@ allowed to read this attribute". Normal searches treat false and undefined the same, so when all table lookups miss, the server can mark the entry as a non-match right away. NOT and VLV are the two places that can tell false and undefined apart, so -under a NOT, and in VLV searches, this shortcut is disabled and the old walk -decides. +under a NOT, and in VLV searches, this shortcut is disabled and the classic +walk decides. -One side effect: the server no longer asks the ACL plugin about branches +One side effect: the server does not ask the ACL plugin about branches whose values the entry does not have. This is only visible with ACL debug -logging enabled, where large OR searches now produce far fewer lines. -Nothing changes about who can see what. +logging enabled, where large OR searches produce far fewer lines than the +classic walk would. Nothing changes about who can see what. Implementation -------------- @@ -139,11 +148,11 @@ search, ACL evaluation, `cn=config` searches) never get tables. The table is read-only after the build, so later pages of a paged search can read it safely. -Memory cost is about 40 bytes per branch for the lifetime of the search. A -10,000-branch OR allocates around 400 KB. There is no upper limit on the -table size beyond the existing limits on incoming requests; in the -measurements below, even a 1000-branch table built for a search with no -candidates cost nothing visible. +The build stays cheap by construction: a search whose candidate list is +empty skips it entirely (no entry would ever probe the table), and key +extraction runs in a single pass per OR node, so DN branches are parsed +and case-normalized once. Memory cost is about 40 bytes per branch for +the lifetime of the search: a 10,000-branch OR allocates around 400 KB. Configuration ------------- @@ -180,34 +189,61 @@ busy server spends more time logging than searching. If the feature does not engage, check that the setting is on, that the OR has at least 16 equality branches on one attribute, that the attribute's -syntax and equality matching rule are in the supported lists above, and that -the search is not a tombstone or RUV search. +syntax and equality matching rule are in the supported lists above, that +the search is not a tombstone or RUV search, and that candidate generation +produced at least one entry (a search with an empty candidate list skips +the build). -Performance - To Be Updated With More Data +Performance ----------- -Measured on Fedora with a 100,000 entry database. Every number is the median -of 15 searches. Both runs used the same installed package and the same -indexes; only `nsslapd-enable-or-filter-lookup` was changed. - -| Search | off | on | -|--------|----:|---:| -| 355 DN branches, 612 candidate entries, no matches | 1.73 s | 0.18 s | -| the same filter over 10,000 candidates | 23.2 s | 0.39 s | -| 1000 branches, more than half values present | 2.92 s | 0.29 s | -| the only match on the last of 355 branches | 3.84 s | 0.20 s | -| 128 branches, all values present | 0.26 s | 0.17 s | +Measured on Fedora with a 100,000 entry database; every number is the +median of 15 searches. The lookup columns come from one installed build +containing this feature and the companion Issue 7655 fix, with only +`nsslapd-enable-or-filter-lookup` changed between them - so "lookup off" +to "lookup on" is a same-binary measurement. Its noise band, taken from +28 strata where the lookup cannot engage, is +/-4.2 % elapsed and ++/-5.1 % CPU. The "main" column comes from a companion run of the same +workload against current main (which contains neither change); the +identical-binary anchor states shared by the two runs agree within 2 %. + +| Search | main | lookup off | lookup on | +|--------|----:|----:|---:| +| 355 DN branches, 612 candidate entries, no matches | 0.61 s | 0.24 s | 0.092 s | +| the same filter over 10,000 candidates | 9.02 s | 2.58 s | 0.19 s | +| the only match on the last of 355 branches | 1.19 s | 0.39 s | 0.11 s | +| 355 branches, nearly all values present | 0.43 s | 0.20 s | 0.10 s | +| 128 branches, all values present | 0.12 s | 0.095 s | 0.088 s | + +The "lookup off" column is what the Issue 7655 fix alone does: it makes +each DN branch test cheaper, while the lookup removes the need to walk +the branches at all. The two address the same hot path from different +sides - the fix helps every DN equality test including sub-threshold ORs +and ANDs; the lookup engages at 16 or more same-type equality branches +for every supported attribute type, not just DNs. Turning the lookup on +takes the 10,000-candidate case a further +93 % (2.58 s to 0.19 s), and +main to lookup-on combined is +97.9 %, about 48x. The saving grows with the number of candidates times the number of -branches. Below about 64 branches the difference is within measurement -noise. Searches with no candidates measure the same even with a 1000-branch -table, so building a table that goes unused costs nothing visible. Nothing -got slower beyond noise, and that's the part we watched most closely. - -For context, packaged OpenLDAP on the same data ran the 10,000 candidate -case in 0.85 seconds; 389 DS with the lookup ran it in 0.39 seconds. -Different servers do different work, so read that as context, not as a -scoreboard. +branches. Below 128 branches the live-value difference is within the +noise band; at 128 it clears it (+6.4 % elapsed / +8.2 % CPU), and from +355 branches up it is decisive. Across the full 134-stratum screen, 47 +strata improve coherently beyond the band and most of the rest sit +within noise. The cost side is one narrow shape: a large OR where only +a handful of entries reach the filter test pays a few milliseconds for +a table that goes mostly unused. Two things bound even that in +practice. Interactive traffic never sees it: logins, lookups, and short +client filters stay under the 16-branch threshold and never engage the +lookup. And the toll and the wins land on the same clients: the batch +job that pays a few milliseconds on a stale-list pass is the job whose +bulk fetch drops from 9 seconds to 0.19, so any realistic mix is +dominated by the wins - with `nsslapd-enable-or-filter-lookup` as the +per-instance opt-out for a workload that somehow is not. + +For context, packaged OpenLDAP on the same data runs the 10,000 candidate +case in about 0.47 seconds; 389 DS with the lookup runs it in 0.19 +seconds, and main without either change in 9.0 seconds. Different servers +do different work, so read that as context, not as a scoreboard. Testing ------- @@ -218,11 +254,13 @@ Testing thresholds 15 and 16, mixed attribute groups, subtypes, CoS, referrals, paging, VLV with server-side sorting, and proxy authorization. - `filter_or_lookup_feature_test.py`: reads the debug log to prove the - lookup engaged, fell back, or was disabled where expected. + lookup engaged, fell back, or was disabled where expected, including a + DN-syntax family (`test_or_lookup_dn_family_engages`). - `filter_or_union_test.py` and `filter_large_filter_interaction_test.py`: OR result sets across `idl_set` consumers, and large DN filters combined with substring and - approximate branches. + approximate branches. `filter_large_filter_support.py` is the shared + workload module these import; it contains no tests itself. - `memory_leaks/filter_not_first_lifecycle_test.py`: filter ownership under ASan across startup, VLV, paged, and cancelled operations. Needs a dedicated ASan build and is not part of the default CI run. @@ -237,7 +275,7 @@ Updates and Upgrades Nothing to migrate. The attribute has a default, needs no schema or `dse.ldif` changes, and upgraded servers get the feature enabled. Setting it -to off restores the previous behavior exactly. +to off restores the classic branch walk exactly. Origin