diff --git a/.claude/board/entries/2026-10-21-dr256-iam-nested-groups.md b/.claude/board/entries/2026-10-21-dr256-iam-nested-groups.md new file mode 100644 index 000000000..b2fdd69e3 --- /dev/null +++ b/.claude/board/entries/2026-10-21-dr256-iam-nested-groups.md @@ -0,0 +1,217 @@ +# 2026-10-21 — IAM nested groups as a DR256 register (arm Q, dir-rmap-probe) + +**Status:** MEASURED, lab only. Nothing admitted. Operator direction (2026-10-21): "IAM nested groups as a DR 256 user 256 group table (global to local register, 'QR code' as group membership matrix)". Answers open decision D3 of the ClassView × DR map (entry `2026-10-21-coresearch-classview-algebra-polyfill.md`) for the case where the security chain fits one register. + +## What landed + +`crates/dir-rmap-probe/src/qr.rs` (arm Q), `tests/qr_register.rs`, `examples/qr_cost.rs`. + +- **Register:** every security group is bound to one of 256 local slots, once per snapshot (global → local). More than 256 security groups is refused (`TooLarge`), never truncated. That is what makes the in-tile closure complete. +- **Nesting:** a 256 × 256 bit tile (child slot → parent slots), closed inside the tile by Warshall (reflexive + transitive). +- **Membership matrix ("QR code"):** users in pages of 256 rows × 256 slot columns. A user's effective groups = its row × `reach`. +- **Roles:** per slot, the grants its reach carries. Per user, they are resolved per page of 256 users as a column fold: the users holding role `r` are the OR of the columns of the slots whose role set contains `r`. +- **Lazy pages (revision 2):** `Qr::bind` builds only the register (slots, nesting tile, closure, slot roles) and resolves no user. A page is resolved when a question names one of its users (`Qr::page`, `Pages`). +- **ClassView wiring:** the class declares `effective_roles` computed from `member_of` (`compute_dag`); the DR table maps that position to `Qr::roles_of`. Same shape as `classview_dr.rs`; no new type. +- **Address rule:** every inner loop runs over slot or user coordinates and tests a bit; no bit scan. The ordinal → slot translations are counted at bind (450 at 8,000 users: the nesting pairs) and per page (each user's direct groups). They are frozen-codebook decoding over one snapshot; the contract-level witness is still missing (ADMISSION B). + +## Correctness + +6 tests. For every user of `Spec::small` (600 users, 174 groups, 164 security groups): +- roles equal dir-sim (arm A), the crepe oracle (arm D) and B+; +- effective security groups match dir-sim's walk slot by slot. + +This holds before and after a nesting add plus a membership removal (rebind). Refusals: >256 security groups (`Spec::iam(2000)`), a stale snapshot (bind and ask), a group outside the chain. Anti-vacuity: some users but not all hold roles, and some user gains a group only through nesting. A further test checks that bind resolves no page and that two users of one page resolve that page once. + +Disable runs (committed first, each reverted; all red): +- Warshall removed; +- reflexive seed removed; +- truncate instead of refuse; +- bind ignores the witness; +- ask ignores the witness; +- grants not folded through reach; +- page column off by one user; +- last page dropped. + +Revision 2 disable runs (all red): `Pages` re-resolving on every call, `page` ignoring the witness, a page dropping its last user, a column bit set at the global user index, a user mapped to the wrong page. + +## Operator correction (2026-10-21) + +Revision 1 resolved every user's roles at bind and measured that. The operator: "Nobody resolves nested groups for 64k users x 1700 groups at once. In real life even with Powershell hashtables that takes 2-12minutes". The unit of work is one team, OU or page asked for on demand, not the population. Revision 2 measures that unit; the eager figure is kept only as a comparison. + +## Measured (release, `examples/qr_cost.rs`, group side of `Spec::small`, every route checked equal to dir-sim first) + +| | 8,000 users | 65,000 users | +|---|---:|---:| +| bind register (slots + closure + slot roles, no user) | 39.9 µs | 41.3 µs | +| resolve one page (256 users), first touch | 14.7 µs | 13.3 µs | +| per user: Q roles from a resolved page | 0.9 ns | 1.0 ns | +| per user: B+ fold | 12.6 ns | 14.1 ns | +| per user: Q roles live (row × slot roles) | 140.1 ns | 148.1 ns | +| per user: Q effective row (M × R) | 167.8 ns | 258.4 ns | +| every page at once (not a workload; comparison only) | 486.6 µs | FAILED GATE (ms) | +| prepare (arm B, dir-sim → CSR; pre-existing) | FAILED GATE (ms) | FAILED GATE (ms) | + +Register 19,108 bytes; one page 10,240 bytes. + +## Reading + +- **Both on-demand costs are independent of the user count:** the register binds in ≈ 40 µs and a page resolves in ≈ 14 µs at 8,000 and at 65,000 users. Asking about one team costs the register plus that team's page. +- **Queries on a resolved page are the cheapest route measured:** ~1 ns per user against ~13–14 ns for the B+ fold. +- **Live per-user products** (a row times the tile, 164 coordinates tested per user) are 10× slower than the B+ fold. The register pays when answers are read from a resolved page. +- **Eager resolve of the whole population** fails the gate at 65,000 users, as revision 1 measured. Per the correction above, that is not a workload. +- **The arm B prepare** (dir-sim → CSR) is a pre-existing millisecond phase, not introduced here. A page resolve reads that CSR; making the page read the source lanes directly is open. + +## Masked count over several tiles (revision 3) + +Operator direction (2026-10-21): measure amortization by masking over multiple QR tiles, with interlaced row activation by selective tile masking. + +A question ("how many users hold role r", or "are effectively in group g") becomes a **slot set**: `Qr::role_slots[r]` (`role` transposed at bind) or `Qr::slots_reaching(g)` (the slots whose reach contains g). It is asked over `k` tiles selected by a **tile mask** (one bit per page), with rows inside each tile selected by a **row mask** (all rows, or interlaced: even rows in even tiles, odd rows in odd tiles). The terminal is a count; no page is resolved and no per-user value is built. + +Routes, all checked equal to a page-by-page resolve (`tests/qr_masked.rs`: every role and a sample of groups × 4 tile masks × 4 row masks; scratch is checked left zeroed), measured in `examples/qr_amortize.rs`: + +- `page`: resolve each selected page on first touch, then fold its roles over the active rows; +- `warm page`: the same fold over pages already resolved; +- `masked` (`Qr::count_masked`): each needed slot's direct-member list is read once, and every member sets its bit in its tile, branch-free (the tile's selection bit is shifted into the OR); popcount of tile AND row mask; +- `runs` (`Runs::count`): a per-slot tile pointer table (`runs[s][p]..runs[s][p+1]` = slot s's members in tile p; member lists are sorted by user ordinal, so each tile is one run). Built for a slot on first use (`cold`), reused after (`warm`). Only selected tiles' runs are visited, OR'd into one 256-bit row per tile. + +65,000 users (254 tiles), all rows. The role is carried by 32 slots (21,042 direct memberships); the group is reached from 26 slots (13,867): + +| k tiles | role: page first touch | role: warm page | role: masked | role: runs warm | group: page first touch | group: masked | group: runs warm | +|---:|---:|---:|---:|---:|---:|---:|---:| +| 1 | 14.0 µs | 0.1 µs | 33.1 µs | 0.7 µs | 56.4 µs | 22.8 µs | 0.7 µs | +| 8 | 116.3 µs | 0.5 µs | 34.2 µs | 2.9 µs | 446.6 µs | 22.0 µs | 2.3 µs | +| 32 | 502.0 µs | 2.1 µs | 34.2 µs | 10.3 µs | ms (gate) | 22.0 µs | 8.8 µs | +| 64 | ms (gate) | 4.2 µs | 34.3 µs | 19.8 µs | ms (gate) | 22.0 µs | 16.2 µs | +| 128 | ms (gate) | 8.4 µs | 34.2 µs | 64.6 µs | ms (gate) | 22.0 µs | 44.7 µs | +| 254 | ms (gate) | 16.6 µs | 33.2 µs | 155.0 µs | ms (gate) | 22.1 µs | 124.6 µs | + +Runs cold (pointer tables built inside the call): 19–32 µs from k = 1 to 32 for the role. The interlaced row mask gives the same costs within noise: row activation is an AND at the terminal and costs nothing extra. At 8,000 users the masked route is 3.1 µs (role) and 2.1 µs (group) for every k. + +### Reading + +- **Amortization across tiles:** the masked route's cost is fixed by the memberships of the question's slots (≈ 1.5 ns per membership), not by k. It beats resolving pages one by one from k = 2–4 for a role and from k = 1 for a group, and reaches 100–700× at all tiles. +- **Selective tile masking:** the per-slot tile pointers make the cost proportional to k (≈ 0.3–0.6 µs per tile at these slot counts). They win while few tiles are selected and lose to the masked scan beyond about a quarter of the tiles (k ≈ 64 of 254): per tile they hop across all of the question's member lists, where the masked route reads each list once in order. +- **Warm pages answer role questions cheapest** (their roles are precomputed). Group questions have no precomputed column, so a warm page still pays a row × tile product per user and is slower than either masked route at every k. +- **The branch matters:** with a branch on the tile bit, the masked route ran 19–20 µs at small k and 64–73 µs at k = 128 (member order is random with respect to tiles, so the branch mispredicts at half selection). Branch-free it is a flat ≈ 33 µs. The sparse case is covered by the pointer tables instead. +- **So the choice is by selected fraction:** pointer runs below about a quarter of the tiles, the masked scan above, warm pages for role questions on already-resolved pages. That is a measured policy, not admitted; the crossover depends on slots per question and members per tile. + +Disable runs (all red): tile mask ignored (masked; runs), scratch left dirty, empty run per tile, role transposition off by one bit, `slots_reaching` taken upward instead of downward. + +## Chained population masking: license groups over all memberships (revision 4) + +Operator direction (2026-10-21): filter license groups over all user memberships, as an example of chained population masking by positional filtering. + +A **population** (`qr::Pop`) is one bit per user, at tile `u / 256`, row `u % 256`, for every tile. `Qr::population(b, slots)` builds one from a slot set by reading each slot's direct-member list once and scattering by position: `slots_reaching(g)` gives the users effectively in group `g` (nesting included), and `role_slots[r]` gives the holders of role `r`. Populations from one register line up by position, so filters chain as word-wise AND / AND-NOT (`Pop::and`, `Pop::and_not`). The answer is a count (`Pop::count`, `Pop::count_and`, which counts an intersection without building it). `Qr::everyone` is the full population. + +Question (`examples/qr_license.rs`): per license group (16), count the users who hold the license, hold role r, and are NOT in location L. Chain: `role AND-NOT location`, then `license AND chain` counted per license. `tests/qr_license.rs` checks all 16 counts, and "no license at all" (`everyone AND-NOT any license`), against dir-sim's own walk per user. Anti-vacuity: the location removes some role holders, the counts differ across licenses, and some users but not all are unlicensed. + +| | 8,000 users | 65,000 users | +|---|---:|---:| +| build role population | 2.3 µs | 24.0 µs | +| build location population | 1.3 µs | 9.3 µs | +| build 16 license populations | 11.6 µs | 66.2 µs | +| chain role AND-NOT location | 0.1 µs | 0.3 µs | +| 16 × (license AND chain), counted | 0.3 µs | 2.8 µs | +| whole question, end to end | 15.1 µs | 123.7 µs | +| per-user route over resolved pages (reference) | 363.0 µs | FAILED GATE (ms) | + +At 65,000 users the populations read 21,615 + 8,349 + 58,521 memberships; one population is 8,128 bytes (254 tiles × 32 B). + +### Reading + +- **Building a population costs about 1.1 ns per membership read.** It is the whole cost: chaining and counting are word operations over 254 tiles (0.3 µs per chain step, about 0.18 µs per license count). +- **A population is reusable across questions.** Once built, the 16 license populations answer any other chain (another role, another location, unlicensed users) for the cost of the AND and the count alone. +- **The per-user route** (one effective row per user, then bit tests) pays the 256-slot row × tile product per user, so it is 24× slower at 8,000 users and fails the gate at 65,000. +- Populations cover every tile. Combining them with the tile mask and row masks of revision 3 (selected OUs or teams only) is not measured yet. + +Disable runs (all red): AND-NOT acting as AND, `count_and` using OR, `everyone` dropping the last user of a tile, a population leaking slot 0. + +## Decision: proxy chains are physical, relationships are semantic (operator, 2026-10-21) + +Recorded as given: + +> Extension proxy chains stay inside one Dn128 OU prefix. Logical memberOf and nested-group relationships may cross OU boundaries. + +- An extension proxy continues a physical team/group chunk within its own OU locality. +- A memberOf or nested-group edge is a logical relationship and may target another OU. +- Cross-OU traversal uses version-attested global-to-local bindings. +- All 256 member slots remain usable. The proxy is separate routing metadata, not a stolen member bit. +- The chunk key must retain stable team/group identity. OU prefix alone is not sufficient. +- A move between OUs invalidates locality mappings, not the group's identity. +- Proxy chains do not replace transitive closure across group chunks. + +Not built yet: the proxy chain itself. The probe below binds a team focus by group identity (`TeamFocus::group`, a group ordinal) with its tile mask as locality. + +## Team focus, versioned populations, method folds, repair (revision 5) + +Operator next probe: bind an OU/team focus without materializing every user; build only the relevant populations; reuse them across several relational method folds; repair them after edits; compare cold and warm with B+. Reuse mask-risc/relalg, no new evaluator. + +Built in `crates/dir-rmap-probe/src/pop.rs`: + +- **`Pop` carries the witness of its version.** `and`, `and_not`, `or`, `count_and` refuse populations of another version (`Refusal::Stale`). A population is a cache of derived state, rebuilt or repaired from the source rows of the version it claims. +- **`TeamFocus`** binds a group's identity (its ordinal) to the tiles and rows of its effective members. It reads only the member lists of the group and the groups nested in it. +- **`Qr::population_in`** fills a population only in the focus's tiles, through the per-slot pointer tables (`Runs`). The tables carry their version too. +- **Methods:** a `TeamAudit` class declares three methods in `compute_dag` (per-license holders of role r outside location L; unlicensed members; role holders). The DR table binds each to a `lance-graph-relalg` query over the populations lent as `Rel::Plane` (`Pop::plane`, a borrow, no copy), lowered by relalg's existing mask-risc path. All three reuse the same bound populations. +- **`Pop::repair`** brings a population to a new version. It diffs the old and new slot sets (the register is rebound). Members of slots that gained the property are scattered in. Members of slots that lost it, and every user whose direct memberships changed, are re-derived from their direct groups. A user or group with a second supporting path keeps the bit. The repair refuses if the slot binding itself changed. `Edit::RemoveNested` was added to the probe's edit set. + +### Correctness (`tests/qr_focus.rs`, 5 tests) + +- The three methods equal dir-sim's walk over the team's members (3,000 users, 16 licenses). +- In a narrow focus, populations are empty outside its tiles and equal to the full population inside. +- Over 8 random edit batches (direct add/remove, nesting add/remove among security groups), every repaired population (all roles, the licenses, the location, the team; full and focus-restricted) equals a fresh build. +- **Multi-support falsifier:** a group under two role-carrying parents keeps the role when one nesting edge is removed, and the population is unchanged. When the second edge goes, the group loses the role, but a member with its own direct path through a third group keeps it. Every user is checked against dir-sim. +- Mixing versions is refused: populations, pointer tables, focus, and repair. + +Disable runs (all red): +- clear members of a removed slot instead of re-deriving (fails the multi-support test and the edit batches); +- skip touched users; +- skip scattering added slots; +- re-derive from only the last direct group; +- `count_and` ignoring the version; +- `population_in` ignoring the pointer tables' version; +- `population_in` ignoring the focus tiles; +- repair ignoring the population's version. + +### Measured (`examples/qr_focus.rs`, release, every route checked equal before timing) + +Question: the three methods for the security team with the most direct members, 16 licenses. + +| | 65,000 users | 8,000 users | +|---|---:|---:| +| team members / focus tiles / all tiles | 77 / 69 / 254 | 244 / 32 / 32 | +| prepare (arm B CSR, shared with B+) | FAILED GATE (ms) | FAILED GATE (ms) | +| bind register | 52.9 µs | 53.6 µs | +| bind team focus | 0.8 µs | 0.8 µs | +| pointer tables + 18 populations in focus | 143.7 µs | 26.4 µs | +| of which populations with tables warm | 45.2 µs | 14.3 µs | +| 3 methods, relalg (18 queries) | 75.0 µs | 54.8 µs | +| **Q cold**, all but prepare | 326.3 µs | 151.4 µs | +| **Q warm**, version checks + 3 methods, relalg | 75.1 µs | 54.9 µs | +| **Q warm**, version checks + 3 methods, native `Pop` folds | 6.6 µs | 1.2 µs | +| after an edit: rebind register | 51.7 µs | 50.5 µs | +| after an edit: repair 18 populations + focus | 5.7 µs | 5.3 µs | +| after an edit: rebuild focus + tables + populations | 152.1 µs | 27.8 µs | +| B+ prepare + compile | FAILED GATE (ms) | FAILED GATE (ms) | +| B+ per focus user, `groups_of` + `roles_of` | 12.3 µs | 37.2 µs | +| B+ `members_of` populations (licenses + location + team) | 72.3 µs | 6.6 µs | + +### Reading + +- **Construction vs reuse holds.** Once the populations are bound, the three methods fold natively in 6.6 µs at 65,000 users. That is below B+'s per-user route at 77 members (12.3 µs) and 31× below it at 244 members. +- **relalg has a fixed cost of about 3 µs per query.** It is the same at 8,000 users (125-word planes) and at 65,000. With 18 queries per call it dominates the warm path (75 µs), so through relalg Q warm is slower than B+ per user for a small team. The algebra is reused as asked. The per-query program build is the cost to remove: one program with several counts, or a prepared program per method, kept across calls. Not built. +- **Repair is 27× cheaper than a rebuild** (5.7 vs 152 µs at 65,000 users) and is correct under multi-support. After an edit, rebinding the register (≈ 52 µs) now dominates; an incremental register update is open. +- **The pointer tables are two thirds of the cold population cost.** Each table reads the slot's whole member list, independent of the focus. +- **Tile locality: the focus saves little while user positions are GUID-ordered.** A 77-member team occupies 69 of 254 tiles; at 8,000 users a 244-member team occupies all 32. The tile restriction only pays when users are positioned by OU/team (Dn128 order), which matches the decision above that chunk locality is physical. The fixture has no such ordering. +- **Prepare** (dir-sim → CSR) is still a millisecond phase on both sides. Reading source lanes directly is not admitted (ADMISSION B), so every cold figure above excludes it. +- **Not done:** ACT-R and Thought-Git consuming the same focus and populations. This revision shows only the IAM side of that claim. + +## Open + +- **Pages by OU/team, not by ordinal:** pages are cut at user ordinal / 256. Cutting them by `Dn128` prefix (OU) or team needs a page key and a pointer table of non-empty tiles (the proxy-chained team design, not built). +- **Chains over 256 security groups** (`Spec::iam(8000)` has 1,439 ≈ 6 pages): a blocked Warshall over register pages. This is the D-RU-6 tile-recursion territory, where the measured win is narrow. +- **Mail and All chains:** only the security chain is bound. +- **The contract-level frozen-codebook witness** (ADMISSION B), shared with D-CMR-5 and D-RU-6. + +## Closeout + +`STATUS: measured | OUTCOME: DR256 register reproduces IAM nested-group roles exactly; versioned populations bound to a team focus answer three ClassView methods in 6.6 µs warm (native folds) or 75 µs through relalg (≈ 3 µs per query); repair after edits 5.7 µs vs 152 µs rebuild, correct under multi-support | OPEN: relalg per-query program cost, incremental register, Dn128 position order for tile locality, proxy chains, >256-group chains, CSR-free reads, codebook witness, ACT-R/Thought-Git consumers` diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 53acbe1a2..b36214e04 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,7 +25,7 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -413 entries, 2026-08-06 .. 2026-10-21. +414 entries, 2026-08-06 .. 2026-10-21. | date | entry id | finding | file | |---|---|---|---| @@ -34,6 +34,7 @@ exactly one of them, never both. | 2026-10-21 | `D-RU-3` | | [2026-10-21-u3-cross-rule-cse.md](2026-10-21-u3-cross-rule-cse.md) | | 2026-10-21 | `D-RU-2` | | [2026-10-21-u2-let-planes.md](2026-10-21-u2-let-planes.md) | | 2026-10-21 | `tuple-zero-copy-council` | | [2026-10-21-tuple-zero-copy-council.md](2026-10-21-tuple-zero-copy-council.md) | +| 2026-10-21 | `dr256-iam-nested-groups` | | [2026-10-21-dr256-iam-nested-groups.md](2026-10-21-dr256-iam-nested-groups.md) | | 2026-10-21 | `D-RU-6-LAB-1` | | [2026-10-21-d-ru-6-tile-walk-probe.md](2026-10-21-d-ru-6-tile-walk-probe.md) | | 2026-10-21 | `D-RU-6-LAB-2` | | [2026-10-21-d-ru-6-sparse-compute.md](2026-10-21-d-ru-6-sparse-compute.md) | | 2026-10-21 | `D-RU-6-LAB-3` | | [2026-10-21-d-ru-6-dirty-frontier.md](2026-10-21-d-ru-6-dirty-frontier.md) | diff --git a/crates/dir-rmap-probe/Cargo.toml b/crates/dir-rmap-probe/Cargo.toml index 02cd55285..268a5910b 100644 --- a/crates/dir-rmap-probe/Cargo.toml +++ b/crates/dir-rmap-probe/Cargo.toml @@ -39,6 +39,11 @@ crepe = { git = "https://github.com/AdaWorldAPI/crepe", branch = "ccr-d3ce1263-i # the contract is used. [dev-dependencies] lance-graph-contract = { path = "../lance-graph-contract" } +# Method folds over bound populations (tests/qr_focus.rs, examples/qr_focus.rs): +# a population is lent as a `Rel::Plane` and folded by relalg's existing +# mask-risc lowering. No evaluator of the probe's own. +lance-graph-relalg = { path = "../lance-graph-relalg" } +lance-graph-mask-risc = { path = "../lance-graph-mask-risc" } [patch."https://github.com/AdaWorldAPI/ndarray"] ndarray = { path = "../../../ndarray" } diff --git a/crates/dir-rmap-probe/examples/qr_amortize.rs b/crates/dir-rmap-probe/examples/qr_amortize.rs new file mode 100644 index 000000000..2274d03d7 --- /dev/null +++ b/crates/dir-rmap-probe/examples/qr_amortize.rs @@ -0,0 +1,208 @@ +//! Amortization of a masked count over several QR tiles (arm Q). +//! +//! ```text +//! cargo run --release --example qr_amortize # 65,000 users +//! cargo run --release --example qr_amortize -- 8000 +//! ``` +//! +//! One question ("how many users hold role r", or "are effectively in group +//! g") over `k` tiles of 256 users, selected by a tile mask spread across +//! the population, with rows inside each tile selected by a row mask (all +//! rows, or interlaced: even rows in even tiles, odd rows in odd tiles). +//! Routes, all checked equal before timing: +//! +//! * `page`: resolve each selected page on first touch, then fold its +//! resolved roles over the active rows; +//! * `warm`: the same fold over pages already resolved; +//! * `masked`: [`Qr::count_masked`]: each needed group's direct members are +//! read once and set bits only in selected tiles; popcount of tile AND row +//! mask. No page is resolved. +//! * `runs (cold)`: [`Runs::count`] with the per-slot tile pointer tables +//! built inside the timed call; `runs (warm)`: tables already built. Only +//! the selected tiles' runs are visited. + +use dir_rmap_probe::World; +use dir_rmap_probe::fixture::{Fixture, Spec}; +use dir_rmap_probe::focus::Row; +use dir_rmap_probe::prep::Prepared; +use dir_rmap_probe::qr::{Page, Qr, Runs}; +use std::hint::black_box; +use std::time::{Duration, Instant}; + +const ALL: Row = [!0; 4]; +const EVEN: Row = [0x5555_5555_5555_5555; 4]; +const ODD: Row = [0xAAAA_AAAA_AAAA_AAAA; 4]; + +fn bit(r: &Row, i: usize) -> bool { + r[i >> 6] >> (i & 63) & 1 == 1 +} + +fn median(reps: usize, mut f: impl FnMut()) -> Duration { + let mut t: Vec = (0..reps) + .map(|_| { + let s = Instant::now(); + f(); + s.elapsed() + }) + .collect(); + t.sort(); + t[reps / 2] +} + +fn fmt(d: Duration) -> String { + if d >= Duration::from_millis(1) { + "FAILED GATE (ms)".into() + } else { + format!("{:.1} us", d.as_secs_f64() * 1e6) + } +} + +/// `k` tiles spread evenly over `pages`. +fn tiles(pages: usize, k: usize) -> Vec { + let mut m = vec![0u64; pages.div_ceil(64)]; + for j in 0..k { + let p = j * pages / k; + m[p >> 6] |= 1 << (p & 63); + } + m +} + +fn selected(t: &[u64], pages: usize) -> Vec { + (0..pages) + .filter(|&p| t[p >> 6] >> (p & 63) & 1 == 1) + .collect() +} + +/// Count via resolved pages: `role` = Some(r) for a role question, else a +/// group slot `t` tested on the effective row. +fn fold(qr: &Qr, pages: &[(usize, &Page)], rows: &[Row], q: Q) -> u64 { + let mut n = 0; + for &(p, page) in pages { + for i in 0..256 { + if bit(&rows[p], i) { + n += u64::from(match q { + Q::Role(r) => page.roles[i] >> r & 1 == 1, + Q::Group(t) => bit(&qr.effective(&page.member[i]), t), + }); + } + } + } + n +} + +#[derive(Clone, Copy)] +enum Q { + Role(u32), + Group(usize), +} + +fn main() { + let users: usize = std::env::args() + .nth(1) + .map_or(65_000, |s| s.parse().expect("user count")); + let mut spec = Spec::small(); + spec.users = users; + let w = World::new(Fixture::generate(spec)); + let b: Prepared = w.prepare(); + let qr = Qr::bind(&b, &w.witness()).unwrap(); + let np = qr.pages(); + let all: Vec = (0..np).map(|p| qr.page(&b, p).unwrap()).collect(); + + // The role carried by the most slots, and the group reached from the + // most slots (the widest questions). + let r = (0..64) + .max_by_key(|&r| qr.role_slots[r].iter().map(|x| x.count_ones()).sum::()) + .unwrap(); + let g = *qr + .ids + .iter() + .max_by_key(|&&g| { + qr.slots_reaching(g) + .unwrap() + .iter() + .map(|x| x.count_ones()) + .sum::() + }) + .unwrap(); + let pop = |row: &Row| row.iter().map(|x| x.count_ones()).sum::(); + let questions = [ + ("role", qr.role_slots[r], Q::Role(r as u32)), + ( + "group", + qr.slots_reaching(g).unwrap(), + Q::Group(qr.slot(g).unwrap()), + ), + ]; + let row_sets: [(&str, Vec); 2] = [ + ("all rows", vec![ALL; np]), + ( + "interlaced", + (0..np) + .map(|p| if p % 2 == 0 { EVEN } else { ODD }) + .collect(), + ), + ]; + println!( + "users {}, tiles {np}, security slots {}", + b.users, + qr.ids.len() + ); + let mut scratch = vec![[0u64; 4]; np]; + for (qname, slots, q) in questions { + let members: usize = qr + .ids + .iter() + .enumerate() + .filter(|&(s, _)| bit(&slots, s)) + .map(|(_, &g)| b.group_users.row(g as usize).len()) + .sum(); + println!( + "\n{qname}: {} slots, {members} direct memberships read by the masked route", + pop(&slots) + ); + for (rname, rows) in &row_sets { + println!(" {rname}:"); + println!( + " {:>4} {:>18} {:>12} {:>12} {:>12} {:>12}", + "k", "page (first touch)", "warm page", "masked", "runs cold", "runs warm" + ); + for k in [1, 2, 4, 8, 16, 32, 64, 128, np] { + let k = k.min(np); + let t = tiles(np, k); + let sel = selected(&t, np); + let warm: Vec<(usize, &Page)> = sel.iter().map(|&p| (p, &all[p])).collect(); + let want = fold(&qr, &warm, rows, q); + let got = qr.count_masked(&b, &slots, &t, rows, &mut scratch).unwrap(); + assert_eq!(got, want, "{qname} {rname} k={k}"); + let t_page = median(9, || { + let fresh: Vec = sel.iter().map(|&p| qr.page(&b, p).unwrap()).collect(); + let v: Vec<(usize, &Page)> = sel.iter().copied().zip(fresh.iter()).collect(); + black_box(fold(&qr, &v, rows, q)); + }); + let t_warm = median(9, || { + black_box(fold(&qr, &warm, rows, q)); + }); + let t_mask = median(9, || { + black_box(qr.count_masked(&b, &slots, &t, rows, &mut scratch).unwrap()); + }); + let mut runs = Runs::new(&qr); + assert_eq!(runs.count(&qr, &b, &slots, &t, rows).unwrap(), want); + let t_cold = median(9, || { + let mut runs = Runs::new(&qr); + black_box(runs.count(&qr, &b, &slots, &t, rows).unwrap()); + }); + let t_runs = median(9, || { + black_box(runs.count(&qr, &b, &slots, &t, rows).unwrap()); + }); + println!( + " {k:>4} {:>18} {:>12} {:>12} {:>12} {:>12}", + fmt(t_page), + fmt(t_warm), + fmt(t_mask), + fmt(t_cold), + fmt(t_runs), + ); + } + } + } +} diff --git a/crates/dir-rmap-probe/examples/qr_cost.rs b/crates/dir-rmap-probe/examples/qr_cost.rs new file mode 100644 index 000000000..b93ca8c64 --- /dev/null +++ b/crates/dir-rmap-probe/examples/qr_cost.rs @@ -0,0 +1,154 @@ +//! Cost of arm Q (the DR256 register) against the compiled closure (B+). +//! +//! ```text +//! cargo run --release --example qr_cost # 8,000 users +//! cargo run --release --example qr_cost -- 65000 +//! ``` +//! +//! The group side is `Spec::small`'s (about 160 security groups, so the whole +//! chain fits one register); only the user count grows. Every route is +//! checked equal to dir-sim's own walk before it is timed. Preparation +//! (prepare, compile, bind) is reported per phase, never folded into a +//! per-user figure. +//! +//! The register resolves no user at bind. A question resolves the user's page +//! of 256 users; "one page" is that first-touch cost. "every page" is shown +//! only to say what an eager resolve of the whole population would cost — +//! nobody asks for that. + +use dir_rmap_probe::fixture::{Fixture, Spec}; +use dir_rmap_probe::prep::{Chain, Compiled, Ordinals}; +use dir_rmap_probe::qr::{Page, Qr}; +use dir_rmap_probe::{World, a_groups}; +use ogar_dir_core::Guid128; +use std::collections::HashMap; +use std::hint::black_box; +use std::time::{Duration, Instant}; + +fn median(mut v: Vec) -> Duration { + v.sort(); + v[v.len() / 2] +} + +fn per_user(n: usize, reps: usize, mut f: impl FnMut()) -> f64 { + let mut t = Vec::with_capacity(reps); + for _ in 0..reps { + let s = Instant::now(); + f(); + t.push(s.elapsed()); + } + median(t).as_nanos() as f64 / n as f64 +} + +fn once(reps: usize, mut f: impl FnMut()) -> Duration { + let mut t = Vec::with_capacity(reps); + for _ in 0..reps { + let s = Instant::now(); + f(); + t.push(s.elapsed()); + } + median(t) +} + +/// P0: a millisecond result is a failed gate, never a measurement. +fn show(name: &str, d: Duration) { + if d >= Duration::from_millis(1) { + println!(" {name:<22} FAILED GATE (millisecond scale)"); + } else { + println!(" {name:<22} {:>8.1} us", d.as_secs_f64() * 1e6); + } +} + +fn main() { + let users: usize = std::env::args() + .nth(1) + .map_or(8_000, |s| s.parse().expect("user count")); + let mut spec = Spec::small(); + spec.users = users; + let w = World::new(Fixture::generate(spec)); + let v = w.view(); + let ord = Ordinals::of(&w.f, &v); + let by_guid: HashMap = + w.f.grants.iter().fold(HashMap::new(), |mut m, &(g, r)| { + *m.entry(w.f.groups[g as usize]).or_default() |= 1 << r; + m + }); + + let witness = w.witness(); + let b = w.prepare(); + let c = Compiled::compile(w.prepare()); + let q = Qr::bind(&b, &witness).expect("security chain fits 256 slots"); + let pages: Vec = (0..q.pages()).map(|p| q.page(&b, p).unwrap()).collect(); + for (i, ug) in w.f.users.iter().enumerate() { + let want = a_groups(&v, ug, Chain::Security) + .iter() + .fold(0u64, |a, g| a | by_guid.get(g).copied().unwrap_or(0)); + let uo = ord.user[i]; + assert_eq!(c.roles_of(uo), want, "B+ {i}"); + let (p, j) = (uo as usize / 256, uo as usize % 256); + assert_eq!(pages[p].roles_of(j), want, "Q page {i}"); + assert_eq!(q.roles_live(&pages[p].member[j]), want, "Q live {i}"); + } + + let t_prepare = once(5, || { + black_box(w.prepare()); + }); + let t_compile = once(5, || { + black_box(Compiled::compile(w.prepare())); + }); + let t_bind = once(15, || { + black_box(Qr::bind(&b, &witness).unwrap()); + }); + let t_page = once(51, || { + black_box(q.page(&b, black_box(0)).unwrap()); + }); + let t_all = once(5, || { + for p in 0..q.pages() { + black_box(q.page(&b, p).unwrap()); + } + }); + + let n = w.f.users.len(); + let reps = 15; + let t_fold = per_user(n, reps, || { + for &uo in &ord.user { + black_box(c.roles_of(black_box(uo))); + } + }); + let at = |uo: u16| &pages[uo as usize / 256]; + let t_q = per_user(n, reps, || { + for &uo in &ord.user { + black_box(at(uo).roles_of(black_box(uo) as usize % 256)); + } + }); + let t_live = per_user(n, reps, || { + for &uo in &ord.user { + black_box(q.roles_live(&at(uo).member[black_box(uo) as usize % 256])); + } + }); + let t_eff = per_user(n, reps, || { + for &uo in &ord.user { + black_box(q.effective(&at(uo).member[black_box(uo) as usize % 256])); + } + }); + + println!( + "users {n}, groups {}, security slots {}, translations {}, all answers checked", + v.groups_len(), + q.ids.len(), + q.translations + ); + println!("preparation, once per version:"); + show("prepare", t_prepare); + show("prepare + compile (B+)", t_compile); + show("bind register (Q)", t_bind); + println!("on demand:"); + show("one page (256 users)", t_page); + show(&format!("every page ({})", q.pages()), t_all); + println!("per user (median of runs):"); + println!(" B+ fold {t_fold:>7.1} ns"); + println!(" Q roles, resolved page {t_q:>7.1} ns"); + println!(" Q roles, row x roles {t_live:>7.1} ns"); + println!(" Q effective row (M x R) {t_eff:>7.1} ns"); + println!("register bytes {}, page bytes {}", q.bytes(), Page::BYTES); +} diff --git a/crates/dir-rmap-probe/examples/qr_focus.rs b/crates/dir-rmap-probe/examples/qr_focus.rs new file mode 100644 index 000000000..5fd146ff6 --- /dev/null +++ b/crates/dir-rmap-probe/examples/qr_focus.rs @@ -0,0 +1,423 @@ +//! Cold and warm cost of a team focus with bound populations (arm Q), +//! against B+. +//! +//! ```text +//! cargo run --release --example qr_focus # 65,000 users +//! cargo run --release --example qr_focus -- 8000 +//! ``` +//! +//! Question, three methods over one focus (the security team with the most +//! members): per license (16), members holding the license and role r, +//! outside location L; members holding no license; members holding r. +//! +//! * Q cold: bind the register, bind the focus, build the pointer tables and +//! the 18 populations in the focus's tiles, check versions, fold the three +//! methods through lance-graph-relalg (populations lent as `Rel::Plane`). +//! * Q warm: version check + the three folds over the bound populations. +//! * Q repair: after one membership removal and one nesting removal, rebind +//! the register and repair the 18 populations and the focus, against +//! rebuilding them. +//! * B+: compile; then per focus user `groups_of` + `roles_of` (its query +//! API), and the populations B+ can build (`members_of`). +//! +//! Every route is checked equal before timing. Phases are reported apart; a +//! millisecond phase is a failed gate, never a number. + +use dir_rmap_probe::fixture::{Fixture, Kind, Spec}; +use dir_rmap_probe::focus::Row; +use dir_rmap_probe::pop::{Pop, TeamFocus}; +use dir_rmap_probe::prep::{Chain, Compiled, Ordinals, Prepared}; +use dir_rmap_probe::qr::{Qr, Runs}; +use dir_rmap_probe::{Edit, World}; +use lance_graph_mask_risc::Planes; +use lance_graph_relalg::{Answer, Measure, Query, Rel, run}; +use std::hint::black_box; +use std::time::{Duration, Instant}; + +fn median(reps: usize, mut f: impl FnMut()) -> Duration { + let mut t: Vec = (0..reps) + .map(|_| { + let s = Instant::now(); + f(); + s.elapsed() + }) + .collect(); + t.sort(); + t[reps / 2] +} + +fn show(name: &str, d: Duration) { + if d >= Duration::from_millis(1) { + println!(" {name:<44} FAILED GATE (millisecond scale)"); + } else { + println!(" {name:<44} {:>8.1} us", d.as_secs_f64() * 1e6); + } +} + +fn bit(r: &[u64], i: usize) -> bool { + r[i >> 6] >> (i & 63) & 1 == 1 +} + +/// Planes: 0 focus, 1 role, 2 location, 3.. licenses. +fn methods(n_rows: usize, focus: &TeamFocus, pops: &[Pop]) -> Vec { + let mut masks: Vec<&[u64]> = vec![focus.rows.plane(n_rows)]; + masks.extend(pops.iter().map(|p| p.plane(n_rows))); + let planes = Planes { + n_rows, + masks: &masks, + lanes: &[], + }; + let count = |rel: Rel| match run( + &Query { + rel, + measure: Measure::Count, + }, + &planes, + None, + ) + .unwrap() + .answer + { + Answer::Count(n) => n, + a => panic!("{a:?}"), + }; + let lic = (pops.len() - 2) as u16; + let mut out: Vec = (0..lic) + .map(|i| { + count(Rel::And(vec![ + Rel::Plane(0), + Rel::Plane(3 + i), + Rel::Plane(1), + Rel::Not(Box::new(Rel::Plane(2))), + ])) + }) + .collect(); + let any = Rel::Or((0..lic).map(|i| Rel::Plane(3 + i)).collect()); + out.push(count(Rel::And(vec![ + Rel::Plane(0), + Rel::Not(Box::new(any)), + ]))); + out.push(count(Rel::And(vec![Rel::Plane(0), Rel::Plane(1)]))); + out +} + +/// The same three methods as [`methods`], folded directly over the bound +/// populations (no relalg program): the reference for relalg's per-query +/// cost. +fn methods_native(focus: &TeamFocus, pops: &[Pop]) -> Vec { + let chain = focus + .rows + .clone() + .and(&pops[0]) + .unwrap() + .and_not(&pops[1]) + .unwrap(); + let mut out: Vec = pops[2..] + .iter() + .map(|l| l.count_and(&chain).unwrap()) + .collect(); + let mut any = pops[2].clone(); + for l in &pops[3..] { + any = any.or(l).unwrap(); + } + out.push(focus.rows.clone().and_not(&any).unwrap().count()); + out.push(focus.rows.count_and(&pops[0]).unwrap()); + out +} + +struct Q { + role: usize, + team: u16, + loc: u16, + lic: Vec, +} + +impl Q { + fn slots(&self, qr: &Qr) -> Vec { + let mut v = vec![ + qr.role_slots[self.role], + qr.slots_reaching(self.loc).unwrap(), + ]; + v.extend(self.lic.iter().map(|&g| qr.slots_reaching(g).unwrap())); + v + } +} + +fn cold(q: &Q, b: &Prepared, w: &dir_rmap_probe::prep::Witness) -> Vec { + let qr = Qr::bind(b, w).unwrap(); + let focus = qr.team_focus(b, q.team).unwrap(); + let mut runs = Runs::new(&qr); + let pops: Vec = q + .slots(&qr) + .iter() + .map(|s| qr.population_in(b, s, &focus, &mut runs).unwrap()) + .collect(); + for p in &pops { + p.check(w).unwrap(); + } + methods(b.users, &focus, &pops) +} + +/// B+ per focus user, through its own query API. +fn bplus(q: &Q, c: &Compiled, focus_users: &[u16], scratch: &mut [u64]) -> Vec { + let n = q.lic.len(); + let mut out = vec![0u64; n + 2]; + for &u in focus_users { + c.groups_of(u, Chain::Security, scratch); + let role = c.roles_of(u) >> q.role & 1 == 1; + let in_loc = bit(scratch, q.loc as usize); + let mut any = false; + for (i, &l) in q.lic.iter().enumerate() { + let has = bit(scratch, l as usize); + any |= has; + out[i] += u64::from(has && role && !in_loc); + } + out[n] += u64::from(!any); + out[n + 1] += u64::from(role); + } + out +} + +fn main() { + let users: usize = std::env::args() + .nth(1) + .map_or(65_000, |s| s.parse().expect("user count")); + let mut spec = Spec::small(); + spec.users = users; + spec.licenses = 16; + let mut w = World::new(Fixture::generate(spec)); + let ord = { + let v = w.view(); + Ordinals::of(&w.f, &v) + }; + let kinds = |w: &World, k: Kind| -> Vec { + (0..w.f.groups.len()) + .filter(|&j| w.f.kind[j] == k) + .collect() + }; + let direct_n = |w: &World, j: usize| w.direct.iter().filter(|&&(_, g)| g as usize == j).count(); + let team_f = *kinds(&w, Kind::Team) + .iter() + .filter(|&&j| w.f.security[j] == Some(true)) + .max_by_key(|&&j| direct_n(&w, j)) + .unwrap(); + let loc_f = *kinds(&w, Kind::Location) + .iter() + .max_by_key(|&&j| direct_n(&w, j)) + .unwrap(); + let wit = w.witness(); + let b = w.prepare(); + let qr = Qr::bind(&b, &wit).unwrap(); + let role = (0..64) + .max_by_key(|&r| qr.population(&b, &qr.role_slots[r]).unwrap().count()) + .unwrap(); + let q = Q { + role, + team: ord.group[team_f], + loc: ord.group[loc_f], + lic: kinds(&w, Kind::License) + .iter() + .map(|&j| ord.group[j]) + .collect(), + }; + + // Q, built once. + let focus = qr.team_focus(&b, q.team).unwrap(); + let mut runs = Runs::new(&qr); + let pops: Vec = q + .slots(&qr) + .iter() + .map(|s| qr.population_in(&b, s, &focus, &mut runs).unwrap()) + .collect(); + let want = methods(b.users, &focus, &pops); + assert_eq!(methods_native(&focus, &pops), want); + assert_eq!(cold(&q, &b, &wit), want); + + // B+. + let c = Compiled::compile(w.prepare()); + let mut focus_users = Vec::new(); + for p in 0..qr.pages() { + if focus.tiles[p >> 6] >> (p & 63) & 1 == 1 { + for i in 0..256 { + if bit(&focus.rows.tiles[p], i) { + focus_users.push((p * 256 + i) as u16); + } + } + } + } + let mut scratch = vec![0u64; b.groups.div_ceil(64)]; + assert_eq!( + bplus(&q, &c, &focus_users, &mut scratch), + want, + "B+ per user" + ); + let mut ubits = vec![0u64; b.users.div_ceil(64)]; + + // Timings. + let t_prepare = median(5, || { + black_box(w.prepare()); + }); + let t_register = median(15, || { + black_box(Qr::bind(&b, &wit).unwrap()); + }); + let t_focus = median(15, || { + black_box(qr.team_focus(&b, q.team).unwrap()); + }); + let slots = q.slots(&qr); + let t_pops = median(15, || { + let mut runs = Runs::new(&qr); + for s in &slots { + black_box(qr.population_in(&b, s, &focus, &mut runs).unwrap()); + } + }); + let t_pops_warm = median(15, || { + for s in &slots { + black_box(qr.population_in(&b, s, &focus, &mut runs).unwrap()); + } + }); + let t_native = median(51, || { + black_box(methods_native(&focus, &pops)); + }); + let t_check = median(51, || { + for p in &pops { + black_box(p.check(&wit)).unwrap(); + } + }); + let t_methods = median(15, || { + black_box(methods(b.users, &focus, &pops)); + }); + let t_cold = median(9, || { + black_box(cold(&q, &b, &wit)); + }); + let t_compile = median(5, || { + black_box(Compiled::compile(w.prepare())); + }); + let t_bplus = median(15, || { + black_box(bplus(&q, &c, &focus_users, &mut scratch)); + }); + let t_bplus_pop = median(15, || { + for &g in q.lic.iter().chain([&q.loc, &q.team]) { + c.members_of(g, Chain::Security, &mut ubits); + black_box(&ubits); + } + }); + + // Repair: one membership removal (a focus member) and one nesting + // removal, both in the security chain. + let sec = |w: &World, g: u32| w.f.security[g as usize] == Some(true); + let (ru, rg) = *w + .direct + .iter() + .find(|&&(u, g)| sec(&w, g) && focus_users.contains(&ord.user[u as usize])) + .unwrap(); + let (nc, np) = *w + .nested + .iter() + .find(|&&(c, p)| sec(&w, c) && sec(&w, p)) + .unwrap(); + w.apply(&[Edit::RemoveDirect(ru, rg), Edit::RemoveNested(nc, np)]); + let wit1 = w.witness(); + let b1 = w.prepare(); + let touched = [ord.user[ru as usize]]; + let qr1 = Qr::bind(&b1, &wit1).unwrap(); + let slots1 = q.slots(&qr1); + let focus_slots = ( + qr.slots_reaching(q.team).unwrap(), + qr1.slots_reaching(q.team).unwrap(), + ); + let repair_all = |pops: &mut Vec, f: &mut Pop| { + for (p, (s0, s1)) in pops.iter_mut().zip(slots.iter().zip(&slots1)) { + p.repair(&qr, &qr1, &b1, (s0, s1), &touched, Some(&focus.tiles)) + .unwrap(); + } + f.repair( + &qr, + &qr1, + &b1, + (&focus_slots.0, &focus_slots.1), + &touched, + None, + ) + .unwrap(); + }; + let mut rp = pops.clone(); + let mut rf = focus.rows.clone(); + repair_all(&mut rp, &mut rf); + let focus1 = qr1.team_focus(&b1, q.team).unwrap(); + assert_eq!(rf, focus1.rows, "repaired focus"); + for (p, s) in rp.iter().zip(&slots1) { + // A fresh population of the new version, kept to the focus's tiles. + let mut fresh = qr1.population(&b1, s).unwrap(); + for (i, t) in fresh.tiles.iter_mut().enumerate() { + if focus.tiles[i >> 6] >> (i & 63) & 1 == 0 { + *t = [0; 4]; + } + } + fresh.kept = Some(focus.tiles.clone()); + assert_eq!(*p, fresh, "repaired pop"); + } + let t_rebind = median(15, || { + black_box(Qr::bind(&b1, &wit1).unwrap()); + }); + let t_repair = median(15, || { + let mut rp = pops.clone(); + let mut rf = focus.rows.clone(); + repair_all(&mut rp, &mut rf); + black_box((rp, rf)); + }); + let t_rebuild = median(15, || { + let f = qr1.team_focus(&b1, q.team).unwrap(); + let mut runs1 = Runs::new(&qr1); + for s in &slots1 { + black_box(qr1.population_in(&b1, s, &f, &mut runs1).unwrap()); + } + }); + let t_clone = median(15, || { + black_box((pops.clone(), focus.rows.clone())); + }); + + println!( + "users {}, tiles {}, team members {}, focus tiles {}, 18 populations, pointer tables {} slots", + b.users, + qr.pages(), + focus.rows.count(), + focus.tiles_selected(), + runs.built + ); + println!("answers {want:?}"); + println!("Q cold (per version):"); + show("prepare (arm B CSR, shared with B+)", t_prepare); + show("bind register", t_register); + show("bind team focus", t_focus); + show("pointer tables + 18 populations in focus", t_pops); + show(" of which populations (tables warm)", t_pops_warm); + show("version checks (18)", t_check); + show("3 methods, relalg folds (18 queries)", t_methods); + show("all of the above but prepare", t_cold); + println!("Q warm (bound populations reused):"); + show("version checks + 3 methods (relalg)", t_check + t_methods); + show( + "version checks + 3 methods (native Pop folds)", + t_check + t_native, + ); + println!("Q after an edit (1 membership + 1 nesting removed):"); + show("rebind register", t_rebind); + show( + "repair 18 populations + focus", + t_repair.saturating_sub(t_clone), + ); + show("rebuild focus + tables + 18 populations", t_rebuild); + println!("B+:"); + show("prepare + compile", t_compile); + show( + &format!( + "per focus user ({}), groups_of + roles_of", + focus_users.len() + ), + t_bplus, + ); + show( + "members_of populations (16 licenses + loc + team)", + t_bplus_pop, + ); +} diff --git a/crates/dir-rmap-probe/examples/qr_license.rs b/crates/dir-rmap-probe/examples/qr_license.rs new file mode 100644 index 000000000..5ca86d632 --- /dev/null +++ b/crates/dir-rmap-probe/examples/qr_license.rs @@ -0,0 +1,186 @@ +//! Chained population masking over license groups (arm Q). +//! +//! ```text +//! cargo run --release --example qr_license # 65,000 users +//! cargo run --release --example qr_license -- 8000 +//! ``` +//! +//! Question: per license group, how many users hold the license (directly or +//! through nesting), hold a given role, and are NOT in a given location? +//! Every filter is a population (one bit per user, by tile and row); the +//! chain is AND / AND-NOT by position; the answer is a count per license. +//! The per-user route (resolved pages, one effective row per user, bit +//! tests) is the reference and is timed for comparison. Both are checked +//! equal before timing. + +use dir_rmap_probe::World; +use dir_rmap_probe::fixture::{Fixture, Kind, Spec}; +use dir_rmap_probe::focus::Row; +use dir_rmap_probe::pop::Pop; +use dir_rmap_probe::prep::Ordinals; +use dir_rmap_probe::qr::{Page, Qr}; +use std::hint::black_box; +use std::time::{Duration, Instant}; + +fn median(reps: usize, mut f: impl FnMut()) -> Duration { + let mut t: Vec = (0..reps) + .map(|_| { + let s = Instant::now(); + f(); + s.elapsed() + }) + .collect(); + t.sort(); + t[reps / 2] +} + +fn fmt(d: Duration) -> String { + if d >= Duration::from_millis(1) { + "FAILED GATE (millisecond scale)".into() + } else { + format!("{:.1} us", d.as_secs_f64() * 1e6) + } +} + +fn bit(r: &Row, i: usize) -> bool { + r[i >> 6] >> (i & 63) & 1 == 1 +} + +fn main() { + let users: usize = std::env::args() + .nth(1) + .map_or(65_000, |s| s.parse().expect("user count")); + let mut spec = Spec::small(); + spec.users = users; + spec.licenses = 16; + let w = World::new(Fixture::generate(spec)); + let v = w.view(); + let ord = Ordinals::of(&w.f, &v); + let b = w.prepare(); + let qr = Qr::bind(&b, &w.witness()).unwrap(); + let of_kind = |k: Kind| -> Vec { + (0..w.f.groups.len()) + .filter(|&j| w.f.kind[j] == k) + .map(|j| ord.group[j]) + .collect() + }; + let licenses = of_kind(Kind::License); + let reach = |g: u16| qr.slots_reaching(g).unwrap(); + let role = (0..64) + .max_by_key(|&r| qr.role_slots[r].iter().map(|x| x.count_ones()).sum::()) + .unwrap(); + let loc = *of_kind(Kind::Location) + .iter() + .max_by_key(|&&g| b.group_users.row(g as usize).len()) + .unwrap(); + let (loc_slots, role_slots) = (reach(loc), qr.role_slots[role]); + let lic_slots: Vec = licenses.iter().map(|&g| reach(g)).collect(); + let lic_bits: Vec = licenses.iter().map(|&g| qr.slot(g).unwrap()).collect(); + + // The chained-mask route. + let masked = || -> Vec { + let chain: Pop = qr + .population(&b, &role_slots) + .unwrap() + .and_not(&qr.population(&b, &loc_slots).unwrap()) + .unwrap(); + lic_slots + .iter() + .map(|s| qr.population(&b, s).unwrap().count_and(&chain).unwrap()) + .collect() + }; + // The per-user route over resolved pages. + let pages: Vec = (0..qr.pages()).map(|p| qr.page(&b, p).unwrap()).collect(); + let (loc_bit, users_n) = (qr.slot(loc).unwrap(), b.users); + let per_user = || -> Vec { + let mut out = vec![0u64; licenses.len()]; + for u in 0..users_n { + let page = &pages[u / 256]; + let i = u % 256; + if page.roles[i] >> role & 1 == 0 { + continue; + } + let eff = qr.effective(&page.member[i]); + if bit(&eff, loc_bit) { + continue; + } + for (o, &s) in out.iter_mut().zip(&lic_bits) { + *o += u64::from(bit(&eff, s)); + } + } + out + }; + let got = masked(); + assert_eq!(got, per_user(), "masked chain vs per-user"); + + let role_pop = qr.population(&b, &role_slots).unwrap(); + let loc_pop = qr.population(&b, &loc_slots).unwrap(); + let chain = role_pop.clone().and_not(&loc_pop).unwrap(); + let lic_pops: Vec = lic_slots + .iter() + .map(|s| qr.population(&b, s).unwrap()) + .collect(); + let t_role = median(15, || { + black_box(qr.population(&b, &role_slots).unwrap()); + }); + let t_loc = median(15, || { + black_box(qr.population(&b, &loc_slots).unwrap()); + }); + let t_lic = median(15, || { + for s in &lic_slots { + black_box(qr.population(&b, s).unwrap()); + } + }); + let t_chain = median(51, || { + black_box(role_pop.clone().and_not(black_box(&loc_pop)).unwrap()); + }); + let t_fold = median(51, || { + for p in &lic_pops { + black_box(p.count_and(black_box(&chain)).unwrap()); + } + }); + let t_all = median(15, || { + black_box(masked()); + }); + let t_user = median(5, || { + black_box(per_user()); + }); + + let members = |s: &Row| -> usize { + qr.ids + .iter() + .enumerate() + .filter(|&(k, _)| bit(s, k)) + .map(|(_, &g)| b.group_users.row(g as usize).len()) + .sum() + }; + println!( + "users {}, tiles {}, {} licenses; role {role} holders {}, location holders {}, chain {}", + b.users, + qr.pages(), + licenses.len(), + role_pop.count(), + loc_pop.count(), + chain.count() + ); + println!("per license (role AND-NOT location): {got:?}"); + println!( + "memberships read: role {}, location {}, licenses {}", + members(&role_slots), + members(&loc_slots), + lic_slots.iter().map(members).sum::() + ); + println!("chained masks:"); + println!(" build role population {}", fmt(t_role)); + println!(" build location population {}", fmt(t_loc)); + println!(" build 16 license populations {}", fmt(t_lic)); + println!(" chain role AND-NOT location {}", fmt(t_chain)); + println!(" 16 x (license AND chain) count {}", fmt(t_fold)); + println!(" whole question, end to end {}", fmt(t_all)); + println!("per-user route over resolved pages (reference):"); + println!(" whole question {}", fmt(t_user)); + println!( + "population bytes {} each", + qr.pages() * std::mem::size_of::() + ); +} diff --git a/crates/dir-rmap-probe/src/lib.rs b/crates/dir-rmap-probe/src/lib.rs index 4699ff972..542ca6f70 100644 --- a/crates/dir-rmap-probe/src/lib.rs +++ b/crates/dir-rmap-probe/src/lib.rs @@ -14,11 +14,15 @@ //! | B | [`prep::Prepared`] | bidirectional membership CSR + nesting CSR | //! | B+ | [`prep::Compiled`] | B + closure rows per group and chain + roles by ordinal | //! | F | [`focus`] | B+ projected onto a 256-slot focus of selected groups | +//! | Q | [`qr`] | DR256 register: the security chain bound to 256 slots; nesting closed inside the tile; users x groups as a bit matrix | +//! | Q | [`pop`] | populations bound to one version: team focus, focus-restricted builds, relalg method folds, repair after edits | pub mod fixture; pub mod focus; pub mod oracle; +pub mod pop; pub mod prep; +pub mod qr; use fixture::{Fixture, Kind}; use lance_graph_dir_sim::{GroupProperty, GroupWhere, View}; @@ -182,6 +186,10 @@ impl World { user: f.groups[c as usize], group: f.groups[p as usize], }, + Edit::RemoveNested(c, p) => Change::RemoveMembership { + user: f.groups[c as usize], + group: f.groups[p as usize], + }, }) .collect(); self.version = self @@ -193,6 +201,7 @@ impl World { Edit::AddDirect(u, g) => self.direct.push((u, g)), Edit::RemoveDirect(u, g) => self.direct.retain(|&x| x != (u, g)), Edit::AddNested(c, p) => self.nested.push((c, p)), + Edit::RemoveNested(c, p) => self.nested.retain(|&x| x != (c, p)), } } self.direct.sort_unstable(); @@ -226,6 +235,8 @@ pub enum Edit { AddDirect(u32, u32), RemoveDirect(u32, u32), AddNested(u32, u32), + /// `(child group, parent group)`: the nesting pair removed. + RemoveNested(u32, u32), } struct Propose(Vec); diff --git a/crates/dir-rmap-probe/src/pop.rs b/crates/dir-rmap-probe/src/pop.rs new file mode 100644 index 000000000..75c03d2d3 --- /dev/null +++ b/crates/dir-rmap-probe/src/pop.rs @@ -0,0 +1,281 @@ +//! Populations over the user axis, bound to one register version (arm Q). +//! +//! A [`Pop`] is one bit per user: bit `i` of tile `p` is user `p * 256 + i`. +//! Populations from the same register line up by position, so filters chain +//! as word-wise AND / AND-NOT and answer with a count. A population carries +//! the witness of the snapshot it was derived from. Combining or folding +//! populations from different snapshots is refused ([`Refusal::Stale`]). +//! +//! A population is a cache of derived state, never a source. It is rebuilt +//! or repaired from the source rows ([`Prepared`]) of the version it claims. +//! +//! * [`Qr::population`] builds one over every tile; [`Qr::population_in`] +//! only over a [`TeamFocus`]'s tiles, through the per-slot tile pointer +//! tables ([`Runs`]). +//! * [`TeamFocus`] binds one group's identity (a group ordinal, stable across +//! versions) to the tiles and rows of its effective members. The tile mask +//! is locality, not identity: a move between tiles changes the mask, never +//! the group. +//! * [`Pop::plane`] lends the bits as one `n_rows`-long mask plane, so +//! `lance-graph-relalg` folds them (`Rel::Plane`) with mask-risc, without a +//! copy. +//! * [`Pop::repair`] brings a population to a new version: the slot set is +//! diffed against the rebound register; slots that gained the property +//! scatter their members in; the members of slots that lost it, and every +//! user whose direct memberships changed, are RE-DERIVED from their direct +//! groups. A user who loses one supporting slot keeps the bit while +//! another remains. Clearing the members of a removed slot instead would be +//! wrong whenever a second path exists. + +use crate::focus::{Row, SLOTS}; +use crate::prep::{Prepared, Refusal, Witness}; +use crate::qr::{Qr, Runs, row_bit, set}; + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Pop { + pub witness: Witness, + pub tiles: Vec, + /// The tile mask the population was built for, `None` for every tile. + /// Outside it the bits are not derived, so a restricted population is + /// exact only within these tiles, and a repair must keep the same mask. + pub kept: Option>, +} + +#[inline] +fn tile_on(tiles: &[u64], p: usize) -> bool { + tiles[p >> 6] >> (p & 63) & 1 == 1 +} + +impl Pop { + pub fn check(&self, current: &Witness) -> Result<(), Refusal> { + if self.witness == *current { + Ok(()) + } else { + Err(Refusal::Stale) + } + } + + pub fn and(mut self, o: &Pop) -> Result { + o.check(&self.witness)?; + for (a, b) in self.tiles.iter_mut().zip(&o.tiles) { + for k in 0..4 { + a[k] &= b[k]; + } + } + Ok(self) + } + + pub fn and_not(mut self, o: &Pop) -> Result { + o.check(&self.witness)?; + for (a, b) in self.tiles.iter_mut().zip(&o.tiles) { + for k in 0..4 { + a[k] &= !b[k]; + } + } + Ok(self) + } + + pub fn or(mut self, o: &Pop) -> Result { + o.check(&self.witness)?; + for (a, b) in self.tiles.iter_mut().zip(&o.tiles) { + for k in 0..4 { + a[k] |= b[k]; + } + } + Ok(self) + } + + pub fn count(&self) -> u64 { + self.tiles + .iter() + .flatten() + .map(|w| u64::from(w.count_ones())) + .sum() + } + + /// `self AND o`, counted, without building the intersection. + pub fn count_and(&self, o: &Pop) -> Result { + o.check(&self.witness)?; + let mut n = 0; + for (a, b) in self.tiles.iter().zip(&o.tiles) { + for k in 0..4 { + n += u64::from((a[k] & b[k]).count_ones()); + } + } + Ok(n) + } + + /// The bits as one mask plane over `n_rows` users (`ceil(n_rows / 64)` + /// words), borrowed. Bits past `n_rows` are zero by construction. + pub fn plane(&self, n_rows: usize) -> &[u64] { + &self.tiles.as_flattened()[..n_rows.div_ceil(64)] + } + + pub fn bytes(&self) -> usize { + self.tiles.len() * 32 + } + + /// Brings this population, built from `slots0` against the previous + /// version, to the version of `qr1` / `b1`, whose register gives the + /// population's slot set as `slots1`. `touched` are the users whose + /// direct memberships changed between the versions. With `tiles`, only + /// those tiles are kept (a focus-restricted population). + /// + /// Refuses when the slot binding changed (the security chain gained or + /// lost a group): the old slot sets do not address the same groups, so + /// the population is rebuilt instead. Refuses ([`Refusal::OutOfFocus`]) + /// a `tiles` other than the mask the population was built for: tiles + /// leaving the mask would keep stale bits and tiles joining it would stay + /// empty under the new version. A different focus is a rebuild. + pub fn repair( + &mut self, + qr0: &Qr, + qr1: &Qr, + b1: &Prepared, + (slots0, slots1): (&Row, &Row), + touched: &[u16], + tiles: Option<&[u64]>, + ) -> Result { + self.check(&qr0.witness)?; + b1.check(&qr1.witness)?; + if qr0.ids != qr1.ids || self.tiles.len() != qr1.pages() { + return Err(Refusal::Stale); + } + if self.kept.as_deref() != tiles { + return Err(Refusal::OutOfFocus); + } + let keep = |u: usize| tiles.is_none_or(|t| tile_on(t, u / SLOTS)); + let mut out = Repair::default(); + for (s, &g) in qr1.ids.iter().enumerate() { + let (was, is) = (row_bit(slots0, s), row_bit(slots1, s)); + if is && !was { + out.slots_added += 1; + for &u in b1.group_users.row(g as usize) { + let u = u as usize; + if keep(u) { + set(&mut self.tiles[u / SLOTS], u % SLOTS); + out.scattered += 1; + } + } + } else if was && !is { + out.slots_removed += 1; + for &u in b1.group_users.row(g as usize) { + if keep(u as usize) { + out.rederived += 1; + self.rederive(qr1, b1, slots1, u as usize); + } + } + } + } + for &u in touched { + if keep(u as usize) { + out.rederived += 1; + self.rederive(qr1, b1, slots1, u as usize); + } + } + self.witness = qr1.witness.clone(); + Ok(out) + } + + /// One user's bit from its direct groups against `slots`. + fn rederive(&mut self, qr: &Qr, b: &Prepared, slots: &Row, u: usize) { + let mut on = false; + for &d in b.user_groups.row(u) { + if let Some(s) = qr.slot(d) { + on |= row_bit(slots, s); + } + } + let w = &mut self.tiles[u / SLOTS][(u % SLOTS) >> 6]; + let m = 1u64 << (u % 64); + if on { + *w |= m; + } else { + *w &= !m; + } + } +} + +/// What a repair did. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct Repair { + pub slots_added: usize, + pub slots_removed: usize, + /// Members of added slots set. + pub scattered: usize, + /// Users re-derived from their direct groups. + pub rederived: usize, +} + +/// One group's identity bound to the tiles and rows of its effective +/// members, for one version. +#[derive(Clone, Debug)] +pub struct TeamFocus { + /// The group's ordinal: the identity, stable across versions. + pub group: u16, + /// Tile mask: the tiles holding at least one member. + pub tiles: Vec, + /// The members (direct or through nesting). + pub rows: Pop, +} + +impl TeamFocus { + pub fn tiles_selected(&self) -> usize { + self.tiles.iter().map(|w| w.count_ones() as usize).sum() + } +} + +impl Qr { + /// Binds the focus of security group `g`: its effective members and the + /// tiles they occupy. Reads only the member lists of `g` and the groups + /// nested in it. Refuses a group outside the chain. + pub fn team_focus(&self, b: &Prepared, g: u16) -> Result { + let slots = self.slots_reaching(g).ok_or(Refusal::OutOfFocus)?; + let rows = self.population(b, &slots)?; + let mut tiles = vec![0u64; self.pages().div_ceil(64)]; + for (p, t) in rows.tiles.iter().enumerate() { + if t.iter().any(|&w| w != 0) { + tiles[p >> 6] |= 1 << (p & 63); + } + } + Ok(TeamFocus { + group: g, + tiles, + rows, + }) + } + + /// The users holding one of `slots` directly, only in the focus's tiles + /// (other tiles stay empty). Reads the selected tiles' runs of each slot + /// through `runs`, building a slot's pointer table on first use. + pub fn population_in( + &self, + b: &Prepared, + slots: &Row, + focus: &TeamFocus, + runs: &mut Runs, + ) -> Result { + b.check(&self.witness)?; + focus.rows.check(&self.witness)?; + self.check(&runs.witness)?; + let mut tiles = vec![[0u64; 4]; self.pages()]; + for (s, &g) in self.ids.iter().enumerate() { + if !row_bit(slots, s) { + continue; + } + let members = b.group_users.row(g as usize); + let off = runs.index(self, b, s); + for (p, t) in tiles.iter_mut().enumerate() { + if tile_on(&focus.tiles, p) { + for &u in &members[off[p] as usize..off[p + 1] as usize] { + set(t, u as usize % SLOTS); + } + } + } + } + Ok(Pop { + witness: self.witness.clone(), + tiles, + kept: Some(focus.tiles.clone()), + }) + } +} diff --git a/crates/dir-rmap-probe/src/qr.rs b/crates/dir-rmap-probe/src/qr.rs new file mode 100644 index 000000000..da922cd9f --- /dev/null +++ b/crates/dir-rmap-probe/src/qr.rs @@ -0,0 +1,490 @@ +//! Arm Q: nested groups as a DR256 register, users x groups as a bit matrix. +//! +//! The security chain's groups are bound to 256 local slots once (the +//! global-to-local register). Everything after that is a 256 x 256 bit +//! matrix over slots, one `Row` (4 words) per row: +//! +//! * `nest`: slot -> parent slots (security child -> security parent); +//! * `reach`: the reflexive transitive closure of `nest`, computed inside the +//! tile by Warshall (`reach[i] |= reach[k]` wherever bit `k` of `reach[i]` +//! is set, for every `k` in order). No closure row is read from outside the +//! register, so the closure is complete only because EVERY security group +//! is bound: a chain with more than 256 security groups is refused +//! ([`Refusal::TooLarge`]), never truncated; +//! * `role`: per slot, the roles its reach carries. +//! +//! [`Qr::bind`] builds only the register. Users are never resolved all at +//! once: nobody asks for every user's nested groups in one go. A page of 256 +//! users (a [`Page`]: those users' rows of the membership matrix, the "QR +//! code", plus their roles) is resolved when a question names it +//! ([`Qr::page`], [`Pages`]). +//! +//! Every inner loop runs over slot or user coordinates and tests a bit; no +//! loop scans set bits for a position (the D-RU-6 address rule). Translating +//! a group ordinal to a slot is counted ([`Qr::translations`], +//! [`Page::translations`]). It is frozen-codebook decoding over the same +//! snapshot, and its contract-level witness is still missing (ADMISSION B, +//! `ISS-ADMISSION-PREP-BINDING`). + +use crate::focus::{Row, SLOTS}; +use crate::pop::Pop; +use crate::prep::{Prepared, Refusal, Witness, bit}; + +#[inline] +pub(crate) fn row_bit(r: &Row, s: usize) -> bool { + r[s >> 6] >> (s & 63) & 1 == 1 +} + +#[inline] +pub(crate) fn set(r: &mut Row, s: usize) { + r[s >> 6] |= 1 << (s & 63); +} + +#[inline] +fn or(a: &mut Row, b: &Row) { + for k in 0..4 { + a[k] |= b[k]; + } +} + +/// The register: the security chain bound to slots and closed in the tile. +pub struct Qr { + pub witness: Witness, + /// Users in the snapshot; pages are `users.div_ceil(256)`. + pub users: usize, + /// Slot -> group ordinal. + pub ids: Vec, + /// Group ordinal -> slot + 1; 0 = not a security group. + slot_of: Vec, + /// Slot -> parent slots. + pub nest: [Row; SLOTS], + /// Slot -> every slot it reaches upward, itself included. + pub reach: [Row; SLOTS], + /// Slot -> roles carried by its reach. + pub role: [u64; SLOTS], + /// Role -> the slots that carry it (`role` transposed). + pub role_slots: [Row; 64], + /// Ordinal -> slot lookups done while binding the register. + pub translations: usize, +} + +/// One page of 256 users: their rows of the membership matrix and their +/// roles. +pub struct Page { + /// Row `i`: user `page * 256 + i`'s direct security groups as slots. + pub member: [Row; SLOTS], + /// Roles of user `page * 256 + i`. + pub roles: [u64; SLOTS], + /// Ordinal -> slot lookups done while resolving this page. + pub translations: usize, +} + +impl Qr { + /// Binds the security chain to slots and closes it in the tile. Resolves + /// no user. Refuses a stale `b` and more than 256 security groups. + pub fn bind(b: &Prepared, current: &Witness) -> Result { + b.check(current)?; + let ids: Vec = (0..b.groups) + .filter(|&g| bit(&b.security, g)) + .map(|g| g as u16) + .collect(); + if ids.len() > SLOTS { + return Err(Refusal::TooLarge); + } + let mut slot_of = vec![0u16; b.groups]; + for (s, &g) in ids.iter().enumerate() { + slot_of[g as usize] = s as u16 + 1; + } + let mut translations = 0; + let mut nest = [[0u64; 4]; SLOTS]; + for &(c, p) in &b.nested { + translations += 2; + let (c, p) = (slot_of[c as usize], slot_of[p as usize]); + if c != 0 && p != 0 { + set(&mut nest[c as usize - 1], p as usize - 1); + } + } + let n = ids.len(); + let reach = closure(&nest, n); + let mut role = [0u64; SLOTS]; + for (i, r) in role.iter_mut().enumerate().take(n) { + for (k, &g) in ids.iter().enumerate() { + if row_bit(&reach[i], k) { + *r |= b.grants[g as usize]; + } + } + } + let mut role_slots = [[0u64; 4]; 64]; + for (s, &rs) in role.iter().enumerate().take(n) { + for (r, row) in role_slots.iter_mut().enumerate() { + if rs >> r & 1 == 1 { + set(row, s); + } + } + } + Ok(Qr { + witness: b.witness.clone(), + users: b.users, + role_slots, + ids, + slot_of, + nest, + reach, + role, + translations, + }) + } + + pub fn check(&self, current: &Witness) -> Result<(), Refusal> { + if self.witness == *current { + Ok(()) + } else { + Err(Refusal::Stale) + } + } + + pub fn slot(&self, g: u16) -> Option { + match self.slot_of[g as usize] { + 0 => None, + s => Some(s as usize - 1), + } + } + + pub fn pages(&self) -> usize { + self.users.div_ceil(SLOTS) + } + + /// Resolves page `p`: its users' membership rows, then their roles as a + /// fold over the page's columns. The users holding role `r` are the OR of + /// the columns of the slots whose role set contains `r`. Refuses a `b` + /// from another snapshot. + pub fn page(&self, b: &Prepared, p: usize) -> Result { + b.check(&self.witness)?; + let base = p * SLOTS; + let end = self.users.min(base + SLOTS); + let mut member = [[0u64; 4]; SLOTS]; + let mut cols = [[0u64; 4]; SLOTS]; + let mut translations = 0; + for u in base..end { + for &d in b.user_groups.row(u) { + translations += 1; + if let Some(s) = self.slot(d) { + set(&mut member[u - base], s); + set(&mut cols[s], u - base); + } + } + } + let mut holders = [[0u64; 4]; 64]; + for (s, c) in cols.iter().enumerate().take(self.ids.len()) { + for (r, h) in holders.iter_mut().enumerate() { + if self.role[s] >> r & 1 == 1 { + or(h, c); + } + } + } + let mut roles = [0u64; SLOTS]; + for (i, o) in roles.iter_mut().enumerate().take(end - base) { + for (r, h) in holders.iter().enumerate() { + *o |= u64::from(row_bit(h, i)) << r; + } + } + Ok(Page { + member, + roles, + translations, + }) + } + + /// Effective security groups as slots: a membership row times `reach`. + pub fn effective(&self, member: &Row) -> Row { + let mut out = [0u64; 4]; + for k in 0..self.ids.len() { + if row_bit(member, k) { + or(&mut out, &self.reach[k]); + } + } + out + } + + /// Roles from a membership row and the slot roles (the row-times-vector + /// product a page resolves by columns). + pub fn roles_live(&self, member: &Row) -> u64 { + let mut out = 0; + for (k, &r) in self.role.iter().enumerate().take(self.ids.len()) { + if row_bit(member, k) { + out |= r; + } + } + out + } + + /// The slots whose reach contains security group `g`: a user is + /// effectively in `g` iff one of its direct slots is in this set. `None` + /// when `g` is outside the chain. + pub fn slots_reaching(&self, g: u16) -> Option { + let t = self.slot(g)?; + let mut out = [0u64; 4]; + for (s, r) in self.reach.iter().enumerate().take(self.ids.len()) { + if row_bit(r, t) { + set(&mut out, s); + } + } + Some(out) + } + + /// Masked count over several tiles at once: how many users, in the tiles + /// selected by `tiles` (one bit per page) and the rows selected by + /// `rows[p]` inside tile `p`, hold one of the slots in `slots` directly. + /// + /// Each selected slot's group is read once (its direct-member list), and + /// every member landing in a selected tile sets its bit in that tile of + /// `scratch`. The answer is the popcount of each selected tile AND its + /// row mask; no page is resolved and no per-user value is built. + /// `scratch` has one `Row` per page and is left zeroed. + pub fn count_masked( + &self, + b: &Prepared, + slots: &Row, + tiles: &[u64], + rows: &[Row], + scratch: &mut [Row], + ) -> Result { + b.check(&self.witness)?; + for (s, &g) in self.ids.iter().enumerate() { + if row_bit(slots, s) { + for &u in b.group_users.row(g as usize) { + let u = u as usize; + let p = u / SLOTS; + // Branchless: the tile's selection bit is shifted into + // place, so an unselected tile receives an OR of zero. + // Member lists are in ordinal order, which is random with + // respect to tiles, so a branch here mispredicts. + let sel = tiles[p >> 6] >> (p & 63) & 1; + let i = u % SLOTS; + scratch[p][i >> 6] |= sel << (i & 63); + } + } + } + let mut n = 0u64; + for (p, (acc, m)) in scratch.iter_mut().zip(rows).enumerate() { + if tiles[p >> 6] >> (p & 63) & 1 == 1 { + for k in 0..4 { + n += u64::from((acc[k] & m[k]).count_ones()); + } + *acc = [0; 4]; + } + } + Ok(n) + } + + /// Bytes retained by the register; pages are counted separately. + pub fn bytes(&self) -> usize { + self.ids.len() * 2 + self.slot_of.len() * 2 + SLOTS * (32 + 32 + 8) + 64 * 32 + } +} + +impl Page { + /// Roles of the page's user `i`. Unchecked hot path. + #[inline] + pub fn roles_of(&self, i: usize) -> u64 { + self.roles[i] + } + + pub const BYTES: usize = SLOTS * (32 + 8); +} + +/// Pages resolved on first use. +pub struct Pages { + /// The register version the pages were resolved against. A cached page + /// never answers for another version. + pub witness: Witness, + pages: Vec>>, + /// Pages resolved so far. + pub resolved: usize, +} + +impl Pages { + pub fn new(qr: &Qr) -> Pages { + Pages { + witness: qr.witness.clone(), + pages: (0..qr.pages()).map(|_| None).collect(), + resolved: 0, + } + } + + /// The page holding user `u`, resolved if it was not. Refuses a register + /// or source rows of another version than the cache's, before returning + /// a cached page or resolving a new one. + pub fn of(&mut self, qr: &Qr, b: &Prepared, u: u16) -> Result<&Page, Refusal> { + qr.check(&self.witness)?; + b.check(&self.witness)?; + let p = u as usize / SLOTS; + if self.pages[p].is_none() { + self.pages[p] = Some(Box::new(qr.page(b, p)?)); + self.resolved += 1; + } + Ok(self.pages[p].as_deref().expect("resolved above")) + } + + /// Roles of user `u`, resolving its page on first use. + pub fn roles_of(&mut self, qr: &Qr, b: &Prepared, u: u16) -> Result { + Ok(self.of(qr, b, u)?.roles_of(u as usize % SLOTS)) + } + + /// Checked: is `u` effectively in security group `g`? + pub fn ask( + &mut self, + qr: &Qr, + b: &Prepared, + current: &Witness, + u: u16, + g: u16, + ) -> Result { + qr.check(current)?; + let s = qr.slot(g).ok_or(Refusal::OutOfFocus)?; + let row = self.of(qr, b, u)?.member[u as usize % SLOTS]; + Ok(row_bit(&qr.effective(&row), s)) + } +} + +/// Reflexive transitive closure of `nest` over the first `n` slots, by +/// Warshall inside the tile. +fn closure(nest: &[Row; SLOTS], n: usize) -> [Row; SLOTS] { + let mut r = *nest; + for (i, row) in r.iter_mut().enumerate().take(n) { + set(row, i); + } + for k in 0..n { + let rk = r[k]; + for row in r.iter_mut().take(n) { + if row_bit(row, k) { + or(row, &rk); + } + } + } + r +} + +/// Per-slot tile pointers into the slot's sorted direct-member list: for +/// slot `s`, `runs[s][p]..runs[s][p + 1]` are its members in tile `p`. +/// Built for a slot the first time a question names it; a query then visits +/// only the selected tiles' runs. +pub struct Runs { + /// The version the tables were read from; a table never outlives it. + pub witness: Witness, + runs: Vec>>, + /// Slots whose pointer table has been built. + pub built: usize, +} + +impl Runs { + pub fn new(qr: &Qr) -> Runs { + Runs { + witness: qr.witness.clone(), + runs: (0..qr.ids.len()).map(|_| None).collect(), + built: 0, + } + } + + pub(crate) fn index(&mut self, qr: &Qr, b: &Prepared, s: usize) -> &[u32] { + if self.runs[s].is_none() { + let pages = qr.pages(); + let mut off = vec![0u32; pages + 1]; + for &u in b.group_users.row(qr.ids[s] as usize) { + off[u as usize / SLOTS + 1] += 1; + } + for p in 0..pages { + off[p + 1] += off[p]; + } + self.runs[s] = Some(off.into_boxed_slice()); + self.built += 1; + } + self.runs[s].as_deref().expect("built above") + } + + /// Same answer as [`Qr::count_masked`], visiting only selected tiles: + /// per tile, the runs of the slots in `slots` are ORed into one row, + /// ANDed with the tile's row mask and counted. + pub fn count( + &mut self, + qr: &Qr, + b: &Prepared, + slots: &Row, + tiles: &[u64], + rows: &[Row], + ) -> Result { + b.check(&qr.witness)?; + qr.check(&self.witness)?; + let n = qr.ids.len(); + for s in 0..n { + if row_bit(slots, s) { + self.index(qr, b, s); + } + } + let mut total = 0u64; + for (p, m) in rows.iter().enumerate().take(qr.pages()) { + if tiles[p >> 6] >> (p & 63) & 1 == 0 { + continue; + } + let mut acc = [0u64; 4]; + for s in 0..n { + if row_bit(slots, s) { + let off = self.runs[s].as_deref().expect("indexed above"); + let members = b.group_users.row(qr.ids[s] as usize); + for &u in &members[off[p] as usize..off[p + 1] as usize] { + set(&mut acc, u as usize % SLOTS); + } + } + } + for k in 0..4 { + total += u64::from((acc[k] & m[k]).count_ones()); + } + } + Ok(total) + } + + pub fn bytes(&self) -> usize { + self.runs.iter().flatten().map(|r| r.len() * 4).sum() + } +} + +impl Qr { + /// Every user of the snapshot. + pub fn everyone(&self) -> Pop { + let mut tiles = vec![[0u64; 4]; self.pages()]; + for (p, t) in tiles.iter_mut().enumerate() { + let n = self.users.saturating_sub(p * SLOTS).min(SLOTS); + for (k, w) in t.iter_mut().enumerate() { + let bits = n.saturating_sub(k * 64).min(64); + *w = if bits == 64 { !0 } else { (1u64 << bits) - 1 }; + } + } + Pop { + witness: self.witness.clone(), + tiles, + kept: None, + } + } + + /// The users holding one of `slots` directly. With `slots` from + /// [`Qr::slots_reaching`] that is the users effectively in the group; + /// from [`Qr::role_slots`], the users holding the role. Each slot's + /// direct-member list is read once and scattered by position. + pub fn population(&self, b: &Prepared, slots: &Row) -> Result { + b.check(&self.witness)?; + let mut tiles = vec![[0u64; 4]; self.pages()]; + for (s, &g) in self.ids.iter().enumerate() { + if row_bit(slots, s) { + for &u in b.group_users.row(g as usize) { + let u = u as usize; + set(&mut tiles[u / SLOTS], u % SLOTS); + } + } + } + Ok(Pop { + witness: self.witness.clone(), + tiles, + kept: None, + }) + } +} diff --git a/crates/dir-rmap-probe/tests/equivalence.rs b/crates/dir-rmap-probe/tests/equivalence.rs index e7921bfa4..be62cc967 100644 --- a/crates/dir-rmap-probe/tests/equivalence.rs +++ b/crates/dir-rmap-probe/tests/equivalence.rs @@ -490,7 +490,7 @@ fn a_refreshed_focus_equals_a_fresh_bind_dir_sim_and_the_oracle() { .iter() .filter_map(|e| match *e { Edit::AddDirect(u, _) | Edit::RemoveDirect(u, _) => Some(ord.user[u as usize]), - Edit::AddNested(..) => None, + Edit::AddNested(..) | Edit::RemoveNested(..) => None, }) .collect(); for (fu, l) in fs.iter_mut().zip(fl.iter_mut()) { diff --git a/crates/dir-rmap-probe/tests/qr_focus.rs b/crates/dir-rmap-probe/tests/qr_focus.rs new file mode 100644 index 000000000..cf124c881 --- /dev/null +++ b/crates/dir-rmap-probe/tests/qr_focus.rs @@ -0,0 +1,642 @@ +//! Arm Q: a team focus, populations bound to one version, methods that fold +//! them, and repair after edits. +//! +//! * A class declares three methods (`compute_dag`); the DR table binds each +//! to a relalg fold over populations lent as mask planes. The populations +//! are built once, only in the focus's tiles, and every method reuses them. +//! Each answer equals dir-sim's own walk over the focus's users. +//! * After membership and nesting edits, repaired populations equal a fresh +//! build, including when a user or a group keeps a property through a +//! second path while one path is removed. +//! * Populations, pointer tables and repairs refuse a mismatched version. + +use dir_rmap_probe::fixture::{Fixture, Kind, Spec}; +use dir_rmap_probe::focus::Row; +use dir_rmap_probe::pop::{Pop, TeamFocus}; +use dir_rmap_probe::prep::{Chain, Ordinals, Prepared, Refusal}; +use dir_rmap_probe::qr::{Qr, Runs}; +use dir_rmap_probe::{Edit, World, a_groups}; +use lance_graph_contract::class_view::{ClassId, ClassView, ComputeEdge, execute_compute_dag}; +use lance_graph_contract::ontology::{DisplayTemplate, FieldRef}; +use lance_graph_mask_risc::Planes; +use lance_graph_relalg::{Answer, Measure, Query, Rel, run}; +use ogar_dir_core::Guid128; +use std::collections::{HashMap, HashSet}; + +// ---------------------------------------------------------------- the class + +const TEAM: ClassId = 2; +const MEMBER_OF: u8 = 0; +/// Per license: members holding the license and the role, outside the +/// location. +const LICENSED_ROLE_OUTSIDE: u8 = 1; +/// Members holding no license. +const UNLICENSED: u8 = 2; +/// Members holding the role. +const ROLE_HOLDERS: u8 = 3; + +static METHODS: [ComputeEdge; 3] = [ + ComputeEdge { + target: LICENSED_ROLE_OUTSIDE, + inputs: &[MEMBER_OF], + }, + ComputeEdge { + target: UNLICENSED, + inputs: &[MEMBER_OF], + }, + ComputeEdge { + target: ROLE_HOLDERS, + inputs: &[MEMBER_OF], + }, +]; + +struct TeamAudit { + fields: Vec, +} + +impl ClassView for TeamAudit { + fn fields(&self, class: ClassId) -> &[FieldRef] { + if class == TEAM { &self.fields } else { &[] } + } + fn template(&self, _: ClassId) -> DisplayTemplate { + DisplayTemplate::Detail + } + fn dolce_category_id(&self, _: ClassId) -> u8 { + 0 + } + fn compute_dag(&self, class: ClassId) -> &[ComputeEdge] { + if class == TEAM { &METHODS } else { &[] } + } +} + +fn team_audit() -> TeamAudit { + TeamAudit { + fields: vec![ + FieldRef::new("urn:iam:team:memberOf", "member_of"), + FieldRef::new("urn:iam:team:licensedRoleOutside", "licensed_role_outside"), + FieldRef::new("urn:iam:team:unlicensed", "unlicensed"), + FieldRef::new("urn:iam:team:roleHolders", "role_holders"), + ], + } +} + +// ------------------------------------------------- the bound populations + +/// Plane indexes in [`Bound::planes`]. +const FOCUS: u16 = 0; +const ROLE: u16 = 1; +const LOC: u16 = 2; +const LIC0: u16 = 3; + +/// The populations a focus binds, for one version, lent as mask planes. +struct Bound<'a> { + n_rows: usize, + planes: Vec<&'a [u64]>, + licenses: usize, +} + +impl Bound<'_> { + fn count(&self, rel: Rel) -> u64 { + let planes = Planes { + n_rows: self.n_rows, + masks: &self.planes, + lanes: &[], + }; + let q = Query { + rel, + measure: Measure::Count, + }; + match run(&q, &planes, None) + .expect("relalg folds the planes") + .answer + { + Answer::Count(n) => n, + a => panic!("not a count: {a:?}"), + } + } +} + +fn licensed_role_outside(b: &Bound) -> Vec { + (0..b.licenses as u16) + .map(|i| { + b.count(Rel::And(vec![ + Rel::Plane(FOCUS), + Rel::Plane(LIC0 + i), + Rel::Plane(ROLE), + Rel::Not(Box::new(Rel::Plane(LOC))), + ])) + }) + .collect() +} + +fn unlicensed(b: &Bound) -> Vec { + let any = Rel::Or( + (0..b.licenses as u16) + .map(|i| Rel::Plane(LIC0 + i)) + .collect(), + ); + vec![b.count(Rel::And(vec![Rel::Plane(FOCUS), Rel::Not(Box::new(any))]))] +} + +fn role_holders(b: &Bound) -> Vec { + vec![b.count(Rel::And(vec![Rel::Plane(FOCUS), Rel::Plane(ROLE)]))] +} + +/// Direct routing: method position -> fold. +type Route = fn(&Bound) -> Vec; +const DR: [Option; 4] = [ + None, + Some(licensed_role_outside), + Some(unlicensed), + Some(role_holders), +]; + +fn call(view: &dyn ClassView, bound: &Bound) -> HashMap> { + let mut out = HashMap::new(); + execute_compute_dag(view.compute_dag(TEAM), &mut out, |out, target| { + let route = DR.get(target as usize).copied().flatten().ok_or(target)?; + out.insert(target, route(bound)); + Ok::<(), u8>(()) + }) + .expect("every declared method has a route"); + out +} + +// ------------------------------------------------------------- the world + +struct Setup { + w: World, + ord: Ordinals, + licenses: Vec, + loc: usize, + team: usize, + role: usize, +} + +fn setup() -> Setup { + let mut spec = Spec::small(); + spec.users = 3000; + spec.licenses = 16; + let w = World::new(Fixture::generate(spec)); + let v = w.view(); + let ord = Ordinals::of(&w.f, &v); + let kinds = |k: Kind| -> Vec { + (0..w.f.groups.len()) + .filter(|&j| w.f.kind[j] == k) + .collect() + }; + let direct_n = |j: usize| w.direct.iter().filter(|&&(_, g)| g as usize == j).count(); + let team = *kinds(Kind::Team) + .iter() + .filter(|&&j| w.f.security[j] == Some(true)) + .max_by_key(|&&j| direct_n(j)) + .expect("a security team"); + let loc = *kinds(Kind::Location) + .iter() + .max_by_key(|&&j| direct_n(j)) + .unwrap(); + let b = w.prepare(); + let qr = Qr::bind(&b, &w.witness()).unwrap(); + let role = (0..64) + .max_by_key(|&r| qr.population(&b, &qr.role_slots[r]).unwrap().count()) + .unwrap(); + let licenses = kinds(Kind::License); + drop(v); + Setup { + w, + ord, + licenses, + loc, + team, + role, + } +} + +/// Per fixture user: effective security groups and roles, from dir-sim. +fn truth(s: &Setup) -> Vec<(u16, HashSet, u64)> { + let v = s.w.view(); + let grants: HashMap = + s.w.f.grants.iter().fold(HashMap::new(), |mut m, &(g, r)| { + *m.entry(s.w.f.groups[g as usize]).or_default() |= 1 << r; + m + }); + s.w.f + .users + .iter() + .enumerate() + .map(|(i, ug)| { + let eff: HashSet = a_groups(&v, ug, Chain::Security).into_iter().collect(); + let roles = eff + .iter() + .fold(0, |a, g| a | grants.get(g).copied().unwrap_or(0)); + (s.ord.user[i], eff, roles) + }) + .collect() +} + +/// The populations a focus binds: role, location, every license. +struct Pops { + role: Pop, + loc: Pop, + lic: Vec, +} + +fn build(s: &Setup, qr: &Qr, b: &Prepared, focus: &TeamFocus, runs: &mut Runs) -> Pops { + let reach = |j: usize| qr.slots_reaching(s.ord.group[j]).unwrap(); + Pops { + role: qr + .population_in(b, &qr.role_slots[s.role], focus, runs) + .unwrap(), + loc: qr.population_in(b, &reach(s.loc), focus, runs).unwrap(), + lic: s + .licenses + .iter() + .map(|&l| qr.population_in(b, &reach(l), focus, runs).unwrap()) + .collect(), + } +} + +fn bound<'a>(n_rows: usize, focus: &'a TeamFocus, p: &'a Pops) -> Bound<'a> { + let mut planes = vec![ + focus.rows.plane(n_rows), + p.role.plane(n_rows), + p.loc.plane(n_rows), + ]; + planes.extend(p.lic.iter().map(|l| l.plane(n_rows))); + Bound { + n_rows, + planes, + licenses: p.lic.len(), + } +} + +#[test] +fn methods_reuse_one_focus_and_its_populations_and_equal_dir_sim() { + let s = setup(); + let b = s.w.prepare(); + let qr = Qr::bind(&b, &s.w.witness()).unwrap(); + let focus = qr.team_focus(&b, s.ord.group[s.team]).unwrap(); + let mut runs = Runs::new(&qr); + let pops = build(&s, &qr, &b, &focus, &mut runs); + let got = call(&team_audit(), &bound(b.users, &focus, &pops)); + + let t = truth(&s); + let g = |j: usize| s.w.f.groups[j]; + let in_focus: Vec<&(u16, HashSet, u64)> = + t.iter().filter(|u| u.1.contains(&g(s.team))).collect(); + let holds_role = |u: &&(u16, HashSet, u64)| u.2 >> s.role & 1 == 1; + let want_lic: Vec = s + .licenses + .iter() + .map(|&l| { + in_focus + .iter() + .filter(|u| u.1.contains(&g(l)) && holds_role(u) && !u.1.contains(&g(s.loc))) + .count() as u64 + }) + .collect(); + let want_unlic = in_focus + .iter() + .filter(|u| s.licenses.iter().all(|&l| !u.1.contains(&g(l)))) + .count() as u64; + let want_role = in_focus.iter().filter(|u| holds_role(u)).count() as u64; + assert_eq!(got[&LICENSED_ROLE_OUTSIDE], want_lic); + assert_eq!(got[&UNLICENSED], vec![want_unlic]); + assert_eq!(got[&ROLE_HOLDERS], vec![want_role]); + + // Anti-vacuity: the focus is a strict subset in few tiles, the role + // filter keeps some members and not all, licenses differ. + assert_eq!(focus.rows.count(), in_focus.len() as u64); + assert!(in_focus.len() > 10 && in_focus.len() < t.len() / 4); + assert!(want_role > 0 && want_role < in_focus.len() as u64); + assert!( + want_lic.iter().collect::>().len() > 2, + "{want_lic:?}" + ); + // Team members are spread over user ordinals (GUID order), so this + // team occupies every tile; the tile restriction is checked on a narrow + // focus below. + // Only the slots the populations name were indexed. + assert!(runs.built > 0 && runs.built < qr.ids.len()); +} + +/// The slot sets of every tracked population at one version: the roles in +/// `roles` (fixed ids, so the order holds across versions), then the named +/// groups. +fn tracked(qr: &Qr, s: &Setup, roles: &[usize]) -> Vec { + let mut v: Vec = roles.iter().map(|&r| qr.role_slots[r]).collect(); + for &j in s.licenses.iter().chain([&s.loc, &s.team]) { + v.push(qr.slots_reaching(s.ord.group[j]).unwrap()); + } + v +} + +fn restrict(p: &Pop, tiles: &[u64]) -> Pop { + let mut q = p.clone(); + q.kept = Some(tiles.to_vec()); + for (i, t) in q.tiles.iter_mut().enumerate() { + if tiles[i >> 6] >> (i & 63) & 1 == 0 { + *t = [0; 4]; + } + } + q +} + +#[test] +fn populations_in_a_narrow_focus_fill_only_its_tiles() { + let s = setup(); + let b = s.w.prepare(); + let qr = Qr::bind(&b, &s.w.witness()).unwrap(); + // The security group with the fewest tiles that still has members. + let focus = qr + .ids + .iter() + .map(|&g| qr.team_focus(&b, g).unwrap()) + .filter(|f| f.rows.count() >= 3) + .min_by_key(TeamFocus::tiles_selected) + .unwrap(); + assert!( + focus.tiles_selected() < qr.pages(), + "a focus narrower than the page" + ); + let mut runs = Runs::new(&qr); + let pops = build(&s, &qr, &b, &focus, &mut runs); + let outside = |p: &Pop| { + p.tiles + .iter() + .enumerate() + .filter(|&(i, _)| focus.tiles[i >> 6] >> (i & 63) & 1 == 0) + .all(|(_, t)| *t == [0; 4]) + }; + assert!(outside(&pops.role) && outside(&pops.loc) && pops.lic.iter().all(outside)); + // Inside the focus's tiles they equal the full population. + let full = qr.population(&b, &qr.role_slots[s.role]).unwrap(); + assert_eq!(pops.role, restrict(&full, &focus.tiles)); + // Anti-vacuity: the role is held outside the focus's tiles, and inside. + assert!(full.count() > pops.role.count() && pops.role.count() > 0); +} + +#[test] +fn repaired_populations_equal_a_fresh_build_across_edit_batches() { + let mut s = setup(); + let sec = |s: &Setup, g: u32| s.w.f.security[g as usize] == Some(true); + let mut b = s.w.prepare(); + let mut qr = Qr::bind(&b, &s.w.witness()).unwrap(); + let focus_tiles = qr.team_focus(&b, s.ord.group[s.team]).unwrap().tiles; + let roles: Vec = (0..64).filter(|&r| qr.role_slots[r] != [0; 4]).collect(); + let mut slots = tracked(&qr, &s, &roles); + let mut full: Vec = slots + .iter() + .map(|x| qr.population(&b, x).unwrap()) + .collect(); + let mut part: Vec = full.iter().map(|p| restrict(p, &focus_tiles)).collect(); + let mut rng = dir_rmap_probe::fixture::Rng::new(0xFEED); + let (mut removed_slots, mut rederived) = (0, 0); + for round in 0..8 { + let n_groups = s.w.f.groups.len(); + let mut edits = Vec::new(); + for _ in 0..3 { + let (u, g) = s.w.direct[rng.below(s.w.direct.len())]; + edits.push(Edit::RemoveDirect(u, g)); + } + let secs: Vec = (0..n_groups as u32).filter(|&g| sec(&s, g)).collect(); + for _ in 0..3 { + edits.push(Edit::AddDirect( + rng.below(s.w.f.users.len()) as u32, + secs[rng.below(secs.len())], + )); + } + let nested_sec: Vec<(u32, u32)> = + s.w.nested + .iter() + .copied() + .filter(|&(c, p)| sec(&s, c) && sec(&s, p)) + .collect(); + for _ in 0..2 { + let e = nested_sec[rng.below(nested_sec.len())]; + edits.push(Edit::RemoveNested(e.0, e.1)); + } + for _ in 0..2 { + let (c, p) = (secs[rng.below(secs.len())], secs[rng.below(secs.len())]); + if c != p { + edits.push(Edit::AddNested(c, p)); + } + } + let touched: Vec = edits + .iter() + .filter_map(|e| match *e { + Edit::AddDirect(u, _) | Edit::RemoveDirect(u, _) => Some(s.ord.user[u as usize]), + _ => None, + }) + .collect(); + s.w.apply(&edits); + let b1 = s.w.prepare(); + let qr1 = Qr::bind(&b1, &s.w.witness()).unwrap(); + let slots1 = tracked(&qr1, &s, &roles); + for i in 0..full.len() { + let r = full[i] + .repair(&qr, &qr1, &b1, (&slots[i], &slots1[i]), &touched, None) + .unwrap(); + removed_slots += r.slots_removed; + rederived += r.rederived; + part[i] + .repair( + &qr, + &qr1, + &b1, + (&slots[i], &slots1[i]), + &touched, + Some(&focus_tiles), + ) + .unwrap(); + let fresh = qr1.population(&b1, &slots1[i]).unwrap(); + assert_eq!(full[i], fresh, "round {round}, population {i}"); + assert_eq!( + part[i], + restrict(&fresh, &focus_tiles), + "round {round}, focus {i}" + ); + } + (b, qr, slots) = (b1, qr1, slots1); + } + let _ = &b; + // Anti-vacuity: slots lost properties and users were re-derived. + assert!(removed_slots > 0, "no slot ever lost a property"); + assert!(rederived > 0); +} + +#[test] +fn a_second_path_keeps_the_property_when_one_path_is_removed() { + let mut s = setup(); + let b = s.w.prepare(); + let qr = Qr::bind(&b, &s.w.witness()).unwrap(); + let r = s.role; + let carries = |qr: &Qr, j: usize| { + let slot = qr.slot(s.ord.group[j]).unwrap(); + qr.role_slots[r][slot >> 6] >> (slot & 63) & 1 == 1 + }; + let fixture_of = |ordinal: u16| s.ord.group.iter().position(|&o| o == ordinal).unwrap(); + // Two parents carrying role r, a third role group for a user-level + // second path, and a child group with members that does not carry r. + let carriers: Vec = qr + .ids + .iter() + .map(|&g| fixture_of(g)) + .filter(|&j| carries(&qr, j)) + .collect(); + let (p1, p2, q) = (carriers[0], carriers[1], carriers[2]); + let c = qr + .ids + .iter() + .map(|&g| fixture_of(g)) + .filter(|&j| !carries(&qr, j) && s.w.f.kind[j] == Kind::Team) + .max_by_key(|&j| s.w.direct.iter().filter(|&&(_, g)| g as usize == j).count()) + .expect("a security team without the role"); + let members: Vec = + s.w.direct + .iter() + .filter(|&&(_, g)| g as usize == c) + .map(|&(u, _)| u) + .collect(); + let u0 = members[0]; + s.w.apply(&[ + Edit::AddNested(c as u32, p1 as u32), + Edit::AddNested(c as u32, p2 as u32), + Edit::AddDirect(u0, q as u32), + ]); + let b1 = s.w.prepare(); + let qr1 = Qr::bind(&b1, &s.w.witness()).unwrap(); + let mut pop = qr1.population(&b1, &qr1.role_slots[r]).unwrap(); + let before = pop.count(); + + // Remove one of two parents: c still carries r. + s.w.apply(&[Edit::RemoveNested(c as u32, p1 as u32)]); + let b2 = s.w.prepare(); + let qr2 = Qr::bind(&b2, &s.w.witness()).unwrap(); + assert!( + carries(&qr2, c), + "second parent keeps the role on the group" + ); + pop.repair( + &qr1, + &qr2, + &b2, + (&qr1.role_slots[r], &qr2.role_slots[r]), + &[], + None, + ) + .unwrap(); + assert_eq!(pop, qr2.population(&b2, &qr2.role_slots[r]).unwrap()); + assert_eq!(pop.count(), before); + + // Remove the second parent: c loses r, u0 keeps it through q. + s.w.apply(&[Edit::RemoveNested(c as u32, p2 as u32)]); + let b3 = s.w.prepare(); + let qr3 = Qr::bind(&b3, &s.w.witness()).unwrap(); + assert!(!carries(&qr3, c)); + let rep = pop + .repair( + &qr2, + &qr3, + &b3, + (&qr2.role_slots[r], &qr3.role_slots[r]), + &[], + None, + ) + .unwrap(); + assert!(rep.slots_removed > 0 && rep.rederived > 0); + let fresh = qr3.population(&b3, &qr3.role_slots[r]).unwrap(); + assert_eq!(pop, fresh); + assert!(pop.count() < before, "some member of c lost the role"); + let bit = + |p: &Pop, u: u16| p.tiles[u as usize / 256][(u as usize % 256) >> 6] >> (u % 64) & 1 == 1; + assert!( + bit(&pop, s.ord.user[u0 as usize]), + "u0 keeps the role through its second path" + ); + // And against dir-sim. + let t = truth(&s); + for (o, _, roles) in &t { + assert_eq!(bit(&pop, *o), roles >> r & 1 == 1, "user ordinal {o}"); + } +} + +#[test] +fn repair_refuses_a_tile_restriction_other_than_the_one_built() { + // Codex review on #1666: a population restricted to some tiles, repaired + // with a different mask, kept the old tiles and stamped the new version. + let mut s = setup(); + let b0 = s.w.prepare(); + let qr0 = Qr::bind(&b0, &s.w.witness()).unwrap(); + let focus = qr0 + .ids + .iter() + .map(|&g| qr0.team_focus(&b0, g).unwrap()) + .filter(|f| f.rows.count() >= 3) + .min_by_key(TeamFocus::tiles_selected) + .unwrap(); + let x0 = qr0.role_slots[s.role]; + let mut pop = qr0 + .population_in(&b0, &x0, &focus, &mut Runs::new(&qr0)) + .unwrap(); + let before = pop.clone(); + // Another mask: every tile the focus does not select. + let other: Vec = focus.tiles.iter().map(|w| !w).collect(); + let (u, g) = s.w.direct[0]; + s.w.apply(&[Edit::RemoveDirect(u, g)]); + let b1 = s.w.prepare(); + let qr1 = Qr::bind(&b1, &s.w.witness()).unwrap(); + let x1 = qr1.role_slots[s.role]; + assert_eq!( + pop.repair(&qr0, &qr1, &b1, (&x0, &x1), &[], Some(&other)) + .err(), + Some(Refusal::OutOfFocus) + ); + assert_eq!( + pop, before, + "a refused repair leaves the population as it was" + ); + // Unrestricted repair of a restricted population is refused too. + assert_eq!( + pop.repair(&qr0, &qr1, &b1, (&x0, &x1), &[], None).err(), + Some(Refusal::OutOfFocus) + ); + // The restriction it was built with is accepted. + pop.repair(&qr0, &qr1, &b1, (&x0, &x1), &[], Some(&focus.tiles)) + .unwrap(); +} + +#[test] +fn a_mismatched_version_is_refused() { + let mut s = setup(); + let b0 = s.w.prepare(); + let qr0 = Qr::bind(&b0, &s.w.witness()).unwrap(); + let mut runs0 = Runs::new(&qr0); + let focus0 = qr0.team_focus(&b0, s.ord.group[s.team]).unwrap(); + let p0 = qr0.population(&b0, &qr0.role_slots[s.role]).unwrap(); + let (u, g) = s.w.direct[0]; + s.w.apply(&[Edit::RemoveDirect(u, g)]); + let b1 = s.w.prepare(); + let qr1 = Qr::bind(&b1, &s.w.witness()).unwrap(); + let p1 = qr1.population(&b1, &qr1.role_slots[s.role]).unwrap(); + assert_eq!(p0.clone().and(&p1).err(), Some(Refusal::Stale)); + assert_eq!(p0.count_and(&p1).err(), Some(Refusal::Stale)); + // Pointer tables and focus from the old version. + let x = qr1.role_slots[s.role]; + assert_eq!( + qr1.population_in(&b1, &x, &focus0, &mut Runs::new(&qr1)) + .err(), + Some(Refusal::Stale) + ); + let focus1 = qr1.team_focus(&b1, s.ord.group[s.team]).unwrap(); + assert_eq!( + qr1.population_in(&b1, &x, &focus1, &mut runs0).err(), + Some(Refusal::Stale) + ); + // A repair whose "before" register is not the population's version. + let mut q = p0.clone(); + assert_eq!( + q.repair(&qr1, &qr1, &b1, (&x, &x), &[], None).err(), + Some(Refusal::Stale) + ); +} diff --git a/crates/dir-rmap-probe/tests/qr_license.rs b/crates/dir-rmap-probe/tests/qr_license.rs new file mode 100644 index 000000000..584966fcd --- /dev/null +++ b/crates/dir-rmap-probe/tests/qr_license.rs @@ -0,0 +1,118 @@ +//! Arm Q, chained population masking: license groups filtered over every +//! user's memberships. Each filter is a population (one bit per user, +//! positioned by tile and row); the chain is AND / AND-NOT by position and +//! the answer is a count. Every count must equal a per-user check against +//! dir-sim's own walk of effective groups. + +use dir_rmap_probe::fixture::{Fixture, Kind, Spec}; +use dir_rmap_probe::prep::{Chain, Ordinals}; +use dir_rmap_probe::qr::Qr; +use dir_rmap_probe::{World, a_groups}; +use ogar_dir_core::Guid128; +use std::collections::{HashMap, HashSet}; + +#[test] +fn license_counts_by_chained_masks_equal_dir_sim() { + let mut spec = Spec::small(); + spec.users = 3000; + spec.licenses = 16; + let w = World::new(Fixture::generate(spec)); + let v = w.view(); + let ord = Ordinals::of(&w.f, &v); + let b = w.prepare(); + let qr = Qr::bind(&b, &w.witness()).unwrap(); + let of_kind = |k: Kind| -> Vec { + (0..w.f.groups.len()) + .filter(|&j| w.f.kind[j] == k) + .collect() + }; + let licenses = of_kind(Kind::License); + let locations = of_kind(Kind::Location); + assert_eq!(licenses.len(), 16); + + // Per user, from dir-sim: effective security groups and roles. + let grants: HashMap = w.f.grants.iter().fold(HashMap::new(), |mut m, &(g, r)| { + *m.entry(w.f.groups[g as usize]).or_default() |= 1 << r; + m + }); + let users: Vec<(u16, HashSet, u64)> = + w.f.users + .iter() + .enumerate() + .map(|(i, ug)| { + let eff: HashSet = a_groups(&v, ug, Chain::Security).into_iter().collect(); + let roles = eff + .iter() + .fold(0, |a, g| a | grants.get(g).copied().unwrap_or(0)); + (ord.user[i], eff, roles) + }) + .collect(); + + // The filters: a widely held role, and the most populated location. + let role = (0..64) + .max_by_key(|&r| users.iter().filter(|u| u.2 >> r & 1 == 1).count()) + .unwrap(); + let loc = *locations + .iter() + .max_by_key(|&&j| { + users + .iter() + .filter(|u| u.1.contains(&w.f.groups[j])) + .count() + }) + .unwrap(); + let pop_of = |j: usize| { + qr.population(&b, &qr.slots_reaching(ord.group[j]).unwrap()) + .unwrap() + }; + let role_pop = qr.population(&b, &qr.role_slots[role]).unwrap(); + let loc_pop = pop_of(loc); + // role holders outside the location: the chain every license is + // filtered through. + // Anti-vacuity: the location removes some role holders. + assert!(role_pop.count_and(&loc_pop).unwrap() > 0); + let chain = role_pop.and_not(&loc_pop).unwrap(); + + let mut any = qr.population(&b, &[0; 4]).unwrap(); + let (mut nonzero, mut distinct) = (0, HashSet::new()); + for &l in &licenses { + let lp = pop_of(l); + let got = lp.count_and(&chain).unwrap(); + let want = users + .iter() + .filter(|(_, eff, roles)| { + eff.contains(&w.f.groups[l]) + && roles >> role & 1 == 1 + && !eff.contains(&w.f.groups[loc]) + }) + .count() as u64; + assert_eq!(got, want, "license {l}"); + // The same answer by building the chain explicitly. + assert_eq!(lp.clone().and(&chain).unwrap().count(), want); + nonzero += usize::from(got > 0); + distinct.insert(got); + for (a, t) in any.tiles.iter_mut().zip(&lp.tiles) { + for k in 0..4 { + a[k] |= t[k]; + } + } + } + // Users with no license at all: everyone AND-NOT any license. + let unlicensed = qr.everyone().and_not(&any).unwrap().count(); + let want = users + .iter() + .filter(|(_, eff, _)| licenses.iter().all(|&l| !eff.contains(&w.f.groups[l]))) + .count() as u64; + assert_eq!(unlicensed, want); + assert_eq!(qr.everyone().count(), b.users as u64); + // Anti-vacuity: the chain filters something, licenses differ, and some + // users are unlicensed. + assert!(chain_filters(&qr, &b, role)); + assert!(nonzero > 8 && distinct.len() > 4, "{nonzero} {distinct:?}"); + assert!(unlicensed > 0 && unlicensed < b.users as u64); +} + +fn chain_filters(qr: &Qr, b: &dir_rmap_probe::prep::Prepared, role: usize) -> bool { + let n = qr.population(b, &qr.role_slots[role]).unwrap().count(); + n > 0 && n < b.users as u64 +} diff --git a/crates/dir-rmap-probe/tests/qr_masked.rs b/crates/dir-rmap-probe/tests/qr_masked.rs new file mode 100644 index 000000000..257d77cbd --- /dev/null +++ b/crates/dir-rmap-probe/tests/qr_masked.rs @@ -0,0 +1,140 @@ +//! Arm Q, masked count over several tiles: selecting tiles (pages of 256 +//! users) with a tile mask and rows inside each tile with a row mask must +//! count exactly the users a page-by-page resolve finds, for roles and for +//! effective group membership. + +use dir_rmap_probe::World; +use dir_rmap_probe::fixture::{Fixture, Spec}; +use dir_rmap_probe::focus::Row; +use dir_rmap_probe::qr::{Pages, Qr, Runs}; + +const ALL: Row = [!0; 4]; +const EVEN: Row = [0x5555_5555_5555_5555; 4]; +const ODD: Row = [0xAAAA_AAAA_AAAA_AAAA; 4]; + +fn bit(r: &Row, i: usize) -> bool { + r[i >> 6] >> (i & 63) & 1 == 1 +} + +/// Tile masks: one tile, every other tile, every third tile, all tiles. +fn tile_masks(pages: usize) -> Vec> { + let words = pages.div_ceil(64); + let by = |f: &dyn Fn(usize) -> bool| { + let mut m = vec![0u64; words]; + for p in (0..pages).filter(|&p| f(p)) { + m[p >> 6] |= 1 << (p & 63); + } + m + }; + vec![ + by(&|p| p == pages / 2), + by(&|p| p % 2 == 0), + by(&|p| p % 3 == 1), + by(&|_| true), + ] +} + +/// Per-page row masks: all rows, even rows, odd rows, or alternating by +/// page (interlaced activation). +fn row_masks(pages: usize) -> Vec> { + vec![ + vec![ALL; pages], + vec![EVEN; pages], + vec![ODD; pages], + (0..pages) + .map(|p| if p % 2 == 0 { EVEN } else { ODD }) + .collect(), + ] +} + +/// Does a user hold the answer, given its roles and effective row? +type Holds = Box bool>; + +#[test] +fn masked_counts_equal_a_page_by_page_resolve() { + let mut spec = Spec::small(); + spec.users = 3000; + let w = World::new(Fixture::generate(spec)); + let b = w.prepare(); + let qr = Qr::bind(&b, &w.witness()).unwrap(); + let mut pages = Pages::new(&qr); + let np = qr.pages(); + let mut scratch = vec![[0u64; 4]; np]; + let mut runs = Runs::new(&qr); + + // Questions: every role, and membership of a few security groups. + let mut questions: Vec<(Row, Holds)> = Vec::new(); + for r in 0..64u32 { + if qr.role_slots[r as usize] != [0; 4] { + questions.push(( + qr.role_slots[r as usize], + Box::new(move |roles: u64, _: &Row| roles >> r & 1 == 1), + )); + } + } + for &g in qr.ids.iter().step_by(17) { + let t = qr.slot(g).unwrap(); + questions.push(( + qr.slots_reaching(g).unwrap(), + Box::new(move |_: u64, eff: &Row| bit(eff, t)), + )); + } + assert!(questions.len() > 10); + + let (mut nonzero, mut checked) = (0, 0); + for (slots, holds) in &questions { + for tiles in tile_masks(np) { + for rows in row_masks(np) { + let got = qr + .count_masked(&b, slots, &tiles, &rows, &mut scratch) + .unwrap(); + let mut want = 0u64; + for p in 0..np { + if tiles[p >> 6] >> (p & 63) & 1 == 0 { + continue; + } + for i in 0..256 { + let u = p * 256 + i; + if u >= b.users || !bit(&rows[p], i) { + continue; + } + let page = pages.of(&qr, &b, u as u16).unwrap(); + let eff = qr.effective(&page.member[i]); + want += u64::from(holds(page.roles[i], &eff)); + } + } + assert_eq!(got, want); + assert_eq!( + runs.count(&qr, &b, slots, &tiles, &rows).unwrap(), + want, + "runs" + ); + assert!(scratch.iter().all(|t| *t == [0; 4]), "scratch left dirty"); + nonzero += usize::from(got != 0); + checked += 1; + } + } + } + // Anti-vacuity: most combinations select somebody. + assert!(nonzero * 2 > checked, "{nonzero} of {checked}"); +} + +#[test] +fn a_stale_snapshot_refuses_the_masked_count() { + let mut w = World::new(Fixture::generate(Spec::small())); + let b = w.prepare(); + let qr = Qr::bind(&b, &w.witness()).unwrap(); + let (u, g) = w.direct[0]; + w.apply(&[dir_rmap_probe::Edit::RemoveDirect(u, g)]); + let b2 = w.prepare(); + let np = qr.pages(); + let mut scratch = vec![[0u64; 4]; np]; + let r = qr.count_masked( + &b2, + &ALL, + &vec![!0; np.div_ceil(64)], + &vec![ALL; np], + &mut scratch, + ); + assert_eq!(r, Err(dir_rmap_probe::prep::Refusal::Stale)); +} diff --git a/crates/dir-rmap-probe/tests/qr_register.rs b/crates/dir-rmap-probe/tests/qr_register.rs new file mode 100644 index 000000000..941a42a89 --- /dev/null +++ b/crates/dir-rmap-probe/tests/qr_register.rs @@ -0,0 +1,266 @@ +//! Arm Q: IAM nested groups as a DR256 register. +//! +//! Every user's roles and effective security groups, read from the +//! users x groups bit matrix and the closure computed inside the 256 x 256 +//! tile, must equal dir-sim (arm A), the crepe oracle (arm D) and the +//! compiled closure (arm B+), before and after edits. The register refuses a +//! chain it cannot hold whole, and a stale snapshot. +//! +//! The ClassView wiring is the same as `classview_dr.rs`: the class declares +//! `effective_roles` computed from `member_of`, and the DR table maps that +//! position to `Page::roles_of` on the user's page, resolved on first use. + +use dir_rmap_probe::fixture::{Fixture, Spec}; +use dir_rmap_probe::prep::{Chain, Compiled, Ordinals, Refusal}; +use dir_rmap_probe::qr::{Page, Pages, Qr}; +use dir_rmap_probe::{Edit, World, a_groups, oracle, templates}; +use lance_graph_contract::class_view::{ClassId, ClassView, ComputeEdge, execute_compute_dag}; +use lance_graph_contract::ontology::{DisplayTemplate, FieldRef}; +use std::collections::HashMap; + +const USER: ClassId = 1; +const MEMBER_OF: u8 = 0; +const EFFECTIVE_ROLES: u8 = 1; + +struct IamUser { + fields: Vec, +} + +static ROLES_FROM_MEMBERSHIP: [ComputeEdge; 1] = [ComputeEdge { + target: EFFECTIVE_ROLES, + inputs: &[MEMBER_OF], +}]; + +impl ClassView for IamUser { + fn fields(&self, class: ClassId) -> &[FieldRef] { + if class == USER { &self.fields } else { &[] } + } + fn template(&self, _: ClassId) -> DisplayTemplate { + DisplayTemplate::Detail + } + fn dolce_category_id(&self, _: ClassId) -> u8 { + 0 + } + fn compute_dag(&self, class: ClassId) -> &[ComputeEdge] { + if class == USER { + &ROLES_FROM_MEMBERSHIP + } else { + &[] + } + } +} + +/// Direct routing over a page: position -> executable. +type Route = fn(&Page, usize) -> u64; +const DR: [Option; 2] = [None, Some(Page::roles_of)]; + +fn run(view: &dyn ClassView, page: &Page, i: usize) -> Option { + let mut roles = None; + execute_compute_dag(view.compute_dag(USER), &mut roles, |out, target| { + let route = DR.get(target as usize).copied().flatten().ok_or(target)?; + *out = Some(route(page, i)); + Ok::<(), u8>(()) + }) + .expect("every declared target has a route"); + roles +} + +fn view() -> IamUser { + IamUser { + fields: vec![ + FieldRef::new("urn:iam:user:memberOf", "member_of"), + FieldRef::new("urn:iam:user:effectiveRoles", "effective_roles"), + ], + } +} + +/// Checks every user against arms A, D and B+. Returns how many users hold a +/// role and how many (user, group) effective memberships were compared. +fn check(w: &World) -> (usize, usize) { + let b = w.prepare(); + let qr = Qr::bind(&b, &w.witness()).expect("security chain fits 256 slots"); + let mut pages = Pages::new(&qr); + let compiled = Compiled::compile(w.prepare()); + let v = w.view(); + let ord = Ordinals::of(&w.f, &v); + let d = oracle::run(&w.f, &w.direct, &w.nested, &templates(&w.f)); + let mut d_roles = vec![0u64; w.f.users.len()]; + for &(u, role) in &d.roles { + d_roles[u as usize] |= 1 << role; + } + let grants: HashMap<_, u64> = w.f.grants.iter().fold(HashMap::new(), |mut m, &(g, r)| { + *m.entry(w.f.groups[g as usize]).or_default() |= 1 << r; + m + }); + let group_ord: HashMap<_, u16> = + w.f.groups + .iter() + .enumerate() + .map(|(i, g)| (*g, ord.group[i])) + .collect(); + let iam = view(); + let (mut holders, mut compared) = (0, 0); + for (i, ug) in w.f.users.iter().enumerate() { + let u = ord.user[i]; + let a = a_groups(&v, ug, Chain::Security); + let a_roles = a + .iter() + .fold(0, |acc, g| acc | grants.get(g).copied().unwrap_or(0)); + let page = pages.of(&qr, &b, u).unwrap(); + let got = run(&iam, page, u as usize % 256).expect("declared"); + let row = page.member[u as usize % 256]; + assert_eq!(qr.roles_live(&row), got, "row x roles, user {i}"); + assert_eq!(got, a_roles, "dir-sim roles, user {i}"); + assert_eq!(got, d_roles[i], "oracle roles, user {i}"); + assert_eq!(got, compiled.roles_of(u), "B+ roles, user {i}"); + holders += usize::from(got != 0); + // Effective security groups: the matrix row times reach, slot by + // slot, against dir-sim's walk. + let eff = qr.effective(&row); + let a_set: Vec = a.iter().map(|g| group_ord[g]).collect(); + for (s, &g) in qr.ids.iter().enumerate() { + let here = eff[s >> 6] >> (s & 63) & 1 == 1; + assert_eq!(here, a_set.contains(&g), "user {i}, group ordinal {g}"); + compared += 1; + } + } + assert_eq!(pages.resolved, qr.pages(), "every page asked for once"); + (holders, compared) +} + +#[test] +fn the_register_answers_every_user_like_dir_sim_the_oracle_and_b_plus() { + let w = World::new(Fixture::generate(Spec::small())); + let (holders, compared) = check(&w); + // Anti-vacuity: roles held by some users and not all; nesting carries + // at least one membership the direct rows do not. + assert!( + holders > 0 && holders < w.f.users.len(), + "holders {holders}" + ); + assert!(compared > 0); + let b = w.prepare(); + let qr = Qr::bind(&b, &w.witness()).unwrap(); + let mut indirect = 0; + for p in 0..qr.pages() { + let page = qr.page(&b, p).unwrap(); + for row in &page.member { + indirect += usize::from(qr.effective(row) != *row); + } + } + assert!(indirect > 0, "no user gains a group through nesting"); +} + +#[test] +fn edits_are_answered_by_a_rebind() { + let mut w = World::new(Fixture::generate(Spec::small())); + check(&w); + let sec = |w: &World, g: u32| w.f.security[g as usize] == Some(true); + // A new nesting pair between two security groups, and a membership + // removed: both change effective groups for someone. + let (c, p) = (0..w.f.groups.len() as u32) + .flat_map(|c| (0..w.f.groups.len() as u32).map(move |p| (c, p))) + .find(|&(c, p)| c != p && sec(&w, c) && sec(&w, p) && !w.nested.contains(&(c, p))) + .expect("two unnested security groups"); + let (u, g) = *w.direct.iter().find(|&&(_, g)| sec(&w, g)).unwrap(); + w.apply(&[Edit::AddNested(c, p), Edit::RemoveDirect(u, g)]); + check(&w); +} + +#[test] +fn a_stale_register_refuses() { + let mut w = World::new(Fixture::generate(Spec::small())); + let b = w.prepare(); + let qr = Qr::bind(&b, &w.witness()).unwrap(); + let mut pages = Pages::new(&qr); + let g = qr.ids[0]; + assert!(pages.ask(&qr, &b, &w.witness(), 0, g).is_ok()); + let (u, gd) = w.direct[0]; + w.apply(&[Edit::RemoveDirect(u, gd)]); + assert_eq!(pages.ask(&qr, &b, &w.witness(), 0, g), Err(Refusal::Stale)); + assert_eq!(Qr::bind(&b, &w.witness()).err(), Some(Refusal::Stale)); + // A page cannot be resolved from a snapshot the register was not bound + // to. + let b2 = w.prepare(); + assert_eq!(qr.page(&b2, 0).err(), Some(Refusal::Stale)); +} + +#[test] +fn a_chain_larger_than_the_register_is_refused_not_truncated() { + let w = World::new(Fixture::generate(Spec::iam(2000))); + let b = w.prepare(); + let security: u32 = b.security.iter().map(|x| x.count_ones()).sum(); + assert!( + security > 256, + "fixture must exceed the register: {security}" + ); + assert_eq!(Qr::bind(&b, &w.witness()).err(), Some(Refusal::TooLarge)); +} + +#[test] +fn a_group_outside_the_security_chain_is_out_of_focus() { + let w = World::new(Fixture::generate(Spec::small())); + let b = w.prepare(); + let qr = Qr::bind(&b, &w.witness()).unwrap(); + let g = (0..b.groups as u16) + .find(|&g| qr.slot(g).is_none()) + .expect("a non-security group"); + let mut pages = Pages::new(&qr); + assert_eq!( + pages.ask(&qr, &b, &w.witness(), 0, g), + Err(Refusal::OutOfFocus) + ); +} + +#[test] +fn binding_resolves_no_user_and_a_question_resolves_one_page() { + let mut spec = Spec::small(); + spec.users = 2000; + let w = World::new(Fixture::generate(spec)); + let b = w.prepare(); + let qr = Qr::bind(&b, &w.witness()).unwrap(); + let mut pages = Pages::new(&qr); + assert_eq!(pages.resolved, 0); + assert!(qr.pages() > 2); + // Two users of one page, then one of the last page. + pages.roles_of(&qr, &b, 3).unwrap(); + pages.roles_of(&qr, &b, 200).unwrap(); + assert_eq!(pages.resolved, 1); + pages.roles_of(&qr, &b, (b.users - 1) as u16).unwrap(); + assert_eq!(pages.resolved, 2); +} + +#[test] +fn a_page_cache_from_an_older_version_refuses() { + // Codex review on #1666: resolve a user's page, remove that user's + // membership, then ask through the same cache with the new register. + let mut w = World::new(Fixture::generate(Spec::small())); + let b0 = w.prepare(); + let qr0 = Qr::bind(&b0, &w.witness()).unwrap(); + let mut pages = Pages::new(&qr0); + let v = w.view(); + let ord = Ordinals::of(&w.f, &v); + drop(v); + let (u, g) = *w + .direct + .iter() + .find(|&&(_, g)| qr0.slot(ord.group[g as usize]).is_some()) + .unwrap(); + let (uo, go) = (ord.user[u as usize], ord.group[g as usize]); + assert_eq!(pages.ask(&qr0, &b0, &w.witness(), uo, go), Ok(true)); + w.apply(&[Edit::RemoveDirect(u, g)]); + let b1 = w.prepare(); + let qr1 = Qr::bind(&b1, &w.witness()).unwrap(); + assert_eq!( + pages.ask(&qr1, &b1, &w.witness(), uo, go), + Err(Refusal::Stale), + "a cached page must not answer for another version" + ); + assert_eq!(pages.roles_of(&qr1, &b1, uo).err(), Some(Refusal::Stale)); + // A cache bound to the new register answers. + assert!( + Pages::new(&qr1) + .ask(&qr1, &b1, &w.witness(), uo, go) + .is_ok() + ); +}