Plan the ProductZipper's factor order from the trie, behind an experimental conjunct_order feature

[MesTTo]

Plan the ProductZipper's factor order from the trie, behind an experimental conjunct_order feature

Space::query_multi hands the conjunction's arguments to the ProductZipper descent in the
order they were written, and that descent order IS the join order of a nested-loop join. A
body written star-first pays the star: bench clique writes its three-, four- and five-clique
bodies with every edge out of $x0 first and defers all the cross edges until every vertex is
bound. This is the shape you diagnosed by hand on chaining's bfc-xp -- 220M transitions for
20K unifications, fixed by switching the join order -- with nothing choosing the order.

The gain is asymptotic, not a factor, and mork bench clique_scaling is here so that is
checkable rather than asserted. It holds a random graph's average degree at 10, grows it, and
prints transitions per size; run it once on a default build and once with the feature:

four-clique |E| = 300 800 1300 2100 3400
written 111,162 316,778 537,529 892,230 1,492,176
planned 10,486 23,473 35,776 56,136 87,746
ratio 10.6x 13.5x 15.0x 15.9x 17.0x

five-clique |E| = 300 800 1300 2100 3400
written 911,195 2,894,603 5,024,101 8,653,338 15,064,427
planned 15,687 28,618 41,984 62,158 93,783
ratio 58.1x 101.1x 119.7x 139.2x 160.6x

Least squares over those five sizes puts the four-clique body's written order at |E|^1.07 and
the planned order at |E|^0.88; five-clique, |E|^1.15 against |E|^0.74. The ratio widens with
the input rather than settling. Answer counts are identical at every point; this is search the
descent no longer does.

REVIEW SHAPE. The routing is behind an experimental conjunct_order feature, default off, the
same shape as the leapfrog dispatch. Without the feature neither this module nor leapfrog
compiles, query_multi_dispatch calls query_multi exactly as it does today, and cargo test
collects the same zero tests it collects on main. Against 06cdcf3 the default build measures
within 0.015% on every one of the twelve benchmarks and 0.0005% on the suite total, with every
output identical, which is the whole of its claim.

One thing in the default build's path did have to change to keep that true. query_multi_raw
forwards to the two-source-list form by handing it the same slice twice, and the compiler cannot
prove two &[ExprEnv] alias, so it kept both pointers live across the descent's hot closure:
bench transitive, which is nothing but that closure, measured +0.033% on a build where neither
this module nor leapfrog is compiled at all. query_multi_raw_sourced takes the identity as a
const parameter instead, in the same shape as MAY_REORDER, so the true instantiation folds
the branch and the second pointer stops existing. That is the -0.005% row below.

THE MODEL. System R (Selinger et al., SIGMOD 1979), and it needs both halves of that paper's
model rather than only the famous one.

The cardinality half estimates how many partial matches reach each step, under uniformity and
independence: a variable's second and later occurrences are equalities that divide out one
domain apiece. The ACCESS PATH half estimates what a step then costs, and that is where a
byte-trie parts company with a relational store. A PathMap indexes prefixes only, so a factor
is entered by seeking its leading columns and scanning the rest: a variable bound in a leading
column makes the step selective, while the same variable bound in a later column only rejects
rows the descent has already walked. leapfrog meets the same fact from the other side --
is_inverted detects a factor whose bound variable is not in leading position, and pays to
re-index it into a private map.

Charging only the cardinality half is not a small error, and it is not a conservative one. On a
3,400-edge graph of average degree 10 the four-clique body's written order walks 1,538,745
transitions; cardinality alone prefers an order walking 4,346,487, which is 2.8x WORSE than
doing nothing, and the error grows with |V| because the conditional domain it reads is a degree
where the marginal is a vertex count. With the access-path term the same body walks 87,537.

The access path has a second edge here that a relational store does not have: A COLUMN CAN HOLD
A VARIABLE. A stored variable unifies with every value, so a query variable read out of such a
column may itself be a variable, and the factor entered next by seeking on it has no key to
seek and scans. This is the degradation first-argument indexing has in a WAM, where a clause
with a variable head argument belongs in every bucket and the index stops discriminating,
carried into the retrieval-as-join setting of Riazanov and Voronkov, "Efficient Instance
Retrieval with Standard and Relational Path Indexing", CADE-19, LNAI 2741, pages 380-396, 2003.

Leaving it out is not a bounded error either, which is why it is priced rather than ignored. Add
one (root $r) beside three ground rows of the same table and the four-conjunct body above is
planned into an order walking 27x the written order's transitions at 300 edges -- and 56x, 116x,
235x, 463x as the graph grows, so the loss widens with the input exactly as the win does.
a_relation_holding_variables_does_not_mislead_the_planner measures both sizes and fails on a
model that leaves this out; differential/corpus/programs/plan_variable_column.mm2 pins the
answers.

The search is the same paper's subset dynamic program. Each step's cost depends only on the
subset before it and the factor appended, so it stays an exact minimisation over the model
rather than a heuristic over it, and it gives way to a greedy order on the same cost function
past twelve conjuncts.

WHERE THE NUMBERS COME FROM. There are no materialised relations and no collected histograms
here, so every count is enumerated off the live byte-trie when a body is first planned. That is
the part you said a satisfying solution needed and did not have. It is admitted in stages, each
refusing to spend where the answer is already known:

• a body whose reordering cannot change its asymptotics is refused from the body alone, with
  no trie access at all: GYO reduction says it is alpha-acyclic, and no prefix of the written
  order is disconnected from the factor that follows it;
• a measured body whose written order walks fewer rows than searching over its orders costs
  keeps that order, on a floor that scales with the 2^n the search would visit;
• the wide reading, which unions a column's domain over several values of the column before
  it, is bought only when the cheap reading leaves the decision inside a band where a better
  measurement could still move it;
• the margin a plan must beat is derived from how much of the reading was cut short rather
  than fixed, so an exactly-counted body is charged no scepticism;
• a run of refusals makes the planner skip a doubling run of bodies, which one acceptance
  clears. bench counter_machine fires 1,813 transforms over 1,368 distinct bodies and
  refuses every one;
• decisions are remembered in a fixed-size set-associative decaying table keyed by a bounded
  prefix of the body, and a refusal is cached as an answer. Collisions are accepted: every
  entry is a permutation and the factor count is checked, so a collision can cost a worse
  order and never a wrong one.

Two ceilings bound one measurement. COLUMN_CAP bounds how many values a column is read for;
COLUMN_BYTE_CAP bounds how many BYTES that walk covers, which is what it actually spends,
since a cursor step walks the value's own bytes. bench exponential_fringe builds exponentially
nested terms whose widest column walked 129,896 bytes under the value ceiling alone.

APPLYING A PLAN re-encodes the factors into the planned order with their variables renumbered
to it, and descends those while unifying against the ORIGINAL factors, so bindings still come
out keyed in the pattern's own namespace and the template instantiation downstream is untouched.
That is what query_multi_raw_with_unification_sources is for. Nothing here re-derives a tag
byte or re-implements unification; the body decomposition, the trie cursor and the unifier are
leapfrog's and mork_expr's, and join order is orthogonal to all three.

Reordering changes which answer arrives first, so it is offered only through
query_multi_planned, which query_multi_dispatch takes -- the space-to-space transform's
entry, and already the point where the leapfrog feature substitutes a different join with a
different order. The sink and source transforms keep query_multi.

bench clique_scaling is the one addition outside the feature. It is a benchmark, not a code
path: bench all does not run it, and it exists so the exponents above can be reproduced
rather than taken on trust. Adding it hoisted clique_query and the graph construction out of
bench_clique_no_unify, which both now share; bench clique's output is unchanged.

[adamv-symbolica]

Right, a few different features are gated by more runtime information in the trie. This is not a feature we want to add until that is merged in (already being worked on), and it can also seriously regress on hand-coded order in its heuristics form, which is not acceptable for a non-gated solution.

[MesTTo]

Rechecked this end to end before you got to it and found something I would rather hand you than
have you find: the cost model had a second asymptotic hole. Fixed, and force-pushed, so the
diff you would have read is not the one that is there now. Still two commits.

What was wrong

A column in a term store can hold a VARIABLE, and a relational cost model has no term for that.
A stored variable unifies with everything, so a query variable read out of such a column may
itself BE a variable, and the factor entered next by seeking on it has no key to seek. It scans.

Put one (root $r) beside three ground rows of a four-row table and the planner takes an order
walking 27x the transitions the written order does. Growing the graph at constant average degree:

|E|	300	600	1200	2400	4800
planned / written, before	27.1x	56.4x	115.7x	235.0x	462.7x
planned / written, after	1.23x	1.16x	1.14x	1.16x	1.15x

Not a constant, which by your own filter makes it something to fix rather than note. Answers were
correct throughout -- every differential passed at every stage, including on the shape that was
463x off -- so no oracle here could have caught it. Only a scaling sweep did.

This is first-argument indexing degrading in a WAM, where a clause with a variable head argument
belongs in every bucket and the index stops discriminating, carried into the retrieval-as-join
setting of Riazanov and Voronkov, Efficient Instance Retrieval with Standard and Relational Path
Indexing, CADE-19, LNAI 2741, 380-396.

The fix, and where it lives

FactorStats.col_wildcard records, per column, whether any value stored there is non-ground --
tested with Expr::is_ground on the cursor key rather than by scanning for tag bytes, which a
symbol's payload could impersonate. It falls out of a walk the measurement already does.

Both consequences are applied where the stats are BUILT, not in the search: the seek path ends at
the first tainted column, and a tainted variable's log_dom becomes 0 so its repeated occurrence
stops dividing. lead_scale, order_cost, dp_order and greedy_order are untouched and the
2^n loop is not one instruction slower. It does force a two-pass build of the body's statistics,
because the column that stores the variable is in one conjunct and the seek it defeats is in
another.

What it also unlocked, which I did not expect

The change was aimed at removing a regression. It also made the planner FIRE where it had been
declining, because the old model priced these bodies so badly they fell outside the admission
band:

probe	written	before	after
a variable-valued row in the joined relation	23,586	23,586 (declined)	7,814
a wholly variable row	506,663	506,663 (declined)	98,107
a fact matching every conjunct	1,253,686	1,253,686 (declined)	205,479

The residual, stated plainly

differential/corpus/programs/plan_variable_column.mm2 is NOT a win: the planner accepts it and
walks 24,979 transitions against the written order's 20,331, because it prices a whole column by
its worst value while three of that table's four rows are in fact seekable. The point is that the
loss is a bounded constant, 1.14x-1.23x across a 16x range of input, where without the term it
was 27x growing to 463x.

I tried charging that coarseness through the machinery already there -- counting a tainted column
into uncertainty so the plan must clear a wider margin -- and measured it as a clear loss: it
did not prevent the one loss and it gave back all three wins above, because the bodies that win
here are exactly the ones with tainted columns. Reverted.

Two corrections to the evidence, not the code

The asymptotic claim is now reproducible. It rested on a sweep that lived only in my notes,
which is a poor way to support the one claim this PR is actually making. mork bench clique_scaling ships it: constant average degree, growing input, transitions per size. Run it
on a default build and on a feature build and read the exponents off. It is a benchmark, not a
code path -- bench all does not run it.

The first gate's small numbers were void and are re-measured. I built base in one worktree
and the branch in another. The compile-time manifest path is baked into the binary and its LENGTH
shifts code layout; I had measured that at 0.114% on a commit that touched only a markdown file,
written the rule down, and then broke it. Everything below comes from three binaries built in one
source directory. (CARGO_TARGET_DIR turns out not to matter at all -- the same commit built
into two very differently named target dirs gives byte-identical binaries.)

Redoing it that way surfaced one more thing worth having: bench transitive, which is nothing but
the descent loop, was +0.033% on the DEFAULT build, where neither this module nor leapfrog
is compiled. query_multi_raw was forwarding to the two-source-list form by passing the same
slice twice, and the compiler cannot prove two &[ExprEnv] alias, so it kept both pointers live
across the hot closure. Making the identity static instead of incidental, with a const parameter
in the same shape as MAY_REORDER, puts it back: +0.033% -> -0.007%.

Gate

mork bench all, min of 3 instructions:u, three binaries built in ONE source directory,
interleaved per repetition, 108 of 108 cells at exit 0, every output identical:

clique              384,237,856,849 -> 2,865,023,996   -99.254%
logic_query                                             -1.739%
bfc                                                     -1.092%
process_calculus                                        -0.689%
counter_machine                                         -0.595%
exponential                                             -0.515%
taxi_lts                                                -0.351%
tile_puzzle_states                                      -0.328%
exponential_fringe                                      -0.233%
finite_domain                                           -0.119%
odd_even_sort                                           -0.088%
transitive                                              -0.071%
suite total                                             -5.508%

Default build against 06cdcf3: max |delta| 0.015%, suite total -0.0005%, every output
identical.

Only clique is the planner, and this time that is measured rather than argued. Same binary,
planner switched off at runtime by an environment variable, so code layout is fixed by
construction -- a second binary that merely never CALLS the planner is destroyed by dead-code
elimination and is not a control:

bench	planner off	planner on
clique	382,926,206,596	2,865,026,637	-99.252%
counter_machine	13,192,495,232	13,195,880,931	+0.026%
bfc	136,354,796,742	136,360,899,809	+0.004%
process_calculus	395,914,772,301	395,918,137,890	+0.001%
logic_query	7,877,083,567,574	7,877,083,718,763	+0.000%

logic_query is the one to read: the planner costs it nothing because it declines on every body
there, so the -1.739% in the table above is leapfrog compiling alongside and moving code, not a
win. Please do not read that column as one. The planner's own cost tops out at +0.026%.

The head of the branch builds BYTE-IDENTICAL binaries to the ones those numbers came from, in
both configurations -- checked, because the provenance commit edits the module header and an
earlier header edit had already moved the binary's hash by shifting the line numbers that
assert! bakes in.

cargo test -p mork: 0 tests on the default build, as on main; 35 with conjunct_order; 21 with
leapfrog; 35 with both; 0 collected under conjunct_order,no_search, which is the cfg exclusion
doing its job.

differential/run.py --slow --generated: 115 ok / 0 failed / 3 skipped over 118, in three
pairings -- feature against default, feature against leapfrog, and feature against the checked-in
expected spaces. Five corpus programs are new and each was verified to be a body the planner
really does reorder, with expected spaces pinned from 06cdcf3.

mork test: byte-identical on all three builds, all exiting 101 on the same pre-existing
sink_hash_expr assertion. Not touched here, and I see you have already fixed it on
proj-aware-join.

[MesTTo]

Our comments crossed -- I pushed an update and posted the long version just after you wrote this,
so here is the short one.

You are right that the heuristics form can seriously regress on hand-coded order, and it was worse
than "can". I hit it independently a few hours ago and it is fixed on the branch now. One
(root $r) beside three ground rows made a four-conjunct body take an order walking 27x the
written order's transitions at 300 edges, 56x at 600, 116x at 1,200, and 463x at 4,800 --
growing with the input rather than settling, so not even a bounded regression. The cause: a column
in a term store can hold a VARIABLE, a stored variable unifies with everything, so a query
variable read out of that column may itself be a variable and the factor entered next by seeking
on it has no key to seek. A cost model borrowed from a relational store has no term for that. With
one, the same shape is 1.14x-1.23x flat across that whole range.

Answers were correct the entire time, which is why nothing in the differential corpus caught it --
it took a scaling sweep. That now ships as mork bench clique_scaling so it is checkable rather
than something I claim, and the regression itself is pinned by a test that reads transitions at
two input sizes and was verified to fail without the fix.

On the scheduling, understood and no argument from me. For what it is worth this one is gated:
conjunct_order is off by default, neither it nor leapfrog compiles without the flag, and the
default build measures within 0.015% of main across all twelve benches with identical output.

I will leave it sitting here rather than push on it. If the extra runtime information in the trie
ends up giving a planner better numbers than enumerating off the live trie at plan time -- which
is all this one does, and the part you said a satisfying solution needed -- then it is better
rebuilt on that than merged ahead of it. Happy to redo it against that when it lands.