Harvest C++ method bodies into the recipe fingerprint

.claude/knowledge/fuzzy-recipe-codebook.md

@@ -52,7 +52,10 @@ on code:
 
 ## 2. The fingerprint — the DTO arm (this is what every frontend must emit)
 
-The fingerprint is FOUR fact sets per method, on `ruff_spo_triplet::Function`:
+The fingerprint is FIVE fact sets per method, on `ruff_spo_triplet::Function`
+(and, since 2026-09-07, on `ruff_spo_triplet::CppMethod` too — see the dated
+note beneath the coverage table below for why C++ needed its own copy of
+these fields rather than reusing `Function`'s):
 
 | field            | predicate emitted | provenance        | what it captures                                                                                                      |
 | ---------------- | ----------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------- |
@@ -62,6 +65,13 @@ The fingerprint is FOUR fact sets per method, on `ruff_spo_triplet::Function`:
 | `calls`          | `calls`           | Inferred          | mutator dispatches `"receiver.method"`                                                                                |
 | `guarded_writes` | `writes_if_blank` | **Authoritative** | writes guarded by a blank/nil test on the same field — the **J1** fact (§5), splits SelfMap into default vs normalize |
 
+The count above read FOUR until 2026-09-07, while the table under it listed
+five rows. That is not a cosmetic slip: `recipe::classify` tests
+`guarded_writes` BEFORE `Compute` and `Normalize`, so a frontend that read the
+sentence rather than the table and harvested only the quartet would classify
+every guarded default as a compute or a normalize. A frontend joins this
+codebook by populating five sets, not four.
+
 Plus the **visibility split**: hook targets are conventionally *private*, so a
 frontend that drops private defs cannot resolve most hooks. `ruff_ruby_spo`
 carries them in `Model::helpers` (walked identically, kept out of the routable
@@ -71,19 +81,46 @@ landed.)
 
 > **This is the "DTO-arm shape" C++ / Python still need.** As of 2026-07-06
 > `ruff_ruby_spo` and `ruff_csharp_spo` emit the full quartet + helpers.
+> **[2026-09-07 correction: no longer true for EITHER. C++'s arm was wired
+> end-to-end that day (see the row below). Python's was already complete and
+> the table had simply not been updated — checking it while correcting the
+> C++ row is what turned that up. The sentence above is retained as the
+> historical claim it was, not as current state.]**
 > Coverage:
 >
 > | frontend          | writes | reads | raises | calls | helpers | verdict                                                                                                                       |
 > | ----------------- | :----: | :---: | :----: | :---: | :-----: | ----------------------------------------------------------------------------------------------------------------------------- |
 > | `ruff_ruby_spo`   |   ✅   |  ✅   |   ✅   |  ✅   |   ✅    | reference — cook here first                                                                                                   |
 > | `ruff_csharp_spo` |   ✅   |  ✅   |   ✅   |  ✅   |   ✅    | syntax-only (SemanticModel upgrade pending); helpers via `has_visibility`; tested end-to-end on a real production C# corpus (~97k triples) |
-> | `ruff_python_spo` |   ~    |  ✅   |   ✅   |   ~   |    ✗    | reads/raises only; **needs writes/calls/helpers**                                                                             |
-> | `ruff_cpp_spo`    |   ~    |   ~   |   ~    |   ~   |    ✗    | scaffolded; **needs the arm populated** (setters/virtuals)                                                                    |
+> | `ruff_python_spo` |   ✅   |  ✅   |   ✅   |  ✅   |   n/a   | **(2026-09-07 correction)** the row read `~ ✅ ✅ ~ ✗ — reads/raises only; needs writes/calls/helpers`, and was stale: `functions.rs`'s `BodyWalker` populates all five sets (writes, the J1 `guarded_writes` across three guard spellings, and `calls` over a closed `ORM_MUTATORS` set with a receiver label), asserted end-to-end by `dto_arm_writes_guarded_writes_and_calls` including the negative case that a local assignment is not a write. The crate's own `lib.rs` quotes this row as the thing it was closing. **`helpers` is n/a here, not missing**: Python has no visibility keyword, so every `Stmt::FunctionDef` in a model body already lands in `Model::functions` and there is no non-routable subset to split off (`functions.rs` module doc). |
+> | `ruff_cpp_spo`    |   ✅   |  ✅   |   ✅   |  ✅   |   n/a   | **(2026-09-07)** wired end-to-end (commit `f068a3f`): harvested from real bodies via libclang 18, same predicates/objects/tiers as `Function`, classified via the now-generic `recipe::classify`; unit- and hermetic-fixture-proven (16 guards, disable-verified) — a corpus census is now re-runnable (`examples/harvest_ladybug.rs` reports the five fact totals plus a full `RecipeCentroid` census; see `.claude/plans/cpp-spo-probes-v1.md` for the numbers) but **nothing asserts it**, so C++ has a repeatable measurement and still not a gate — a regression would print quietly and fail nothing; J1 hit-rate is still open, because that corpus contains no lazy-init writes to hit (helpers reads `n/a` for the reason the note under the table gives: C++ keeps every method in one collection with its own `access` field, so there is no separate pool to be missing) |
+>
+> **Reading the `helpers` column:** it does NOT mean the same thing in every
+> row. For Ruby it is a real capability — private defs live in a separate
+> `Model::helpers` pool and would otherwise be dropped, which is the ~80%
+> hook loss this document opens with. For Python and C++ there is nothing to
+> split: Python has no visibility keyword at all, and C++ keeps every method
+> in one `methods` collection carrying its own `access` field (and has emitted
+> `has_visibility` since before the arm landed). Those two rows read `n/a`
+> rather than `✗` as of 2026-09-07, because a `✗` there invited exactly the
+> misreading that a frontend was losing hooks when it was not.
 >
 > The fingerprint predicates are already in the shared IR
 > (`ruff_spo_triplet::Function`) and `expand()` already emits them — a frontend
 > "adds the arm" purely by *populating* those Vecs from its AST. Zero IR change.
 > Do it per-frontend, then this codebook runs unchanged on that language.
+> **[2026-09-07 correction: "zero IR change" held for Ruby/Python/C# only
+> because those three frontends already route their methods through
+> `ruff_spo_triplet::Function` itself — there was nothing to add a field to
+> but the one struct all three share. `ruff_cpp_spo` does NOT share it: its
+> methods are a distinct IR struct, `CppMethod`, so wiring its arm required a
+> real (additive) IR change — the same five fields added a second time, by
+> name, on `CppMethod` (`ir.rs`) — plus a new `BodyFacts` trait so
+> `recipe::classify` could stay a single function generic over either struct
+> instead of forking into a `classify_function`/`classify_cpp_method` pair.
+> Read "populate the Vecs, zero IR change" as the fast path for a frontend
+> that already reuses `Function`; a frontend with its own method type pays a
+> one-time IR/trait-impl cost instead, then joins the same classifier.]**
 >
 > **`ruff_csharp_spo` also needed a configurability step the reference
 > frontend didn't**: Ruby's `AR_MUTATORS` is a closed, ORM-shaped set, but a
@@ -92,7 +129,27 @@ landed.)
 > `--mutator-prefixes`/`--mutator-receivers` generalise the mutator predicate
 > from "closed name set" to "name set OR prefix, optionally receiver-scoped"
 > — the same recipe centroids then classify EF Core (`SaveChanges`) and
-> bespoke ADO.NET (`main.mysql.add_x`) call sites identically.
+> bespoke ADO.NET (`main.mysql.add_x`) call sites identically. **`ruff_cpp_spo`
+> got the same generalization on 2026-09-07**: `BodyArmConfig::with_mutators`/
+> `with_mutator_prefixes` replace the shipped closed set (`Save`/`Update`/
+> `Commit`/…) for a corpus with its own persistence vocabulary — unit-tested,
+> not yet measured against a real non-STL-shaped C++ DAL.
+>
+> **On the `helpers` column for C++, specifically:** unlike Ruby (which keeps
+> routable actions and private hook targets in two separate `Model` slots,
+> `functions` vs `helpers`) or C# (one slot, `has_visibility` distinguishes),
+> `ruff_cpp_spo` has always kept EVERY method — public, protected, and
+> private alike — in the ONE `CppMethod`/`methods` collection, carrying its
+> own `access: CppAccess` field and already emitting a `has_visibility`
+> predicate before this commit. So the "a frontend without helpers loses
+> ~80% of its hooks to no-facts" failure mode this doc opens with may simply
+> not apply to C++ in the same shape — a private method's body facts were
+> never at risk of being dropped the way Ruby's private defs originally were.
+> This is an observation about the existing (pre-2026-09-07) shape of the
+> walker, not something this commit changed or measured; the ✗ is left as-is
+> above rather than reinterpreted, and a real corpus run (the next probe,
+> `.claude/plans/cpp-spo-probes-v1.md`) would be the way to confirm private
+> C++ methods actually resolve as hook targets in practice.
 
 ## 3. The recipe codebook — the centroids (pure fact-set predicates, GENERIC)
 

.claude/plans/behavioral-ore-v1.md

@@ -0,0 +1,178 @@
+# Ordered behavioral ore for C++ — plan v1
+
+**Status:** IN PROGRESS. The extractor is the deliverable here; the tokenizer
+that consumes it is a separate, downstream experiment and nothing in this plan
+asserts a result for it.
+
+## 1. The baseline defect (this is a FINDING, not plumbing)
+
+> **The C++ harvest representation cannot carry behavior, because it is a set.**
+
+Two independent causes, both in `ruff_cpp_spo`:
+
+1. **`method_body_arm` ends by collapsing the traversal into sets.**
+   `clang_walker.rs:1011-1025` runs `facts.sort(); facts.dedup();` over all
+   five vectors, unconditionally, in the one function that produces a
+   `BodyArm`. Adjacency, repetition and interleaving are gone before any
+   consumer sees a fact. `expand.rs:904-943` then emits by fixed category
+   (reads → raises → writes → guarded → calls), alphabetical within category,
+   so nothing downstream can recover source order either.
+
+2. **`walk_body` gives the structured constructs no match arm.**
+   Only `ThrowExpr`, `BinaryOperator`, `CompoundAssignOperator`,
+   `UnaryOperator`, `CallExpr`, `IfStmt` and `MemberRefExpr` are matched
+   (`clang_walker.rs:1035-1110`). `ForStmt`, `WhileStmt`, `DoStmt`,
+   range-for, `SwitchStmt`, `CXXTryStmt`, `CXXCatchStmt`, `LambdaExpr`,
+   `ReturnStmt` fall through `_ => walk_body(...)`: their CONTENTS are
+   visited, their BOUNDARIES are not recorded anywhere. A write inside a loop
+   and a write in straight-line code are the same fact. `IfStmt` is matched
+   only to thread the J1 guard; it emits no event of its own.
+
+**Consequence.** The recurring shapes a behavioral vocabulary would be made of
+— an iterator loop, a lock/mutate/unlock triple, an RAII scope, a
+dispatch-then-propagate chain, a try/catch compensation — are precisely the
+ones that require order and scope, and every one of them is unrepresentable
+today. A sequence learner run against this output is learning words from a bag
+of letters.
+
+**This is not an argument against the five sets.** They are the right input to
+`recipe::classify`, which is a set-membership classifier by design and works.
+The defect is that the harvest DISCARDS the sequence rather than emitting both.
+
+### Measured on the control fixture
+See `§4`. The measurement is reported by `ore.py`'s loss table and is repeated
+in the commit that lands the extractor; it is not asserted here in advance.
+
+## 2. What this adds — three things, preserved SEPARATELY
+
+| preserved | artefact | deduped? |
+|---|---|---|
+| event identity | `symbols.tsv` | yes — the only place |
+| event order | `events.tsv` | **never** |
+| structural context | `scopes.tsv` | n/a |
+
+The same fact occurring twice is two rows in `events.tsv` pointing at one row
+in `symbols.tsv`. Keeping the canonical symbol table and the occurrence stream
+apart is what lets a consumer choose its own granularity without the extractor
+having chosen for it.
+
+## 3. Neutrality rules — the ore must not pre-solve the experiment
+
+1. **No inferred motifs in the alphabet.** The 20 kinds are primitive
+   (`ScopeEnter`/`Read`/`Call`/`Condition`/…). There is no `RAII`,
+   `IteratorLoop`, `VirtualDispatch` or `Move` kind. A `lock_guard` local is
+   `Decl` + a type symbol; `std::move(x)` is `Call` + a callee symbol. Those
+   concepts are candidate LEARNED tokens; emitting them would be the
+   tokenizer's answer written into its input.
+2. **No consumer vocabulary.** Nothing about classids, facets, rails, masks or
+   any 96-bit layout appears in the ore or in this crate. The ore does not
+   know those exist.
+3. **Identities are verbatim, never hashed.** Different consumers hash
+   differently; a hash in the ore forces one of them.
+4. **Lexical order and control flow stay distinguishable.** Every event carries
+   its source byte `anchor`. `control` carries only structurally-certain
+   relations. **libclang's C API exposes no CFG** — successor/predecessor
+   edges are not available and are never fabricated. A later probe may ask
+   whether CFG adjacency beats lexical adjacency; it cannot ask it from these
+   files, and must say so rather than assume the two are the same.
+
+## 4. The control
+
+`for (…) { if (…) foo(); else bar(); }` — the expected event sequence was
+hand-written BEFORE the extractor existed. Where libclang disagrees (AST child
+order for `ForStmt` is init/cond/inc/body, which is not source order), the
+disagreement is reported and the `anchor` column keeps both orders separable.
+It is not resolved by editing the expectation to match the code.
+
+## 5. Fences
+
+- **Additive only.** `method_body_arm`, `BodyArm`, `CppMethod`, `expand` and
+  the emitted ndjson are byte-identical after this change. A test asserts that
+  collapsing the new events to the five sets reproduces `method_body_arm`
+  exactly — proving the change is about what is PRESERVED, not about what a
+  fact is.
+- **No new predicate.** The closed vocabulary is untouched; the ore is TSV
+  beside the manifest, not a triple.
+- The extractor is a second walk over the same cursor, in its own module.
+
+## 6. Measured — 227-TU ladybug corpus, 2026-09-07
+
+The §1 defect, quantified on real code rather than the control fixture
+(2,217 methods, 95,365 ordered events):
+
+| | |
+|---|---|
+| ordered events | 95,365 |
+| facts the shipped set arm keeps | 2,574 |
+| duplicate occurrences destroyed | 2,093 |
+| scope boundaries destroyed | 19,902 |
+| adjacency pairs destroyed | 93,148 |
+| **retained** | **2.7% of ordered events, 0 adjacency, 0 scope** |
+
+The control fixture said 11.2%; at corpus scale it is 2.7%. The gap is call
+sites — 22,748 of them, of which the configured-mutator rule keeps 171.
+
+**What the loss costs, in bits.** Charging every representation for describing
+the same held-out events (`bits/ore-event`, uncovered events costed at
+`log2|alphabet|`):
+
+| representation | bits/ore-event |
+|---|---|
+| shipped sets | 4.0987 |
+| ordered ore | 2.7455 |
+| ordered ore + behavioural BPE (256 merges) | 2.6411 |
+
+The upstream fix is 13× the tokenizer's contribution. It decomposes into two
+roughly equal, slightly sub-additive halves — **order** (same five kinds,
+duplicates and adjacency restored) +0.7776, and **alphabet** (the 13 kinds the
+set arm has no representation for, order still destroyed) +0.7001. Neither
+alone explains it, so "restoring order is the wound" is half the story.
+
+**Provenance (the §15 check).** Events are 58% libclang-answered overall, but
+`ScopeEnter` / `ScopeExit` / `Condition` / `Branch` are **0%** — they are
+walk-derived. Every high-consistency motif the tokenizer learns is built from
+those, so the recurring structure is contributed by the ordered walk, not by
+the compiler.
+
+Full write-up, ladder and falsifier verdicts live with the experiment, not in
+this repo; what belongs here is the defect and its size.
+
+## 7. Re-measured on the whole corpus — 1,081 TUs, 2026-09-07
+
+§6 was the 227-TU sample. The full harvest supersedes its figures: **14,106
+methods, 646,411 ordered events, 36,224 symbols.**
+
+The defect gets WORSE with scale, because call sites dominate and the
+configured-mutator rule keeps almost none of them:
+
+| | 227 TUs | 1,081 TUs |
+|---|---|---|
+| ordered events | 95,365 | 646,411 |
+| facts the set arm keeps | 2,574 | 12,062 |
+| call sites (kept) | 22,748 (171) | 109,603 (745) |
+| adjacency pairs destroyed | 93,148 | 632,305 |
+| **retained** | 2.7% | **1.9%** |
+
+The bits cost, same common denominator:
+
+| representation | 227 TUs | 1,081 TUs |
+|---|---|---|
+| shipped sets | 4.0987 | 4.2684 |
+| ordered ore | 2.7455 | 2.7181 |
+| + BPE (256 merges) | 2.6411 | 2.4721 |
+
+The two-halves decomposition holds at 6.8× the data — order +0.8254, alphabet
++0.6651, together +1.5503 — so neither alone explains the gap, and the upstream
+fix remains several times the tokenizer's contribution.
+
+**Provenance, re-checked:** 54.1% of events are libclang-answered overall, and
+`ScopeEnter` / `ScopeExit` / `Branch` / `Condition` / `Return` / `Cast` are still
+exactly **0%**. The recurring structure is contributed by the ordered walk.
+
+**One measurement the sample got wrong.** At 227 TUs the ordered ore's
+override-pair retrieval looked like a tie among representations, because only 5
+of 291 declared override targets had both sides present. At 1,081 TUs there are
+**446** usable pairs, and the shipped set arm retrieves **0 of 446** while the
+ordered ore retrieves 26 — a 5.3σ separation. A representation that cannot find
+a method's override partner even once is the sharpest single statement of what
+sort+dedup costs.