Skip to content

Selectively load rules in the same TU as the translated code - #451

Merged
nunoplopes merged 81 commits into
Cpp2Rust:masterfrom
lucic71:rules-in-tu
Oct 10, 2026
Merged

Selectively load rules in the same TU as the translated code#451
nunoplopes merged 81 commits into
Cpp2Rust:masterfrom
lucic71:rules-in-tu

Conversation

@lucic71

@lucic71 lucic71 commented Oct 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Part of #447. This only loads the rules into cpp2rust, AST matching comes in a future PR:

To implement this, rules are included in the translated TU. Only the rules that parse are kept. If the TU never included then there are no rules for map. This is faster than the string matching approach.

cpp_rule_indexer runs at build time to build an index of rules. This index is used by rule_loader to lazily load the rules into the converter for doing AST matching.

There are 2 index files: one for C code and one for C++ code. Both live in build/rules_index. C translated code only loads the C index, C++ translated code loads both indexes.

Example of index:

key:      std/vector/push_back
requires: std/vector
dir:      vector
text:
        template <typename T1> void f14(std::vector<T1> &o, T1 &&value) {
          return o.push_back(std::move(value));
        }

key:      std/basic_ifstream/rdbuf
requires: std/basic_ifstream, std/basic_filebuf
dir:      ifstream
text:
        std::filebuf *f4(const std::ifstream &o) { return o.rdbuf(); }

key is generated using the dependent name of the rules for templates and using the concrete name for non-templates. text is extracted from the file and appended into the translated TU if it needs the rule. requires only loads the rule in the TU if the translated code uses all the dependencies, basic_ifstream and basic_fielbuf in the example.

This makes the translation fast. With 100K dummy rules, it takes on avergage 20ms per invocation to selectively load the rules. With full loading of all the 100K dummy rules, it takes 5 seconds on average.

@lucic71 lucic71 mentioned this pull request Oct 5, 2026
Closed
@lucic71 lucic71 changed the title Compile rules in the same TU as user code Selectively load rules in the same PR as the translated code Oct 6, 2026
@lucic71 lucic71 changed the title Selectively load rules in the same PR as the translated code Selectively load rules in the same TU as the translated code Oct 6, 2026
@lucic71
lucic71 marked this pull request as ready for review October 6, 2026 15:09
@lucic71

lucic71 commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

@nunoplopes please take a look

@nunoplopes

nunoplopes commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Why do you need to store those namespaces in the index? Can be injected when loading a rule.

Also, what's the end-to-end slowdown when translating brunsli or sqlite?

@lucic71

lucic71 commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

Why do you need to store those namespaces in the index? Can be injected when loading a rule.

Right, they can.

Also, what's the end-to-end slowdown when translating brunsli or sqlite?

I only have numbers for unit tests, woff2 and brunsli. Sqlite is on a 2-3 months diverging branch.

branch unit (500 TUs) woff2 (20 TUs) brunsli (17 TUs) description
master (unsafe + refcount) (s) 33 11.3 24.6
rules-in-tu (unsafe + refcount) (s) 33.2 11.9 26.5
slowdown per TU (ms) 0.2 15 55.9 (rules-in-tu - master) / 2 / number-of-TUs * 1000
total slowdown (%) 0.6 5.3 7.7 (rules-in-tu - master) / master * 100

@lucic71

lucic71 commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

Why do you need to store those namespaces in the index? Can be injected when loading a rule.

Right, they can.

Done

@nunoplopes
nunoplopes merged commit 42ba985 into Cpp2Rust:master Oct 10, 2026
9 checks passed
@lucic71
lucic71 deleted the rules-in-tu branch October 10, 2026 21:49
@lucic71 lucic71 mentioned this pull request Oct 12, 2026
Merged
nunoplopes pushed a commit that referenced this pull request Oct 14, 2026
@lucic71
Replace string matcher with AST matcher (#527)
e65e265 
Closes #447, depends on #529

Rules are matched on the AST instead of on printed strings.
string_matcher and ir_src.json are deleted. cpp-rule-prepocessor is
replaced by cpp-rule-indexer which walks the rules, compiles them, and
creates an index (#451).

---

The bodies of the used rules are loaded in the TU to match against their
usages.

A rule matches a usage if calling the rule with the operands of the
usage would end up calling the same declaration as the usage.
Concretely:

1. The usages is keyed by its declaration: `Member{class, member_name}`,
`Constructor{class}`, `Unqualified{name}`. In Member we keep: data
members, static members, function members. In Unqualified we keep
everything outside a class: free function, macro, enumerators, etc.
Every loaded rule with the same key is a candidate.
2. Between a usage and a candidate rule, let clang deduce the template
arguments of a rule by calling the rule with the usage's operands. If
this does not work (T1, T2, not used in the argument list), deduce using
the return type.
3. The deduced arguments are subtituted in the rule's return expression.
If the resolved expressoin has the same declaration as the usage, then
it's a match.
4. If several rules mtach, non-template rules win over template rules.
Otherwise let clang's partial ordering pick the most specialized one

For example:

```cpp
std::vector<int> v;
v.push_back(3);

template <typename T1> void f1(std::vector<T1> &a0, const T1& a1) {
  return a0.push_back(a1);
}

template <typename T1> void f2(std::vector<T1> &a0, T1&& a1) {
  return a0.push_back(std::move(a1));
}
```
At load time: both f1 and f2 are stored under `Member{vector,
push_back}`.

1. Mapper calls `RuleRegistry::Search(expr)` which reaches
`RuleRegistry::findExpr`. The decl of the usage
(`vector<int>::push_back(int&&)`) is also keyed under `Member{vector,
push_back}`.
2. `Match` takes the operands `[v, 3]` from the usage and gets:
  i.  `f1(vector<T1>&, const T1&)` -> `T1 = int`
  ii. `f2(vector<T1>&, T1&&)` -> `T1 = int`
3. The deduced tempalate arguments are inserted in the return
expression:
i. f1: `a0.push_back(a1)` calls `push_back(const int&)`. Does not match
the usage decl.
ii. f2: `a0.push_back(std::move(a1))` calls `push_back(int&&)`. It
matches the usage decl.
4. Only f2 is matches so there is no need for disambiguation. 

Type rules work the same. The usage type is deduced against the rule's
alias with sugar preferred over the canonical type.

---

Not all rules have deducible arguments, for example `std::map<T1,
T2>::const_iterator`, in general every template argument that sits on
the left of `::` is non-deducible. Such rules need changes so they
become deducible:

```diff
- template <typename T1, typename T2>
- using t2 = typename std::map<T1, T2>::const_iterator;
+ template <typename It, typename T1 = KeyOf<It>, typename T2 = MappedOf<It>,
+          typename = SameAs<It, typename std::map<T1, T2>::const_iterator>>
+ using t2 = It;
```

Both T1 and T2 originally sat in non-deducible contexts. The new syntax
works like this:
1. The iterator is taken as plain `It` which is always deducible
2. `T1` and `T2` are computed from `It` using iterator traits: `T1 =
It::value_type::first_type` and `T2 = It::value_type::second_type`
3. `SameAs` constraints the rule only for const iterators of `std::map`
so that they don't conflict with the rules for non-cosnt iterators for
`std::map` or other map types

---

Performance

| branch | unit (500 TUs) | woff2 (20 TUs) | brunsli (17 TUs) |
description |
| --- | --- | --- | --- | --- | 
| master (unsafe + refcount) (s) | 33.2 | 11.9 | 26.5 | |
| ast-matcher (unsafe + refcount) (s) | 32.7 | 11.6 | 25.6 | |
| **speedup** per TU (ms)| 0.5 | 7.5 | 26.5 | (master - ast-matcher) / 2
/ number-of-TUs * 1000 |
| total **speedup** (%) | 1.5 | 2.5 | 3.4 | (master - ast-matcher) /
master * 100 |

---

Some nice implications for rule writing:
1. deque_cxx11 and vector_cxx11 disappear. Instead we change the
definition of functions:

```cpp
auto f12(std::deque<T1> &o, Init<T1, Args> &&...args)
    // in C++11 the return type is deduced as void
    // post-C++11 the return type is deduced as reference type
    -> decltype(o.emplace_back(std::forward<Args>(args)...)) {
  return o.emplace_back(std::forward<Args>(args)...);
}
```

2. We don't need to write rules for `T*`, `T**`, `T&`, `const T*`, etc.
We only write one rule for T and the converter takes care of the rest,
see `rules/builtin_types/src.c` and `rules/iostream/src.cpp`
3. We only write one constructor for std::pair that forwards the
arguments, see `rules/pair/src.cpp`
4. We only write rules for `vector<T, A>`, rules for `vector<T>` are not
needed. This drops a lot of useless rules in `rules/vector/src.cpp`

This opens another new interesting opportunity: writing rules for
multiple statements at once. Now that we match AST subtrees we can match
an entire subtree at once and write more idiomatic rules. Also we can
move some of the converter translations in rules: for exmaple binary
operators which have a very complicated visitor today.

---

Next steps after this PR:
1. The AST type matcher stops walking the sugar typedefs and starts
translating rules for each typedef: off_t, value_type, time_t, etc
2. Stop masking NTTPs in rules (#435)
3. Add a cache on top of mapper to avoid deduction for results we
already mapped
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lucic71 @nunoplopes