Skip to content

Replace string matcher with AST matcher - #527

Merged
nunoplopes merged 3 commits into
Cpp2Rust:masterfrom
lucic71:ast-matcher
Oct 14, 2026
Merged

nunoplopes merged 3 commits into
Cpp2Rust:masterfrom
lucic71:ast-matcher

Conversation

@lucic71

@lucic71 lucic71 commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Closes #447, depends on #529

Rules are matched on the AST instead of on printed strings. string_matcher and ir_src.json are deleted. cpp-rule-prepocessor is replaced by cpp-rule-indexer which walks the rules, compiles them, and creates an index (#451).


The bodies of the used rules are loaded in the TU to match against their usages.

A rule matches a usage if calling the rule with the operands of the usage would end up calling the same declaration as the usage. Concretely:

  1. The usages is keyed by its declaration: Member{class, member_name}, Constructor{class}, Unqualified{name}. In Member we keep: data members, static members, function members. In Unqualified we keep everything outside a class: free function, macro, enumerators, etc. Every loaded rule with the same key is a candidate.
  2. Between a usage and a candidate rule, let clang deduce the template arguments of a rule by calling the rule with the usage's operands. If this does not work (T1, T2, not used in the argument list), deduce using the return type.
  3. The deduced arguments are subtituted in the rule's return expression. If the resolved expressoin has the same declaration as the usage, then it's a match.
  4. If several rules mtach, non-template rules win over template rules. Otherwise let clang's partial ordering pick the most specialized one

For example:

std::vector<int> v;
v.push_back(3);

template <typename T1> void f1(std::vector<T1> &a0, const T1& a1) {
  return a0.push_back(a1);
}

template <typename T1> void f2(std::vector<T1> &a0, T1&& a1) {
  return a0.push_back(std::move(a1));
}

At load time: both f1 and f2 are stored under Member{vector, push_back}.

  1. Mapper calls RuleRegistry::Search(expr) which reaches RuleRegistry::findExpr. The decl of the usage (vector<int>::push_back(int&&)) is also keyed under Member{vector, push_back}.
  2. Match takes the operands [v, 3] from the usage and gets:
    i. f1(vector<T1>&, const T1&) -> T1 = int
    ii. f2(vector<T1>&, T1&&) -> T1 = int
  3. The deduced tempalate arguments are inserted in the return expression:
    i. f1: a0.push_back(a1) calls push_back(const int&). Does not match the usage decl.
    ii. f2: a0.push_back(std::move(a1)) calls push_back(int&&). It matches the usage decl.
  4. Only f2 is matches so there is no need for disambiguation.

Type rules work the same. The usage type is deduced against the rule's alias with sugar preferred over the canonical type.


Not all rules have deducible arguments, for example std::map<T1, T2>::const_iterator, in general every template argument that sits on the left of :: is non-deducible. Such rules need changes so they become deducible:

- template <typename T1, typename T2>
- using t2 = typename std::map<T1, T2>::const_iterator;
+ template <typename It, typename T1 = KeyOf<It>, typename T2 = MappedOf<It>,
+          typename = SameAs<It, typename std::map<T1, T2>::const_iterator>>
+ using t2 = It;

Both T1 and T2 originally sat in non-deducible contexts. The new syntax works like this:

  1. The iterator is taken as plain It which is always deducible
  2. T1 and T2 are computed from It using iterator traits: T1 = It::value_type::first_type and T2 = It::value_type::second_type
  3. SameAs constraints the rule only for const iterators of std::map so that they don't conflict with the rules for non-cosnt iterators for std::map or other map types

Performance

branch unit (500 TUs) woff2 (20 TUs) brunsli (17 TUs) description
master (unsafe + refcount) (s) 33.2 11.9 26.5
ast-matcher (unsafe + refcount) (s) 32.7 11.6 25.6
speedup per TU (ms) 0.5 7.5 26.5 (master - ast-matcher) / 2 / number-of-TUs * 1000
total speedup (%) 1.5 2.5 3.4 (master - ast-matcher) / master * 100

Some nice implications for rule writing:

  1. deque_cxx11 and vector_cxx11 disappear. Instead we change the definition of functions:
auto f12(std::deque<T1> &o, Init<T1, Args> &&...args)
    // in C++11 the return type is deduced as void
    // post-C++11 the return type is deduced as reference type
    -> decltype(o.emplace_back(std::forward<Args>(args)...)) {
  return o.emplace_back(std::forward<Args>(args)...);
}
  1. We don't need to write rules for T*, T**, T&, const T*, etc. We only write one rule for T and the converter takes care of the rest, see rules/builtin_types/src.c and rules/iostream/src.cpp
  2. We only write one constructor for std::pair that forwards the arguments, see rules/pair/src.cpp
  3. We only write rules for vector<T, A>, rules for vector<T> are not needed. This drops a lot of useless rules in rules/vector/src.cpp

This opens another new interesting opportunity: writing rules for multiple statements at once. Now that we match AST subtrees we can match an entire subtree at once and write more idiomatic rules. Also we can move some of the converter translations in rules: for exmaple binary operators which have a very complicated visitor today.


Next steps after this PR:

  1. The AST type matcher stops walking the sugar typedefs and starts translating rules for each typedef: off_t, value_type, time_t, etc
  2. Stop masking NTTPs in rules (NTTP integers are masked in rules #435)
  3. Add a cache on top of mapper to avoid deduction for results we already mapped

@lucic71
lucic71 marked this pull request as draft October 12, 2026 09:45
v2.push(0);
v2.push(1);
v2.push(3);
{

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old string matcher was matching the push_back(const T&) version, which was wrong. The new matcher matches push_back(T&&), which is ok. I a future PR we can rewrite the body of the rules so that they don't generate useless temporaries. The ideal output is: v2.push(0)

Comment thread rules/map/src.cpp Outdated
typename It,
typename T1 = typename std::remove_const<
typename std::iterator_traits<It>::value_type::first_type>::type,
typename T2 = typename std::iterator_traits<It>::value_type::second_type,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need all of this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is not ready for review yet

@lucic71
lucic71 force-pushed the ast-matcher branch 3 times, most recently from a329484 to 983c30d Compare October 12, 2026 22:51
@lucic71
lucic71 marked this pull request as ready for review October 13, 2026 10:22
@lucic71

lucic71 commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

@nunoplopes this is ready for review. But please take a look at #529 first

Comment thread rules/brotli/src.c
typedef BrotliDecoderErrorCode t5;

BrotliEncoderMode f1() { return BROTLI_MODE_FONT; }
BrotliEncoderMode f1(void) { return BROTLI_MODE_FONT; }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replacing () with (void) is needed because C++ considers () prototype-less

Comment thread rules/map/src.cpp
}

template <typename T1, typename T2>
std::map<T1, T2> f6(const std::map<T1, T2> &&o) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const && is a wrong rule that was not caught in the old matcher. Same for vector, it had some wrong const && rules

@nunoplopes
nunoplopes merged commit e65e265 into Cpp2Rust:master Oct 14, 2026
9 checks passed
@lucic71
lucic71 deleted the ast-matcher branch October 14, 2026 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Match rules on AST instead of strings

2 participants