[dimpase]

Description

Allow CMake to use installed memtailor, mathic and mathicgb through their CMake configs or pkg-config metadata. MEMTAILOR_PROVIDER, MATHIC_PROVIDER and MATHICGB_PROVIDER default to BUNDLED; SYSTEM requires a compatible installed library, and AUTO opts into installed libraries with bundled fallback. Existing caches retain their selected providers.

Configuration reports the selected versions and providers and checks debug ABI flags, transitive dependency flags and mathicgb's TBB setting with compile/link probes. System mathic requires system memtailor; system mathicgb requires both. Minimum versions are memtailor 1.4, mathic 1.5 and mathicgb 1.4, with metadata describing their installed ABI.

The engine and interpreter inherit the selected libraries' headers and ABI definitions. A Debug M2 can use a complete Release system stack; bundled Debug consumers still require Debug dependency layouts. Headers from a custom prefix take precedence over headers in other dependency prefixes, preventing accidental use of Homebrew headers with different selected libraries. Only bundled libraries are installed/exported or have their own upstream unit-test targets added.

Selection remains in M2/Macaulay2/e/CMakeLists.txt. Autotools behavior and submodule acquisition are unchanged. General build CI selects bundled providers and runs their library tests. The CMake wiki guide staged on wiki-mathic-providers documents configuration and ABI constraints; it should be published to the wiki after merge.

Related issues and pull requests

Related metadata improvements: Macaulay2/memtailor#23, Macaulay2/mathic#29 and Macaulay2/mathicgb#84.

This PR supplies external engine discovery for #4758; the no-submodules policy is kept in that separate PR.

Testing

Latest header fixes were validated on the integration branch in #4758:

• Built M2-core, M2-unit-tests and editor grammars with all three libraries installed under a custom prefix, using both CMake metadata and pkg-config-only discovery.
• All 275 engine unit tests and M2 -q --check 1 pass for both discovery paths.
• The system-prefix and custom-prefix Linux integration jobs in Support CMake builds with deinitialized submodules #4758 also pass.
• Audited compiler dependency records: every consumed memtailor/mathic/mathicgb header comes from the selected prefix, despite competing Homebrew installations. Populated submodules with invalid public headers are not consumed.

Earlier validation of provider selection: 19 selection checks, a Debug integration build against Release system libraries, and a fresh bundled Debug build without TBB (273 library tests plus an API/link smoke test) passed. Bundled exports and exclusion of system libraries from bundled exports were checked. These bundled tests were not repeated for the latest header fixes; bundled CI covers them. A full package/documentation installation was not run locally.

Existing Homebrew bottles of these versions lack the required metadata; their versions alone are insufficient. Current upstream source builds supply it.

Release notes

Use PR title.

AI disclosure

• 5 — AI produced the content with minimal human review.

Tools and models used: OpenAI Codex (earlier work also used GPT-6 Astra).
How AI was used: implementation, validation, documentation and PR description.
Verification: the builds, tests and dependency audits above were run by Codex; human review remains pending.

License

• My changes use the existing license of the affected files or package.
• I release my new material under the GNU GPL, version 2 or later.

[d-torrance]

I like the basic idea of this PR - the cmake build should absolutely support system-installed mathic and friends, which the autotools build has done since the beginning (or at least, for a very long time).

A couple quick initial comments for now from my phone. I'll dig into the actual code later

[d-torrance]

Any build docs should go on the wiki.

You brought up a good point on Zoom yesterday, by the way, about the wiki not being reviewed. So this is definitely something to consider. But for the time being, let's stick with current practice.

[dimpase]

I think installation docs should be a part of M2 proper; anyway, I'll proceed to move docs to wiki here. Which means, move them to a separate branch on my fork, and do not forget to publish them once this PR is merged.

[mahrud]

The CMake build intentionally uses the submodule mathic/etc to make development easier. From my perspective this PR is counterproductive and adds unnecessary complexity to the CMake build.

[dimpase]

There is nothing easier than external dependencies. Vendoring is bad, git submodules are awful, and 1% of developers and 0.01% of users who need it changed can go an extra hoop of installing an external dependency from source.

Needless to say, the same argument applies to all the other M2 dependencies - let's bring them all in as submodules, sure, it will only make them more convenient for everybody

[mahrud]

There is nothing easier than external dependencies. Vendoring is bad, git submodules are awful,

I completely agree, but mathic/etc are not typical external libraries.

and 1% of developers and 0.01% of users who need it changed can go an extra hoop of installing an external dependency from source.

Unless things have significantly changed in the past few months, the CMake build is really meant for the 1% of developers that are likely to want to change something in M2's engine and mathic simultaneously, so the extra hoop is really for a much bigger portion of people who use the CMake build. Note that the autotools build already uses external mathic/etc libraries.

Needless to say, the same argument applies to all the other M2 dependencies - let's bring them all in as submodules, sure, it will only make them more convenient for everybody

mathic/etc in particular have a complicated history. They used to be external libraries, then in #2160 Mike made them entirely internal to the engine, and #2173 was a compromise which allowed them (and in particular mathicgb) to continue being usable independently while making changes easy.

Digression: I don't think it's very productive to request to drop the whole thing without an initial discussion of relative merits and shortcomings. It took me several days to work this and other cmake and CI-related PRs out, and it was done in discussion and with help of other participants of the Cornell workshop.

I'm just one voice here, pointing out that the discussion goes back 5+ years, but Doug/Michael/others can disagree and ultimately accept this change. Maybe you already discussed the history at the workshop but I didn't see it addressed in your proposal.

[dimpase]

@mahrud - cmake is winning over autotools, there is no question about it. So more and more people (and 🤖s)
prefer it over autotools. Not that I have anything about autotools personally, and I actually contribute code to it. But it's slow, it lacks options, lacks support of newer hardware, etc.

(And autotools development pace is glacial; e.g. anything that depends on GNU libtool there needs first to get into libtool, and so I taught GNU libtool to understand Apple clang OpenMP options, which have made it into libtool 2.6.2, but the corresponding changes to autoconf needed the fixed libtool to come out, and so I submitted the fix for it only yesterday. Hopefully it will make it into autoconf 2.73.4...)

And I think @d-torrance @mikestillman and @antonleykin were on board with this change back in Cornell.

[dimpase]

@mahrud note also that Macaulay2/memtailor#23, Macaulay2/mathic#29 (and Macaulay2/mathic#30), and Macaulay2/mathicgb#84
provide full cmake support for memtailor and mathic(gb), so dealing with their building out of tree from source should get as easy as it gets, with cmake.

[d-torrance]

The default should probably be BUNDLED for consistency with the current behavior, and with the autotools build. Let users opt in for SYSTEM.

I wonder if AUTO is even necessary? If the default is to bundle, and users opt-in for SYSTEM, then presumably they already have the libraries on their system. If not, then raise an error, or maybe fall back on the bundled build.

[dimpase]

OK, let's keep BUNDLED. Let's keep SYSTEM strict: missing or incompatible installed libraries should produce an error. And AUTO would remain an explicit opt-in for users who want system discovery with bundled fallback.

How about this?

[d-torrance]

What's the advantage of this? I think it makes sense to run the unit tests for mathic & friends when we build them, but if they're already available in the system, then presumably they already passed the tests when the packages were built.

[dimpase]

I’ll remove this module, retain the library tests for bundled builds, and use M2’s own tests to validate system-library integration.

[d-torrance]

Why these minimum versions? M2 supports building with far older versions of the system libraries

[dimpase]

Are you saying that there are a bunch of systems out there with old versions of memtailor and mathic(gb) ?

Even if it's the case, these old versions are most probably cmake-deficient in some or another way.

[d-torrance]

Yes, absolutely. From https://launchpad.net/ubuntu/+source/mathicgb:

• Ubuntu 18.04: 1.0~git20170606
• Ubuntu 20.04: 1.0~git20211216
• Ubuntu 24.04: 1.0~git20240206
• Ubuntu 26.04: 1.2

1.4 didn't hit Ubuntu until 26.10, which was just released 4 days ago

[dimpase]

I guess one can have a fallback, checking system libraries directly, without cmake scaffolding.

Another question is whether we want this extra bloat, which will have to stay in for some time.

[d-torrance]

That's a good question. I care about supporting older versions for my Debian/Ubuntu packaging work, but I use the autotools build there.

[dimpase]

the CI erros is just a connection failure. A restart (which I can't trigger without another push) should fix it

[mahrud]

Since you requested my review, I'll share my honest feedback, which you can disregard if you wish.

I see several red flags here:

1. files like mathic-providers.yml, select-mathic-libraries.cmake, and test_mathic_providers.py seem over-engineered to the point that I can't understand their purpose on my own;
2. you directly quoted an LLM in response to Doug's question, which gives me the impression that you don't 100% understand them on your own either;
3. I can accept a complex kludge that I don't fully understand, even one written by AI, if it fixed an important problem I couldn't solve otherwise, but besides appealing to general virtues (vendoring bad, submodules aweful), I don't see what actual issue you're solving in exchange for the imposed technical debt.

Again, you can disregard all of this, and if anyone who wanted this change at Cornell can say that they 100% understand it and take responsibility for its maintenance, I don't mind at all if they go ahead and merge it.

[dimpase]

@mahrud - apologies, I got sloppy indeed. All these files you wanted gone are gone now.

[d-torrance]

In addition to the other comments, I'm still confused by the 3 separate loops we have in e/CMakeLists.txt. It looks like we:

• Detect what we have
• Check our debug/tbb flags
• Make sure everything will play nicely with each other

Is that the basic idea?

[d-torrance]

Do we need this? BUNDLED is the default, right?

[d-torrance]

I'm still not sure about these minimum versions. If the goal is to make sure that we have, e.g., Macaulay2/memtailor#23, then they're not sufficient. None of those PR's have been included in a release yet. (They will soon -- I'll release new versions prior to M2 1.26.11.)

[d-torrance]

Why not libmathicgbIsPresent() for consistency with the other two?