refactor: migrate Botan backend crypto-refresh/PQC code to FFI

[ronaldtse]

Summary

Migrates the Botan backend off Botan's C++ API and onto the stable C FFI (botan_*), matching rnp's existing FFI discipline (which is what keeps rnp working across Botan 2.x/3.x). Stacks on #2392 (pqc-openssl). Fixes the Botan 3.12 build breakage on that branch (native include-hygiene errors + deprecations).

All migrated files contain zero Botan:: references:

• Crypto-refresh / PQC (CMake-gated to Botan 3.6+): ec.cpp (ec_generate_generic_native), ed25519_ed448.cpp, x25519_x448.cpp, exdsa_ecdhkem.cpp, dilithium.cpp (ML-DSA), kyber.cpp (ML-KEM), sphincsplus.cpp (SLH-DSA), kyber_ecdh_composite.cpp (RFC 3394 keywrap → botan_nist_kw_*), botan_utils.hpp (view callback + KEM op RAII wrappers)
• Always-compiled core (Botan 2.14+): hash.cpp/hash_botan.hpp (incl. CRC24 → botan_hash_*), hkdf_botan.cpp (botan_kdf), s2k.cpp (Argon2id → botan_pwdhash), elgamal.cpp (botan_mp_*/botan_mp_powmod), backend_version.cpp (botan_version_*), rng.cpp/rng.h (dropped the Botan::RandomNumberGenerator exposure; opaque botan_rng_t handle), mem.h (rnp::secure_allocator replacing Botan::secure_vector), and the always-compiled ec.cpp/eddsa.cpp keygen paths → botan_*_view_raw

OpenSSL-side *_ossl.cpp files are untouched.

Version compatibility

rnp supports Botan 2.14 → 3.x, so any FFI symbol added after 2.14 is behind a version gate with an older-Botan fallback — the same pattern rnp already uses (e.g. ecdh.cpp for botan_nist_kw_*):

• ec.cpp Key::generate_x25519 and eddsa.cpp generate: #if BOTAN_VERSION_CODE >= BOTAN_VERSION_CODE_FOR(3,6,0) → botan_*_view_raw; #else → the legacy botan_*_get_* getters (2.x-safe).
• botan_utils.hpp: view callback guarded >= 3.6.0; KEM op wrappers guarded #if defined(ENABLE_PQC).
• Everything else in the always-compiled core uses FFI available since ≤ 2.14 — no guard needed. The crypto-refresh/PQC files only compile under ENABLE_CRYPTO_REFRESH/ENABLE_PQC (which require 3.6+).

cipher_botan is intentionally left native (NOT a blocker)

cipher_botan.cpp/.hpp stays on Botan::Cipher_Mode. The Botan FFI exposes only the streaming botan_cipher_update — there is no equivalent of Botan::Cipher_Mode::process() (raw per-block, immediate output) which rnp's cipher streaming is built on, and CBC-decrypt inherently buffers a block. A correct FFI cipher therefore needs a 1-block-delay buffering rewrite on rnp's side.

This does not block the PR. The native cipher compiles and works correctly everywhere; everything else in the Botan backend is now FFI-only. The cipher migration is a self-contained follow-up — and, like every other FFI symbol, it will be version-gated regardless of what upstream does, so waiting on Botan to expose a process-equivalent wouldn't change rnp's gating architecture. (Filed as a question to @randombit in a separate comment, out of genuine curiosity — not a merge dependency.)

Test plan

• cmake -DCRYPTO_BACKEND=botan3 -DENABLE_CRYPTO_REFRESH=ON -DENABLE_PQC=ON builds clean on Botan 3.12.
• Full rnp_tests: 296/296 pass locally (no regressions); all PQC tests pass — this also fixes two PQC tests that failed against 3.12 on the native API (kyber_ecdh_roundtrip, test_ffi_pqc_gen_enc_sign).
• All migrated files contain zero Botan:: references (cipher_botan excepted, by design).
• clang-format (v11) clean on all changed files.
• CI across the Botan version matrix (2.x system, 3.3.0, 3.6+, head).

Notes

• pqc-openssl at origin is a staging copy of kaie's fork branch, created only so this PR can target Add an automatically generated OpenSSL backend for the Crypto Refresh and PQC code #2392. It can be deleted once Add an automatically generated OpenSSL backend for the Crypto Refresh and PQC code #2392 and this PR are handled.
• Supersedes refactor: migrate crypto-refresh EC/Ed code from Botan native API to FFI #2447.

[codecov]

Codecov Report

❌ Patch coverage is 87.30159% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.39%. Comparing base (b562146) to head (330878e).
⚠️ Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
src/lib/crypto/hash.cpp	81.81%	4 Missing ⚠️
src/lib/crypto/mem.h	75.00%	3 Missing ⚠️
src/lib/crypto/elgamal.cpp	96.15%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2448      +/-   ##
==========================================
- Coverage   85.40%   85.39%   -0.01%     
==========================================
  Files         126      126              
  Lines       22924    23008      +84     
==========================================
+ Hits        19578    19648      +70     
- Misses       3346     3360      +14     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ronaldtse]

@randombit Botan remains to be RNP's primary cryptographic library, and we're grateful for your support and commitment!

Since the PQC merge, which used a number of functions from Botan's internal API, we are trying to migrate to Botan's official FFI so RNP can use the same code across Botan 2.x/3.x without per-version #ifdef maintenance. This PR (and #2392 from @kaie) are part of that effort.

Most needs map cleanly to FFI functions except for Cipher_Mode::process() which is used for cipher streaming.

It seems that FFI exposes only the streaming botan_cipher_update but not a raw/per-block process-equivalent in ffi.h. The per-block process is used to process update_granularity-sized blocks with immediate output and no buffering.

What we observe (Botan 3.12.0, IDEA/CBC):

• botan_cipher_update with a single update_granularity block, decrypt direction -> consumed = 0, written = 0 (it buffers).
• feeding two granularity-blocks at once -> it consumes and produces both.
• encrypt direction consumes/produces a single block immediately.

So for CBC decrypt we'd need to buffer on our side (hold ≥2 blocks before we can emit the first plaintext block) to preserve RNP's immediate-output streaming contract.

A few questions:

1. Is there an FFI entry point for the raw, no-buffering per-block path that we missed? Or is botan_cipher_update the intended surface for this kind of streaming?
2. If streaming-only is the intended design (totally fine), is there a recommended pattern for consumers that want process-style immediate per-block output? We could implement a small buffering layer -- just don't want to do it the wrong way.

If the answer is "no raw path exists, please buffer," that's a perfectly good answer and will do it. If there's ever appetite to expose something like botan_cipher_process for the raw case, we'd find it useful (if that's a correct solution). In any case though, we would have to version gate.

Thanks again for any pointers. Really appreciated.

[randombit]

@ronaldtse good question and indeed right now we only support streaming via FFI. I am planning some extensions to cipher mode API in the relatively near future to make them easier to use, I will keep this use case in mind during the design process. But for now (and of course for all versions before this updated API lands), yes you'll have to buffer internally to handle this. In general you'll want to make inputs as large as possible (say ~~ 128 bytes to 4 Kb range) since there are many optimizations [SIMD implementations, interleaving, etc] that trigger when multiple blocks are processed at once.

[ni4]

LGTM, thanks!

[ronaldtse]

@randombit thanks — that matches what we see in practice. Our FFI layer passes the caller's buffer sizes straight to botan_cipher_update: OpenPGP AEAD data arrives as whole chunks (≥ 64 B, typically KiB-scale) and the stream layer reads KB-sized blocks, so we're naturally in the 128 B–4 KB+ range you recommend; only the final fragment of each message is small. We'll keep the updated cipher-mode API in mind and adopt it when it lands.