Skip to content

Free decoded keys before decoding them again at login - #261

Merged
philljj merged 3 commits into
wolfSSL:masterfrom
LinuxJedi:fix-rsa-bigint-leak
Oct 20, 2026
Merged

philljj merged 3 commits into
wolfSSL:masterfrom
LinuxJedi:fix-rsa-bigint-leak

Conversation

@LinuxJedi

@LinuxJedi LinuxJedi commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

C_Login decodes every token object again, including keys that are already decoded in memory. Public keys are not scrubbed at logout, and with an empty user PIN every object is decoded at load and again at login. The RSA, ECC, ML-DSA and ML-KEM decoders re-initialise the wolfCrypt key over the decoded one, which drops whatever the key had allocated without freeing it.

Nothing leaks with wolfSSL built from the documented flags. It shows up when the decoded key owns heap memory:

  • With HAVE_WOLF_BIGINT (e.g. wolfSSL --enable-pkcs11), each re-decoded RSA public key leaks its n and e bigint buffers. valgrind reports 61,383 bytes in 474 blocks definitely lost over one tests/pkcs11test run. In the empty-PIN flow, RSA private components and the ECC private scalar leak the same way.
  • With WOLFSSL_MLDSA_DYNAMIC_KEYS / WOLFSSL_MLKEM_DYNAMIC_KEYS, ML-DSA and ML-KEM keys leak their key buffers.

This was found by the valgrind step of the 2.2.0 release prep (#260). 2.1.0 has the same leak, so it is not a regression.

Changes:

  • Free the key before re-initialising it in wp11_Object_Decode_RsaKey, _EccKey, _MldsaKey and _MlKemKey, unless the object is marked encoded (in which case it has already been freed). The free on a failed decode is unchanged.
  • Add an ASAN-only "wolfSSL Bigint PQC Build" entry to the sanitizer matrix. It builds wolfSSL with --enable-pkcs11 and the ML-DSA/ML-KEM dynamic key buffers, then runs make check on the v3.2 PQC build, so LeakSanitizer catches this kind of leak. The other matrix entries are unchanged.
  • Build that entry's wolfSSL with -Wno-error=shorten-64-to-32. wolfSSL master's wc_pkcs11.c narrows CK_ULONG to word32 in its ML-DSA code (lines 4922 and 5234), and the -Werror that wolfSSL's configure adds in a git checkout turns that into a build failure. The warning is still printed. Fix -Wshorten-64-to-32 in the PKCS#11 ML-DSA callbacks wolfssl#11819 fixes it; once that merges, this flag can go.

Not changed, as they do not leak: DH and symmetric keys decode into fixed buffers; LMS/XMSS objects are public-only and their keys hold no heap memory; C_CopyObject, WP11_Rsa_PrivKey2PubKey, WP11_Rsa_ParsePrivKey and WP11_Object_SetRsaKey only initialise keys of new objects (key components are read-only once an object exists).

Testing:

  • make check on macOS: 75 pass, 9 skip, 0 fail. The pre-commit hook make test passes (current and default configs).
  • valgrind tests/pkcs11test, Debian trixie, wolfSSL 5.9.4 with --enable-pkcs11: 61,383 bytes lost before, no leaks after; 161/161 pass both times.
  • valgrind with wolfSSL master built with --enable-pkcs11 --enable-mldsa --enable-mlkem and both dynamic-key defines: no leaks in pkcs11test, pkcs11v3test, pkcs11str, empty_pin_store_test and private_object_empty_pin_test. A probe that creates token RSA/EC/ML-DSA/ML-KEM key pairs, logs in again and reloads lost 12.7 KB (and 29.5 KB with an empty PIN) before the change and nothing after.
  • The new CI job, reproduced locally with clang 19 ASAN and wolfSSL master 5f9ab72a from a git checkout: without the warning flag wolfSSL fails to build as in CI; with it the job fails on master (pkcs11test and pkcs11v3test, LeakSanitizer) and passes with the fix (75 pass, 9 skip, 0 fail).

C_Login decodes every token object, including keys that are already
decoded in memory: public keys are not scrubbed at logout, and with an
empty user PIN every object is decoded at load and again at login. The
RSA, ECC, ML-DSA and ML-KEM decoders re-initialise the wolfCrypt key
over the decoded one, which drops what it allocated without freeing it.

With a wolfSSL built with HAVE_WOLF_BIGINT (e.g. --enable-pkcs11) each
re-decoded RSA public key leaked its n and e bigint buffers; valgrind
reported 61,383 bytes in 474 blocks definitely lost over one pkcs11test
run. RSA and ECC private keys leaked their bigints the same way in the
empty-PIN flow, and ML-DSA and ML-KEM keys leaked their key buffers
when wolfSSL uses dynamic key buffers.

Free the key before re-initialising it unless it is marked encoded, in
which case it has already been freed. A failed decode still frees the
key and marks it encoded.
No CI job builds wolfSSL with HAVE_WOLF_BIGINT, so a decoded wolfCrypt
key there holds no heap memory and re-initialising it without a free
goes unnoticed. That is how the C_Login key re-decode leak survived.

Add an ASAN-only entry to the sanitizer matrix that builds wolfSSL with
--enable-pkcs11 (HAVE_WOLF_BIGINT) and ML-DSA/ML-KEM dynamic key
buffers, and runs make check on the PKCS#11 v3.2 PQC build, so
LeakSanitizer reports such leaks. Other entries pass empty wolfSSL
options and are unchanged.
Copilot AI balanced review requested due to automatic review settings October 20, 2026 13:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The cleanup follows the existing key-ownership model, and no blocking issues were identified.

No open issues 🎉

What changed in this PR

Fixes key-allocation leaks during repeated login decoding and adds sanitizer coverage for heap-backed keys.

Changes:

  • Frees already-decoded RSA, ECC, ML-DSA and ML-KEM keys before reinitialization.
  • Adds an ASAN matrix entry enabling bigint and dynamic post-quantum key allocations.
File Description
src/​internal.c Frees existing key allocations before decoding again.
.github/​workflows/​sanitizer-tests.yml Adds leak-detection coverage for heap-backed keys.

Review effort: Balanced

wolfSSL master does not build with --enable-pkcs11 --enable-mldsa
--enable-debug from a git checkout: wc_pkcs11.c narrows CK_ULONG to
word32 in its ML-DSA code, and the -Werror that configure adds in a git
checkout turns -Wshorten-64-to-32 into an error. Keep that one
diagnostic a warning in the bigint build so the job reaches the tests.

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #261

Scan targets checked: wolfpkcs11-src, wolfpkcs11-bugs
Not reviewed (no scan target covers them): .github/workflows/sanitizer-tests.yml
Coverage: 1 of 1 in-scope changed file(s) opened by the reviewer

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

Review tier: Lite

@philljj philljj self-assigned this Oct 20, 2026
@philljj
philljj merged commit c2c9274 into wolfSSL:master Oct 20, 2026
100 checks passed
LinuxJedi added a commit to LinuxJedi/wolfPKCS11 that referenced this pull request Oct 21, 2026
Add the C_Login key re-decode leak fix and move the release date to
Oct 21, 2026.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants