[LinuxJedi]

Description

With WOLFSSL_DYNAMIC_PEER_CERT (#11990, on by default with WOLFSSL_SMALL_STACK), a connection now keeps only the DER of the peer's leaf certificate when it arrives. The WOLFSSL_X509 is built from it the first time something asks for it, for example wolfSSL_get_peer_certificate(). This follows #11914, #11991, #11992 and #11990 (all merged), is based on master, and is not stacked on another PR.

On an established compat-layer client connection, most of the heap above a non-compat build was the decoded peer certificate kept by KEEP_PEER_CERT. Host harness, ECC leaf, OPENSSL_EXTRA minus no compat layer = 7128 bytes, of which 5320 is the peer certificate:

• the WOLFSSL_X509 struct, with its two embedded names: 2936
• one ASN1_STRING per name entry, 14 of them: 1456
• the DER copy: 712
• the public key, signature and key ID copies: 216

OPENSSL_ALL adds a larger struct (4032), a stack node per name entry and an ASN1_OBJECT. An application that never looks at the peer certificate paid all of this for the life of the connection.

Changes

• ssl->peerCertDer holds the leaf's encoding until ssl->peerCert is built from it, then is freed: the object has its own copy. At most one of the two is set.
• SetPeerCertDer() keeps the encoding. It is used when ProcessPeerCerts() keeps the leaf and when the certificate is restored from a session ticket. Without WOLFSSL_DYNAMIC_PEER_CERT it is SetPeerCert(), so those builds are unchanged.
• GetPeerCert() is now a function in dynamic builds. When only the encoding is kept it builds the object as d2i_X509() does (NO_VERIFY, names added straight to the X509) and keeps it in its place. When memory runs out the encoding is kept so a later call can try again. A copy that fails part way is still kept, as SetPeerCert() does.
• GetPeerCertDer() returns the encoding without building the object. Session ticket creation and wolfSSL_cmp_peer_cert_to_file() use it.
• Verify callback. When the callback runs for the leaf, SetupStoreCtxCallback() builds the object from the already decoded leaf, not from the encoding. The callback's current certificate is still the kept object, so pointer identity and ex_data set in the callback behave as before.
• Leaves accepted despite a decode error. Only a leaf whose own ParseCertRelative() returned no error is kept as its encoding alone (new ProcPeerCertArgs.leafDecodeErr). Some errors come back even with NO_VERIFY, such as KEYUSAGE_E for keyCertSign on a non-CA certificate, and the handshake can still accept such a leaf: the chain verify callback clears content errors, ProcessPeerCertCb can override the result, and a verify callback can accept it in builds without OPENSSL_EXTRA. Such a leaf is copied from the decoded certificate as before, since decoding it again would fail the same way.
• PushPeerCertToChain() and CreatePeerCertChain() take a non-const WOLFSSL*, since getting the peer certificate can now build it.

No public structure changes. WOLFSSL gains one pointer.

Results

STM32U585 (Cortex-M33 at 160 MHz, newlib-nano malloc), TLS 1.3 client and server over memory I/O, bytes unless noted, master -> this PR. Heap kept by the established client connection (its CTX and CA included), no verify callback:

Chain	No compat layer	OPENSSL_EXTRA	OPENSSL_ALL
ECC leaf	3445	9409 -> 5697	11846 -> 6652
ECC leaf + 2 intermediates	3932	10325 -> 6434	14152 -> 8779
RSA leaf + 2 intermediates	4461	11596 -> 7346	16819 -> 11086

For the ECC leaf the compat layer's extra cost goes from 5964 to 2252 bytes (OPENSSL_EXTRA) and from 8401 to 3207 (OPENSSL_ALL).

ECC leaf	OPENSSL_EXTRA	OPENSSL_ALL
Client handshake heap peak	14822 -> 12925	16304 -> 13294
Client handshake allocations	387 -> 365	403 -> 365
Client handshake time	69.6 -> 69.1 ms	70.1 -> 69.3 ms
First wolfSSL_get_peer_certificate()	1.2 ms, 35 allocations	1.3 ms, 51 allocations
Flash	-112	-128

With a verify callback set, or once the certificate has been asked for, the numbers are the same as before (+8 bytes for the pointer).

Host (macOS arm64), same harness, ECC leaf, no verify callback: resident 11201 -> 6601 (OPENSSL_EXTRA) and 14558 -> 7724 (OPENSSL_ALL); handshake peak 14134 -> 10498 and 17352 -> 11482.

Not in this PR

An application that sets a verify callback or asks for the peer certificate still holds the whole WOLFSSL_X509, about 4 KB on the M33 with OPENSSL_EXTRA. The remaining large parts are each name's 330-byte one-line string, the fixed entry[MAX_NAME_ENTRIES] array and a heap ASN1_STRING per name entry. Shrinking those touches much of x509.c and is left for a separate change.

Testing

New tests:

• test_wolfSSL_peer_cert_on_demand (TLS 1.3 and 1.2, dynamic builds):
  • only the encoding is kept after the handshake, and cmp_peer_cert_to_file() builds nothing;
  • out of memory, no object is returned and the encoding stays;
  • get_peer_certificate() builds an object with the same DER, names and contents as the certificate sent, and the encoding is dropped;
  • wolfSSL_clear() drops the encoding;
  • with a verify callback, the callback's leaf is the kept object.
• test_wolfSSL_peer_cert_kept_decode_err: a generated leaf with keyCertSign but not a CA, accepted by the chain verify callback, is kept as an object. It fails with the decode-error handling disabled.

test_wolfSSL_get_peer_certificate_ref no longer calls GetPeerCert() from the test, as it is now a hidden symbol in dynamic builds. The ticket re-verify test also checks that no encoding is kept.

make check passes on macOS (clang):

• --enable-opensslextra, --enable-opensslall, --enable-all, --enable-opensslextra=x509small, default;
• with the new path on: --enable-opensslextra, --enable-opensslall and --enable-all with -DWOLFSSL_DYNAMIC_PEER_CERT, --enable-opensslextra=x509small with -DKEEP_PEER_CERT -DWOLFSSL_DYNAMIC_PEER_CERT, --enable-opensslextra --enable-smallstack, and --enable-opensslextra --enable-asn=original with -DWOLFSSL_DYNAMIC_PEER_CERT.

Also:

• Linux (gcc 16): --enable-all -DWOLFSSL_DYNAMIC_PEER_CERT builds with -Wall -Wextra -Wdeclaration-after-statement -Werror, and make check passes for --enable-opensslextra --enable-smallstack.
• ASan + UBSan, --enable-all -DWOLFSSL_DYNAMIC_PEER_CERT: the whole unit test passes. The only UBSan report is a misaligned load in wolfssl/test.h, which this PR does not touch.

[github-actions]

MemBrowse Memory Report · Explore what changed →

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text +320 B (+0.0%, 800,868 B / 1,048,576 B, total: 76% used)

No memory changes detected for:

• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• stm32-sim-stm32h753
• zephyr-frdm_rw612

---

_{Catch memory regressions in CI with MemBrowse. Free for open source.}

[Copilot]

🟡 Changes recommended

The tests do not exercise wolfSSL_clear() while peerCert is NULL and peerCertDer is non-NULL, leaving DER-only cleanup unverified.

1 open issue

• Add clear test for DER-only peer certificate cleanup · New

What changed in this PR

Reduces per-connection memory use in dynamic peer-certificate builds by retaining only the certificate’s DER encoding until an X509 object is requested.

Changes:

• Adds lazy X509 construction, with retry support after allocation failure.
• Uses DER directly for certificate comparison and session tickets while preserving verification-callback behavior.
• Adds tests for on-demand decoding, allocation failure, callback identity, and certificates accepted despite decode errors.

File	Description
wolfssl/​internal.h	Adds DER storage, decode-error tracking, and internal accessors.
tests/​api/​test_tls13.c	Checks that ticket re-verification leaves no retained DER.
tests/​api/​test_ssl_cert.h	Registers the new peer-certificate tests.
tests/​api/​test_ssl_cert.c	Tests lazy construction and accepted decode-error handling.
src/​ssl_api_cert.c	Compares DER directly and adapts chain helpers to lazy construction.
src/​internal.c	Implements DER retention, lazy decoding, cleanup, and handshake/ticket integration.

_{Review effort: Balanced}

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #12020

Scan targets checked: wolfssl-src, wolfssl-bugs
Not reviewed (no scan target covers them): tests/api/test_ssl_cert.h
Coverage: 5 of 5 in-scope changed file(s) opened by the reviewer

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Review tier: Max

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #12020

Scan targets checked: wolfssl-src, wolfssl-bugs
Not reviewed (no scan target covers them): tests/api/test_ssl_cert.h
Coverage: 1 of 1 in-scope changed file(s) opened by the reviewer

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

Review tier: Lite

Fenrir's latest completed scan found no issues; clearing the prior automated change request.