[LinuxJedi]

What and why

With OPENSSL_ALL and KEEP_PEER_CERT (and without NO_CERT_IN_TICKET) a session ticket can carry the peer certificate, so SESSION_TICKET_LEN includes MAX_TICKET_PEER_CERT_SZ (2048 bytes). Every buffer sized from it was sized for the largest possible ticket:

• each WOLFSSL_SESSION had a ~2.3 KB staticTicket, including every row of the static session cache;
• MAX_PSK_ID_LEN followed the same size, so Arrays carried two ~2.3 KB identity buffers and WOLFSSL_CTX one.

Most tickets carry no certificate, so this PR sizes those buffers for a ticket without one. A longer ticket uses the existing dynamic ticket buffer (ticket / ticketLenAlloc).

• WOLFSSL_SESSION.staticTicket is SESSION_TICKET_STATIC_LEN, the length of a ticket without a certificate. Builds that do not put the certificate in the ticket keep their sizes (SESSION_TICKET_STATIC_LEN == SESSION_TICKET_LEN there).
• CreateTicket() selects the peer certificate first and sizes the ticket from it. The encrypt callback gets WOLFSSL_TICKET_ENC_BASE_SZ plus the certificate length, which is WOLFSSL_TICKET_ENC_SZ minus the certificate space the ticket does not use. The ticket format and server-side decryption are unchanged.
• MAX_PSK_ID_LEN defaults to SESSION_TICKET_STATIC_LEN, as in builds without OPENSSL_ALL. The new MAX_PSK_ID_WIRE_LEN lets a TLS 1.3 server still accept its own longer tickets as PSK identities. A longer identity is not copied into Arrays and is not offered to the PSK callbacks, on either the TLS 1.3 or the DTLS 1.3 stateless path (the bound is in FindPskSuite()).
• "Fits in the static buffer" checks use sizeof(staticTicket) in the session cache, wolfSSL_SESSION_dup(), d2i and wolfSSL_set_SessionTicket().

Visible change: MAX_PSK_ID_LEN default

With OPENSSL_ALL and certificates in tickets, MAX_PSK_ID_LEN used to be the full ticket size (about 2.3 KB). It is now the size of a ticket without a certificate (a few hundred bytes), the same limit builds without OPENSSL_ALL already have. As a result:

• a TLS 1.3 server no longer offers a longer external PSK identity to its PSK callback (a debug message says so);
• a longer TLS 1.2 PSK identity fails the handshake;
• identity hints are cut to the new length.

Define MAX_PSK_ID_LEN to keep a larger limit.

Fixes found along the way

• CreateTicket(): when the encrypt callback reported too long an output, BAD_TICKET_KEY_CB_SZ was set but the code did not jump to the error path, so ticketLen could point past the end of the buffer. It now fails cleanly. ticketLen is also cleared until the new ticket is complete.
• SetTicket() (client receiving a NewSessionTicket): when the allocation for a longer ticket failed after the old dynamic ticket was freed, ticketLen kept the old length on the static buffer. It is now cleared. Certificate-carrying tickets make this path reachable for ordinary tickets.
• GetSessionFromCache(): it recorded the 512-byte scratch size as ticketLenAlloc for a buffer it had allocated with ticketLen bytes. It now records the size it allocated.
• wolfSSL_set_SessionTicket(): it decided whether to reallocate from the previous ticketLen rather than from ticketLenAlloc, and left ticketLen set when the allocation failed.
• Restoring a persisted session cache (SessionSanityPointerSet()): an oversize ticketLen was clamped to the static size, which would send a truncated ticket of stale bytes. The ticket is now dropped. Trade-off: a saved cache only holds the static buffer, so with OPENSSL_ALL a cert-carrying ticket is no longer kept across a session cache save and restore (file or memory). The session itself is kept.

Measured savings

Cortex-M33 (STM32U585), OPENSSL_ALL, before -> after:

	Before	After
Arrays	4976 B	880 B
WOLFSSL_SESSION	2588 B	540 B
WOLFSSL_CTX	2888 B	840 B
Static session cache	85536 B (85.5 KB)	17952 B (18 KB)
TLS 1.3 client handshake heap peak	59040 B	50848 B
Client resumption heap peak	27773 B	21629 B

Handshake times did not change. Builds without OPENSSL_ALL (or with NO_CERT_IN_TICKET) have the same sizes as before.

Tests

New in tests/api/test_tls13.c:

• test_tls13_ticket_peer_cert_resume: mutual auth with the server session cache off, over TLS 1.3 and TLS 1.2. It checks that both sides hold the cert-carrying ticket in the dynamic buffer, then resumes from the session, from a wolfSSL_SESSION_dup() copy and from an i2d/d2i copy, and compares the resumed peer certificate with the client certificate.
• test_tls13_ticket_enc_too_long: an encrypt callback that reports one byte more than it was given. It expects BAD_TICKET_KEY_CB_SZ and ticketLen == 0, and it fails without the CreateTicket() fix.
• test_tls13_ticket_set_alloc_fail: a client holding a cert-carrying ticket fails to allocate the buffer for a second one. It expects the session to be left without a ticket, and it fails without the SetTicket() fix.

Existing tests that hard-coded SESSION_TICKET_LEN or MAX_PSK_ID_LEN use the new limits.

make check passes on Linux/gcc for --enable-opensslall, --enable-all, --enable-opensslextra and --enable-session-ticket, and on macOS/clang for --enable-opensslall and --enable-all. Before the rebase onto current master it also passed under ASan+UBSan with --enable-all, with a small MAX_PSK_ID_LEN=128, and with --enable-savesession + WOLFSSL_SESSION_INDEX.

Related

This PR is based on current master and does not depend on any other PR. The peer certificate pointer change (#11990, WOLFSSL_DYNAMIC_PEER_CERT) is a separate PR, and the two touch the same line. Whichever lands second changes CreateTicket()'s ssl->peerCert.derCert to GetPeerCert(ssl)->derCert.

[github-actions]

MemBrowse Memory Report · Explore what changed →

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text -128 B (-0.0%, 798,020 B / 1,048,576 B, total: 76% used)
• RAM: .bss -12,288 B (-9.2%, 121,532 B / 262,144 B, total: 46% used)

No memory changes detected for:

• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• stm32-sim-stm32h753
• zephyr-frdm_rw612

---

_{Catch memory regressions in CI with MemBrowse. Free for open source.}

[Copilot]

🟡 Changes recommended

The session layout change lacks the required cache-version bump, and failed ticket replacement can leave a stale length that causes out-of-bounds reads.

2 open issues

• Clear stale ticket length when replacement allocation fails · New
• Bump cache version after changing persisted session layout · New

What changed in this PR

This PR reduces wolfSSL’s session and PSK buffer footprint by sizing static storage for tickets without peer certificates, while keeping larger tickets in dynamic storage.

Changes:

• Separates static ticket and PSK buffer limits from the maximum accepted ticket size.
• Updates ticket allocation, copying, restoration, and encryption error handling.
• Adds certificate-carrying ticket resumption and oversized callback-output tests.

File	Description
wolfssl/​internal.h	Defines separate static and wire limits; shrinks session ticket storage.
tests/​api/​test_tls13.h	Registers the new ticket tests.
tests/​api/​test_tls13.c	Tests certificate-bearing ticket resumption and callback output limits.
tests/​api/​test_tls_msgtype.c	Updates PSK parsing tests to use the wire limit.
tests/​api/​test_ssl_ext.c	Updates a comment naming the static ticket limit.
tests/​api/​test_session.c	Adjusts dynamic-ticket thresholds and cache restoration expectations.
src/​tls13.c	Keeps long ticket identities out of static identity buffers and PSK callbacks.
src/​tls.c	Accepts PSK identities up to the separate wire limit.
src/​ssl_sess.c	Updates ticket storage checks and cache allocation bookkeeping.
src/​ssl_api_ext.c	Uses allocated capacity for ticket replacement and clears length on allocation failure.
src/​internal.c	Sizes ticket creation buffers from certificate content and rejects invalid callback lengths.

_{Review effort: Balanced}

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11991

Scan targets checked: wolfssl-src, wolfssl-bugs
Not reviewed (no scan target covers them): tests/api/test_tls13.h
Coverage: 6 of 10 in-scope changed file(s) opened by the reviewer; not opened: src/tls.c, tests/api/test_ssl_ext.c, tests/api/test_tls13.c, tests/api/test_tls_msgtype.c

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

Review tier: Lite