Harden DTLS handshake fragment reassembly and rx message handling by julek-wolfssl · Pull Request #11757 · wolfSSL/wolfssl

julek-wolfssl · 2026-10-15T12:43:46Z

This series tightens how DTLS 1.2 and 1.3 store, order and reassemble buffered handshake fragments, so spoofed or malformed plaintext fragments can no longer stall or abort a handshake.

Fragment validation and limits

Reject WOLFSSL_DTLS_FRAG_POOL_SZ above 255, since fragBucketListCount is a byte and a larger value wraps the counter and disables the per-message fragment limit.
Reject the no_shake sentinel (255) as a wire type, which previously let a type 255 fragment skip the per-fragment type, epoch and seq check.
Make the fragment bounds check wrap-safe.
Do not store zero-length fragments, which used to take a bucket slot.
Return BAD_STATE_E and drop the buckets on a fragment assembly inconsistency, instead of leaving the message unready forever.

Rx list handling and robustness against spoofing

Order the rx message list by sequence number alone, since message_seq is one counter across epochs. The complete expected message is now found wherever it is in the list.
Process buffered messages in epoch order, skipping and purging stale entries from an epoch below the last processed one.
Purge stored messages the handshake has moved past. A stale partial copy at the head used to block later buffered messages.
Let the latest fragment describe a plaintext message. An unauthenticated partial entry is replaced when a fragment disagrees with it, so one spoofed fragment can't reserve a sequence number.
DTLS 1.3: drop unauthenticated fragments that fall outside their message or arrive when the rx list is full, instead of failing fatally. Authenticated fragments and the next in-order message take the room of out-of-order plaintext entries.
DTLS 1.3: only a complete ServerHello acknowledges the ClientHello, so a spoofed fragment can't flush it from the retransmit buffer.
Keep a buffered message across a handshake suspend (e.g. OCSP_WANT_READ). DtlsMsgDrain used to free it while ProcessPeerCerts still pointed into it.

Other fixes

DTLS 1.3: Dtls13HashHandshake took a word16 length, so reassembled messages above 65535 bytes (possible with PQ certificate chains) were hashed with a wrapped length. It now takes a larger length.
Keep the original block when wolfSSL_Realloc fails with WOLFSSL_CHECK_MEM_ZERO. The copy and free previously ran on a NULL result.
Guard the fragment bucket copy fallback with WOLFSSL_NO_REALLOC rather than #ifdef XREALLOC, which is always defined.
Fix XFREE with WOLFSSL_STATIC_MEMORY and WOLFSSL_XFREE_NO_NULLNESS_CHECK, where the macro referenced xp, which only the nullness-checking variant declares.

Tests

Add a randomized differential DTLS fragment bucket test against a coverage bitmap. It covers zero-length fragments, the bucket limit, type 255, allocation failures, the assembly failure and BAD_STATE_E recovery, type mismatch on an existing message, inserting before the first bucket, message allocation failure with an empty list, and per-fragment allocation counts. Allocation failure injection is skipped when XMALLOC bypasses wolfSSL_Malloc.

fragBucketListCount is a byte. A larger value wraps the counter and disables the per-message fragment limit.

Reject the no_shake sentinel as a wire type. It marks an unpopulated message, so a type 255 fragment skipped the per-fragment type, epoch and seq check. Make the fragment bounds check wrap-safe.

They carry no data but used to take a fragment bucket slot.

Return BAD_STATE_E and drop the buckets instead of leaving the message unready forever.

Dtls13HashHandshake took a word16 length. A reassembled message above 65535 bytes, possible with PQ certificate chains, was hashed with a wrapped length.

Differential test against a coverage bitmap. Covers zero-length fragments, the bucket limit, type 255 and allocation failures.

…_MEM_ZERO The copy and free ran on a NULL result. Callers rely on realloc leaving the old pointer valid on failure.

…cases Accept the bucket limit as an allocation-free outcome in the injected pass. Skip injection when XMALLOC bypasses wolfSSL_Malloc. Exercise the BAD_STATE_E recovery path.

Type mismatch on an existing message, the bucket limit when inserting before the first bucket, and message allocation failure with an empty list.

The fallback was guarded by #ifdef XREALLOC, which is always defined because settings.h undefines it before types.h defines it. Use the WOLFSSL_NO_REALLOC guard like the other XREALLOC call sites.

One allocation per message and bucket, one growth per combined fragment.

A complete in-order message is processed from the record buffer. A partial copy of it stored earlier stayed at the head of the rx list and blocked every later buffered message.

A fragment outside its message or a full rx list was a fatal error even for plaintext records, so a few spoofed datagrams could abort a handshake. Drop them instead. Authenticated fragments and the next message in order take the room of out of order plaintext entries.

The first fragment pinned the size and type of a stored message. One spoofed plaintext fragment could reserve a sequence number and block the real message for the rest of the handshake. Replace an unauthenticated partial entry when a fragment disagrees with it.

message_seq is one counter across epochs. An entry with a lower epoch and a higher number at the head hid every later message appended behind it, and the consumers only looked at the head. Find the complete expected message wherever it is.

DtlsMsgDrain freed the message on OCSP_WANT_READ while ProcessPeerCerts still pointed into it, and only async builds processed the list again on resume. A message suspended from the list has its fragment consumed, so the record is finished; a message suspended in place keeps the rewound index and is read from the record again.

The macro referenced xp, which only the nullness checking variant declares.

A spoofed ServerHello fragment flushed the ClientHello from the retransmit buffer before anything was validated, so a lost ServerHello then stalled the client silently.

message_seq is one counter per handshake and epochs only go up, so a stored message from an epoch below the last processed one is stale. Track that epoch, skip and purge such entries.

github-actions · 2026-10-15T12:48:07Z

MemBrowse Memory Report · Explore what changed →

gcc-arm-cortex-m0plus

FLASH: .text +824 B (+1.2%, 70,459 B / 262,144 B, total: 27% used)

gcc-arm-cortex-m3

FLASH: .text +996 B (+0.8%, 130,769 B / 262,144 B, total: 50% used)
RAM: .bss +12 B, .data +4 B (+2.0%, 836 B / 65,536 B, total: 1% used)

gcc-arm-cortex-m4

FLASH: .rodata.CSWTCH.1 +8 B, .rodata.str1.1 +83 B, .text +1,024 B (+0.5%, 209,948 B / 262,144 B, total: 80% used)
RAM: .bss +12 B, .data +4 B (+1.9%, 856 B / 65,536 B, total: 1% used)

gcc-arm-cortex-m4-baremetal

FLASH: .text +832 B (+1.2%, 73,015 B / 262,144 B, total: 28% used)

gcc-arm-cortex-m4-crypto-only

FLASH: .rodata.CSWTCH.1 +8 B, .rodata.str1.1 +83 B, .text +960 B (+0.6%, 182,591 B / 262,144 B, total: 70% used)

gcc-arm-cortex-m4-dtls13

FLASH: .rodata +8 B, .text +1,344 B (+0.7%, 194,456 B / 1,048,576 B, total: 19% used)
RAM: .bss +12 B, .data +4 B (+1.9%, 872 B / 262,144 B, total: 0% used)

gcc-arm-cortex-m4-min-ecc

FLASH: .text +832 B (+1.3%, 67,673 B / 262,144 B, total: 26% used)

gcc-arm-cortex-m4-openssl-compat

FLASH: .rodata +64 B, .text +2,688 B (+0.3%, 796,140 B / 1,048,576 B, total: 76% used)
RAM: .bss +16 B (+0.0%, 133,820 B / 262,144 B, total: 51% used)

gcc-arm-cortex-m4-pkcs7

FLASH: .rodata.CSWTCH.1 +8 B, .rodata.str1.1 +83 B, .text +896 B (+0.4%, 223,552 B / 262,144 B, total: 85% used)

gcc-arm-cortex-m4-pq

FLASH: .rodata +104 B, .text +1,856 B (+0.6%, 311,716 B / 1,048,576 B, total: 30% used)
RAM: .bss +16 B, .data +4 B (+0.1%, 16,680 B / 262,144 B, total: 6% used)

gcc-arm-cortex-m4-rsa-only

FLASH: .rodata +104 B, .text +1,280 B (+0.4%, 343,128 B / 1,048,576 B, total: 33% used)
RAM: .bss +16 B, .data +4 B (+0.1%, 16,676 B / 262,144 B, total: 6% used)

gcc-arm-cortex-m4-sp-math

FLASH: .text +832 B (+1.3%, 67,673 B / 262,144 B, total: 26% used)

gcc-arm-cortex-m4-tls12

FLASH: .text +960 B (+0.7%, 131,537 B / 262,144 B, total: 50% used)
RAM: .bss +12 B, .data +4 B (+2.0%, 836 B / 65,536 B, total: 1% used)

gcc-arm-cortex-m4-tls13

FLASH: .rodata.CSWTCH.1 +8 B, .rodata.str1.1 +83 B, .text +1,088 B (+0.5%, 249,230 B / 262,144 B, total: 95% used)
RAM: .bss +16 B, .data +4 B (+0.1%, 16,628 B / 65,536 B, total: 25% used)

gcc-arm-cortex-m7

FLASH: .rodata.CSWTCH.1 +8 B, .rodata.str1.1 +83 B, .text +1,024 B (+0.5%, 209,948 B / 262,144 B, total: 80% used)
RAM: .bss +12 B, .data +4 B (+1.9%, 856 B / 65,536 B, total: 1% used)

gcc-arm-cortex-m7-pq

FLASH: .rodata +104 B, .text +1,856 B (+0.6%, 312,676 B / 1,048,576 B, total: 30% used)
RAM: .bss +16 B, .data +4 B (+0.1%, 16,680 B / 262,144 B, total: 6% used)

gcc-arm-cortex-m7-tls13

FLASH: .rodata.CSWTCH.1 +8 B, .rodata.str1.1 +83 B, .text +1,088 B (+0.5%, 249,294 B / 262,144 B, total: 95% used)
RAM: .bss +16 B, .data +4 B (+0.1%, 16,628 B / 65,536 B, total: 25% used)

linuxkm-standard

Data: __patchable_function_entries +136 B (+0.3%, 52,072 B)

stm32-sim-stm32h753

FLASH: .data +4 B, .text +9,096 B (+4.6%, 205,720 B / 2,097,152 B, total: 10% used)

zephyr-frdm_rw612

FLASH: datas +8 B, rodata +6,780 B, text +4,168 B (+4.0%, 282,448 B / 67,108,864 B, total: 0% used)
RAM: bss -28 B, datas +8 B (-0.0%, 58,358 B / 983,040 B, total: 6% used)

_{Catch memory regressions in CI with MemBrowse. Free for open source.}

Copilot

🟡 Changes recommended

Epoch truncation can discard valid DTLS 1.3 messages, and missing test guards break DTLS-disabled builds with MTU support enabled.

2 open findings

What changed in this PR

Hardens wolfSSL’s DTLS handshake reassembly and buffered-message handling against malformed or spoofed fragments.

Changes:

Tightens fragment validation, message ordering, cleanup, and handshake suspension handling.
Fixes large-message hashing and memory-management edge cases.
Adds randomized fragment tests and DTLS/OCSP regressions.

File	Description
wolfssl/wolfcrypt/types.h	Fixes static-memory free macros.
wolfssl/internal.h	Adds epoch tracking, helpers, and fragment limits.
wolfcrypt/src/memory.c	Preserves allocations when reallocation fails.
tests/api/test_ocsp.h	Declares DTLS OCSP regression test.
tests/api/test_ocsp.c	Tests nonblocking DTLS OCSP handling.
tests/api/test_dtls.h	Declares and registers DTLS tests.
tests/api/test_dtls.c	Adds fragment and buffered-message regressions.
tests/api.c	Registers the OCSP regression test.
src/internal.c	Hardens reassembly and buffered-message lifecycle.
src/dtls13.c	Updates buffering, acknowledgments, and transcript hashing.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Keep the full epoch in DtlsMsg and in the epoch floor (keys.dtls_peer_hs_epoch) so a DTLS 1.3 epoch past 65535 is not purged as stale. DTLS 1.3 stores the reconstructed epoch of the record that carried the message (keys.curEpoch64), not dtls13PeerEpoch, since a buffered message can come from an older epoch. DTLS 1.2 widens its 16-bit record epoch. keys.curEpoch stays 16-bit. DtlsMsg and the floor also exist in DTLS 1.2-only builds, where only TLS 1.3 turned on WOLFSSL_W64_WRAPPER. Turn it on for WOLFSSL_DTLS in settings.h. The w64wrapper type is always defined, so this only adds the misc.c helpers. On targets using the 32-bit Poly1305 code, a DTLS build without TLS 1.3 now also uses the w64wrapper Poly1305 path, as TLS 1.3 builds already do. Tests: cover epochs across 65535/65536 and 2^32, epochs more than 32768 apart, and KeyUpdates across the boundary with all handshake messages buffered. Require WOLFSSL_DTLS for the stale partial head test, since WOLFSSL_DTLS_MTU does not imply it. Uppercase literal suffixes, drop a dead store and rename prevEnd for clang-tidy and codespell. Spell out USE_WOLFSSL_MEMORY at the allocator calls for check-api-guards.

Crypto callback only builds have no software fallback for an unregistered device, so key updates failed with the made up device id. Without swdev the id is unregistered and crypto falls back to software.

rizlik

The PR contains correctness fixes and DoS protection. I agree with the correctness fixes.
I'm not convinced about the DoS protection: it closes some edge cases, but complete DoS
protection against an active attacker during the plaintext phase of the handshake cannot
be achieved. A well-formed bogus ServerHello (or HelloRetryRequest, or ClientKeyExchange
on a 1.2 server) still kills the handshake with one packet, with or without this PR, so
I'm not convinced all these changes are worth it.

What I think is worth keeping is the robustness side: bogus packets that can arise
normally, from a reconnection on the same port or from reordering, must not disrupt the
connection if possible. But let's consolidate that in DtlsShouldDrop() and
DtlsShouldDropHandshake() (src/internal.c), which already exist for exactly this
decision, instead of adding IsEncryptionOn() checks inside _Dtls13HandshakeRecv().

Widening DtlsShouldDropHandshake() to any
unauthenticated record (not only a stateless server) and adding BUFFER_ERROR,
DTLS_TOO_MANY_FRAGMENTS_E and DUPLICATE_MSG_E to its list covers both DTLS versions in one place.

For the rest of the DoS changes (the replacement of plaintext entries, the purge modes,
the epoch floor), I'd ask to consider whether they add any value, and
maintain them only if they cover a case that can arise from normal usage, not to protect
against an adversary. An adversary has a cheaper option that still works, so against them
these changes add complexity without protection. The description should also say that a
well-formed spoofed message during the plaintext phase still ends the handshake, as that
is a property of DTLS, not of this implementation.

On the 64-bit epochs (c2cffb3): an epoch above 65535 can never happen, because
message_seq overflows first.. This was pointed out by David Benjamin on the TLS list
("DTLS 1.3 epochs vs message_seq overflow", April 2024,
https://www.mail-archive.com/tls@ietf.org/msg17479.html) . So I'd keep the 16-bit epoch and add a comment explaining why it is enough.

Review feedback: during the plaintext phase any well-formed spoofed message ends the handshake, with or without these changes, so the defences that only an adversary could trigger add complexity without protection. Keep what covers records that arise in normal use, from reordering, re-fragmented retransmissions, duplication on the path or a stale peer on the same port, and consolidate the drop decision where it already lives. - DtlsShouldDropHandshake() now covers any unauthenticated record on either side, not only a stateless server, and also drops BUFFER_ERROR, DTLS_TOO_MANY_FRAGMENTS_E and DUPLICATE_MSG_E. _Dtls13HandshakeRecv() returns its errors instead of checking the encryption state itself. - Epoch 0 records are neither checked against nor counted in the replay window. They carry no MAC, and one plaintext record with a sequence number far ahead shut out every genuine record of the handshake until the retransmission timer gave up, in both DTLS 1.2 and DTLS 1.3. RFC 9147 Section 4.5.1: the window MUST NOT be updated due to a received record until that record has been deprotected successfully. Duplicate handshake messages in epoch 0 are already caught by their message_seq, and a repeated ChangeCipherSpec is already ignored. - Drop the epoch floor and the 64-bit message epochs. A message with a higher number in a lower epoch cannot come from a conforming peer, and without the floor the 16-bit epoch is only part of the lookup key. WOLFSSL_W64_WRAPPER is no longer forced on for DTLS builds. - Drop the plaintext entry replacement and the purge of plaintext entries when the list is full. Only an adversary produces those. - Flush the ClientHello on any ServerHello fragment again. RFC 9147 Section 7.2 makes any record of the responding flight an implicit acknowledgement. - Keep the lint fixes and the WOLFSSL_DTLS guard on the stale partial head test. - Tests: a plaintext record with a fragment outside its message, one far ahead in record sequence, and an exact duplicate of the first record, each ahead of the server's first flight; the handshake must complete for a client of either version. test_dtls_certreq_order and the DTLS 1.2 half of test_dtls13_ignore_legacy_record_version now expect the bad plaintext record to be dropped rather than to fail the connection, and test_dtls_old_seq_number expects a flight with a reused record sequence number to be processed rather than dropped.

julek-wolfssl · 2026-10-28T15:12:27Z

Drop decision consolidated in DtlsShouldDropHandshake(): any unauthenticated record, either side, plus BUFFER_ERROR, DTLS_TOO_MANY_FRAGMENTS_E, DUPLICATE_MSG_E. _Dtls13HandshakeRecv() just returns its errors.
Removed: plaintext entry replacement, plaintext eviction from a full list, epoch floor, 64-bit message epochs (without the floor the 16-bit epoch is only part of the lookup key and message_seq never repeats), ServerHello gating (RFC 9147 7.2 makes any responding record an implicit ACK).
Kept: purge of entries the handshake moved past, ordering by message_seq alone, the suspend fix, the bucket and allocator fixes.
New: epoch 0 records no longer touch the replay window. One plaintext record far ahead in sequence stalled the whole handshake in both versions. RFC 9147 4.5.1: no window update before successful deprotection. Duplicates in epoch 0 are already handled by message_seq.

julek-wolfssl added 19 commits October 15, 2026 12:08

DTLS: reject WOLFSSL_DTLS_FRAG_POOL_SZ above 255

1449ef1

fragBucketListCount is a byte. A larger value wraps the counter and disables the per-message fragment limit.

DTLS: tighten DtlsMsgSet argument checks

7140448

Reject the no_shake sentinel as a wire type. It marks an unpopulated message, so a type 255 fragment skipped the per-fragment type, epoch and seq check. Make the fragment bounds check wrap-safe.

DTLS: do not store zero-length fragments

411fe63

They carry no data but used to take a fragment bucket slot.

DTLS: fail on fragment assembly inconsistency

ac79a6f

Return BAD_STATE_E and drop the buckets instead of leaving the message unready forever.

DTLS 1.3: hash handshake messages longer than 64K

ed9b61f

Dtls13HashHandshake took a word16 length. A reassembled message above 65535 bytes, possible with PQ certificate chains, was hashed with a wrapped length.

Add randomized DTLS fragment bucket test

129a54a

Differential test against a coverage bitmap. Covers zero-length fragments, the bucket limit, type 255 and allocation failures.

Keep the original block when wolfSSL_Realloc fails with WOLFSSL_CHECK…

4b71f26

…_MEM_ZERO The copy and free ran on a NULL result. Callers rely on realloc leaving the old pointer valid on failure.

DTLS bucket test: cover assembly failure and gate allocation failure …

5ddb9ac

…cases Accept the bucket limit as an allocation-free outcome in the injected pass. Skip injection when XMALLOC bypasses wolfSSL_Malloc. Exercise the BAD_STATE_E recovery path.

DTLS bucket test: cover remaining error returns

2a5bc4e

Type mismatch on an existing message, the bucket limit when inserting before the first bucket, and message allocation failure with an empty list.

DTLS: guard the fragment bucket copy fallback with WOLFSSL_NO_REALLOC

5dbb899

The fallback was guarded by #ifdef XREALLOC, which is always defined because settings.h undefines it before types.h defines it. Use the WOLFSSL_NO_REALLOC guard like the other XREALLOC call sites.

DTLS bucket test: count allocations per fragment

5cdfc9f

One allocation per message and bucket, one growth per combined fragment.

DTLS: purge stored messages the handshake moved past

7b9131d

A complete in-order message is processed from the record buffer. A partial copy of it stored earlier stayed at the head of the rx list and blocked every later buffered message.

Fix XFREE with WOLFSSL_STATIC_MEMORY and WOLFSSL_XFREE_NO_NULLNESS_CHECK

3947226

The macro referenced xp, which only the nullness checking variant declares.

DTLS 1.3: only a complete ServerHello acknowledges the ClientHello

7de41b7

A spoofed ServerHello fragment flushed the ClientHello from the retransmit buffer before anything was validated, so a lost ServerHello then stalled the client silently.

DTLS: process buffered handshake messages in epoch order

e6a0d10

message_seq is one counter per handshake and epochs only go up, so a stored message from an epoch below the last processed one is stale. Track that epoch, skip and purge such entries.

Copilot AI balanced review requested due to automatic review settings October 15, 2026 12:43

julek-wolfssl self-assigned this Oct 15, 2026

julek-wolfssl requested a review from rizlik October 15, 2026 12:44

Copilot started reviewing on behalf of julek-wolfssl October 15, 2026 12:44

View session

Copilot AI reviewed Oct 15, 2026

View reviewed changes

Comment thread tests/api/test_dtls.c Outdated

Comment thread wolfssl/internal.h Outdated

julek-wolfssl added 2 commits October 16, 2026 17:15

Fix CI: use the swdev device in the DTLS 1.3 epoch boundary test

adef246

Crypto callback only builds have no software fallback for an unregistered device, so key updates failed with the made up device id. Without swdev the id is unregistered and crypto falls back to software.

julek-wolfssl assigned wolfSSL-Bot and rizlik and unassigned julek-wolfssl Oct 20, 2026

rizlik requested changes Oct 27, 2026

View reviewed changes

julek-wolfssl assigned julek-wolfssl and unassigned rizlik and wolfSSL-Bot Oct 28, 2026

julek-wolfssl requested a review from rizlik October 28, 2026 15:12

julek-wolfssl assigned rizlik and wolfSSL-Bot and unassigned julek-wolfssl Oct 28, 2026

Navigation Menu

Repository navigation

Harden DTLS handshake fragment reassembly and rx message handling - #11757

Harden DTLS handshake fragment reassembly and rx message handling#11757
julek-wolfssl wants to merge 22 commits into
wolfSSL:masterfrom
julek-wolfssl:dtls-frag-bucket-hardening

julek-wolfssl commented Oct 15, 2026

Uh oh!

github-actions Bot commented Oct 15, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

rizlik left a comment

Uh oh!

julek-wolfssl commented Oct 28, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

julek-wolfssl commented Oct 15, 2026

Uh oh!

github-actions Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MemBrowse Memory Report · Explore what changed →

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

rizlik left a comment

Choose a reason for hiding this comment

Uh oh!

julek-wolfssl commented Oct 28, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Oct 15, 2026 •

edited

Loading