[twcook86]

Description

Fixes F-16278 to handle keyupdate flood

Testing

new test

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m4-dtls13

• FLASH: .text +64 B (+0.0%, 194,392 B / 1,048,576 B, total: 19% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text +128 B (+0.0%, 796,268 B / 1,048,576 B, total: 76% used)

gcc-arm-cortex-m4-pq

• FLASH: .text +128 B (+0.0%, 311,844 B / 1,048,576 B, total: 30% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .text +128 B (+0.0%, 343,256 B / 1,048,576 B, total: 33% used)

gcc-arm-cortex-m4-tls13

• FLASH: .text +128 B (+0.1%, 249,358 B / 262,144 B, total: 95% used)

gcc-arm-cortex-m7-pq

• FLASH: .text +128 B (+0.0%, 312,804 B / 1,048,576 B, total: 30% used)

gcc-arm-cortex-m7-tls13

• FLASH: .text +128 B (+0.1%, 249,422 B / 262,144 B, total: 95% used)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m7
• linuxkm-pie
• linuxkm-standard
• stm32-sim-stm32h753

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11884

Scan targets checked: wolfssl-src, wolfssl-bugs
Not reviewed (no scan target covers them): tests/api/test_tls13.h
Coverage: 2 of 3 in-scope changed file(s) opened by the reviewer; not opened: tests/api/test_tls13.c

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

Review tier: Lite

[Copilot]

🟡 Changes recommended

Deferred-response state remains inconsistent across mixed KeyUpdates, explicit updates, and shutdown.

3 open issues

• Clear deferred KeyUpdate state when write is closed · New
• Clear deferred response state for update_not_requested · New
• Clear deferred marker after explicit KeyUpdate response · New

What changed in this PR

This PR addresses KeyUpdate-flood denial of service in wolfSSL’s TLS 1.3 handling by coalescing responses while output is blocked.

Changes:

• Defer repeated KeyUpdate responses until pending output drains.
• Send deferred responses before new application data while respecting update limits.
• Add regression coverage for blocked writes and deferred-response handling.

File	Description
tests/​api/​test_tls13.h	Declare and register the blocked-write regression test.
tests/​api/​test_tls13.c	Exercise KeyUpdate floods, write retries, and explicit updates.
src/​tls13.c	Defer responses when TLS output is already buffered.
src/​internal.c	Allow reads with deferred responses and adjust write-side response ordering.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Security-sensitive key-update state and nonblocking record ordering require human validation across supported configurations.

No open issues 🎉

3 resolved since last review

• Clear deferred marker after explicit KeyUpdate response
• Clear deferred response state for update_not_requested
• Clear deferred KeyUpdate state when write is closed

_{Review effort: Balanced}

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11884

Scan targets checked: wolfssl-src, wolfssl-bugs
Not reviewed (no scan target covers them): tests/api/test_tls13.h
Coverage: 2 of 3 in-scope changed file(s) opened by the reviewer; not opened: tests/api/test_tls13.c

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

Review tier: Lite

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11884

Fenrir already completed a review of this PR at commit 6d024a391d5e (run 3521); its findings are the review threads on the PR. Push a new commit to request another review.

[Copilot]

🟢 Approval recommended

There are no active review issues for the changes that defer TLS 1.3 KeyUpdate responses when writes are blocked.

No open issues 🎉

_{Review effort: Balanced}