Skip to content

fix(mqtt): 8 KB MQTT task stack when TLS verifies a CA certificate - #3301

Merged
proddy merged 1 commit into
emsesp:devfrom
csutcliff:pr/mqtt-task-stack
Oct 20, 2026
Merged

proddy merged 1 commit into
emsesp:devfrom
csutcliff:pr/mqtt-task-stack

Conversation

@csutcliff

@csutcliff csutcliff commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem

On boards with PSRAM the MQTT client runs in its own task with a 5120 byte stack (EMC_TASK_STACK_SIZE). With a CA certificate configured, BearSSL verifies the broker's certificate chain inside that task. For an EC chain, such as Let's Encrypt's ISRG Root X2, that needs about 6 KB, so the task overflows and the ESP32 panics on every connect: a boot loop that only stops when MQTT or TLS is turned off.

Fix

Size the MQTT task stack from the TLS settings, as suggested in review:

  • the espMqttClient / espMqttClientSecure constructors take a stack size, defaulting to EMC_TASK_STACK_SIZE (5120)
  • MqttSettingsService::startClient() passes EMSESP_MQTT_STACKSIZE_CA (8192) when TLS is on and rootCA is a certificate, and keeps the default for insecure TLS and plain MQTT. The client is recreated when that changes, not only when TLS is toggled
  • the stack usage log in MqttClient.cpp reports against the size the task was created with

Boards without PSRAM don't create the task (the client runs on the loop task, 8 KB), so nothing changes for them.

Testing

Peak stack use, each read after a fresh boot whose only handshake used the listed chain:

Chain (CA verified) Free of 8192 (bytes) Peak use (bytes)
Let's Encrypt ISRG Root X2 (P-384 root and intermediate) 2152 ~6040
P-521 CA and server, ecdsa-with-SHA512 2212 ~5980
RSA-4096 root, intermediate and server 3268 ~4924

With rootCA set to "insecure" (one handshake, test server):

Server leaf key Peak use (bytes)
P-384 ~2616
P-521 ~5736, over 5120

So a P-521 broker certificate would still overflow the default in insecure mode. That's rare (Let's Encrypt doesn't issue P-521) and left as it is.

With this change, on a Gateway E32 V2 against a Let's Encrypt broker (P-256 leaf), switching rootCA in the settings without a restart, twice each way:

Setting Task stack Free (bytes)
X2 CA (after boot, and after each switch back) 8192 2144 to 2152
insecure 5120 624, flat over 5 minutes

Every switch recreated the client and reconnected, with no restart. Insecure mode against this broker peaks at ~4.5 KB of the 5120, which is the same as before this change.

  • Cost: 3 KB more internal RAM, only on boards with PSRAM and a CA set. Free internal heap 195 KB with the CA, 197 to 198 KB insecure.
  • Builds for s_16M_P, c3_mini_4M, native and native-test, clang-format clean in src/.

Related MQTT TLS fixes, independent of each other: #3300, #3302, #3303.

Checklist

  • Against the dev branch
  • Formatted per .clang-format
  • New logic is commented
  • One fix only, only relevant files touched
  • CHANGELOG_LATEST.md updated
  • I accept the CLA in CONTRIBUTING.md

@proddy

proddy commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

this is also a valid bug. I haven't tested but can see on board with PSRAM + TLS + CA using BearSSL EC , it will crash EMS-ESP. It does use up an additional 3kb of heap though. @MichaelDvP do you approve this change?

@proddy
proddy requested a review from MichaelDvP October 19, 2026 19:36
@csutcliff

Copy link
Copy Markdown
Contributor Author

This was the original bug I was chasing when I found the others. With an EC CA set for certificate validation (Let's Encrypt ISRG Root X2), my BBQKees Gateway E32 V2 boot looped, crashing on every MQTT connect. It has run with this change since 2026-10-13 without a crash.

@proddy

proddy commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Thanks for the solid diagnosis and measurements - the EC-chain overflow on the 5120-byte MQTT task is real, and 8192 is the right size when CA verification is in play.

One thought on the cost: the +3 KB of internal RAM hits every PSRAM board, while most users run insecure TLS or no TLS and never need that stack. The task size is fixed at create time today, but we could size it from the config instead:

  • keep the library default at 5120
  • add an optional stackSize arg to the espMqttClient / espMqttClientSecure constructors (defaulting to EMC_TASK_STACK_SIZE)
  • in MqttSettingsService::startClient(), pass 8192 only when TLS is on and rootCA is a real cert (not "insecure"), and recreate the client when that changes (today we only recreate on TLS on/off)

That way insecure / no-CA stays at 5120, and only CA users pay the extra 3 KB. Same peak headroom you measured (~2.1 KB on ISRG Root X2).

would that work if we implement it that way?

@csutcliff

Copy link
Copy Markdown
Contributor Author

Yes, that works. I checked the insecure case first, as BearSSL still verifies the server's key exchange signature against the leaf key when rootCA is "insecure". Peak MQTT task stack use after one handshake with rootCA "insecure":

Server leaf key Peak use
P-256 (Let's Encrypt) ran at 5120 without a crash
P-384 ~2.6 KB
P-521 ~5.7 KB, over 5120

So with a CA it's the chain validation that needs the bigger stack, and insecure is fine for P-256 and P-384 server keys. A P-521 server key would still overflow 5120 in insecure mode. That's rare (Let's Encrypt doesn't issue P-521), but a self-signed broker cert could use it.

Two ways to cover it:

  1. Size from "TLS on" rather than "CA set". Every TLS user pays the 3 KB, plain MQTT stays at 5120, and startClient() already recreates the client when TLS is toggled, so no new recreate condition is needed.
  2. Your version as proposed, accepting that an insecure connection to a P-521 broker would overflow.

I'd lean towards 1 as it's simpler and has no gap, but either is fine by me. Happy to update this PR whichever way you prefer.

@proddy

proddy commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

how about:

1. lib/espMqttClient/src/MqttClient.h

// change the protected ctor:
explicit MqttClient(espMqttClientTypes::UseInternalTask useInternalTask,
                    uint8_t priority = 1,
                    uint8_t core = 1,
                    uint32_t stackSize = EMC_TASK_STACK_SIZE);

2. lib/espMqttClient/src/MqttClient.cpp

MqttClient::MqttClient(espMqttClientTypes::UseInternalTask useInternalTask,
                       uint8_t priority,
                       uint8_t core,
                       uint32_t stackSize)
: _useInternalTask(useInternalTask)
// ... existing member init unchanged ...
{
  EMC_GENERATE_CLIENTID(_generatedClientId);
#if defined(ARDUINO_ARCH_ESP32)
  _xSemaphore = xSemaphoreCreateMutex();
  EMC_SEMAPHORE_GIVE();
  if (_useInternalTask == espMqttClientTypes::UseInternalTask::YES) {
#ifdef CONFIG_FREERTOS_UNICORE
    xTaskCreate((TaskFunction_t)_loop, "mqttclient", stackSize, this, priority, &_taskHandle);
#else
    if (core > 1) {
      xTaskCreate((TaskFunction_t)_loop, "mqttclient", stackSize, this, priority, &_taskHandle);
    } else {
      xTaskCreatePinnedToCore((TaskFunction_t)_loop, "mqttclient", stackSize, this, priority, &_taskHandle, core);
    }
#endif
  }
#else
  (void) useInternalTask;
  (void) priority;
  (void) core;
  (void) stackSize;
#endif
  _clientId = _generatedClientId;
  _core = core;
}

3. lib/espMqttClient/src/MqttClientSetup.h

explicit MqttClientSetup(espMqttClientTypes::UseInternalTask useInternalTask,
                         uint8_t priority = 1,
                         uint8_t core = 1,
                         uint32_t stackSize = EMC_TASK_STACK_SIZE)
: MqttClient(useInternalTask, priority, core, stackSize) {
  // ... existing callback wiring unchanged ...
}

4. lib/espMqttClient/src/espMqttClient.h + .cpp

// espMqttClient.h — ESP32 ctors
explicit espMqttClient(uint8_t priority = 1, uint8_t core = 1,
                       uint32_t stackSize = EMC_TASK_STACK_SIZE);
explicit espMqttClientSecure(uint8_t priority = 1, uint8_t core = 1,
                             uint32_t stackSize = EMC_TASK_STACK_SIZE);
// espMqttClient.cpp
espMqttClient::espMqttClient(uint8_t priority, uint8_t core, uint32_t stackSize)
    : MqttClientSetup(espMqttClientTypes::UseInternalTask::YES, priority, core, stackSize)
    , _client() {
    _transport = &_client;
}

espMqttClientSecure::espMqttClientSecure(uint8_t priority, uint8_t core, uint32_t stackSize)
    : MqttClientSetup(espMqttClientTypes::UseInternalTask::YES, priority, core, stackSize)
    , _client() {
    _transport = &_client;
}

(UseInternalTask::NO overloads stay as they are — no task, no stack.)

5. src/ESP32React/MqttSettingsService.cpp — the EMS-ESP side

#ifndef EMSESP_MQTT_STACKSIZE_CA
#define EMSESP_MQTT_STACKSIZE_CA 8192 // BearSSL EC chain verify needs ~6 KB
#endif

static bool mqtt_needs_ca_stack(const MqttSettings & s) {
    return s.enableTLS && !s.rootCA.isEmpty() && s.rootCA != "insecure";
}

void MqttSettingsService::startClient() {
    static bool     isSecure  = false;
    static uint32_t stackSize = EMC_TASK_STACK_SIZE; // 5120 default

    const uint32_t wantStack =
        mqtt_needs_ca_stack(_state) ? EMSESP_MQTT_STACKSIZE_CA : EMC_TASK_STACK_SIZE;

    if (_mqttClient) {
        const bool sameTls   = (isSecure == _state.enableTLS);
        const bool sameStack = (stackSize == wantStack);
        if (sameTls && sameStack) {
            return;
        }
        delete _mqttClient;
        _mqttClient = nullptr;
    }

    isSecure  = _state.enableTLS;
    stackSize = wantStack;

    if (_state.enableTLS) {
        if (emsesp::EMSESP::system_.PSram() == 0) {
            _mqttClient = new espMqttClientSecure(espMqttClientTypes::UseInternalTask::NO);
        } else {
            _mqttClient = new espMqttClientSecure(
                EMSESP_MQTT_PRIORITY, EMSESP_MQTT_RUNNING_CORE, wantStack);
        }
        // ... existing onConnect / onDisconnect / onMessage ...
        return;
    }

    // plain MQTT — default 5120 is enough
    if (emsesp::EMSESP::system_.PSram() == 0) {
        _mqttClient = new espMqttClient(espMqttClientTypes::UseInternalTask::NO);
    } else {
        _mqttClient = new espMqttClient(EMSESP_MQTT_PRIORITY, EMSESP_MQTT_RUNNING_CORE);
    }
    // ... existing callbacks ...
}

On boards with PSRAM the MQTT client runs in its own task with a 5120
byte stack. Verifying an EC certificate chain against a CA, such as
Let's Encrypt ISRG Root X2, needs about 6 KB, so the task overflowed
and the board boot looped on every connect.

The espMqttClient constructors now take the stack size, and
startClient() passes EMSESP_MQTT_STACKSIZE_CA (8192) when TLS is on
with a CA, keeping the 5120 default for insecure TLS and plain MQTT.
The client is recreated when that changes. The stack usage log
reports against the size the task was created with.
@csutcliff csutcliff changed the title fix(mqtt): raise MQTT task stack to 8 KB for TLS certificate validation fix(mqtt): 8 KB MQTT task stack when TLS verifies a CA certificate Oct 20, 2026
@csutcliff

Copy link
Copy Markdown
Contributor Author

Implemented your suggestion and force-pushed: the espMqttClient constructors take a stack size, startClient() passes 8192 when TLS is on with a CA and 5120 otherwise, and the client is recreated when that changes. Two additions: the EMSESP_MQTT_STACKSIZE=8192 line in platformio.ini is gone (it overrode the 5120 default for every client), and the stack usage log now reports against the size the task was created with. I dropped the isEmpty() check, as update() already turns an empty rootCA into "insecure" when TLS is on.

Tested on the Gateway E32 V2 by switching rootCA between the X2 CA and "insecure" without a restart, twice each way: 8192 with 2152 bytes free on the CA, 5120 with 624 free on insecure, a clean reconnect every time and no restarts. Also rebased on dev; the details are in the description.

@MichaelDvP MichaelDvP left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is imho a very good solution.

@proddy
proddy merged commit 960e737 into emsesp:dev Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants