[Bug]: EVP_DigestSignFinal() with an EC key returns success but writes no signature when the buffer is too small #11904

Contact Details

Here on GitHub.

Version

master @ 12b2b04 (2026-10-26)

Description

Configure: ./configure --enable-opensslextra, with no other options.

In wolfSSL_EVP_DigestSignFinal() (wolfcrypt/src/evp.c:5254-5270), the ECDSA branch only DER-encodes the signature when it fits in *siglen. When it does not fit, the code still sets *siglen to the needed length and returns WOLFSSL_SUCCESS, and nothing is written to sig:

len = wolfSSL_i2d_ECDSA_SIG(ecdsaSig, NULL);
if (len > 0 && (size_t)len <= *siglen)
    len = wolfSSL_i2d_ECDSA_SIG(ecdsaSig, &sig);   /* skipped if too small */
wolfSSL_ECDSA_SIG_free(ecdsaSig);
if (len == 0)
    break;
*siglen = (size_t)len;
ret = WOLFSSL_SUCCESS;                              /* success anyway */

The caller then sends or stores whatever was in its buffer before the call, which may be uninitialised memory, as a "signature".

OpenSSL fails in this case. EVP_DigestSignInit(3) says *siglen must hold the size of sig and that the signature and its length are written "if the call is successful". OpenSSL 3.0.13 returns 0 and leaves both sig and *siglen unchanged (log below). wolfSSL's own RSA branch also fails when the buffer is too small.

This is easy to hit with ordinary code. A DER ECDSA signature varies in length (70-72 bytes for P-256). A caller that sizes its buffer from an earlier signature's length, rather than from the sig == NULL size query, will now and then get a longer signature: wolfSSL reports success with no signature written.

Suggested fix: fail when the buffer is too small. A test is added to test_wolfSSL_EVP_MD_ecc_signing.

--- a/wolfcrypt/src/evp.c
+++ b/wolfcrypt/src/evp.c
@@ -5258,8 +5258,10 @@ int wolfSSL_EVP_DigestSignFinal(WOLFSSL_EVP_MD_CTX *ctx, unsigned char *sig,
             len = wolfSSL_i2d_ECDSA_SIG(ecdsaSig, NULL);
             if (len > 0 && (size_t)len <= *siglen)
                 len = wolfSSL_i2d_ECDSA_SIG(ecdsaSig, &sig);
+            else
+                len = 0; /* buffer too small: fail, don't report success */
             wolfSSL_ECDSA_SIG_free(ecdsaSig);
-            if (len == 0)
+            if (len <= 0)
                 break;
             *siglen = (size_t)len;
             ret = WOLFSSL_SUCCESS;
--- a/tests/api/test_evp_pkey.c
+++ b/tests/api/test_evp_pkey.c
@@ -2365,6 +2365,10 @@ int test_wolfSSL_EVP_MD_ecc_signing(void)
                                           (unsigned int)XSTRLEN(testData)), 1);
     checkSz = sizeof(check);
     ExpectIntEQ(wolfSSL_EVP_DigestSignFinal(&mdCtx, NULL, &checkSz), 1);
+    /* A buffer too small for the DER signature must be an error. */
+    checkSz = 8;
+    ExpectIntEQ(wolfSSL_EVP_DigestSignFinal(&mdCtx, check, &checkSz), 0);
+    ExpectIntEQ(checkSz, 8);
     checkSz = sizeof(check);
     ExpectIntEQ(wolfSSL_EVP_DigestSignFinal(&mdCtx, check, &checkSz), 1);
     ret = wolfSSL_EVP_MD_CTX_cleanup(&mdCtx);

Testing with --enable-opensslextra at 12b2b04:

• The new check fails without the evp.c change (test_evp_pkey.c line 2370) and passes with it.
• tests/unit.test (all tests) passes with the patch.

I can open a PR instead if you prefer. I would first need to sort out the contributor agreement.

Reproduction steps

Build wolfSSL with ./configure --enable-opensslextra && make && make install, then compile and run this program. The same source builds against OpenSSL without -DUSE_WOLFSSL.

/* cc -DUSE_WOLFSSL repro.c -lwolfssl      |   cc repro.c -lcrypto */
#ifdef USE_WOLFSSL
#include <wolfssl/options.h>
#include <wolfssl/openssl/evp.h>
#include <wolfssl/openssl/ec.h>
#include <wolfssl/openssl/obj_mac.h>
#else
#include <openssl/evp.h>
#include <openssl/ec.h>
#include <openssl/obj_mac.h>
#endif
#include <stdio.h>
#include <string.h>

int main(void) {
  unsigned char sig[128];
  size_t siglen;
  int i, ret, untouched = 1;
  EC_KEY *ec = EC_KEY_new_by_curve_name(NID_X9_62_prime256v1);
  EVP_PKEY *pkey = EVP_PKEY_new();
  EVP_MD_CTX *ctx = EVP_MD_CTX_new();

  EC_KEY_generate_key(ec);
  EVP_PKEY_assign_EC_KEY(pkey, ec);
  EVP_DigestSignInit(ctx, NULL, EVP_sha256(), NULL, pkey);
  EVP_DigestSignUpdate(ctx, "hello", 5);

  memset(sig, 0xAA, sizeof(sig));
  siglen = 16;                       /* too small for a 70-72 byte signature */
  ret = EVP_DigestSignFinal(ctx, sig, &siglen);
  for (i = 0; i < (int)sizeof(sig); i++)
    if (sig[i] != 0xAA) untouched = 0;
  printf("ret=%d siglen=%zu buffer %s\n", ret, siglen,
         untouched ? "untouched" : "written");

  EVP_MD_CTX_free(ctx);
  EVP_PKEY_free(pkey);
  return 0;
}

Relevant log output

OpenSSL 3.0.13:        ret=0 siglen=16 buffer untouched
wolfSSL master:        ret=1 siglen=71 buffer untouched   <- success, no signature (siglen 70-72, varies per run)
wolfSSL + patch:       ret=0 siglen=16 buffer untouched

This issue was found by an LLM-Enhanced Static Analysis Engine and verified manually.