Description
Hi NVIDIA team,
I am running into an unexpected behavior when encoding large images using nvJPEG, and I would appreciate your help in confirming if this might be a bug.
When the image dimensions increase slightly beyond a certain point, nvjpegEncodeGetBufferSize returns an unusually large size_t value. This then seems to cause nvjpegEncodeImage to fail with error code 6 (NVJPEG_STATUS_EXECUTION_FAILED) and destroyed other resource context (like TensorRT).
Based on the returned value, I suspect there might be a 32-bit signed integer overflow happening internally during the buffer size calculation. For example, when encoding an image of 26400 x 13272, the required size calculates as expected (~2.1 GB). However, when the width is slightly increased to 27200 x 13272, the required size jumps to 18446744071580576768.
Interestingly, this massive value corresponds to 0xFFFFFFFF8122C000 in hex, which behaves exactly like a negative 32-bit integer (-2128437248) that has been cast to a 64-bit unsigned size_t.
Related issue: #15 #6407
Below I post logs of two images demonstrating the sudden jump in the required buffer size and the subsequent failure.
Request for Confirmation
Could you kindly help confirm if this is indeed a bug caused by integer overflow, or if there is a documented maximum resolution limit for nvjpegEncodeImage that I might have overlooked?
If it is an overflow issue, would it be possible to consider updating the internal calculations to use 64-bit integers in a future release to better support high-resolution images?
Thank you for your time and support!
Minimum reproducible example
#include "nvjpeg.h"
#include "logger.h"
#include <iostream>
#include <cstdio>
#include <cstring>
#include <vector>
bool encode_device_to_jpeg(
NVJPEGBuffer &nv,
const uint8_t *d_image,
int width, int height, int channels,
std::vector<char> &out_file_data,
cudaStream_t stream)
{
nvjpegEncoderState_t encoder_state;
nvjpegEncoderParams_t encoder_params;
nvjpegEncoderStateCreate(nv.nv_handle, &encoder_state, stream);
nvjpegEncoderParamsCreate(nv.nv_handle, &encoder_params, stream);
nvjpegEncoderParamsSetQuality(encoder_params, 100, stream);
nvjpegEncoderParamsSetOptimizedHuffman(encoder_params, 1, stream);
nvjpegEncoderParamsSetSamplingFactors(encoder_params, NVJPEG_CSS_444, stream);
nvjpegImage_t img_desc = {0};
img_desc.channel[0] = (unsigned char *)d_image;
img_desc.pitch[0] = width * channels;
size_t max_stream_length = 0;
nvjpegStatus_t get_size_status = nvjpegEncodeGetBufferSize(
nv.nv_handle, encoder_params, width, height, &max_stream_length);
if (get_size_status == NVJPEG_STATUS_SUCCESS)
{
char buf[128];
snprintf(buf, sizeof(buf), "nvjpegEncodeGetBufferSize required size: %zu bytes for image (%d x %d)", max_stream_length, width, height);
CU_LOG_INFO(buf);
}
nvjpegStatus_t status = nvjpegEncodeImage(
nv.nv_handle, encoder_state, encoder_params,
&img_desc,
NVJPEG_INPUT_BGRI,
width, height,
stream);
if (status != NVJPEG_STATUS_SUCCESS)
{
char buf[128];
snprintf(buf, sizeof(buf), "nvjpegEncodeImage failed with status %d", static_cast<int>(status));
CU_LOG_ERROR(buf);
nvjpegEncoderParamsDestroy(encoder_params);
nvjpegEncoderStateDestroy(encoder_state);
return false;
}
size_t length = 0;
nvjpegEncodeRetrieveBitstream(nv.nv_handle, encoder_state, NULL, &length, stream);
out_file_data.resize(length);
nvjpegEncodeRetrieveBitstream(nv.nv_handle, encoder_state, (unsigned char *)out_file_data.data(), &length, stream);
cudaStreamSynchronize(stream);
nvjpegEncoderParamsDestroy(encoder_params);
nvjpegEncoderStateDestroy(encoder_state);
return true;
}Relevant log output
[2026-06-25 15:40:25.613] [info] [testNVJPEG.cpp:34] nvjpegEncodeGetBufferSize required size: 2102286848 bytes for image (26400 x 13272)
[2026-06-25 15:41:20.208] [info] [testNVJPEG.cpp:34] nvjpegEncodeGetBufferSize required size: 18446744071580576768 bytes for image (27200 x 13272)
[2026-06-25 15:41:22.589] [error] [testNVJPEG.cpp:48] nvjpegEncodeImage failed with status 6
Other/Misc.
Environment
- CUDA Version: [cuda toolkit 12.9]
- GPU Model: [RTX 4060, RTX 4070S, A800 40GB]
- OS: [win11]
- Driver Version: [nvJpeg 12.4.0.16]
Description
Hi NVIDIA team,
I am running into an unexpected behavior when encoding large images using nvJPEG, and I would appreciate your help in confirming if this might be a bug.
When the image dimensions increase slightly beyond a certain point,
nvjpegEncodeGetBufferSizereturns an unusually largesize_tvalue. This then seems to causenvjpegEncodeImageto fail with error code6(NVJPEG_STATUS_EXECUTION_FAILED) and destroyed other resource context (like TensorRT).Based on the returned value, I suspect there might be a 32-bit signed integer overflow happening internally during the buffer size calculation. For example, when encoding an image of
26400 x 13272, the required size calculates as expected (~2.1 GB). However, when the width is slightly increased to27200 x 13272, the required size jumps to18446744071580576768.Interestingly, this massive value corresponds to
0xFFFFFFFF8122C000in hex, which behaves exactly like a negative 32-bit integer (-2128437248) that has been cast to a 64-bit unsignedsize_t.Related issue: #15 #6407
Below I post logs of two images demonstrating the sudden jump in the required buffer size and the subsequent failure.
Request for Confirmation
Could you kindly help confirm if this is indeed a bug caused by integer overflow, or if there is a documented maximum resolution limit for
nvjpegEncodeImagethat I might have overlooked?If it is an overflow issue, would it be possible to consider updating the internal calculations to use 64-bit integers in a future release to better support high-resolution images?
Thank you for your time and support!
Minimum reproducible example
Relevant log output
Other/Misc.
Environment