fix: reallocate RAM decode buffer on resolution changes

[21pages]

Related to rustdesk/rustdesk#16275

The RAM hardware decoder reuses a software frame whose dimensions can become stale when the stream resolution changes.

RustDesk normally masks this issue: when the controlled device's resolution changes, it sends a SwitchDisplay message that resets and recreates the decoder on the controlling side. However, hwcodec should also handle resolution changes without requiring callers to recreate the decoder, consistent with the other tested decoder paths.

To reproduce in RustDesk without this fix, use RAM hardware decoding and connect to a controlled device with at least two displays:

1. View Display 1 only.
2. Switch to Display 2 only.
3. While still viewing Display 2, change Display 1's resolution on the controlled device.
4. Select Use all my displays.

Display 1 shows a black screen. Switching back to single-display mode restores the picture.

Unref the cached software frame when its width or height changes, allowing av_hwframe_transfer_data() to allocate a buffer with the correct dimensions.

The change is limited to the hardware-to-RAM transfer path in cpp/ffmpeg_ram/ffmpeg_ram_decode.cpp.

Validation

Resolution changes were tested while reusing a single decoder instance:

Decoder path	H.264 / H.265
RAM software — Intel/NVIDIA and AMD machines	Passed
VRAM FFmpeg/D3D11VA — Intel UHD Graphics 730, both enumerated adapter entries	Already passed
VRAM FFmpeg/D3D11VA — NVIDIA GTX 1650	Already passed
VRAM FFmpeg/D3D11VA — AMD Radeon(TM) Graphics, all three enumerated adapter entries	Passed
RAM D3D11VA — default adapter on both machines	Passed with this fix
RAM VideoToolbox — Apple M1	Passed with this fix
RAM VAAPI — AMD and Intel Linux machines	Passed with this fix
RAM NVDEC/CUDA — NVIDIA GTX 1650 on Linux	Passed with this fix

Passed with this fix denotes dimension checks run with this RAM hardware-transfer fix applied. Software decoding does not use the changed transfer path.

An additional resolution-switching check reused each decoder for 100 rounds, without resetting or recreating it between frames or rounds:

1280x720 -> 640x360 -> 1920x1080 -> 1280x720

• Intel/NVIDIA machine: 10/10 available decoder configurations passed.
• AMD machine: 10/10 available decoder configurations passed, including RAM hardware/software decoding and VRAM H.264/H.265 on all three enumerated adapter entries. AMD driver: 31.0.12044.47003.
• These runs decoded 4,000 frames per machine, 8,000 frames total, with every frame checked for the expected dimensions.
• Adapter entries are counted as separate configurations; they do not imply separate physical GPUs.

Earlier stress testing on the Intel/NVIDIA machine also covered:

• All 10 locally available decoder configurations.
• Repeated resolution switching up to 3840×2160 and decoder creation/destruction.
• All 60,840 decoded frames had the expected dimensions.
• Process memory, handle counts, and GPU memory were monitored during stress tests.

macOS (Apple M1)

Tested commit df3a224 on Apple M1 (arm64), macOS 14.5 (23F79) with the checked-in decode_resolution example. Each configuration reused one hwcodec decoder instance for the entire resolution sequence and all rounds, without an explicit reset or recreation.

Available decoder	100 rounds	1,000 rounds with memory checking
RAM VideoToolbox H.264	Passed with this fix	Passed with this fix
RAM VideoToolbox H.265/HEVC	Passed with this fix	Passed with this fix
RAM software H.264	Passed	Passed
RAM software H.265/HEVC	Passed	Passed

• 4/4 available decoder configurations passed in both runs.
• 1,600 + 16,000 = 17,600 frames decoded in total. Every packet produced exactly one frame with the expected dimensions for 1280x720 -> 640x360 -> 1920x1080 -> 1280x720.
• The 1,000-round run used macOS leaks with allocation stack logging and reported 0 leaks, 0 total leaked bytes. Physical footprint was 21.5M, with a 28.0M peak, as reported by the tool. No leaks were detected in this run.

cargo build --offline --example decode_resolution
./target/debug/examples/decode_resolution 100
MallocStackLogging=1 /usr/bin/leaks --atExit -- ./target/debug/examples/decode_resolution 1000

Linux (AMD VAAPI)

Tested commit df3a224 on AMD Ryzen 7 5800H with Radeon Graphics, Ubuntu 22.04.5 (x86_64), kernel 7.2.6-070206-generic, Mesa 23.2.1-1ubuntu3.1~22.04.4 (radeonsi), and libavcodec 61.19.101.

The checked-in example reused one hwcodec decoder instance per configuration for the full 1280x720 -> 640x360 -> 1920x1080 -> 1280x720 sequence and all rounds.

Available decoder	100 rounds: frame dimensions	1,000 rounds: frame dimensions
RAM VAAPI H.264	Passed with this fix	Passed with this fix
RAM VAAPI H.265/HEVC	Passed with this fix	Passed with this fix
RAM software H.264	Passed	Passed
RAM software H.265/HEVC	Passed	Passed

All 4/4 available configurations passed the dimension checks in both runs: 1,600 + 16,000 = 17,600 frames, with exactly one correctly sized frame per packet.

The Linux VAAPI memory check reported leaks. GCC 11 LeakSanitizer reported 944,184 bytes in 13,019 allocations after 1,000 rounds; the process exited with the configured sanitizer status 23 after all dimension checks passed. The allocation stacks point into radeonsi_drv_video.so and libdrm_amdgpu.so.1. The detector was validated with a known leaking control program.

The changing-resolution workload reported 9,120 bytes after 1 round, 101,784 bytes after 100 rounds, and 944,184 bytes after 1,000 rounds. Additional 100-round controls produced:

Workload	Reported leaked bytes
hwcodec, changing resolution	101,784
hwcodec, fixed 720p	8,328
Direct FFmpeg VAAPI, changing resolution	97,656
Direct FFmpeg VAAPI, fixed 720p	4,200

Both comparisons show the same 93,456-byte increase over their fixed-resolution controls. The direct FFmpeg control does not use hwcodec, sw_frame_, or av_hwframe_transfer_data(), and releases its packets, frames, codec contexts, and device references. This reproduces the growing leak report independently of this RAM-transfer fix. The precise cause within the FFmpeg/VAAPI/driver path remains unresolved.

A software-only fixed-720p control, bypassing hardware capability probing, completed 1,000 rounds with no reported leaks and exit status 0. One initial fixed-size direct FFmpeg control exited with SIGSEGV without diagnostics; a repeat completed both codecs and produced the 4,200-byte result above. The cause of that initial failure was not established.

LeakSanitizer build and run:

cargo rustc --offline --example decode_resolution -- \
  -C linker=gcc \
  -C default-linker-libraries=yes \
  -C link-arg=-fsanitize=leak \
  -C force-frame-pointers=yes
LSAN_OPTIONS=exitcode=23:report_objects=0 ./target/debug/examples/decode_resolution 1000

Linux (Intel VAAPI + NVIDIA NVDEC)

Tested commit df3a224 on Intel Core i5-12400, Intel iGPU 8086:4682, and NVIDIA GeForce GTX 1650, running Ubuntu 22.04.4 (x86_64), kernel 6.5.0-26-generic. The NVIDIA driver was 535.161.07; VAAPI used libva 2.15.0 and Intel iHD media driver 22.3.1. hwcodec linked against vcpkg FFmpeg 7.1.1 (libavcodec 61.19.101).

The checked-in example reused one decoder instance per backend/codec configuration for every packet in all rounds, without an explicit reset or recreation. Every packet was required to produce exactly one frame with the expected dimensions for:

1280x720 -> 640x360 -> 1920x1080 -> 1280x720

On Linux, available_decoders() probes CUDA before VAAPI and removes later entries for a codec format that already succeeded. The normal pass therefore enumerated NVDEC H.264/HEVC plus software H.264/HEVC. A second pass blocked only the dynamic loading of libcuda.so, without changing hwcodec, so the same available_decoders() API exercised its VAAPI fallback plus software H.264/HEVC. Explicit VAAPI contexts were also tested independently and produced the same result.

Decoder path	RAM hardware result	100 rounds	1,000 rounds	Native peak RSS at 1,000 rounds
RAM NVDEC H.264/HEVC + RAM software H.264/HEVC	Passed with this fix	4/4 passed	4/4 passed	155,672 KiB
RAM VAAPI H.264/HEVC + RAM software H.264/HEVC	Passed with this fix	4/4 passed	4/4 passed	40,504 KiB

Across the six unique backend/codec configurations, this checked 2,400 packets at 100 rounds and 24,000 packets at 1,000 rounds. Every packet produced exactly one correctly sized frame.

Valgrind 3.18.1 was first validated with a known leaking program; it correctly reported a deliberately leaked 12,345-byte allocation as definitely lost and returned the configured nonzero status.

Valgrind workload	Definite	Indirect	Possible	Still reachable
NVDEC + software, 100 rounds	0 B	0 B	2,424 B	1,079,227 B
NVDEC + software, 1,000 rounds	0 B	0 B	2,424 B	1,079,227 B
VAAPI + software, 100 rounds	0 B	0 B	0 B	6,277 B
VAAPI, 1,000 rounds	0 B	0 B	0 B	6,269 B

The CUDA possibly lost and still reachable records point into NVIDIA libcuda.so.535.161.07 and have identical byte/block totals at 100 and 1,000 rounds. The CUDA run also contains one glibc dynamic-loader malloc(0) record classified as a zero-byte definitely-lost block, so its configured Valgrind error exit was nonzero despite zero positive bytes in the definite and indirect categories. The VAAPI runs exited with status 0 and zero Valgrind errors.

This machine did not reproduce the 944,184-byte growing leak seen on the AMD VAAPI system. Because no positive-byte definite or indirect leak was found, the conditional fixed-720p and direct-FFmpeg comparisons were not needed. Pure software H.264/HEVC was nevertheless included in the 1,000-round Valgrind run. RSS values above are recorded only as operational data; the leak conclusion is based on Valgrind's end-of-process heap scan.

Linux RAM decoding was built without --features vram; enabling that Windows-oriented feature on Linux currently attempts to compile D3D11 code and fails on the missing d3d11.h header.

Summary by CodeRabbit

• Bug Fixes

  • Improved hardware-accelerated video decoding when stream resolutions change.
  • Prevented unnecessary frame-buffer reinitialization during initial decoding.
  • Improved decoder stability when switching repeatedly between video resolutions without resetting.

• Tests

  • Added a resolution-change validation example covering supported RAM decoders and, where available, VRAM decoders across multiple video resolutions and repeated decode rounds.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 5c3e7a97-270b-48e1-9acc-494bd318904a

📥 Commits

Reviewing files that changed from the base of the PR and between df3a224 and b36f05a.

📒 Files selected for processing (1)

• cpp/ffmpeg_ram/ffmpeg_ram_decode.cpp

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The hardware-acceleration decoder now preserves a newly allocated software frame on the first decode. A new example tests repeated 720p, 360p, and 1080p transitions across available decoders.

Changes

Hardware frame resolution handling

Layer / File(s)	Summary
Software frame reallocation guard cpp/ffmpeg_ram/ffmpeg_ram_decode.cpp	The decoder calls av_frame_unref only when sw_frame_ has initialized dimensions and those dimensions differ from frame_.
Resolution change validation example examples/decode_resolution.rs	The example decodes repeated H.264 and H.265 resolution sequences with available RAM decoders and supported VRAM decoders. It checks frame counts, dimensions, argument parsing, and exit status.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: reallocating the RAM decode buffer when the stream resolution changes.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[21pages]

For a release that needs this resolution-change fix while retaining the SDK backends and leaving out repeat encoding, create a separate branch in rustdesk-org/hwcodec, for example fix/ram-decoder-resolution-change. No revert of hwcodec master is needed.

This proposed branch should start from the SDK-preserving code before #43 and include #46's RAM resolution-change fix and validation example, plus 76ca0d7 — fix: close MFX decoder before reallocating surfaces. The additional Intel SDK fix is separate from #46.

After that branch is created, RustDesk can reference it in libs/scrap/Cargo.toml:

[dependencies.hwcodec]
git = "https://github.com/rustdesk-org/hwcodec"
branch = "fix/ram-decoder-resolution-change"
optional = true

Update RustDesk's Cargo.lock to the selected commit on this branch when preparing the release.

Native SDK VRAM decoding was tested on Windows by explicitly selecting each SDK backend and retaining one hwcodec decoder instance throughout 1280x720 -> 640x360 -> 1920x1080 -> 1280x720, including all repeated rounds. Every output texture was checked for the expected dimensions.

SDK decoder	H.264 / H.265 result
AMD AMF — Radeon Graphics	Both passed without additional SDK changes. All three enumerated adapter entries passed 100 rounds; one entry additionally passed 1,000 rounds / 4,000 frames per codec.
Intel MFX — UHD Graphics 730	Both initially failed at 1080p -> 720p with MFX_ERR_MEMORY_ALLOC (-4). With 76ca0d7, both enumerated adapter entries passed 1,000 rounds / 4,000 frames per codec. Including ascending, descending, and fixed-size controls, 25,840 frames passed all dimension checks.

The Intel fix closes the SDK decoder before freeing and reallocating its surface pool, releasing the SDK's references so the allocator cannot reuse an old pool with too few surfaces.

AMF H.265 is disabled in normal SDK availability enumeration; its result above comes from explicitly selecting that backend and does not change that policy. Adapter entries are not counts of physical GPUs.