Robustness and aperture problem

[valentinkiryanski]

Hello! I implemented not yours but IPOL's handheld multi frame super resolution which is closer to google's implementation but uses an FFT full res gray to align the images which is more accurate than google's decimate. My question is how do i fix the misalignments that the robustness mask doesnt catch because it creates thicker edges on some places and if motion has rotation it creates shifted object edges due to the aperture problem. How can i make areas with aperture problem caused misalignments to be rejected and not merged. Do you have an idea and does your implementation produce such stairstepping shifted edges artefacts caused by merging because of the aperture problem and if not how did you solve it? I will appreciate very much if you could get me more insight on this because i am trying to improve my algorithm. Thank you in advance!

[kunzmi]

Hi, it has been some time that I was working on that, so my memory might be a bit weak on the details :)
I also had a quick look at the IPOL paper - and I cannot see any major difference to my implementation. They do a lowpass filter on the full res gray image as I do a band-pass filter, but that is just a minor detail.
I can't follow your thoughts on misalignments due to rotation and aperture problem: We do patch tracking on relatively small patches so that we can neglect rotations to some extend. And if we cannot determine a precise shift for a given patch due to missing features (or aperture) then no shift is applied at all. I can't recall if that is something I came up with or if that is also done in the other attemps, but that seems to me like the trivial way to tackle this.
Finally, even if a patch is misaligned due to the aperture problem, it shouldn't create shifted edges: In case no clear precise shift can be determind, then there are no features that would show up in the average, so it doesn't matter if a patch is averaged on the "correct" position or with a wrong shift.

[valentinkiryanski]

I implemented IPOL algorithm but in cpp and it produces exactly the same results as it but the problem is Wronski (the author of the Google paper) answered me:,, Hi Valentin, This is an old project from over 7 years ago, and I am no longer affiliated with Google, so I can only answer from my recollection/memory. Yes, alignment is fundamentally unreliable, will suffer from misalignment, scene changes (water, light changes, flicker, camera movement, object movement), occlusions and disocclusions, and the aperture problem. The aperture problem is not as big as other alignment problems - I'd say the algorithm handles it well, unlike the other problems. If the region is truly ambigous, like in the aperture problem, merging something along the line is producing roughly correct results, even if the optical flow is locally incorrect and the iterative Lucas Kanade reflinement after the patch-matching helped with the thickedges . The robustness was enough to keep those other problems in check, but as a heads-up: tuning and tweaking it (along with the optical flow improvements) was 90% of the project time and took us many months to get right. And generally, due to the possibility of e.g. scene changes, alignment has to be treated as fundamentall unreliable - and the robustness needs to be tuned for that issue. We generally preferred to be robust and merge fewer frames - visual artifacts from incorrect merge are very noticeable and the quality loss from over-rejection was not too bad. I believe the paper explains that the thresholds etc. were based on the local noise model and even includes some of our "magic numbers" in the paper's supplement (if it doesn't, I have no way of helping you as I don't have access to the algorithm's code). Getting the correct noise model - both the Poisson-like and read noise - is absolutely crucial and requires very accurate camera metadata. For every new released phone, we had a team in the hardware organization to measure and calibrate those noise models. Using noise model had the advantage that we could automatically handle both low light scenes (as with Night Sight) - more merging, more blurry, but denoised - and in good light, where we'd sacrifice some of denoising for more super-resolution. Nowadays, I would use an NN, at least for the robustness part, to avoid this manual tuning and over-reliance on camera metadata. In either case, I hope you figure it out - itmight require quite a bit of tuning, but thenshould "just work" and the results can be great - the algorithm powered many generations of Pixel phones without any code changes, other than some performance optimizations. Good luck and thanks for the interest in the algorithm, Bart” - but my and IPOL algorithm which implements the Google algorithm 1:1 but uses a full res FFT gray alignment which is even better than the half res decimate Google uses suffers from uncaught by the robustness mask misalignments when for example i rotate the camera during burst capture bit has no problems rejecting motion if the motion is in one direction. The misalignments i am talking about you can see in this crop: https://drive.google.com/file/d/1-Fc9p5GR9zkrI6hn9oRM1HQp7lKAgGz5/view?usp=drivesdk which is exactly what Wronski paper describes their algorithm sometimes produces they call it ,,shifted object edges” in their paper. And i dont know how to fix this issue nor why these misalignments look almost white in the robustness mask.

…

On Mon, Aug 17, 2026 at 00:27 Michael Kunz ***@***.***> wrote: *kunzmi* left a comment (kunzmi/ImageStackAlignator#38) <#38 (comment)> Hi, it has been some time that I was working on that, so my memory might be a bit weak on the details :) I also had a quick look at the IPOL paper - and I cannot see any major difference to my implementation. They do a lowpass filter on the full res gray image as I do a band-pass filter, but that is just a minor detail. I can't follow your thoughts on misalignments due to rotation and aperture problem: We do patch tracking on relatively small patches so that we can neglect rotations to some extend. And if we cannot determine a precise shift for a given patch due to missing features (or aperture) then no shift is applied at all. I can't recall if that is something I came up with or if that is also done in the other attemps, but that seems to me like the trivial way to tackle this. Finally, even if a patch is misaligned due to the aperture problem, it shouldn't create shifted edges: In case no clear precise shift can be determind, then there are no features that would show up in the average, so it doesn't matter if a patch is averaged on the "correct" position or with a wrong shift. — Reply to this email directly, view it on GitHub <#38?email_source=notifications&email_token=AWNR2QNIP5P65YB3ACMSE5L5KIRK5A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMZQHE3TKNJVGAYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5309755500>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWNR2QO4UYXWF6V5BC73H7L5KIRK5AVCNFSNUABFKJSXA33TNF2G64TZHMZDGMZSGEZTGNJXHNEXG43VMU5TKMJVGQZTCMJTGEZKC5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AWNR2QPCECEYBZLXFX2JNEL5KIRK5A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMZQHE3TKNJVGAYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/AWNR2QL7V4DC4QYD75DW6KT5KIRK5A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMZQHE3TKNJVGAYKM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>