Skip to content

Fix subpixel misalignment when an image dimension is not a multiple of 4 - #19

Merged
Parskatt merged 1 commit into
Parskatt:mainfrom
f-dy:fix/subpixel-alignment
Oct 12, 2026
Merged

Parskatt merged 1 commit into
Parskatt:mainfrom
f-dy:fix/subpixel-alignment

Conversation

@f-dy

@f-dy f-dy commented Sep 23, 2026 •

Copy link
Copy Markdown

Closes #18

Keypoint coordinates are misaligned by up to half a pixel when an image dimension is not a multiple of 4.

Cause

forward_impl upsamples the logits back through the sizes recorded by the encoder. Those are floor-halvings, so a dimension that does not halve exactly makes an F.interpolate ratio non-integral, and with align_corners=False the score map lands off the pixel grid.

Only the two finest levels move a keypoint measurably, so the condition is % 4, not % 8:

width %8 %4 upsample ratios (fine to coarse) biased
497 1 1 2.0040, 2.0000, 2.0000 yes
498 2 2 2.0000, 2.0081, 2.0000 yes
499 3 3 2.0040, 2.0081, 2.0000 yes
500 4 0 2.0000, 2.0000, 2.0161 no
508 4 0 2.0000, 2.0000, 2.0161 no
504 0 0 2.0000, 2.0000, 2.0000 no

500 and 508 are the discriminating cases: not multiples of 8, but unbiased, because only the coarsest ratio is non-integral and that level is smoothed away.

load_image floors both dimensions with (scale * W) // 8 * 8. A multiple of 8 is also a multiple of 4, so that path satisfies the condition and never exposes this. A caller passing its own tensor to detect gets no such guarantee.

Fix

Pad the bottom-right up to a multiple of 4, then crop the score map back. Padding after the origin leaves the top-left alignment unchanged, so sizes that were already correct produce identical output.

Padding to 8 instead would also make the coarsest halving exact. Measured against padding to 4, that moves keypoints by 0.0025 px and is bit-identical wherever both round to the same size, so 4 is used as the smallest sufficient modulus. Overhead is under 0.2% of a detect call (0.04 ms of ~23 ms on a T4).

Measured

Trimmed-mean flip-equivariance residual, which must be 0. to_pixel_coords maps index i to i + 0.5 on a [0, W] axis, so a flip maps x to W - x exactly. One image cropped to each size, so content is constant.

dim % 4 x before x after y before y after
0 +0.001 +0.008 -0.004 -0.003
1 -0.430 +0.014 -0.506 -0.002
2 -0.152 +0.007 -0.191 -0.004
3 -0.570 +0.008 -0.612 -0.002

x follows W % 4 and y follows H % 4 independently: across 16 widths the y residual stays near -0.002, and across 16 heights the x residual stays near +0.010.

More keypoints also survive the flip, since they no longer move off their true position (width 503: 929 to 1191 correspondences).

Tests

tests/flip_equivariance_test.py covers one size per residue class on both axes. Without the fix 12 of 16 cases fail and the 4 that pass are exactly the % 4 == 0 sizes, so it is not vacuous. With the fix all 16 pass, and tests/smoke_test.py is unchanged at 1 passed.

Found while measuring keypoint coordinate conventions through LoMa, which carries a copy of this detector and shows the same behaviour; davnords/LoMa#16 is the same patch there.

Draft because you may prefer the padding in detect rather than forward_impl.

The encoder halves three times and the decoder upsamples back through the
sizes it recorded. Those are floor-halvings, so a dimension that does not
halve exactly makes an F.interpolate ratio non-integral, and with
align_corners=False the score map lands off the pixel grid.

Only the two finest levels move a keypoint measurably, so detections break
flip equivariance by up to half a pixel whenever a dimension is not a
multiple of 4, following W % 4 on x and H % 4 on y independently. Widths 500
and 508 are unbiased despite not being multiples of 8.

load_image() floors both dimensions to a multiple of 8, which is also a
multiple of 4, so the documented path is unaffected; a caller that passes its
own tensor to detect() is not.

Pad the bottom-right up to a multiple of 4, then crop the score map back.
Padding after the origin leaves the top-left alignment unchanged, so sizes
that were already correct produce identical output. Padding to 8 instead
would also make the coarsest halving exact; that was measured to move
keypoints by 0.0025 px, so 4 is used as the smallest sufficient modulus.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Keypoints are misaligned by up to half a pixel when an image dimension is not a multiple of 4

2 participants