Keypoints are misaligned by up to half a pixel when an image dimension is not a multiple of 4 #18

to_pixel_coords maps a score-map index i to i + 0.5 on a [0, W] axis, so a horizontal flip maps x to W - x exactly, and detections on an image and on its mirror must agree to the subpixel. They do not, unless the dimension is a multiple of 4: the error reaches 0.58 px, following W % 4 on x and H % 4 on y independently.

load_image floors both dimensions with (scale * W) // 8 * 8, so the documented path and the published benchmark numbers are unaffected. A caller that passes its own tensor to detect gets no such guarantee.

Cause

forward_impl upsamples the logits back through the sizes recorded by the encoder. Those are floor-halvings, so a dimension that does not halve exactly makes an F.interpolate ratio non-integral, and with align_corners=False the score map lands off the pixel grid. Only the two finest levels matter in practice, which is why the condition is % 4 and not % 8:

500 and 508 are the discriminating cases: not multiples of 8, but unbiased, because only the coarsest ratio is non-integral and that level is smoothed away.

Reproduction

"""Keypoints are not flip-equivariant unless the image dimensions are multiples of 4."""

import numpy as np
import torch
from scipy.spatial import cKDTree
from scipy.stats import trim_mean
from skimage import data

import dad as dad_pkg

detector = dad_pkg.load_DaD().eval().cuda()


def pixel_keypoints(image):
    tensor = torch.from_numpy(image.transpose(2, 0, 1)[None] / 255.0).float().cuda()
    with torch.inference_mode():
        kpts = detector.detect({"image": tensor}, num_keypoints=2048)["keypoints"][0]
    coords = detector.to_pixel_coords(kpts, image.shape[0], image.shape[1])
    return coords.cpu().numpy().astype(np.float64)


base = np.asarray(data.astronaut())[..., :3]
print(" W    W%4   x offset (expected +0.000)")
for width in range(497, 513):
    crop = np.ascontiguousarray(base[:512, :width])
    original = pixel_keypoints(crop)
    mirrored = pixel_keypoints(np.ascontiguousarray(crop[:, ::-1]))
    mirrored[:, 0] = width - mirrored[:, 0]
    distance, index = cKDTree(original).query(mirrored, k=1)
    near = distance < 2.0
    offset = (mirrored[near] - original[index[near]])[:, 0]
    flag = "" if abs(trim_mean(offset, 0.2)) < 0.05 else "   <-- biased"
    print(f" {width}   {width % 4}     {trim_mean(offset, 0.2):+.3f}{flag}")

Output on a Tesla T4, torch 2.14.0+cu130, dad at c2ee9e1:

 W    W%4   x offset (expected +0.000)
 497   1     -0.430   <-- biased
 498   2     -0.152   <-- biased
 499   3     -0.570   <-- biased
 500   0     +0.001
 501   1     -0.457   <-- biased
 502   2     -0.165   <-- biased
 503   3     -0.555   <-- biased
 504   0     +0.011
 505   1     -0.440   <-- biased
 506   2     -0.146   <-- biased
 507   3     -0.576   <-- biased
 508   0     +0.001
 509   1     -0.450   <-- biased
 510   2     -0.158   <-- biased
 511   3     -0.559   <-- biased
 512   0     +0.011

Sweeping the height instead gives the same pattern on y (-0.506, -0.191, -0.612 for H % 4 of 1, 2, 3). The axes are independent: across 16 widths the y residual holds near -0.002, and across 16 heights the x residual holds near +0.010.

Suggested fix

Padding the bottom-right up to a multiple of 4, then cropping the score map back, removes the bias at every size and leaves already-correct sizes bit-identical. Padding to 8 instead would also make the coarsest halving exact, but that was measured to move keypoints by only 0.0025 px, so 4 is the smallest sufficient modulus. Overhead is under 0.2% of a detect call (0.04 ms of ~23 ms on a T4). #19 does that, with a test covering one size per residue class on both axes.

Found while measuring keypoint coordinate conventions through LoMa, which carries a copy of this detector and shows the same behaviour.