podresapi: Get() disabled fallback to List() never triggers — sentinel string does not match kubelet, so device locality hints silently no-op

[mzyczyns]

Version

v0.14.0 (also present on main as of today), pkg/agent/podresapi.

What happens

On any cluster whose kubelet runs with default feature gates, every pod sandbox produces one ERROR line:

level=ERROR msg="failed to get pod resources for <ns>/<pod>: failed to get pod resources: rpc error: code = Unknown desc = PodResources API Get method disabled"

and the List() fallback that Client.Get implements for exactly this case never runs. The PodResources API Get() disabled, falling back to List()... warning is never printed either, and c.noGet never latches, so the failing call is retried for every pod for the lifetime of the process.

Why

The fallback is selected by comparing the kubelet's error text against a locally declared sentinel:

pkg/agent/podresapi/client.go

var (
	errGetDisabled = fmt.Errorf("pod resource API Get method disabled")
)
...
	if !strings.Contains(fmt.Sprintf("%v", err), fmt.Sprintf("%v", errGetDisabled)) {
		return nil, fmt.Errorf("failed to get pod resources: %w", err)
	}

	log.Warnf("PodResources API Get() disabled, falling back to List()...")

The kubelet's actual text is different — PodResources and resource, and the capitalisation:

pkg/kubelet/apis/podresources/server_v1.go, unchanged across at least release-1.30 … release-1.34:

	return nil, fmt.Errorf("PodResources API Get method disabled")

strings.Contains("... desc = PodResources API Get method disabled", "pod resource API Get method disabled") is false, so the branch that would have fallen back to List() is unreachable and the error is returned instead.

Effect

Two things, one cosmetic and one not.

1. One ERROR per pod created, forever, on every default cluster. Combined with balloons: dismissContainer logs ERROR on every container release when pinMemory is false #800 this was 100% of the ERROR output we saw from the balloons plugin over a measured 24 hours (40 of 78 lines were this message).

2. preferCloseToDevices: ["podresourceapi:*/*"] silently does nothing for pods created after the plugin started. GoGetPodResources logs the error and closes the channel without a value, so pod.PodResources stays nil, container.GetPodResources() returns nil, and containerDeviceCpus returns false — the balloon is still created, the container is still pinned, but its CPUs are not steered to the granted device's NUMA node. Containers that existed at Synchronize() time are unaffected, because RefreshPods goes through GoListPodResources. So device locality works for pods that predate the plugin and not for pods created after it, which is a confusing failure mode to debug.

The KubeletPodResourcesGet gate that enables Get is alpha and off by default, so this is the default path, not an edge case. With the fallback working, podresourceapi: hints would resolve on a default cluster without anyone having to enable an alpha kubelet gate.

Suggested fix

Do not match on the message. Get fails with codes.Unknown here, so a text comparison is hard to avoid entirely, but it can at least be made robust — compare case-insensitively against both spellings, or key off "method disabled". Alternatively fall back to List() on any Get error, since List() is the strictly more available call and the result is the same data.

Happy to send a PR.