Version
v0.14.0 (also present on main as of today), balloons policy.
What happens
dismissContainer logs at ERROR every time a container leaves a balloon on a policy configured with pinMemory: false:
level=ERROR msg="dismissContainer: failed to release memory for <ns>/<pod>/<container>: libmem: unknown allocation: no request with ID <id>"
It is not an occasional race — it is one line per terminating container, for every container, forever. On a cluster where the balloons policy pins CPUs but not memory this is the plugin's only ERROR output: over 24 hours we measured 78 ERROR lines from the plugin, 38 of them this one, and excluding this message plus one other (issue filed separately) left zero. The rate is simply pod churn on the node.
Why
dismissContainer releases unconditionally:
https://github.com/containers/nri-plugins/blob/main/cmd/plugins/balloons/policy/balloons-policy.go — dismissContainer
func (p *balloons) dismissContainer(c cache.Container, bln *Balloon) {
if err := p.memAllocator.Release(c.GetID()); err != nil {
log.Errorf("dismissContainer: failed to release memory for %s: %v", c.PrettyName(), err)
}
but an allocation is only ever made in pinCpuMem, and only when memory pinning is on:
pinMemory := p.bpoptions.PinMemory == nil || *p.bpoptions.PinMemory
if blnDefPinMemory != nil {
pinMemory = *blnDefPinMemory
}
if pinMemory {
...
zone := p.allocMem(c, mems, effMemTypeMask, false)
So with pinMemory: false no container ever has an allocation, every release fails with libmem.ErrUnknownRequest, and every failure is logged as an error. pinMemory: false is a documented, supported configuration, so the error is guaranteed by configuration rather than by anything going wrong.
Independently of pinMemory: "there is no allocation to release" is not a failure to release. ErrUnknownRequest on a release path means there was nothing to do.
Effect
The plugin's ERROR level carries no signal on such a cluster. A real error from the plugin is indistinguishable from the noise, and no alert can be built on the level — we ended up filtering the message out of our Grafana log panels to keep them readable, which also hides any real error that arrives later.
Suggested fix
Treat ErrUnknownRequest as nothing to do rather than as a failure, e.g.
if err := p.memAllocator.Release(c.GetID()); err != nil && !errors.Is(err, libmem.ErrUnknownRequest) {
log.Errorf(...)
}
or skip the release entirely when the container was never given an allocation. Happy to send a PR if you agree on which of the two you prefer.
Version
v0.14.0(also present onmainas of today), balloons policy.What happens
dismissContainerlogs atERRORevery time a container leaves a balloon on a policy configured withpinMemory: false:It is not an occasional race — it is one line per terminating container, for every container, forever. On a cluster where the balloons policy pins CPUs but not memory this is the plugin's only
ERRORoutput: over 24 hours we measured 78ERRORlines from the plugin, 38 of them this one, and excluding this message plus one other (issue filed separately) left zero. The rate is simply pod churn on the node.Why
dismissContainerreleases unconditionally:https://github.com/containers/nri-plugins/blob/main/cmd/plugins/balloons/policy/balloons-policy.go —
dismissContainerbut an allocation is only ever made in
pinCpuMem, and only when memory pinning is on:So with
pinMemory: falseno container ever has an allocation, every release fails withlibmem.ErrUnknownRequest, and every failure is logged as an error.pinMemory: falseis a documented, supported configuration, so the error is guaranteed by configuration rather than by anything going wrong.Independently of
pinMemory: "there is no allocation to release" is not a failure to release.ErrUnknownRequeston a release path means there was nothing to do.Effect
The plugin's
ERRORlevel carries no signal on such a cluster. A real error from the plugin is indistinguishable from the noise, and no alert can be built on the level — we ended up filtering the message out of our Grafana log panels to keep them readable, which also hides any real error that arrives later.Suggested fix
Treat
ErrUnknownRequestas nothing to do rather than as a failure, e.g.or skip the release entirely when the container was never given an allocation. Happy to send a PR if you agree on which of the two you prefer.