Skip to content

balloons: dismissContainer logs ERROR on every container release when pinMemory is false #800

Description

@mzyczyns

Version

v0.14.0 (also present on main as of today), balloons policy.

What happens

dismissContainer logs at ERROR every time a container leaves a balloon on a policy configured with pinMemory: false:

level=ERROR msg="dismissContainer: failed to release memory for <ns>/<pod>/<container>: libmem: unknown allocation: no request with ID <id>"

It is not an occasional race — it is one line per terminating container, for every container, forever. On a cluster where the balloons policy pins CPUs but not memory this is the plugin's only ERROR output: over 24 hours we measured 78 ERROR lines from the plugin, 38 of them this one, and excluding this message plus one other (issue filed separately) left zero. The rate is simply pod churn on the node.

Why

dismissContainer releases unconditionally:

https://github.com/containers/nri-plugins/blob/main/cmd/plugins/balloons/policy/balloons-policy.godismissContainer

func (p *balloons) dismissContainer(c cache.Container, bln *Balloon) {
	if err := p.memAllocator.Release(c.GetID()); err != nil {
		log.Errorf("dismissContainer: failed to release memory for %s: %v", c.PrettyName(), err)
	}

but an allocation is only ever made in pinCpuMem, and only when memory pinning is on:

	pinMemory := p.bpoptions.PinMemory == nil || *p.bpoptions.PinMemory
	if blnDefPinMemory != nil {
		pinMemory = *blnDefPinMemory
	}
	if pinMemory {
		...
		zone := p.allocMem(c, mems, effMemTypeMask, false)

So with pinMemory: false no container ever has an allocation, every release fails with libmem.ErrUnknownRequest, and every failure is logged as an error. pinMemory: false is a documented, supported configuration, so the error is guaranteed by configuration rather than by anything going wrong.

Independently of pinMemory: "there is no allocation to release" is not a failure to release. ErrUnknownRequest on a release path means there was nothing to do.

Effect

The plugin's ERROR level carries no signal on such a cluster. A real error from the plugin is indistinguishable from the noise, and no alert can be built on the level — we ended up filtering the message out of our Grafana log panels to keep them readable, which also hides any real error that arrives later.

Suggested fix

Treat ErrUnknownRequest as nothing to do rather than as a failure, e.g.

	if err := p.memAllocator.Release(c.GetID()); err != nil && !errors.Is(err, libmem.ErrUnknownRequest) {
		log.Errorf(...)
	}

or skip the release entirely when the container was never given an allocation. Happy to send a PR if you agree on which of the two you prefer.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions