Skip to content

main doesn't start with the chart's defaults: maps sized from max_sessions need more than the 128 Mi memory limit #670

Closed

Description

@martin-mat

What happens

An image built from main, installed with the chart's default values, fails at startup:

ERR Failed to create map pdr_map_teid_ip4: creating map: map create: cannot allocate memory
FTL Loading bpf objects failed: creating map: map create: cannot allocate memory

(run 38937121384: main, root, chart defaults, kind.) The released 0.7.1 image, which the chart ships as appVersion, starts under the same defaults. So the next image release would not start with the chart's defaults.

Why

Since #622 and #624, an unset map size is derived from max_sessions, whose default is 65535: PDR, FAR and URR maps of 131,070 entries and a QER map of 65,535. They're allocated at startup and charged to the pod's memory cgroup. The chart's default resources.limits.memory is 128 Mi (#649), set for 0.7.1's fixed 1024-entry maps.

Measured with the kubelet's accounting (stats/summary, run 38941479851):

working set starts under
main defaults, 65535 sessions ~140 MiB (process ~27 MiB, maps ~115 MiB) 192 Mi and above; not 128 Mi
main, UPF_MAX_SESSIONS=1024 ~30 MiB 128 Mi

About 1.8 MiB per 1024 sessions on top of the process.

Options

Either side can move, and the chart and the code should agree before the release:

  1. The chart sets the session count its limit was sized for: UPF_MAX_SESSIONS: "1024" in defaultEnv, keeping 0.7.1's capacity and the 128 Mi limit, with a README note on how max_sessions and the limit scale together. Verified: the pod starts and passes the same tests as 0.7.1 (run 38942371982).
  2. The chart raises the default limit to carry 65535 sessions, 256 Mi for headroom, so the code's default capacity becomes the chart's.
  3. The code's max_sessions default moves to a value whose maps fit a small deployment, with production sizing documented.

What let this through

Nothing in CI installs the chart with its defaults against the code being shipped: test.yml runs the unit tests; deploy-to-dev installs the chart with the commit's image, but by hand and with values/dev.yaml. The nightly job proposed in #668 (#669) installs the chart's defaults on a kind cluster and would have caught this at the next appVersion bump; the same job with image.tag set to the commit, as deploy-to-dev does, would catch it at the pull request.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions