Linear memory growth (reclaimable kernel slab) with storage.type: filesystem. POOL proposal. #12504

Describe the bug

There have been quite a few issues reported against Fluent Bit about steady,
linear pod/container memory growth. The curve looks alarming - for example, the working set
climbs day after day toward the pod limit while the application's own memory
(RSS) stays flat. In the worst case the pod ends up pinned at its limit with no
headroom, where a large or fast anon spike (a burst, an output backlog/retry
storm) that arrives faster than the kernel can reclaim the cached slab can
trigger a pod-level OOM kill.

The growth is not a heap leak. It is reclaimable kernel slab (dentry +
inode objects) created by filesystem chunk storage: storage.type: filesystem
creates a new chunk file per flush per stream (<pid>-<sec>.<nsec>.flb) and
unlinks it after delivery, so every flush allocates a dentry+inode and leaves a
negative dentry cached. Those objects are charged to the container cgroup
(memory.stat: slab_reclaimable on cgroup v2; memory.kmem.* on cgroup v1,
and included in memory.usage_in_bytes on modern v1 kernels), which is why
they count against the container limit and appear as working-set growth.

The memory is reclaimable, so reaching the cap does not by itself kill the pod:
the kernel reclaims and the workload continues. The real problem is how this
behaviour presents when monitored: the container's charged memory (working set)
climbs steadily toward its limit while RSS stays flat, so it looks exactly like
an unbounded leak and creates unnecessary investigations, alerts, restarts and
capacity questions.

This issue is a continuation of #11958
(#11958) and an attempt to draw
attention to the situation and to propose a way to remove this monitoring
uncertainty: a bounded file-slot pool in chunkio. Today every flush creates
a unique chunk file and unlinks it after delivery, so the kernel allocates a
new dentry+inode every time and keeps the old objects cached. With a pool, each
stream keeps a fixed set of files (slot-NNNNNN.flb); a new chunk takes a free
slot and a delivered chunk is truncated and returned to the pool instead of
being unlinked. The same kernel objects are then reused for the lifetime of the
process, so the charged memory stays flat instead of growing - while delivery,
durability and crash recovery are unchanged. The pool is opt-in (a pool size
setting; disabled by default), so stock behaviour is untouched. This is the
direction requested in #11958 ("reusing from the same chunk file path pool").

To Reproduce

• Rubular link: N/A
• Example log message: any; the reproducer generates synthetic log lines.

Minimal Docker reproducer is attached: unpack and follow lab/README.md.

lab.zip

Steps:

1. Run stock Fluent Bit 5.1.2 with the lab config in a container with a
   memory limit (how to run it is documented in lab/README.md).

2. Let the bundled load generator run (100 log files at 15 MB/s).

3. Watch the container memory grow linearly:

   # inside the container (or on the node):
   grep -E 'slab_reclaimable|^slab' /sys/fs/cgroup/memory.stat
   # or from monitoring:
   container_memory_working_set_bytes{name=~".*fluent-bit.*"}

Result on the reproducer host (ext4): stock grows ~35 MB/h; on XFS (RHEL 9 VM,
800 flushes/s) +533.4 MB/h.

Expected behavior

When monitoring a pod's working memory at a constant workload, there should not
be a steady, linear growth.

A proposed way to improve - pool mechanism (created and tested as per community guidelines)

Instead of creating a unique
chunk file per flush and unlinking it after delivery, chunkio can keep a bounded
set of reusable chunk files per stream: a new chunk takes a free slot
(slot-NNNNNN.flb), and a delivered chunk is truncated and returned to the pool
instead of being deleted. The kernel dentry/inode objects are then allocated
once and reused, so the charged slab stays bounded by the pool size instead of
growing with flush count - while delivery, durability and crash recovery are
unchanged. The pool size would be a setting (disabled by default), so stock
behaviour is untouched. A working prototype is included in the attached lab
(pool-fix.patch), and the lab runs both variants side by side under the same
load so the difference can be seen directly on one graph: stock slab_reclaimable
climbs linearly, patched stays flat (on XFS +533.4 MB/h vs +2.3 MB/h at 800
flushes/s; on ext4 ~35 vs ~13 MB/h). See lab/README.md for how to run it.

Your Environment

• Version used: 5.1.2 (same behaviour in 4.2.0, 5.0.5-5.0.8, 5.1.1; the
  relevant chunkio code is unchanged across these versions)
• Configuration: filesystem chunk storage (storage.type: filesystem),
  Flush 1, many tail streams, 128 KB chunks, cgroup v2; null / Splunk HEC
  output (output type does not matter)
• Environment name and version: Docker on Linux (cgroup v2); also
  OpenShift with RHCOS 9 nodes (customer environment)
• Server type and version: KVM VM / bare metal
• Operating System and version: Ubuntu (kernel 6.x, ext4) and Rocky Linux
  9.8 / RHEL 9 (kernel 5.14, XFS, cgroup v2)
• Filters and plugins: tail (filesystem storage), kubernetes filter
  (irrelevant to the issue), null / splunk output

Additional context

Related existing reports:

• Excessive Kernel Memory Usage by fluent-bit #7732 "Excessive Kernel Memory Usage by fluent-bit" (2023, cgroup v1,
  memory.kmem.usage_in_bytes, closed as stale)
• Linear memory increase after update to 5.0.6 #11846 "Linear memory increase after update to 5.0.6" (mixed with the
  unrelated http_server regression fixed in 5.0.7)
• Persistent linear memory increase even after upgrading to 5.0.5 #11958 memory.stat proof: slab_reclaimable 18.35 GiB of a 19 GiB limit,
  anon 148 MiB, oom_kill 0
• Persistent memory growth with Tail filesystem storage on Fluent Bit 5.1.2, significantly mitigated by storage.type: memory #12443 "Persistent memory growth with Tail filesystem storage on 5.1.2,
  significantly mitigated by storage.type: memory"