Repository navigation
Fluent-bit crashes with a coredump when running on RHEL10 #11068
Description
Bug Report
Describe the bug
Fluent-bit 4.0.x and 4.1.x crashes with a coredump when running on RHEL10.
The bug seems to be related to the systemd input plugin. When started, the agent works fine for a while before crashing with a coredump. When this happen, any subsequent attempts to start the agent will result in an immediate crash with another coredump.
If we delete all contents from storage.path (systemd.0/ and systemd.db), the agent will start without problems, and run for a while before crashing again with a coredump.
It seems to me that the systemd chunk file gets corrupted for some reason, and when this happens, the agent crashes.
This happens with packages (4.0.13, 4.1.0, 4.1.1) from the almalinux repo at packages.fluentbit.io on multiple servers. But I have compiled 4.1.0 and 4.1.1 from source to activate FLB_DEBUG and I get the same problem.
To Reproduce
- Journald logs related to the crash:
Oct 24 15:11:22 hostname.domain fluent-bit[939013]: [2025/10/24 15:11:22] [engine] caught signal (SIGSEGV)
Oct 24 15:11:22 hostname.domain fluent-bit[939013]: #0 0x7f2e9f071608 in ???() at ???:0
Oct 24 15:11:22 hostname.domain fluent-bit[939013]: #1 0x7f2ea0057b08 in ???() at ???:0
Oct 24 15:11:22 hostname.domain fluent-bit[939013]: #2 0x7f2e9ffb4d32 in ???() at ???:0
Oct 24 15:11:22 hostname.domain fluent-bit[939013]: #3 0x5c899e in ???() at ???:0
Oct 24 15:11:22 hostname.domain fluent-bit[939013]: #4 0x55488e in ???() at ???:0
Oct 24 15:11:22 hostname.domain fluent-bit[939013]: #5 0x577e2b in ???() at ???:0
Oct 24 15:11:22 hostname.domain fluent-bit[939013]: #6 0x7f2e9f2bbb67 in ???() at ???:0
Oct 24 15:11:22 hostname.domain fluent-bit[939013]: #7 0x7f2e9f32c6bb in ???() at ???:0
Oct 24 15:11:22 hostname.domain fluent-bit[939013]: #8 0xffffffffffffffff in ???() at ???:0
Oct 24 15:11:22 hostname.domain systemd-coredump[946015]: Process 939013 (fluent-bit) of user 0 terminated abnormally with signal 6/ABRT, processing...
Oct 24 15:11:22 hostname.domain systemd[1]: Started systemd-coredump@11317-946015-0.service - Process Core Dump (PID 946015/UID 0).
Oct 24 15:11:22 hostname.domain systemd-coredump[946016]: Removed old coredump core.fluent-bit.0.dfd7beb07d594c77bef0090bd555891f.834839.1761112301000000.zst.
Oct 24 15:11:22 hostname.domain systemd-coredump[946016]: [🡕] Process 939013 (fluent-bit) of user 0 dumped core.
Module libzstd.so.1 from rpm zstd-1.5.5-9.el10.x86_64
Module libpcre2-8.so.0 from rpm pcre2-10.44-1.el10.3.x86_64
Module libcrypt.so.2 from rpm libxcrypt-4.4.36-10.el10.x86_64
Module libselinux.so.1 from rpm libselinux-3.8-2.el10_0.x86_64
Module libsasl2.so.3 from rpm cyrus-sasl-2.1.28-27.el10.x86_64
Module libevent-2.1.so.7 from rpm libevent-2.1.12-16.el10.x86_64
Module libkeyutils.so.1 from rpm keyutils-1.6.3-5.el10.x86_64
Module libkrb5support.so.0 from rpm krb5-1.21.3-8.el10_0.x86_64
Module libcom_err.so.2 from rpm e2fsprogs-1.47.1-3.el10.x86_64
Module libk5crypto.so.3 from rpm krb5-1.21.3-8.el10_0.x86_64
Module libkrb5.so.3 from rpm krb5-1.21.3-8.el10_0.x86_64
Module libgssapi_krb5.so.2 from rpm krb5-1.21.3-8.el10_0.x86_64
Module libz.so.1 from rpm zlib-ng-2.2.3-1.el10.x86_64
Module libcap.so.2 from rpm libcap-2.69-7.el10.x86_64
Module libcrypto.so.3 from rpm openssl-3.2.2-16.el10_0.4.x86_64
Module libssl.so.3 from rpm openssl-3.2.2-16.el10_0.4.x86_64
Module libsystemd.so.0 from rpm systemd-257-9.el10_0.1.x86_64
Module libyaml-0.so.2 from rpm libyaml-0.2.5-16.el10.x86_64
Stack trace of thread 939016:
#0 0x00007f2e9f2bd9dc __pthread_kill_implementation (libc.so.6 + 0x969dc)
#1 0x00007f2e9f267a96 raise (libc.so.6 + 0x40a96)
#2 0x00007f2e9f24f8fa abort (libc.so.6 + 0x288fa)
#3 0x00000000004b1498 n/a (n/a + 0x0)
#4 0x313a35312034322f n/a (n/a + 0x0)
ELF object binary architecture: AMD x86-64
Oct 24 15:11:22 hostname.domain systemd[1]: systemd-coredump@11317-946015-0.service: Deactivated successfully.
Oct 24 15:11:22 hostname.domain systemd[1]: systemd-coredump@11317-946015-0.service: Consumed 268ms CPU time, 106.8M memory peak.
Oct 24 15:11:22 hostname.domain systemd[1]: fluent-bit.service: Main process exited, code=dumped, status=6/ABRT
Oct 24 15:11:22 hostname.domain systemd[1]: fluent-bit.service: Failed with result 'core-dump'.
Oct 24 15:11:22 hostname.domain systemd[1]: fluent-bit.service: Consumed 22.833s CPU time, 48.4M memory peak.
Oct 24 15:11:22 hostname.domain systemd[1]: fluent-bit.service: Scheduled restart job, restart counter is at 1.
-
Trace logs from fluent-bit, before, during, after the crash: fluent-bit.log.txt
-
I have been able to generate this backtrace from the coredump generated by the subsequent attempts to start the agent :
(gdb) backtrace
#0 0x00007fd5df4bd9dc in __pthread_kill_implementation () from /lib64/libc.so.6
#1 0x00007fd5df467a96 in raise () from /lib64/libc.so.6
#2 0x00007fd5df44f8fa in abort () from /lib64/libc.so.6
#3 0x00000000004ae47d in flb_signal_handler (signal=11) at /root/rhel10-test/fluent-bit-4.1.1/src/fluent-bit.c:636
#4 <signal handler called>
#5 0x00000000011d0e84 in ZSTD_freeDDict (ddict=0x1) at /root/rhel10-test/fluent-bit-4.1.1/lib/zstd-1.5.7/lib/decompress/zstd_ddict.c:215
#6 0x00007fd5df9ad609 in ZSTD_freeDCtx () from /lib64/libzstd.so.1
#7 0x00007fd5e01e5b09 in journal_file_data_payload.isra () from /lib64/libsystemd.so.0
#8 0x00007fd5e0142d33 in sd_journal_enumerate_data () from /lib64/libsystemd.so.0
#9 0x00000000007466ea in in_systemd_collect (ins=0x390d8d60, config=0x390a7490, in_context=0x7fd5d0001090) at /root/rhel10-test/fluent-bit-4.1.1/plugins/in_systemd/systemd.c:387
#10 0x0000000000746b07 in in_systemd_collect_archive (ins=0x390d8d60, config=0x390a7490, in_context=0x7fd5d0001090) at /root/rhel10-test/fluent-bit-4.1.1/plugins/in_systemd/systemd.c:512
#11 0x0000000000504fcd in input_collector_fd (fd=39, ins=0x390d8d60) at /root/rhel10-test/fluent-bit-4.1.1/src/flb_input_thread.c:166
#12 0x0000000000505b0a in engine_handle_event (fd=39, mask=1, ins=0x390d8d60, config=0x390a7490) at /root/rhel10-test/fluent-bit-4.1.1/src/flb_input_thread.c:181
#13 input_thread (data=0x7fd5d801f4e0) at /root/rhel10-test/fluent-bit-4.1.1/src/flb_input_thread.c:420
#14 0x000000000057ca77 in step_callback (data=0x7fd5d8024b70) at /root/rhel10-test/fluent-bit-4.1.1/src/flb_worker.c:43
#15 0x00007fd5df4bbb68 in start_thread () from /lib64/libc.so.6
#16 0x00007fd5df52c6bc in clone3 () from /lib64/libc.so.6
- Steps to reproduce the problem:
The first crash after the agent has been working without problems for a while is random, I have not been able to identify the reason. After the first crash (and when the chunk file probably gets corrupted) the crash will be reproducible if you used the attached chunk/db files under storage.path/systemd.0/ and storage.path/systemd.db
Expected behavior
The agent should not crash with a coredump.
Screenshots
Your Environment
- Version used:
fluent-bit 4.0.13, 4.1.0 and 4.1.1 - Configuration:
[SERVICE]
# Flush
# =====
# set an interval of seconds before to flush records to a destination
flush 10
# Daemon
# ======
# instruct Fluent Bit to run in foreground or background mode.
daemon Off
# Log_file
# ========
# Absolute path for an optional log file. By default all logs are
# redirected to the standard error interface (stderr).
log_file /var/log/fluent-bit/fluent-bit.log
# Log_Level
# =========
# Set the verbosity level of the service, values can be:
#
# - error
# - warning
# - info
# - debug
# - trace
#
# by default 'info' is set, that means it includes 'error' and 'warning'.
log_level trace
# Parsers File
# ============
# specify an optional 'Parsers' configuration file
parsers_file parsers.conf
# Plugins File
# ============
# specify an optional 'Plugins' configuration file to load external plugins.
plugins_file plugins.conf
# HTTP Server
# ===========
# Enable/Disable the built-in HTTP Server for metrics
http_server On
http_listen 127.0.0.1
http_port 2020
# Storage
# =======
# Fluent Bit can use memory and filesystem buffering based mechanisms
#
# - https://docs.fluentbit.io/manual/administration/buffering-and-storage
#
# storage metrics
# ---------------
# publish storage pipeline metrics in '/api/v1/storage'. The metrics are
# exported only if the 'http_server' option is enabled.
#
storage.metrics on
# storage.path
# ------------
# absolute file system path to store filesystem data buffers (chunks).
#
storage.path /var/lib/fluent-bit/storage
# storage.sync
# ------------
# configure the synchronization mode used to store the data into the
# filesystem. It can take the values normal or full.
#
storage.sync normal
# storage.checksum
# ----------------
# enable the data integrity check when writing and reading data from the
# filesystem. The storage layer uses the CRC32 algorithm.
#
storage.checksum off
# storage.backlog.mem_limit
# -------------------------
# if storage.path is set, Fluent Bit will look for data chunks that were
# not delivered and are still in the storage layer, these are called
# backlog data. This option configure a hint of maximum value of memory
# to use when processing these records.
#
storage.backlog.mem_limit 100M
# storage.max_chunks_up
# ---------------------
# If the input plugin has enabled filesystem storage type, this
# property sets the maximum number of chunks that can be up in
# memory. Use this setting to control memory usage when you enable
# storage.type filesystem.
#
storage.max_chunks_up 128
# storage.delete_irrecoverable_chunks
# -----------------------------------
# When enabled, irrecoverable chunks will be deleted during
# runtime, and any other irrecoverable chunk located in the
# configured storage path directory will be deleted when
# Fluent-Bit starts. Accepted values: 'Off, 'On.
#
storage.delete_irrecoverable_chunks on
# scheduler.base
# ---------------
# Set a base of exponential backoff in seconds.
scheduler.base 5
# scheduler.cap
# -------------
# Set a maximum retry time in seconds.
scheduler.cap 900
[INPUT]
Name systemd
Tag logs_5000_systemd
db /var/lib/fluent-bit/storage/systemd.db
db.Sync Normal
Mem_Buf_Limit 100MB
storage.type filesystem
storage.pause_on_chunks_overlimit on
Read_From_Tail On
Lowercase On
Threaded true
[FILTER]
Name modify
Match logs_5000_systemd
Add dataops.data_processor dataops-logs-systemd
Add event.module systemd
Add event.provider systemd
Add event.dataset systemd.journald
Add data_stream.namespace prod
Add data_stream.dataset systemd.journald
Add service.name linux-systemd
[FILTER]
Name nest
Match *
Operation nest
Wildcard dataops.*
Nest_under dataops
Remove_prefix dataops.
[FILTER]
Name nest
Match *
Operation nest
Wildcard event.*
Nest_under event
Remove_prefix event.
[FILTER]
Name nest
Match *
Operation nest
Wildcard data_stream.*
Nest_under data_stream
Remove_prefix data_stream.
[FILTER]
Name nest
Match *
Operation nest
Wildcard service.*
Nest_under service
Remove_prefix service.
[FILTER]
Name modify
Match *
Add agent.type fluent-bit
[FILTER]
Name sysinfo
Match *
Fluentbit_version_key agent.version
Os_name_key os.name
Os_version_key os.version
Kernel_version_key os.kernel
Hostname_key host.name
[FILTER]
Name nest
Match *
Operation nest
Wildcard agent.*
Nest_under agent
Remove_prefix agent.
[FILTER]
Name nest
Match *
Operation nest
Wildcard os.*
Nest_under os
Remove_prefix os.
[FILTER]
Name nest
Match *
Operation nest
Wildcard host.*
Wildcard os*
Nest_under host
Remove_prefix host.
[OUTPUT]
Name http
Match logs_5000_*
Host server-receiver.example.org
Port 5000
Format json
Workers 1
storage.total_limit_size 100M
Retry_Limit no_limits
tls On
tls.verify On
tls.ca_file /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
tls.crt_file /path/my.crt
tls.key_file /path/my.key
fluent-bit.conf.txt / fluent-bit-systemd.conf.txt
- Server type and version:
Linux 6.12.0-55.38.1.el10_0.x86_64 x86_64 GNU/Linux - Operating System and version:
Red Hat Enterprise Linux release 10.0 (Coughlan) - Filters and plugins:
systemd(input), modify, nest, sysinfo, http(output)
Additional context
- Fluent-bit systemd chunk file and systemd db after the crash: fluent-bit-storage_path_files.zip
Activity
[-]Fluent-bit crash with a coredump when running on RHEL10[/-][+]Fluent-bit crashes with a coredump when running on RHEL10[/+]edsiper commentedon Oct 27, 2025
@rafaelma thanks for providing the bug with useful info.
I have pushed a "potential fix" for this issue here: #11073, would you please give it a try ?
rafaelma commentedon Oct 27, 2025
Hello, thank you very much for your replay.
I have patched the 4.1.1 source code with your commit 0bd94dd , compiled the source and started the agent.
The agent worked without problems for 30min and crashed again.
We wonder if maybe the systemd input plugin is having problems parsing multiline logs in RHEL10? These multiline logs are processed without problems in rhel7,8,9.
The last logs in journald before the crash are:
And the last log sent via the output plugin is:
I have checked the other crashes and many of them have these multiline logs from selinux right before the crash
And these are the trace logs from fluent-bit right before the crash:
edsiper commentedon Oct 28, 2025
@rafaelma I did another check and found that I needed another restart of the cursor in the collector function, I have updated the branch replacing the old fix with a new one, would you please retry it ?, thanks
rafaelma commentedon Oct 28, 2025
Hello Eduardo, thanks for the patch.
Same procedure as last time, and another crash after 1 hour running without problems, just after the same type of logs:
I have installed some debug packages and I think the backtrace output from the coredump is more complete now. I hope you will get more out of it.
Maybe is not important but at thread 3 there are 2 errors of this type:
error: Cannot access memory at addressAnd these are the trace logs from fluent-bit right before the crash:
edsiper commentedon Oct 28, 2025
It seems to be related to having a bundled version of zstd, which conflicts when the external library uses the system’s zstd.
@rafaelma I have added another change to the PR/branch, please give it a try, thanks again for your help and patience on this.
rafaelma commentedon Oct 29, 2025
Hello @edsiper, I am sorry for bringing bad news once again. We experience a new core dump crash with the new patch when these multiline logs from selinux are generated. The positive aspect is that we have identified when these logs are created, allowing us to provoke a crash at will, rather than waiting for one to occur unexpectedly.
Here you have the new backtraces. Tell me if you need some other information. Thank you very much for looking at this.
rafaelma commentedon Oct 30, 2025
Hei @edsiper
I have run fluent-bit with valgrind and triggered a crash, I hope this new information will help debugging this issue. Good luck:
Here is the logfile generated by valgrind for this crash: valgrind.log
And here the backtrace from this crash:
Given that the issue is recurring and the invalid pointers are different, it's likely that the decompression context (dctx) is being used after it has been freed (use-after-free) or the memory is being overwritten by something else?
cosmo0920 commentedon Oct 31, 2025
Hi, I tried to use system's libzstd and linking it for unifying the using libzstd between in_systemd plugin and the system provided libsystemd.
This could unify the symbols for using systemd related code.
#11088
This could eliminate your issue?
rafaelma commentedon Oct 31, 2025
@cosmo0920 Should this patch be used in addition to the ones sent by @edsiper, or instead of them?
rafaelma commentedon Oct 31, 2025
I have applied your patch @cosmo0920 in addition to the patches from @edsiper and recompiled with the same result:
Here is the backtrace for this crash:
16 remaining items
cosmo0920 commentedon Dec 16, 2025
Unfortunately, the official Fluent Bit v4.2.1 does not include this fix.
rafaelma commentedon Dec 16, 2025
This was very unfortunate for us :( ...... We hope 4.2.2 will be released soon
patrick-stephens commentedon Dec 16, 2025
Thanks, I've not been able to trigger it with our Alma/Rocky Linux 10 packages but have done so with UBI 10 container so does seem to be something weird.
mike-somethingnooneelsehasusedyet commentedon Jan 15, 2026
So is the fix in 4.2.2? I've got 4.2.2 on a RHEL 10 server and it's running rather than consistently core dumping straight away as earlier versions did. But in the absence of a change log in the rpm and not being able to see anything at https://fluentbit.io/announcements/ which indicates the fix for this issue is in 4.2.2, maybe the lack of core dumping is by chance.
henri9813 commentedon Feb 24, 2026
Hello,
Do you have news about it ? i also have the same issue.
Thanks !
I501307 commentedon Mar 2, 2026
#11268
does this fix ? @cosmo0920 can you please check the PR status now.
mike-somethingnooneelsehasusedyet commentedon Mar 3, 2026
Well I've now got 4.2.3 on a second RHEL 10 server and it's constantly core dumping straight away. So when's the fix going for this issue going to make it in to a release??
mike-somethingnooneelsehasusedyet commentedon Mar 11, 2026
Is there actually a fix for this issue? It seems like there is, but it also seems like there have been three release of Fluent Bit, spread over more than two months, since the fix was created, and that those releases don't include the fix. This seems like a very long time to go not including a fix if it exists. (Though there is still no mention of 4.2.3 release at https://fluentbit.io/announcements/ so I don't know what fixes were in that release.)
If there isn't actually a fix for this issue, then maybe I can provide some additional useful information to help with one being created.
henri9813 commentedon Mar 12, 2026
Hello, i'm not on Kubernetes, i'm on bare RHEL 10, i still have the issue.
patrick-stephens commentedon Mar 13, 2026
Reading through the comments, there are some potential fixes that people thought may resolve the issue but looks like do not - they may have resolved some issues but there are potentially others. I reopened the issue a while back to try to indicate this and it is not targeted at a release yet. @cosmo0920 has been looking at some of these issues.
If you are seeing failures it is always useful to provide information against the latest release and if possible a nightly build from
master, we provide these automatically but you can also build for a specific target using thepackaging/build.sh <target>wheretargetcan be one of the supported ones listed inbuild-config.json, i.e.centos/10,almalinux/10,rockylinux/10. A build frommasterwill include all latest changes (for FB 5.0.0 in the future), you can also build from the4.2branch.Please note that we do not explicitly build for RHEL 10, we build for the open-source versions we can access, i.e. CentOS, Alma and RockyLinux but RHEL now no longer matches CentOS (and there are potentially other fun times with backporting kernel fixes, etc.). If you can build and test on your actual RHEL 10 target that is super helpful for the OSS project to provide information we may not be able to get easily otherwise.
Anecdotally, I personally am not seeing failures in our commercial Rocky/Alma builds but I am with the UBI container image we build so it does seem to be something RHEL-specific potentially.
mike-somethingnooneelsehasusedyet commentedon Mar 19, 2026
There is "No-cost RHEL for developers subscription" available.
https://developers.redhat.com/products/rhel/download
I wanted to make the environment in which I got a RHEL 10 core dump to upload here as generic and easy to reproduce as possible, so how I got a core dump was:
Download the RHEL 10.1 ISO and create a new virtual machine using that. There's a list of pre-defined options for packages, I used the one for a server with no GUI. (I didn't know and can't remember the exact name.)
Decide to see what happens if I do nothing to enable installing updates which will have been released since the ISO was made.
Install the
libpqrpm from the RHEL 10.1 iso.Install
fluent-bit-4.2.3-1.x86_64.rpmStart
fluent-bit.serviceand confirm it's running./etc/fluent-bit/fluent-bit.confand add inimmediately before
Start
fluent-bit.serviceand confirm it's core dumped.Pick a PID from the list
Core dump and the output of
are attached. If there's something else that would be useful for me to try or provide, let me know and I'll see what I can do.
fluent-bit-coredump.gz
fluent-bit-journal.gz
mike-somethingnooneelsehasusedyet commentedon Apr 13, 2026
EDIT : What is described below is bad information, it was done on a RHEL 10.1 server which I have now realised was using the repo
https://packages.fluentbit.io/almalinux/9/which is wrong. Having corrected this oversight and installed Fluent-bit fromhttps://packages.fluentbit.io/almalinux/10/the core dump does not occur.I've been looking at this again to see if I can think of any other useful information I can provide, and I might have found something. I can make the latest release
fluent-bit-5.0.2-1core dump on demand but not thefluent-bit-4.2.1-1rpm from the link mentioned here #11068 (comment)Sometimes
fluent-bit-5.0.2-1runs fine with thesystemdinput enable, I think if it doesn't encounter a multiline message. I spent some time failing to work out a way to dump an arbitrary multiline message into the journal before coming up with a way to easily trigger an SELinux denial that cause a multiline message.Install
setroubleshoot-serverandhttpdfrom RHEL repos if not already insatlled and starthttpd.service.As a non-root user in their home directory make a file
Then move (not copy, move) that file in to
/var/www/html/. It should have an SELinux type which is something nothttpd_sys_content_te.g.Then make a HTTP request for that file. That should trigger an SELinux denial (check with
aureport -a | tailand write a multiline log message to the journal.I've found that if running
fluent-bit-4.2.1-1rpm it doesn't core dump when an SELiunx denial is triggered, it keeps running and sending what it's getting from thesystemdinput to where it's been told to. If I then upgrade tofluent-bit-5.0.2-1and restartfluent-bit.serviceit runs fine, until I trigger and SELinux denial then it core dumps. I've repeated this multiple times, swapping between the two versions.I can see #11111 has been merged, but it's not evident to me if it has ever made it in to a release. And that the
fluent-bit-4.2.1-1rpm doesn't core dump for me, but the latest release version does, makes me think it didn't get in to a release.Could some clarify whether #11111 has got into a release? And if it didn't, could it do so, please?
cosmo0920 commentedon Apr 15, 2026
eb9f9bd
#11111 is already shipped as v4.2.1.
mike-somethingnooneelsehasusedyet commentedon Apr 15, 2026
Oh. I wonder what is different about the package linked in #11068 (comment) that means I can't make it core dump on demand in the way I can the 4.x and 5.x releases then. 🤷
mike-somethingnooneelsehasusedyet commentedon Apr 27, 2026
So what's in #11068 (comment) can be ignored because it was done whilst using packages from
https://packages.fluentbit.io/almalinux/9/. Sorry for the erroneous information. Thanks to @cosmo0920 whose comment elsewhere made me realise I had done something wrong.