Exploring On-Demand Log Upload Solutions

[zshuang0316]

Hi,

We are currently using fluent-bit to process logs and have received requests from customers who want the ability to upload logs on-demand. This is aimed at reducing resource usage and managing output storage capacity. Our customers would like to control an existing input in the configuration to enable or disable logging via a job.

To address this, we're considering using the pause mechanism. We propose exposing a POST API to set the pause/resume status for an input plugin, and providing a GET API to check the status for a specific pod.

Alternatively, changing the configuration could be a solution (like comment the "path" field in tail and change back later), but it might be less elegant and potentially prone to issues if not executed correctly.

What are your thoughts on this approach? Do you think on-demand log uploading is a common requirement for fluent-bit, and do you have any other suggestions for this requirement?

Looking forward to your feedback!

[patrick-stephens]

I'd probably just do it via a simple enable/disable mechanism and hot reload using the existing implementation - easy enough to make an output not match anything to effectively disable it which is what I've done in the past (be nice to have a simple enable/disable flag) or use a grep processor to drop everything.

I mean it can have issues sure, e.g. making sure you flush or don't trigger a crash loop, but those can be managed and I think the approach you suggest also has quite a few issues I can see. Reloading configuration either via hot-reload or rollout is a well understood approach I would argue.

Are you suggesting a per-pod approach for this? That doesn't seem right to me.
Why would you want some pods doing it but not others?
What happens with pod eviction/re-scheduling and everything else the scheduler could do? E.g. you "pause" the pod but then it gets restarted and has no idea it was paused.
How would you target the "right" pod in a safe way?

An alternative would be an intermediate filter/processor (e.g. LUA if you don't want a custom one) you can control that gates the outputs, e.g. it checks with an API whether to allow output or not and then you control that API however you want.