Skip to content

out_file: Mitigated the File output cardinality gap - #12676

Open
cosmo0920 wants to merge 3 commits into
masterfrom
cosmo0920-malformed-tag-on-tag-derived-mode
Open

cosmo0920 wants to merge 3 commits into
masterfrom
cosmo0920-malformed-tag-on-tag-derived-mode

Conversation

@cosmo0920

@cosmo0920 cosmo0920 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Fixes File output ignoring explicitly configured max_dynamic_files limits when filenames come from tags. For example, with max_dynamic_files 8, additional tag destinations now follow on_limit_reached instead of creating more files.

Destination lookup and reservation are synchronized across output workers. Destinations are reserved before rotation or file creation, so failed opens also consume slots and cannot accumulate uncounted rotation entries.

For compatibility, tag-based configurations that omit max_dynamic_files retain unlimited behavior. Setting it to 0 explicitly remains unlimited.

Validation:

  • File runtime suite: passed.
  • Focused rotation tests (dynamic_destination, fallback_destination, open_failure_releases_lock): all passed.
  • Integration tests: 8 passed, covering four workers, fallback/error/drop actions, unlimited settings, destination reuse, directory creation, and failed opens.
  • The same integration tests under strict Valgrind: 8 passed, memory checks clean.
  • Commit-prefix linter: passed for all three signed-off commits.
ctest --test-dir build -R '^flb-rt-out_file$' --output-on-failure
tests/integration/.venv/bin/python -m pytest tests/integration/scenarios/out_file -q
VALGRIND=1 VALGRIND_STRICT=1 tests/integration/.venv/bin/python -m pytest tests/integration/scenarios/out_file -q

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

  • New Features
    • Dynamic file destinations derived from tags now respect the configured destination limit when a limit is set and no fixed output path is configured. When the limit is reached, the configured error, drop, or fallback behavior is applied.
  • Bug Fixes
    • Destinations count toward the limit before files are opened, so failed file opens also use a slot. Already admitted destinations remain available for reuse.

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@cosmo0920
cosmo0920 requested a review from edsiper as a code owner October 28, 2026 07:55
@cosmo0920 cosmo0920 added this to the Fluent Bit v5.2.0 milestone Oct 28, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T07:57:09.578719Z 4674281 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 363875b4-1bcc-47e2-8bbe-656291dbb91c
Commits

Reviewing files that changed from the base of the PR and between f68b7b1 and 4674281.

Files selected for processing: 3
  • plugins/out_file/file.c
  • tests/integration/scenarios/out_file/tests/test_destination_limit.py
  • tests/runtime/out_file.c

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

Walkthrough

The file output reserves destinations under a lock before opening files. Configured limits apply to dynamic destinations and, when explicitly set, tag-derived destinations without a fixed output file. New tests cover limit actions, destination counts, fallback output, and failed opens.

Changes: Destination limits

Layer / File(s) Summary
Locked destination reservation
plugins/out_file/file.c
The file context adds a lock for destination-table reservations. Destinations are reserved during resolution, before file opening.
Limit behavior across destination modes
plugins/out_file/file.c
Explicit max_dynamic_files limits now also apply to tag-derived paths when out_file is unset. Without that explicit setting, tag-derived paths retain their prior unlimited behavior. Failed opens count toward the limit.
Limit and failure tests
tests/integration/scenarios/out_file/tests/test_destination_limit.py, tests/runtime/out_file.c
Tests cover tag and accessor destinations, configured limits and overflow actions, fallback output, and failed opens.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Suggested reviewers: sagiroosto

Merge Risk: Minimal · up to 46742

No merge-blocking issue was identified in the destination-limit changes. The reviewed reservation path preserves the configured cap and overflow behavior.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 10.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly identifies the out_file change and summarizes the main objective: enforcing dynamic destination limits to address the file output cardinality gap.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 10.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 10 functions across 3 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches: 💡 1
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant