Skip to content

Flush before exit! so a job's last log lines are not discarded - #1

Merged
facundofarias merged 6 commits into
masterfrom
fix/flush-before-child-exit
Sep 16, 2026
Merged

facundofarias merged 6 commits into
masterfrom
fix/flush-before-child-exit

Conversation

@facundofarias

@facundofarias facundofarias commented Sep 16, 2026 •

Copy link
Copy Markdown

Problem

The forked child ends with exit!(0):

result = perform_job(incoming_message.params)
result_handler = Leveret::ResultHandler.new(incoming_message)
result_handler.handle(result)                   # logs "Job returned ...", then acks

log.info "[...] Exiting child process #{pid}"
exit!(0)                                        # skips at_exit -- every buffer discarded

exit! is deliberate: it skips at_exit handlers registered in the parent and inherited across the fork, which must not run once per job. But it skips all of them, and every buffered write goes with them.

That silently discards the last lines a job writes — which are exactly the lines saying whether the job succeeded.

Evidence

This gem loses its own diagnostics. On one production host over two days:

line written by count
Forked to child ... parent (exits normally, flushes) 4,769
Job returned ... child (microseconds before exit!) 14

0.3%. log_file defaults to STDOUT, and a redirected STDOUT is block-buffered, so the child's lines are almost never written out. The log cannot answer the question it is most often asked — did that job succeed? — for virtually any job this gem has run.

That is also what makes the defect self-concealing: the missing lines are precisely the ones you go looking for.

It bites host applications the same way. A nightly sweep in our app logged its start and its every-2,000-records progress lines, but its completion line appeared once in thirty hours across thirteen runs. It was written up as a job that never finishes. It had been finishing all along. The false signal cost about two days of investigation and produced a written conclusion that was simply wrong.

Fix

Two changes, both immediately before exit!:

1. Configuration#before_child_exit — a hook mirroring the existing after_fork, so a host application can flush its own sinks (an HTTP log shipper, a metrics client):

Leveret.configure do |config|
  config.before_child_exit = proc { MyLogSink.flush! }
end

2. #flush_own_log — flushes this gem's own logger. This is plain IO buffering, reproducible with the default STDOUT configuration and no host application involvement, so it is fixed here rather than delegated to the hook.

Deliberate properties:

  • Both run after the acknowledgement, so a hook that hangs can never cause redelivery.
  • The hook is bounded by Worker::CHILD_EXIT_HOOK_TIMEOUT (5s) and rescued. An unreachable sink must never stop a child exiting, or forks accumulate until the host runs out of processes. A failed flush costs log lines; a wedged child costs the worker.
  • The hook rescues Exception, not StandardError. Timeout::Error does not descend from StandardError on the rubies this gem supports, so a bare rescue StandardError would let precisely the timeout case escape and defeat the bound.
  • flush_own_log runs last, so it also flushes whatever the hook logged, and rescues without reporting — reporting is what has just failed.

Default hook is proc {}, so behaviour is unchanged for anyone who does not set one.

Scope note — an unrelated one-line fix included

lib/leveret.rb now also requires forwardable and timeout.

Queue, Worker and DelayQueue all extend Forwardable, but nothing ever required it — so the gem's own spec suite could not load standalone (uninitialized constant Leveret::DelayQueue::Forwardable). Under Rails it only worked because ActiveSupport requires forwardable first.

I'd normally split this out, but the suite could not be run at all to verify the actual fix without it. Flagging it rather than burying it. timeout is required by the fix itself.

Testing

9 new examples: the hook is called; the default is a no-op; a raising hook is swallowed; a non-StandardError raising hook is swallowed; a hanging hook is abandoned rather than blocking the exit; our own log is still flushed when the hook blows up; the logger's IO is flushed; no raise when the logger exposes no flushable device; no raise when the flush itself fails.

spec/worker_spec.rb spec/configuration_spec.rb    12 examples, 0 failures

Full suite unchanged: 14 pre-existing failures before, 14 after (verified by stashing). Those are unrelated gem rot on modern RSpec/Ruby.

Backward compatibility

Additive. New config attribute with a no-op default; no existing behaviour changes. The only observable difference is that log lines which were previously discarded now appear.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added a configurable hook that runs in worker child processes immediately before they exit.
    • Hook execution is limited to five seconds; errors are logged while allowing the worker to exit safely.
    • Worker log output is flushed before child processes exit when supported.
  • Bug Fixes

    • Improved standalone loading by ensuring required runtime dependencies are available.
  • Chores

    • Added automated testing through GitHub Actions across supported Ruby environments.

The forked child ends with `exit!(0)`. That is deliberate -- it skips at_exit
handlers registered in the PARENT and inherited across the fork, which must not
run once per job -- but it skips ALL of them, and any in-memory buffer that an
at_exit handler would have flushed goes with them.

An HTTP log shipper is the common case. A batching sink relies on
`at_exit { close }` to deliver its tail, so the LAST lines a job writes are the
ones most reliably lost -- which are exactly the lines reporting whether the job
succeeded. Long jobs partially hide this, because a periodic flush ships
everything except the final batch; short jobs can lose their entire output.

Observed in production: a nightly sweep logged its start and its every-2,000
progress lines, but its completion line appeared once in thirty hours across
thirteen runs. It was reported as a job that never finished. It had been
finishing all along; only the line saying so was being dropped. The false signal
cost about two days of investigation.

Adds Configuration#before_child_exit, mirroring the existing after_fork hook, and
calls it from the child immediately before exit!. It runs AFTER the
acknowledgement, so a hook that hangs can never cause redelivery, and it is
bounded by Worker::CHILD_EXIT_HOOK_TIMEOUT and rescued: an unreachable sink must
never stop a child exiting, or forks accumulate until the host runs out of
processes. A failed flush costs log lines; a wedged child costs the worker.

The rescue is `Exception`, not `StandardError`, because Timeout::Error does not
descend from StandardError on the rubies this gem supports -- a bare
`rescue StandardError` would let precisely the timeout case escape and defeat the
bound. There is nothing left to protect microseconds before exit!.

Also requires 'forwardable' and 'timeout' at the top level. Queue, Worker and
DelayQueue all `extend Forwardable` but nothing ever required it, so the gem's
own spec suite could not load standalone; under Rails it only worked because
ActiveSupport requires forwardable first. Unrelated to the fix above, but the
suite could not be run to verify it otherwise.

Suite: 5 new examples, all passing. The 14 pre-existing failures elsewhere are
unchanged (14 before, 14 after).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Essentials

Run ID: 6a98a3f3-c47e-41e0-a29c-1a41a9f68889

📥 Commits

Reviewing files that changed from the base of the PR and between 5c56b21 and c6fbb17.

📒 Files selected for processing (3)
  • .github/workflows/ci.yml
  • .travis.yml
  • spec/spec_helper.rb
💤 Files with no reviewable changes (1)
  • .travis.yml

Included review availability: 3 reviews are currently available. Your included PR review attempts over the past 7 days set your current allowance at 4 reviews per hour.


Walkthrough

The library now loads required dependencies explicitly. Configuration exposes a before_child_exit hook. Workers run the hook with a five-second timeout, rescue failures, flush logs, and then exit. CI now runs RSpec through GitHub Actions with RabbitMQ.

Changes

Child exit hook and CI

Layer / File(s) Summary
Hook configuration and loading
lib/leveret.rb, lib/leveret/configuration.rb, spec/configuration_spec.rb
The library requires forwardable and timeout. Leveret::Configuration exposes before_child_exit and defaults it to an empty Proc.
Child hook execution and validation
lib/leveret/worker.rb, spec/worker_spec.rb
Worker runs the hook after acknowledgment, limits it to five seconds, rescues failures, flushes its logger when supported, and exits. Specs cover normal, failing, timing-out, and unsupported flush behavior.
CI and test environment updates
.github/workflows/ci.yml, .travis.yml, spec/spec_helper.rb
GitHub Actions runs RSpec on Ruby 2.7 and Ruby 3.4 with RabbitMQ. The AMQP URL can come from LEVERET_AMQP_URL, and the Travis configuration is removed.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant Worker
  participant Configuration
  participant before_child_exit
  participant Logger
  Worker->>Configuration: Read before_child_exit
  Worker->>before_child_exit: Run hook after acknowledgment
  before_child_exit-->>Worker: Complete or raise
  Worker->>Logger: Log rescued failure
  Worker->>Logger: Flush logger device
  Worker->>Worker: exit!(0)
Loading

Merge Risk: ⚪ Minimal · up to c6fbb

The CI broker credentials and AMQP configuration match the test setup, with no remaining merge-blocking issue identified.

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: flushing worker output before child exit to prevent the final job log lines from being discarded.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/flush-before-child-exit

Warning

Git: CodeRabbit could not clone the repository, so clone-backed analysis was skipped and this review may be incomplete. Verify repository clone access, such as SSH credentials, before requesting another full review. If clone access is intentionally unavailable, use path_filters to narrow the review scope.


Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-16T06:32:37.661927Z e4643c7 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e4643c7ccc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread Gemfile.lock Outdated
The same defect applies to this gem's own diagnostics, and the evidence is
stark. On one production host over two days the log held 4,769 "Forked to child"
lines against 14 "Job returned" lines -- 0.3%.

"Forked to child" is written by the PARENT, which exits normally and flushes.
"Job returned" and "Exiting child process" are written by the CHILD, microseconds
before exit!. log_file defaults to STDOUT and a redirected STDOUT is
block-buffered, so the child's lines are almost never written out.

The practical effect is that the log cannot answer the one question it is most
often asked -- did that job succeed? -- for virtually any job this gem has run.
That is also what makes the defect self-concealing: the missing lines are exactly
the ones you would go looking for.

Note this is plain IO buffering, not anything specific to a log vendor. It
reproduces with the default STDOUT configuration and no host application
involvement, which is why it is fixed here rather than left to
before_child_exit.

Ordered after the hook so it also flushes whatever the hook logged, and rescues
Exception without reporting -- reporting is precisely what has failed, and exit!
is the next statement.

Suite: 12 examples in the touched files, 0 failures. Full suite unchanged at 14
pre-existing failures.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@facundofarias facundofarias changed the title Add a before_child_exit hook so a job's last log lines are not discarded Flush before exit! so a job's last log lines are not discarded Sep 16, 2026
Not tracked on master, and committed here only because it was generated by a
local `bundle install` and swept up by `git add -A`. It does not belong in this
PR on either count.

It would also break CI. .travis.yml runs Ruby 2.2.3, and the generated lockfile
pins rake 13.4.2, which requires Ruby >= 2.3 -- so bundler would install the
incompatible pin instead of resolving a compatible release, and dependency
installation would fail before any spec ran.

A library should not ship an application-style lockfile regardless; it pins
resolution for every consumer.

Caught by Codex review.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
coderabbitai[bot]
coderabbitai Bot previously approved these changes Sep 16, 2026
This repo has never run tests. The only CI config was a .travis.yml pinning Ruby
2.2.3 and bundler 1.10.6, and it was dead three times over: Travis was never
connected to this fork (created 2026-08-20; a fork does not inherit the
upstream's CI integration), travis-ci.org shut down in 2021, and Ruby 2.2.3 is
long EOL. The PR that prompted this had zero check runs and zero registered
workflows.

The consequence was not theoretical. The suite could not even load -- `extend
Forwardable` with nothing requiring forwardable -- and once that was fixed, 14 of
73 examples failed. Nobody could have known.

Adds a workflow that runs rspec on two rubies:

  2.7  (ubuntu-22.04) -- what DeployHQ runs today. Gates.
  3.4  (ubuntu-24.04) -- where the app is heading. Advisory via
       continue-on-error, so a failure there cannot block a fix for the Ruby
       actually in production. Promote to required once consumers have moved.

ubuntu-22.04 is required for 2.7 specifically: setup-ruby ships no 2.7 build for
ubuntu-24.04.

Two details the workflow has to get right:

  - The suite is NOT hermetic. spec_helper purges real queues through Bunny, so a
    RabbitMQ service container has to be healthy before any example runs.
  - The broker user is deliberately NOT `guest`. RabbitMQ restricts guest to
    loopback, and a service container is reached over the docker bridge, so guest
    authentication is refused. The workflow provisions a `leveret` user instead.

That last point needs spec_helper to be configurable, so it now reads
LEVERET_AMQP_URL and falls back to the same amqp://guest:guest@localhost:5672
developers already use. Local runs are unchanged; verified both paths.

bundler-cache is off on purpose: this gem ships no Gemfile.lock, so there is no
stable key to cache against, and the dependency set is small.

Verified locally on 2.7.8 with and without LEVERET_AMQP_URL set: 73 examples,
0 failures both ways.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 2.7 job failed at Install dependencies, before a single spec ran:

    The last version of bundler (>= 0) to support your Ruby & RubyGems was 2.4.22
    bundler requires Ruby version >= 3.2.0. The current ruby version is 2.7.8.225

`gem install bundler` with no constraint resolves to the newest release, which
now requires Ruby >= 3.2. On 2.7 it cannot install at all.

Pin it per Ruby through setup-ruby's own `bundler` input rather than a bare
`gem install`, so the version is fixed by the matrix instead of resolved at run
time: 2.4.22 on 2.7, latest on 3.4. That 2.4.22 is the same pin the consuming
app documents as the last line supporting Ruby 2.7 -- and specifically not
1.17.3, which has a default-gem activation bug that has caused production deploy
outages there.

Also prints `bundle --version` before installing, so the next failure of this
shape is one line of log rather than an inference.

The 3.4 job already passed on the previous run -- 73 examples, 0 failures against
the RabbitMQ service container -- so the service, the non-guest broker user and
LEVERET_AMQP_URL are all confirmed working. Only bundler was wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
thdurante
thdurante previously approved these changes Sep 16, 2026

@thdurante thdurante left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!

Replace dead Travis config with GitHub Actions
@facundofarias
facundofarias merged commit dec57fd into master Sep 16, 2026
3 checks passed
@facundofarias
facundofarias deleted the fix/flush-before-child-exit branch September 16, 2026 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants