Flush before exit! so a job's last log lines are not discarded - #1
Conversation
The forked child ends with `exit!(0)`. That is deliberate -- it skips at_exit
handlers registered in the PARENT and inherited across the fork, which must not
run once per job -- but it skips ALL of them, and any in-memory buffer that an
at_exit handler would have flushed goes with them.
An HTTP log shipper is the common case. A batching sink relies on
`at_exit { close }` to deliver its tail, so the LAST lines a job writes are the
ones most reliably lost -- which are exactly the lines reporting whether the job
succeeded. Long jobs partially hide this, because a periodic flush ships
everything except the final batch; short jobs can lose their entire output.
Observed in production: a nightly sweep logged its start and its every-2,000
progress lines, but its completion line appeared once in thirty hours across
thirteen runs. It was reported as a job that never finished. It had been
finishing all along; only the line saying so was being dropped. The false signal
cost about two days of investigation.
Adds Configuration#before_child_exit, mirroring the existing after_fork hook, and
calls it from the child immediately before exit!. It runs AFTER the
acknowledgement, so a hook that hangs can never cause redelivery, and it is
bounded by Worker::CHILD_EXIT_HOOK_TIMEOUT and rescued: an unreachable sink must
never stop a child exiting, or forks accumulate until the host runs out of
processes. A failed flush costs log lines; a wedged child costs the worker.
The rescue is `Exception`, not `StandardError`, because Timeout::Error does not
descend from StandardError on the rubies this gem supports -- a bare
`rescue StandardError` would let precisely the timeout case escape and defeat the
bound. There is nothing left to protect microseconds before exit!.
Also requires 'forwardable' and 'timeout' at the top level. Queue, Worker and
DelayQueue all `extend Forwardable` but nothing ever required it, so the gem's
own spec suite could not load standalone; under Rails it only worked because
ActiveSupport requires forwardable first. Unrelated to the fix above, but the
suite could not be run to verify it otherwise.
Suite: 5 new examples, all passing. The 14 pre-existing failures elsewhere are
unchanged (14 before, 14 after).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Essentials Run ID: 📒 Files selected for processing (3)
💤 Files with no reviewable changes (1)
Included review availability: 3 reviews are currently available. Your included PR review attempts over the past 7 days set your current allowance at 4 reviews per hour. WalkthroughThe library now loads required dependencies explicitly. Configuration exposes a ChangesChild exit hook and CI
Priority: ⬇️ Low Estimated code review effort: 3 (Moderate) | ~20 minutes Change: Bug fix Sequence Diagram(s)sequenceDiagram
participant Worker
participant Configuration
participant before_child_exit
participant Logger
Worker->>Configuration: Read before_child_exit
Worker->>before_child_exit: Run hook after acknowledgment
before_child_exit-->>Worker: Complete or raise
Worker->>Logger: Log rescued failure
Worker->>Logger: Flush logger device
Worker->>Worker: exit!(0)
Merge Risk: ⚪ Minimal · up to The CI broker credentials and AMQP configuration match the test setup, with no remaining merge-blocking issue identified. 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Git: CodeRabbit could not clone the repository, so clone-backed analysis was skipped and this review may be incomplete. Verify repository clone access, such as SSH credentials, before requesting another full review. If clone access is intentionally unavailable, use Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e4643c7ccc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The same defect applies to this gem's own diagnostics, and the evidence is stark. On one production host over two days the log held 4,769 "Forked to child" lines against 14 "Job returned" lines -- 0.3%. "Forked to child" is written by the PARENT, which exits normally and flushes. "Job returned" and "Exiting child process" are written by the CHILD, microseconds before exit!. log_file defaults to STDOUT and a redirected STDOUT is block-buffered, so the child's lines are almost never written out. The practical effect is that the log cannot answer the one question it is most often asked -- did that job succeed? -- for virtually any job this gem has run. That is also what makes the defect self-concealing: the missing lines are exactly the ones you would go looking for. Note this is plain IO buffering, not anything specific to a log vendor. It reproduces with the default STDOUT configuration and no host application involvement, which is why it is fixed here rather than left to before_child_exit. Ordered after the hook so it also flushes whatever the hook logged, and rescues Exception without reporting -- reporting is precisely what has failed, and exit! is the next statement. Suite: 12 examples in the touched files, 0 failures. Full suite unchanged at 14 pre-existing failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Not tracked on master, and committed here only because it was generated by a local `bundle install` and swept up by `git add -A`. It does not belong in this PR on either count. It would also break CI. .travis.yml runs Ruby 2.2.3, and the generated lockfile pins rake 13.4.2, which requires Ruby >= 2.3 -- so bundler would install the incompatible pin instead of resolving a compatible release, and dependency installation would fail before any spec ran. A library should not ship an application-style lockfile regardless; it pins resolution for every consumer. Caught by Codex review. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This repo has never run tests. The only CI config was a .travis.yml pinning Ruby
2.2.3 and bundler 1.10.6, and it was dead three times over: Travis was never
connected to this fork (created 2026-08-20; a fork does not inherit the
upstream's CI integration), travis-ci.org shut down in 2021, and Ruby 2.2.3 is
long EOL. The PR that prompted this had zero check runs and zero registered
workflows.
The consequence was not theoretical. The suite could not even load -- `extend
Forwardable` with nothing requiring forwardable -- and once that was fixed, 14 of
73 examples failed. Nobody could have known.
Adds a workflow that runs rspec on two rubies:
2.7 (ubuntu-22.04) -- what DeployHQ runs today. Gates.
3.4 (ubuntu-24.04) -- where the app is heading. Advisory via
continue-on-error, so a failure there cannot block a fix for the Ruby
actually in production. Promote to required once consumers have moved.
ubuntu-22.04 is required for 2.7 specifically: setup-ruby ships no 2.7 build for
ubuntu-24.04.
Two details the workflow has to get right:
- The suite is NOT hermetic. spec_helper purges real queues through Bunny, so a
RabbitMQ service container has to be healthy before any example runs.
- The broker user is deliberately NOT `guest`. RabbitMQ restricts guest to
loopback, and a service container is reached over the docker bridge, so guest
authentication is refused. The workflow provisions a `leveret` user instead.
That last point needs spec_helper to be configurable, so it now reads
LEVERET_AMQP_URL and falls back to the same amqp://guest:guest@localhost:5672
developers already use. Local runs are unchanged; verified both paths.
bundler-cache is off on purpose: this gem ships no Gemfile.lock, so there is no
stable key to cache against, and the dependency set is small.
Verified locally on 2.7.8 with and without LEVERET_AMQP_URL set: 73 examples,
0 failures both ways.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 2.7 job failed at Install dependencies, before a single spec ran:
The last version of bundler (>= 0) to support your Ruby & RubyGems was 2.4.22
bundler requires Ruby version >= 3.2.0. The current ruby version is 2.7.8.225
`gem install bundler` with no constraint resolves to the newest release, which
now requires Ruby >= 3.2. On 2.7 it cannot install at all.
Pin it per Ruby through setup-ruby's own `bundler` input rather than a bare
`gem install`, so the version is fixed by the matrix instead of resolved at run
time: 2.4.22 on 2.7, latest on 3.4. That 2.4.22 is the same pin the consuming
app documents as the last line supporting Ruby 2.7 -- and specifically not
1.17.3, which has a default-gem activation bug that has caused production deploy
outages there.
Also prints `bundle --version` before installing, so the next failure of this
shape is one line of log rather than an inference.
The 3.4 job already passed on the previous run -- 73 examples, 0 failures against
the RabbitMQ service container -- so the service, the non-guest broker user and
LEVERET_AMQP_URL are all confirmed working. Only bundler was wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replace dead Travis config with GitHub Actions
c6fbb17
Problem
The forked child ends with
exit!(0):exit!is deliberate: it skipsat_exithandlers registered in the parent and inherited across the fork, which must not run once per job. But it skips all of them, and every buffered write goes with them.That silently discards the last lines a job writes — which are exactly the lines saying whether the job succeeded.
Evidence
This gem loses its own diagnostics. On one production host over two days:
Forked to child ...Job returned ...exit!)0.3%.
log_filedefaults to STDOUT, and a redirected STDOUT is block-buffered, so the child's lines are almost never written out. The log cannot answer the question it is most often asked — did that job succeed? — for virtually any job this gem has run.That is also what makes the defect self-concealing: the missing lines are precisely the ones you go looking for.
It bites host applications the same way. A nightly sweep in our app logged its start and its every-2,000-records progress lines, but its completion line appeared once in thirty hours across thirteen runs. It was written up as a job that never finishes. It had been finishing all along. The false signal cost about two days of investigation and produced a written conclusion that was simply wrong.
Fix
Two changes, both immediately before
exit!:1.
Configuration#before_child_exit— a hook mirroring the existingafter_fork, so a host application can flush its own sinks (an HTTP log shipper, a metrics client):2.
#flush_own_log— flushes this gem's own logger. This is plain IO buffering, reproducible with the default STDOUT configuration and no host application involvement, so it is fixed here rather than delegated to the hook.Deliberate properties:
Worker::CHILD_EXIT_HOOK_TIMEOUT(5s) and rescued. An unreachable sink must never stop a child exiting, or forks accumulate until the host runs out of processes. A failed flush costs log lines; a wedged child costs the worker.Exception, notStandardError.Timeout::Errordoes not descend fromStandardErroron the rubies this gem supports, so a barerescue StandardErrorwould let precisely the timeout case escape and defeat the bound.flush_own_logruns last, so it also flushes whatever the hook logged, and rescues without reporting — reporting is what has just failed.Default hook is
proc {}, so behaviour is unchanged for anyone who does not set one.Scope note — an unrelated one-line fix included
lib/leveret.rbnow also requiresforwardableandtimeout.Queue,WorkerandDelayQueueallextend Forwardable, but nothing ever required it — so the gem's own spec suite could not load standalone (uninitialized constant Leveret::DelayQueue::Forwardable). Under Rails it only worked because ActiveSupport requiresforwardablefirst.I'd normally split this out, but the suite could not be run at all to verify the actual fix without it. Flagging it rather than burying it.
timeoutis required by the fix itself.Testing
9 new examples: the hook is called; the default is a no-op; a raising hook is swallowed; a non-
StandardErrorraising hook is swallowed; a hanging hook is abandoned rather than blocking the exit; our own log is still flushed when the hook blows up; the logger's IO is flushed; no raise when the logger exposes no flushable device; no raise when the flush itself fails.Full suite unchanged: 14 pre-existing failures before, 14 after (verified by stashing). Those are unrelated gem rot on modern RSpec/Ruby.
Backward compatibility
Additive. New config attribute with a no-op default; no existing behaviour changes. The only observable difference is that log lines which were previously discarded now appear.
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Chores