Repository navigation
Handle clock stepping back in mailbox prune - #2006
Manuel1234477 wants to merge 2 commits into
Conversation
codaMW
left a comment
There was a problem hiding this comment.
tACK 70aec62
Reviewed the clock-step prune panic fix, including a mutation to confirm the test pins the exact crash.
The bug: next_prune computes ttl minus created.elapsed(), and SystemTime::elapsed() returns Err whenever the stored timestamp is in the future relative to now. That is precisely what a backward clock step produces (NTP correction, suspend or resume, VM snapshot restore): an entry written before the step now sits in the future. The old expect("system clock moved back") turned that into a panic. It is reached from prune(), which runs both in the request path via maybe_prune and in the background task via spawn_background_prune (there wrapped in expect("disk storage failed")), so a single backward step took out pruning.
The fix: unwrap_or(Duration::ZERO) treats a future timestamp as zero elapsed, so next_prune returns the full ttl and the entry is handled as if just written. That is the right semantics. The entry is not yet expired, so it should stay, and the scheduler simply waits a full ttl while the clock catches up.
Scope check: I went through the rest of the mailroom time handling. next_prune was the only fallible elapsed() guarded by an expect. gateway_prober uses Instant, which is monotonic and cannot error, and metrics.rs has a separate expect for a pre-1970 clock, which is a different condition, not a backward step. So this is the complete fix for the backward-step panic class. The prune() loop itself compares created + ttl < now via SystemTime ordering, which does not panic under a backward clock and just prunes less until the clock catches up, consistent with the fix.
Tested on 70aec62, repo-pinned nightly, --all-features:
- test_prune_with_future_timestamp: pass
The test pushes the stored insert_order timestamps 60s into the future to simulate the clock stepping back after a write, asserts prune() returns ttl rather than panicking and that the entry is retained (len 1), then posts a second payload and asserts len 2, proving the db keeps serving while the clock catches up.
Mutation: restoring created.elapsed().expect("system clock moved back") makes the test panic at files.rs:491 with SystemTimeError(59.998...s), i.e. the precise crash the fix removes. So the test is load-bearing.
One optional, non-blocking thought: unwrap_or(Duration::ZERO) swallows the SystemTimeError silently. A trace or debug log on the error path would let an operator notice a genuine backward step, though in the prune hot path that is a judgment call and I would not hold the PR for it.
caarloshenriq
left a comment
There was a problem hiding this comment.
ACK 70aec62
Reproduced locally: the new test passes, and panics at files.rs:491 with the original expect restored. Treating a future timestamp as zero elapsed is the right call, since the entry is kept and nothing expires early
|
Thanks for the reviews, @codaMW and @caarloshenriq. I took the optional suggestion from @codaMW. The change is a separate commit (d882db7) so the original fix is unchanged. It was rebased onto the latest branch tip, since |
|
tACK e98f2a3, once the two points below are addressed. This also covers d882db7, which landed after the earlier ACKs on 70aec62.
Tested: fmt, clippy ( AI-assisted review; I ran the tests and mutation myself. |
next_prune() called created.elapsed().expect(...), which panics when a mailbox timestamp is later than the current time. That happens if the system clock is stepped back after a mailbox was written, e.g. NTP correcting the time on a host without an RTC. Timestamps loaded from file metadata at startup can be affected the same way. prune() runs on every post and read through has_capacity(), so while the clock is behind, every mailbox request panics. The background prune task panics too and is never restarted. Treat a future timestamp as zero elapsed time instead. The entry still expires once the clock passes created + ttl.
next_prune treats a mailbox timestamp in the future as just created so a backward clock step no longer panics. That fallback was silent, leaving operators no way to notice the step happened. Emit a debug message with the size of the step when elapsed() fails. Debug level keeps the request-path prune from flooding warn logs while the clock catches up.
e98f2a3 to
98eb9bd
Compare
|
Thanks @Xoulomon, I've made the requested changes:
|
AdamuAbba
left a comment
There was a problem hiding this comment.
Thanks @Manuel1234477 for putting this together!
tACK 98eb9bd for the code. The branch needs a rebase onto master before it can merge, and I've left a comment inline.
I built the binary before and after the change and started each one against a mailbox file with a creation time in 2030, with [v1] enabled, then sent a V1 POST. Before the change, the background prune task panicked at startup and the POST panicked too, dropping the connection. With the change there were no panics, the POST returned the normal 503 timeout, and with RUST_LOG=debug the debug line was logged. I also confirmed with a temporary test that a future-dated entry expires once the clock passes it.
- 28a978a: clippy and tests pass
- 98eb9bd: fmt, clippy, codespell, lib tests and integration tests pass
Since the earlier ACKs are on older hashes, this head will need fresh ones after the rebase. The PR description also still needs the named reviewer and local AI review lines from the template.
|
|
||
| - Let relay operators accept gateway purposes besides BIP77 with `[ohttp_relay] extra_allowed_purposes` (#2024) | ||
| - Reject mailbox payloads too large to return through the OHTTP gateway with 413, and V1 requests too large to relay with `original-psbt-rejected`, instead of truncating the response (#1983) | ||
| - Stop mailbox requests and the background prune task from panicking when the system clock steps back |
There was a problem hiding this comment.
#2038 moved changelog entries into per-PR fragments, so this line now conflicts with master, where ## Unreleased no longer exists. When you rebase, could you drop this line from 28a978a and add a new file, payjoin-mailroom/changelog.d/prune-clock-step.md, in that same commit, with this content:
- Stop mailbox requests and the background prune task from panicking when the system clock steps back (#2006)I tested this on current master: the fragment passes contrib/release/changelog-fragments.sh and the crate's clippy and lib tests pass.
| - Stop mailbox requests and the background prune task from panicking when the system clock steps back |
next_prune()doescreated.elapsed().expect("system clock moved back"). If the clock gets stepped back after a mailbox was written (NTP correcting the time on a box without an RTC is the usual case), that timestamp is in the future andelapsed()errors.prune()runs on every post and read viahas_capacity(), so until the clock catches up every mailbox request panics. The background prune task also panics and never comes back.This treats a future timestamp as zero elapsed instead of panicking. The entry still expires normally once the clock passes
created + ttl.Added
test_prune_with_future_timestamp, which panics on master and passes with the change.