feat(monitor): appendfsync=always main-thread blocking advisory (valkey#3515) - #375
Open
jamby77 wants to merge 2 commits into
Open
feat(monitor): appendfsync=always main-thread blocking advisory (valkey#3515)#375jamby77 wants to merge 2 commits into
jamby77 wants to merge 2 commits into
Conversation
…ey#3515) - New config-hazard evaluator: appendfsync=always with AOF on raises a low-severity advisory on config alone, escalating to a warning hazard when symptoms confirm blocking (aof_delayed_fsync rising across probes, aof-fsync-always/aof-write LATENCY events, or a failing aof_last_write_status) - everysec is flagged only when aof_delayed_fsync climbs on two consecutive probes (the background fsync backing up), never on config - Service probes appendfsync + INFO persistence + LATENCY LATEST on the existing TTL-cached path; symptom probe failures degrade to the config-only advisory instead of suppressing it - Dashboard banner gains an advisory presentation (info icon, muted) alongside the existing hazard/unverified states
Collaborator
Author
|
@claude review |
LATENCY LATEST entries persist until LATENCY RESET, so a single past aof-fsync-always/aof-write spike would escalate the advisory to a hazard forever. Filter entries by their spike timestamp (5-minute freshness window) so escalation mirrors the fresh-rise discipline of the aof_delayed_fsync path.
Collaborator
Author
|
@BugBot review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 8f96986. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the AOF fsync-policy advisory from valkey-io/valkey#3515 to the config-hazard subsystem (the sibling of #337's valkey#3983 hazard): with
appendfsync always, every write fsyncs synchronously on the main thread, so disk latency becomes command latency. The broader upstream AOF-modernization work (WAL headers, io_uring, direct I/O) is unshipped and not pollable — the misconfiguration and its symptom are observable today.Detection (per the issue's guardrails)
appendonly=no→ no-op (managed/ephemeral instances with AOF intentionally off never fire).appendfsync=alwayswith no symptoms → low-severity advisory (severity: info,status: advisory): the config is a latency risk, considereverysecunless per-write durability is a hard requirement.aof_delayed_fsyncrising across probes,aof-fsync-always/aof-writeLATENCY events, oraof_last_write_status != ok.appendfsync=everysecstays quiet unlessaof_delayed_fsyncclimbs on two consecutive probes (the once-per-second background fsync itself backing up).Changes
evaluateAppendfsyncHazardpure evaluator inconfig-hazard.ts;ConfigHazardFindingwidened (new ids,infoseverity,advisorystatus).ConfigHazardServiceprobesappendfsync, INFO persistence, and LATENCY LATEST on the existing TTL-cached path, tracking a per-connection rising streak foraof_delayed_fsync. Symptom probe failures degrade to config-only evaluation instead of suppressing the advisory.ConfigHazardBannergains an advisory presentation (info icon, muted border) alongside hazard/unverified.Test plan
tsc --noEmitclean for api and web.Closes #368
Note
Medium Risk
Touches health-polling probe paths and shared health types; logic is well-tested but adds more Redis commands per poll when AOF is enabled and changes what the dashboard surfaces.
Overview
Extends the config-hazard monitor with valkey#3515: when AOF is on and
appendfsync=always, every write fsyncs on the main thread, so disk latency can stall commands.Detection behavior:
appendonly=nostays silent.appendfsync=alwayswith no symptoms yields a low-severity advisory (info/advisory); escalation to a warning hazard uses risingaof_delayed_fsyncacross probes, freshaof-fsync-always/aof-writeLATENCY events (5‑minute freshness), oraof_last_write_statusnotok.everyseconly fires after two consecutive risingaof_delayed_fsyncprobes (background fsync backlog), not on config alone.Implementation: Pure
evaluateAppendfsyncHazardinconfig-hazard.ts;ConfigHazardServiceadds CONFIGappendfsync, INFO persistence, and LATENCY LATEST on the existing 60s TTL cache, with per-connection streak tracking. Failed INFO/LATENCY probes degrade to config-only advisory rather than hiding findings. SharedConfigHazardFindingtypes add new ids,infoseverity, andadvisorystatus. ConfigHazardBanner shows advisories with muted styling and an info icon alongside hazard/unverified.Reviewed by Cursor Bugbot for commit 8f96986. Bugbot is set up for automated code reviews on this repo. Configure here.