Skip to content

feat(monitor): appendfsync=always main-thread blocking advisory (valkey#3515) - #375

Open
jamby77 wants to merge 2 commits into
masterfrom
feature/368-appendfsync-always-advisory
Open

feat(monitor): appendfsync=always main-thread blocking advisory (valkey#3515)#375
jamby77 wants to merge 2 commits into
masterfrom
feature/368-appendfsync-always-advisory

Conversation

@jamby77

@jamby77 jamby77 commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds the AOF fsync-policy advisory from valkey-io/valkey#3515 to the config-hazard subsystem (the sibling of #337's valkey#3983 hazard): with appendfsync always, every write fsyncs synchronously on the main thread, so disk latency becomes command latency. The broader upstream AOF-modernization work (WAL headers, io_uring, direct I/O) is unshipped and not pollable — the misconfiguration and its symptom are observable today.

Detection (per the issue's guardrails)

  • appendonly=no → no-op (managed/ephemeral instances with AOF intentionally off never fire).
  • appendfsync=always with no symptoms → low-severity advisory (severity: info, status: advisory): the config is a latency risk, consider everysec unless per-write durability is a hard requirement.
  • Escalates to a warning hazard when symptoms confirm blocking, and the message names the specific counters: aof_delayed_fsync rising across probes, aof-fsync-always/aof-write LATENCY events, or aof_last_write_status != ok.
  • appendfsync=everysec stays quiet unless aof_delayed_fsync climbs on two consecutive probes (the once-per-second background fsync itself backing up).
  • Rate limiting comes from the subsystem's design: findings are TTL-cached polled state (60s) surfaced via health → dashboard banner, not repeated events.

Changes

  • evaluateAppendfsyncHazard pure evaluator in config-hazard.ts; ConfigHazardFinding widened (new ids, info severity, advisory status).
  • ConfigHazardService probes appendfsync, INFO persistence, and LATENCY LATEST on the existing TTL-cached path, tracking a per-connection rising streak for aof_delayed_fsync. Symptom probe failures degrade to config-only evaluation instead of suppressing the advisory.
  • ConfigHazardBanner gains an advisory presentation (info icon, muted border) alongside hazard/unverified.

Test plan

  • 9 pure evaluator tests (acceptance matrix: AOF off, always clean/rising/latency-event/write-status, everysec healthy/single-rise/steady-climb, unknown policies).
  • 7 service tests (probe wiring, cross-probe streak escalation, everysec two-probe gate, LATENCY failure tolerance, coexistence with the valkey#3983 finding); cache tests updated to count probes rather than raw CONFIG GET calls.
  • 2 new banner tests (advisory rendering, escalated hazard rendering).
  • monitor suite 346/346, health + MCP health 7/7, banner 6/6; tsc --noEmit clean for api and web.

Closes #368


Note

Medium Risk
Touches health-polling probe paths and shared health types; logic is well-tested but adds more Redis commands per poll when AOF is enabled and changes what the dashboard surfaces.

Overview
Extends the config-hazard monitor with valkey#3515: when AOF is on and appendfsync=always, every write fsyncs on the main thread, so disk latency can stall commands.

Detection behavior: appendonly=no stays silent. appendfsync=always with no symptoms yields a low-severity advisory (info / advisory); escalation to a warning hazard uses rising aof_delayed_fsync across probes, fresh aof-fsync-always / aof-write LATENCY events (5‑minute freshness), or aof_last_write_status not ok. everysec only fires after two consecutive rising aof_delayed_fsync probes (background fsync backlog), not on config alone.

Implementation: Pure evaluateAppendfsyncHazard in config-hazard.ts; ConfigHazardService adds CONFIG appendfsync, INFO persistence, and LATENCY LATEST on the existing 60s TTL cache, with per-connection streak tracking. Failed INFO/LATENCY probes degrade to config-only advisory rather than hiding findings. Shared ConfigHazardFinding types add new ids, info severity, and advisory status. ConfigHazardBanner shows advisories with muted styling and an info icon alongside hazard/unverified.

Reviewed by Cursor Bugbot for commit 8f96986. Bugbot is set up for automated code reviews on this repo. Configure here.

…ey#3515)

- New config-hazard evaluator: appendfsync=always with AOF on raises a
  low-severity advisory on config alone, escalating to a warning hazard
  when symptoms confirm blocking (aof_delayed_fsync rising across
  probes, aof-fsync-always/aof-write LATENCY events, or a failing
  aof_last_write_status)
- everysec is flagged only when aof_delayed_fsync climbs on two
  consecutive probes (the background fsync backing up), never on config
- Service probes appendfsync + INFO persistence + LATENCY LATEST on the
  existing TTL-cached path; symptom probe failures degrade to the
  config-only advisory instead of suppressing it
- Dashboard banner gains an advisory presentation (info icon, muted)
  alongside the existing hazard/unverified states
Comment thread apps/api/src/monitor/config-hazard.service.ts
@jamby77

jamby77 commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator Author

@claude review

LATENCY LATEST entries persist until LATENCY RESET, so a single past
aof-fsync-always/aof-write spike would escalate the advisory to a
hazard forever. Filter entries by their spike timestamp (5-minute
freshness window) so escalation mirrors the fresh-rise discipline of
the aof_delayed_fsync path.
@jamby77

jamby77 commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator Author

@BugBot review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 8f96986. Configure here.

@jamby77
jamby77 requested a review from KIvanow August 11, 2026 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Advisory: AOF appendfsync=always main-thread blocking

1 participant