Repository navigation
feat(storage): opt-in blocking write backpressure when the head cannot drain #146
Description
Follow-up to #141 / #145. The head-size flush trigger now fires as soon as an engine's head reaches Options.FlushThresholdBytes, but nothing makes the writer wait when flushes cannot keep up. If ingestion outruns the flush loop — a bulk backfill, a slow object-store backend, a merge-heavy cycle — the head keeps growing past the threshold and the process still OOMs, just later.
Today the only thing that reacts to a too-large head is tenant.Limits.MaxInFlightBytes, which sheds records (rejectBytes). That is the wrong tool for a durable ingest path or a migration: dropping a log record breaks a stream, and a backfill must not silently lose data. So an operator's real choice is currently "lose records" or "OOM".
Proposal
An opt-in mode where a write blocks until the head drains below a high-water mark instead of shedding — the VictoriaMetrics/VictoriaLogs pattern:
// lib/mergeset/table.go
func (tb *Table) addToInmemoryParts(pw *partWrapper, isFinal bool) {
// Wait until the number of in-memory parts goes below maxInmemoryParts.
select {
case tb.inmemoryPartsLimitCh <- struct{}{}:
default:
tb.inmemoryPartsLimitReachedCount.Add(1)
select {
case tb.inmemoryPartsLimitCh <- struct{}{}:
case <-tb.stopCh:
}
}
...
}Sketch for this engine:
- New option (e.g.
WithWriteBackpressure(highWaterBytes)), off by default — blocking changes the write contract, so it must be explicit. - On the write path, after the append that crossed the threshold: poke the flush loop (already done), then wait until the engine's head drops below the mark, or until
ctxis done, or the store closes. Returningctx.Err()on timeout gives the embedder a retryable signal (OTLP throttling) instead of a silent drop. - Counters: time spent blocked and how often, next to
MaintenanceStats.PressureFlushes, so the condition is visible inInspect.
Constraints
- Durability is not affected — records are WAL-logged on append (
logWALatflushedEpoch+1; flush publishes the index watermark, then checkpoints). Blocking only adds latency. - Must not deadlock: the writer waits on the maintenance loop, so the wait needs
stopChandctxescapes, andClosemust not block behind a stalled writer. - Flush must not queue behind merges for this to relieve anything — see the scheduling note in perf(recordengine): flush amplifies the head 2-3x in memory #140/perf(recordengine): merge working set is unbounded on two paths #143:
maintaincurrently runsflush(); merge()in the sameparallel.ForEachslot, so a multi-second merge delays the very flush the writer is waiting on.flushPressured(from fix(storage): honor FlushThresholdBytes — wire the head-size flush trigger #145) sidesteps that today; a full fix separates the flush and merge scheduling classes the way VM does with its per-tier concurrency channels. - Interaction with
MaxInFlightBytes: shedding should remain the last-resort valve (hard cap), with backpressure as the earlier, lossless one.
Activity
tdakkota commentedon Jul 24, 2026
Re-verified on
cd9847b— not implemented. NoWithWriteBackpressure/ blocking-wait option exists;rg -i backpressurefinds only doc comments describing the shedding behaviour (admission.go,tenant/tenant.go,recordengine/admission.go:14,engine/admission.go:18,31). The head-size trigger from #141/#145 is in place (storage.go:1090flushPressured), so the premise of this issue holds exactly as written.One correction to the "Constraints" section, from the review pass: it says "Durability is not affected — records are WAL-logged on append." That is not currently true in the failure case this feature exists for. Per #160, a flush that fails after
head.detachloses the detached records permanently, and the WAL does not recover them becauseWAL.Checkpointdeletes segments by sequence regardless of epoch. A slow or failing object-store backend is precisely the scenario that triggers both, so #160 should land before backpressure is relied on as the "don't lose data" answer for backfills.Also relevant to sizing the high-water mark: #177 —
HeadBytes()reports 0 while a flush is in flight, becausedetachzeroes the counter before the buffers are released. Any watermark logic built on it today would read the wrong value at exactly the moment it matters.tdakkota commentedon Sep 15, 2026
Triage @ 15f1742: still not implemented. The #160/#177 blockers are closed, so the durability premise holds. Flush and merge still share one maintenance slot (
storage.gomaintainOneEngine; see #155).flushPressuredis now atstorage.go:1460.🤖 Generated with Claude Code
tdakkota commentedon Sep 28, 2026
#747 (part of #725) adds another way a flush can fail to drain the head. A new owner must commit a tenure-establishing index write before its first flush, and until that write lands, flushes write nothing and keep their rows in the head and WAL. The causes are the same as for any failing flush: the backend is down, or index CAS conflicts persist. So with
MaxInFlightBytesunset (the default), head and WAL growth stays unbounded, as it already is on main during a backend outage. Codex review of #747 flagged this, and it belongs here: a mandatory, lossless bound that covers every cause of not draining, applied the same way in both engines.🤖 Generated with Claude Code
https://claude.ai/code/session_01KcLq1G8KpWBvM53JaQb1tS