app/vlagent/remotewrite: add flag to remote writes getting 404 responses

[angelofallars]

Adds a new flag to vlagent -remoteWrite.retryOn404 that retries writing blocks of data to -remoteWrite.url if the response returned 404 instead of dropping them.

This flag allows users with a certain setup where requests routed to VictoriaLogs sometimes get a transient 404 status code to have more reliability in receiving complete logs without dropped blocks.

Closes #1499

Flags entry:

  -remoteWrite.retryOn404 array
    	Whether to retry sending a block of data instead of dropping when a 404 status code is received from the corresponding -remoteWrite.url.
    	Supports array of values separated by comma or specified via multiple flags.
    	Empty values are set to false.

[cubic-dev-ai]

1 issue found across 3 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="app/vlagent/remotewrite/client.go">

<violation number="1" location="app/vlagent/remotewrite/client.go:361">
P2: With -remoteWrite.retryOn404 enabled, a 404 is now routed into the unbounded retry loop in sendBlockHTTP (the '// Unexpected status code returned' branch), which retries until c.stopCh is closed with no cap on total retry time. If the 404 is permanent rather than transient (for example a misconfigured -remoteWrite.url, or a proxy/LB returning 404 for a nonexistent endpoint), the worker goroutine spins forever in exponential backoff, never drops the block, and stops draining further blocks from the queue for that URL. A 404 usually indicates a permanent routing/misconfiguration problem rather than a transient one, so this flag can turn a previously fast-failing/dropped block into an indefinite stall. Consider bounding the 404 retries (e.g. give up after -remoteWrite.retryMaxTime total elapsed time for the 404 case) or documenting this behavior in the flag help.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: With -remoteWrite.retryOn404 enabled, a 404 is now routed into the unbounded retry loop in sendBlockHTTP (the '// Unexpected status code returned' branch), which retries until c.stopCh is closed with no cap on total retry time. If the 404 is permanent rather than transient (for example a misconfigured -remoteWrite.url, or a proxy/LB returning 404 for a nonexistent endpoint), the worker goroutine spins forever in exponential backoff, never drops the block, and stops draining further blocks from the queue for that URL. A 404 usually indicates a permanent routing/misconfiguration problem rather than a transient one, so this flag can turn a previously fast-failing/dropped block into an indefinite stall. Consider bounding the 404 retries (e.g. give up after -remoteWrite.retryMaxTime total elapsed time for the 404 case) or documenting this behavior in the flag help.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At app/vlagent/remotewrite/client.go, line 361:

<comment>With -remoteWrite.retryOn404 enabled, a 404 is now routed into the unbounded retry loop in sendBlockHTTP (the '// Unexpected status code returned' branch), which retries until c.stopCh is closed with no cap on total retry time. If the 404 is permanent rather than transient (for example a misconfigured -remoteWrite.url, or a proxy/LB returning 404 for a nonexistent endpoint), the worker goroutine spins forever in exponential backoff, never drops the block, and stops draining further blocks from the queue for that URL. A 404 usually indicates a permanent routing/misconfiguration problem rather than a transient one, so this flag can turn a previously fast-failing/dropped block into an indefinite stall. Consider bounding the 404 retries (e.g. give up after -remoteWrite.retryMaxTime total elapsed time for the 404 case) or documenting this behavior in the flag help.</comment>

<file context>
@@ -355,7 +358,7 @@ again:
 
 	metrics.GetOrCreateCounter(fmt.Sprintf(`vlagent_remotewrite_requests_total{url=%q, status_code="%d"}`, c.sanitizedURL, statusCode)).Inc()
-	if statusCode == 400 || statusCode == 404 {
+	if statusCode == 400 || (!c.retryOn404 && statusCode == 404) {
 		logBlockRejected(block, c.sanitizedURL, resp)
 		_ = resp.Body.Close()
</file context>

[angelofallars]

default max retry time is 1 minute from what i can tell, so i think retrying a permanent 404 is negligible. it won't "run forever" in that situation