output: shared helper to split and resend payloads rejected with 413 Payload Too Large #12746

Is your feature request related to a problem? Please describe.

Many HTTP-based output plugins send a flushed chunk as a single request. When the destination enforces a maximum request size and answers 413 Payload Too Large, no output plugin reacts to that specifically today, so the data is eventually lost:

Two plugins already try to avoid the problem on their own, each with a different approach:

• out_azure_logs_ingestion: split payloads over the 1 MiB request limit #12696 (open, fixes Azure Logs Ingestion API output doesn't split chunks into multiple requests to stay under payload size limit #12409) makes out_azure_logs_ingestion split a payload over 1 MiB in two halves, recursively, before sending.
• out_chronicle splits a chunk into batches by an estimated size and, if a formatted batch is still too large, re-formats with a shrinking threshold up to 8 times. A single record larger than the limit uses up all attempts and fails the whole flush, which also drops the other records still pending in that chunk. It also does not react to an actual 413 from the server.

Both are proactive only (they rely on a hard-coded limit) and the logic is not reusable by other plugins.

Describe the solution you'd like

A small, opt-in core helper that any output plugin can use to send a log chunk as one or more requests, and split and resend a request when the endpoint rejects it as too large, bounded by a maximum number of splits.

The helper would only own record-range bookkeeping and the split/retry decision. Each plugin keeps its own HTTP client, auth and compression and plugs in through callbacks:

/* plugin-reported outcome of one send attempt */
#define FLB_OUTPUT_SPLIT_SENT       0  /* accepted                       */
#define FLB_OUTPUT_SPLIT_TOO_LARGE  1  /* 413 (or vendor equivalent)     */
#define FLB_OUTPUT_SPLIT_RETRY      2  /* transient, retry whole chunk   */
#define FLB_OUTPUT_SPLIT_ERROR      3  /* permanent for this range       */

int flb_output_split_send(struct flb_output_instance *ins,
                          const char *data, size_t size,
                          struct flb_output_split_options *opts,
                          flb_output_split_format_cb format_cb, /* records -> body */
                          flb_output_split_send_cb send_cb,     /* body -> status  */
                          flb_output_split_free_cb free_cb,
                          void *plugin_ctx,
                          struct flb_output_split_stats *stats);

Proposed behavior:

1. Index the end offset of every log record in the chunk (same technique as out_azure_logs_ingestion: split payloads over the 1 MiB request limit #12696).
2. Format and send the full range.
3. On TOO_LARGE, halve the record range and send each half, up to max_depth levels (e.g. default 4, at most 16 sub-requests per original request) so a misbehaving endpoint cannot cause unbounded request amplification.
4. Optionally, if the plugin knows a hard limit (Azure 1 MiB, Chronicle 1 MiB / 4 MB), split proactively before sending, using the same code path.
5. Stop at the first RETRY (the engine retries the whole chunk anyway); keep sending the other half after a permanent ERROR and report it at the end.
6. If a single record is still rejected as too large, log it (size only, not content) and drop just that record (configurable), instead of failing the whole chunk.

Possible shared config keys for adopting plugins: split_on_payload_too_large, split_max_depth, split_drop_oversized_record.

Suggested adoption order: out_http first (widest reach), then out_chronicle (replacing its shrinking-threshold loop and fixing the oversized-record case), then out_azure_logs_ingestion (the #12696 recursion maps almost 1:1 onto the helper). Others such as out_splunk, out_loki, out_datadog and out_es would be evaluated individually, since Loki batches per stream and ES bulk responses carry per-item errors.

Testing would include internal unit tests with fake callbacks (empty chunk, split on 413, depth limit, single oversized record, retry on left half stops the right half, every body freed once), runtime tests per adopter with a fake endpoint that returns 413 above N bytes, and integration scenarios run with Valgrind/Leaks.

Describe alternatives you've considered

• Keep per-plugin implementations like out_azure_logs_ingestion: split payloads over the 1 MiB request limit #12696 and Chronicle's loop. That works, but every plugin re-implements the same offset indexing and recursion, with differing edge-case behavior, and none of them reacts to an actual 413.
• Shrinking-threshold retries (Chronicle's current approach). A single oversized record cannot be isolated, so the whole chunk fails after N attempts. Halving isolates it in O(log n) requests.
• Engine-level splitting. More general, but the engine doesn't know each plugin's serialization or how its status codes map, so a plugin-driven helper seems simpler and lower risk.

Additional context

Open questions where maintainer input would help:

1. OTLP group markers. Splitting a raw byte range can separate group start markers from their records. A simple first version could be limited to plugins that don't serialize group metadata; alternatives are splitting only on group boundaries or re-emitting the group header per sub-range.
2. Should proactive (known-limit) splits count against max_depth, or only server-driven 413 splits?
3. Some APIs signal "too large" with a 400 and a vendor error body; the proposal lets the plugin's send callback decide what counts as TOO_LARGE.
4. Partial success followed by a retry re-sends records that were already accepted. That is already the case for Chronicle and out_azure_logs_ingestion: split payloads over the 1 MiB request limit #12696; the helper would keep it and document it.
5. A later version could remember the largest accepted body size per instance and split proactively on the next flushes to avoid a guaranteed 413 every time.
6. Metrics/traces (OTLP/HTTP can also return 413) are out of scope for a first version; input_metric: split large metric contexts into chunk-sized batches #11554 and out_opentelemetry: support metrics max datapoints #12363 are related prior art for those signals.

Is there interest in a shared helper like this, versus continuing with per-plugin fixes? If so, I'd be happy to help with an implementation starting with the core helper, its unit tests and out_http.