Repository navigation
Add doc counts to BulkIndexerResponseStat - #326
Merged
Merged
Conversation
elastic-observability-automation
Bot
added
the
safe-to-test
The bulk response decoder reads `_index` for every item, but it only keeps it for failed documents, which end up in FailedDocs with their Index. Successful documents just bump Indexed, so a caller has no way to know which indices they were written to. We would like to expose the destination index as an opt-in metric attribute. This PR adds `BulkIndexerResponseStat.IndexStats`, mapping a destination index to a `BulkIndexerIndexStat` holding Indexed / Failed counts. It's opt-in with a new `BulkIndexerConfig.PopulateIndexStats`, so existing users don't incur the cost unless they're intested in the stat. When it's on, it costs one map allocation per flush, which grows with the number of distinct indices in a bulk request. Since `_index` is already part of the response, it doesn't change the size either way. The Indexed count is what you'd expect, so it includes the documents indexed to the failure store like the response-level one does, and Failed is filled in at the end of Flush from the final FailedDocs instead of in the decoder, so we avoid counting retries or document-level retries as Failed. A document redirected to the failure store returns a 201 with the `.fs-` index in `_index`, so it would be indistinguishable from any other success. To tell them apart, each index also carries `FailureStoreDocs` with the used / failed / not_enabled counts. They're counted once the whole item is decoded, since the response doesn't guarantee that `_index` comes before `failure_store`. Signed-off-by: Marc Lopez Rubio <marc5.12@outlook.com>
marclop
force-pushed
the
indexed-per-index
branch
from
October 13, 2026 01:49
6fb3b91 to
f95a48a
Compare
marclop
changed the title
BulkIndexerResponseStatBulkIndexerResponseStat
marclop
enabled auto-merge (squash)
October 14, 2026 04:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bulk response decoder reads
_indexfor every item, but it only keeps it for failed documents, which end up in FailedDocs with their Index. Successful documents just bump Indexed, so a caller has no way to know which indices they were written to. We would like to expose the destination index as an opt-in metric attribute.This PR adds
BulkIndexerResponseStat.IndexStats, mapping a destination index to aBulkIndexerIndexStatholding Indexed / Failed counts. It's opt-in with a newBulkIndexerConfig.PopulateIndexStats, so existing users don't incur the cost unless they're intested in the stat.When it's on, it costs one map allocation per flush, which grows with the number of distinct indices in a bulk request. Since
_indexis already part of the response, it doesn't change the size either way.The Indexed count is what you'd expect, and Failed is filled in at the end of Flush from the final FailedDocs instead of in the decoder, so we avoid counting retries or document-level retries as Failed.