Repository navigation
Use <aggregate> label values for distributor-sampled profiles #5600
Description
Background
Distributor sampling can remove detailed information from sampled-out profiles while retaining their totals. For labels derived from removed data, this can cause label values to appear and disappear over time without explaining why.
Proposal
For sampled-out profiles, use <aggregate> as the value of labels whose original values can no longer be preserved.
For example, a time series grouped by function_name could show concrete function names for retained profiles and function_name="<aggregate>" for sampled-out profiles.
Motivation
This would:
- Explain why specific label values appear and disappear.
- Make the effect of distributor sampling visible in query results.
- Preserve sampled totals without attributing them to an incorrect label value.
- Help users distinguish missing data from intentionally aggregated data.
This does not need to block the current sampling work and can be implemented as a follow-up, potentially at query time rather than storage time.
Activity
subhramit commentedon Sep 20, 2026
Hey, just wanted to ask, why query time rather than storage? Wouldn't storage time be more precise?
Reason being, the distributor strips the profile before the sample-label split, so at that point it still knows which sample-label keys the profile had. It could add
key="<aggregate>"to the series labels for those keys before stripping.At query time, a stripped row that is missing a
groupBylabel could mean the label was stripped, or that the profile never had it. So a query-time fix would sometimes show where no data was aggregated.Although the tradeoff I see with storage time are that it changes stored data (existing stripped profiles wouldn't get it) and adds a label value to the index. Is that why you propose query time?
subhramit commentedon Sep 20, 2026
(Some more related follow-up questions)
LabelValuesreturn<aggregate>, so users can filter on it from the UI? (If yes, this also favors storage time as<aggregate>would be in the index whereas query time would need more work)withExcludeSampled()and ignoreinclude_stripped_profiles. meaning, would we leave heatmap out of scope?