feat(distributor): make profile stripping recording-rules aware

[alsoba13]

What

Follows #5441. Makes the ProfileStripper recording-rules aware: when a profile is sampled out but kept (keep-stripped-profiles), the parts a tenant's recording rules need survive stripping so the rules keep producing correct metrics at compaction, instead of collapsing to a single label-less total.

Two kinds of rule are handled ("exceptions" to the strip):

• Function-filter rules (stacktrace_filter.function_name): the stacktraces reaching the targeted function are kept, trimmed to the frames that resolve to it, carrying the labels the rule matches/groups on.
• Total rules (no function filter): these match and group on labels that are promoted from pprof sample labels at split time. So the totals are split per distinct combination of those labels — rather than one label-less total — so the rules can still match and group.

ProfileStripper gains a Ruler dependency (declared in the sampling package, RecordingRules(tenant) ([]*RecordingRule, error); satisfied structurally by metrics.CachedRemoteRuler / the static-overrides ruler). initDistributor builds it and passes it to distributor.New, and RecordingRulesClient is added to the Distributor module deps. distributor.go still just calls d.stripper.Strip(...).

StripToTotals → Strip, now Strip(tenantID, seriesLabels, p) — it keeps rule-targeted stacktraces too, so "ToTotals" was only half-true.

Structure

func (s *ProfileStripper) Strip(tenantID, seriesLabels, p) (rulesUnavailable bool) {
    exc := s.computeExceptions(...)  // rules -> per-location function rules + total rules
    stripSamples(p, exc)             // per sample: keep (trimmed) or fold into a total
    pruneUnreferencedSymbols(p)
    compactStringTable(p)
    return exc.rulesUnavailable
}

• computeExceptions matches the rules once: __profile_type__ is resolved from the pprof's sample types, base-label matchers are checked now, and matchers on labels not yet present (promoted sample labels) are deferred to per-sample checks.
• A rule's labels/stacktraces are kept for a sample only if it satisfies that rule's deferred matchers. A matcher that also matches "" (e.g. !=) keeps its label with the real value, so a dropped label doesn't read as "" downstream and wrongly match. This guard is shared by the function and total paths (keepRuleLabels) so they can't drift.

Configuration

The distributor's rule-aware stripping must use the same rules the compaction observer uses, or stripping drops the stacktraces the observer later needs. The rules source used to live only under compaction-worker.metrics-exporter, so the distributor silently fell back to the static-overrides ruler even when the compaction worker read rules from the remote client — diverging.

This introduces a shared, single source of truth:

• New recording-rules.enabled and recording-rules.client-address (empty ⇒ static rules from per-tenant overrides; set ⇒ dynamic client). Read by the query-frontend (feature flag), the distributor (rule-aware stripping) and the compaction-worker (metrics generation).
• The distributor and compaction-worker resolve their ruler from the same value, so they cannot diverge.
• The old compaction-worker.metrics-exporter.{enabled,rules-source.client-address} flags are deprecated but honored as a fallback (RecordingRulesConfig.ReconcileWithExporter: shared flags win, effective values written back), so existing configs keep working unchanged. A warning is logged at startup when either deprecated flag is set. remote-write-address stays compaction-worker-only and mandatory when enabled.
• The distributor builds a rule-aware ruler only when recording-rules.enabled; otherwise it's a plain strip-to-totals. Whether stripping happens at all remains the per-tenant keep-stripped-profiles setting.

Rules loading and availability

metrics.Ruler.RecordingRules now returns ([]*RecordingRule, error). The error means a tenant's rules have never been loaded successfully; CachedRemoteRuler keeps serving the last good rules (stale) when a refresh fails, so it only errors on a cold failure. The static ruler never errors.

Since the distributor now reads rules on the push path, CachedRemoteRuler never lets a slow rules service stall pushes:

• The first load per tenant is synchronous and shared by concurrent callers; the client call has a 5s timeout.
• Once rules expire (1m), the stale rules keep being served while a single background refresh runs.
• A failed fetch is not retried for 10s.

Writes take priority over rule data: if the rules are unavailable, the profile is still stripped to plain totals and pushed, and Strip reports it. The distributor counts these in pyroscope_distributor_stripped_profiles_without_rules_total{tenant} and adds recording_rules_unavailable=true to the push log line. Those profiles lose the stacktraces the rules would have needed.

The compaction worker shares this ruler, so it also gets stale-while-revalidate and the 10s failure backoff; the observer treats a rules error as no rules, as before.

When recording-rules.enabled is set, distributors now also load tenant rules from recording-rules.client-address, so the settings service sees their requests as well as the compaction workers'.

Label aliasing

Sample labels are reduced in place, which can leave dropped labels live in a slice's spare capacity — the pprof split's restoreRemovedLabels reads labels[len:cap] and resurrects them with their pre-compaction string indices, indexing the rebuilt (smaller) string table out of range (the panic #5409 originally worked around by collapsing to a single sample). Fixed in both paths:

• Totals: grouped with pprof.GroupSamplesByLabels, group labels copied into fresh slices (per the fix/stripped-profile-label-aliasing approach).
• Kept function samples: retainLabels clamps the returned slice's capacity to its length, so nothing lingers past len().

Trade-off

Totals are grouped in-strip but not merged with the function-kept samples; identical trimmed stacks are deduped by the downstream Normalize. So keptSize (measured right after strip for the discarded-bytes accounting) is pre-dedup and slightly over-reports what's kept / under-reports discarded. Stored data is unaffected.

Tests

• sampling/profile_stripper_test.go: a readable end-to-end reference (TestProfileStripper_Strip_Overview) exercising, in one profile, function rules with group-by / base matchers / deferred sample-label matchers, a scoped-out rule, total rules with group-by and !=, a stack hitting two functions, recursion, and non-deduped kept samples — with an in/out diagram. Plus focused cases: function-targeted keep + symbol prune + re-intern; nil ruler == plain strip; total split by group-by; matcher gating; != keeps its label on both total and function paths; a kept function sample also carries total-rule labels; TestProfileStripper_Strip_NoAliasedLabelCapacity (no sample retains aliased spare capacity), and TestProfileStripper_Strip_RulesUnavailable (ruler error → plain totals, reported).
• Distributor: an end-to-end aliasing guard (TestPushSeries_KeepStrippedProfiles_SampleSeries) driving a labeled total through pushSeries → split, and TestPushSeries_KeepStrippedProfiles_RulesUnavailable (the counter increments when the ruler errors).
• metrics/recording_rules_test.go: config resolution — shared-only, deprecated-only fallback, shared-wins-over-deprecated, and static (enabled, no address) — plus the deprecation warning.
• metrics/ruler_cache_test.go: CachedRemoteRuler — single-flight first load, stale served during refresh, failure backoff, stale kept when a refresh fails, recovery after a failure.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit e1a0af3. Configure here.}