docs: document trace exemplars on the duration histograms (MAPCO-11496)

[NivGreenstein]

Docs half of MAPCO-11496. Code: MapColonies/shigola#24

The duration histograms now carry the active trace as a Prometheus exemplar, so a latency spike is one click from the trace behind it.

What changed, and one retraction

docs/tracing.md said [tracing] and [observer] "have nothing to say to each other". Exemplars falsify that, so the claim is now scoped to what remains true: neither section implies the other, and tracing still publishes no metric family of its own. The page's new ### Trace exemplars and ### Wiring it up in Grafana sections are the substance.

Everything an operator has to get right is stated, because all three requirements fail quietly:

1. the scraper must ask for OpenMetrics — the only format that encodes exemplars;
2. the server must store them — --enable-feature=exemplar-storage, or Mimir's equivalent;
3. the Grafana Prometheus datasource needs an exemplar link on trace_id targeting Tempo, or the dots render with nothing behind them.

The exemplarTraceIdDestinations snippet is included, so this is copy-pasteable rather than a description of a thing to go and find.

The asymmetry that looks like a bug

Exemplars are attached only for sampled traces; log records carry ids either way. Stated as a :::warning on the tracing page and cross-referenced from the logging page, because encountering one convention after the other reads as an inconsistency until the reason is given: an exemplar is only ever a link, and Prometheus keeps one per bucket, overwritten by the next observation — so unfiltered, clicking a bucket would open nothing ~99 times out of 100. A log line's trace id still groups that request's lines whether or not Tempo kept the trace.

Breaking change, recorded twice

Negotiating OpenMetrics respells boundaries that render as whole numbers: le="1" is now le="1.0". The tracing page carries the full table under ### Two limits, and docs/layered-cache.md gets a second instalment of its bucket-identity warning — where the MAPCO-11495 bucket change already warned that le series do not line up across the upgrade. Someone reading only that warning would otherwise think it was over.

Two things in that table are counterintuitive, and the first draft of these docs got both wrong: 2.5 is not affected (the client appends .0 only when the shortest rendering contains neither . nor e), and the response-size families are, despite carrying no exemplars, because the format is negotiated per scrape rather than per family. Both are called out explicitly, and the code repo now derives the list in a test so these pages cannot drift from it.

Cross-references

Four other pages describe something exemplars touch: logging.md (same two ids, now says where they differ), layered-cache.md (its tier-latency section is what exemplars are most useful on), http-endpoints.md (/metrics answers in a different format), configuration.md (the tracing section claimed independence from [observer]).

Verified

npm run build clean, which is the real check — it throws on broken links and anchors, and this adds several of both, including a deep link to layered-cache.md#tier-latency-and-why-it-used-to-look-identical-everywhere.

Correction after review

The page originally said a push_url deployment "pushes through the classic text format to a Pushgateway, which has no notion of them". That was wrong on the mechanism — the push client sends protobuf, which does carry exemplars — and asserted a conclusion about Pushgateway that this repo cannot verify. It now states only what is checkable, and says explicitly that Shigola does not test it.

🤖 Generated with Claude Code