[APMSVLS-469] feat(traces): rescue errored traces via agent-side error sampler

[lucaspimentel]

Overview

On the lambda_extension_compute_stats path, the extension drops every trace marked P0 (priority <= 0) after computing its stats. That means an errored trace sampled away by the tracer is invisible in the UI, even though its stats are counted.

This adds an agent-side error sampler that gives those dropped traces a second look: errored traces are rescued up to DD_APM_ERROR_TPS traces/sec (default 10), distributed fairly across trace signatures, with _dd.errors_sr stamped on the rescued root span. The result is guaranteed error visibility even under aggressive sampling, matching the Go trace agent's ScoreSampler / ErrorTPS behavior.

Which traces are candidates, matching the Go agent:

• An error on any span in the trace makes it a candidate, not just on the root span. A trace whose failure happened deeper (for example a failed downstream call the handler caught) is rescued too.
• Only traces dropped by automatic sampling are candidates. Traces dropped on purpose, by a tracer sampling rule or an explicit MANUAL_DROP, are honored and never rescued.
• Non-errored P0 traces are still dropped, and stats still count all traces.
• The sampler's per-signature rate limits are keyed on the env the tracer reported with the trace, so distinct services do not compete for one shared budget.

The sampling logic itself lives in the shared, dependency-free datadog-agent-trace-sampler crate added in DataDog/serverless-components#141. It takes primitives in (SpanView / TraceView) and returns a SampleDecision, exposing no protobuf Span type, so consumers pinning different libdatadog revisions can share it.

New configuration:

Variable	Default	Notes
DD_APM_ERROR_TPS	10.0	Target rescued error traces/sec. 0 disables the rescue. Go agent equivalent: apm_config.errors_per_second.
DD_APM_EXTRA_SAMPLE_RATE	1.0	Extra raw sample rate applied on top of the computed rate. Go agent equivalent: apm_config.extra_sample_rate.

Both are flat env/YAML keys (apm_error_tps, apm_extra_sample_rate), unlike the Go agent's nested apm_config.* form.

Blocked on

DataDog/serverless-components#141 must merge first. The four serverless-components pins in bottlecap/Cargo.toml currently point at that PR's branch rev (3759fae) and need to be repinned to the merged main rev before this can land. Draft until then.

Testing

• New unit test test_error_sampler_rescues_errored_p0_chunks in traces/trace_processor.rs: an errored auto-dropped chunk is rescued and stamped with _dd.errors_sr; a non-errored one is still dropped; an errored chunk marked as an explicit user drop is not rescued.
• New unit test test_error_sampler_rescues_chunk_with_errored_child_span: a trace whose root is fine but whose child span errored is still rescued.
• New config tests covering the defaults and env-var overrides for both settings.
• apm_integration_test.rs wires the sampler into the processor pipeline, so the existing end-to-end APM assertions run with it enabled.
• Full suite locally: cargo test --workspace — 564 passed, 0 failed. cargo clippy --workspace --all-targets --features default and cargo fmt --all -- --check clean.

🤖

[datadog-datadog-us1-prod]

✨ Unblock PR with BitsAI

⚠️ Warnings

🚦 2 Pipeline jobs failed

DataDog/datadog-lambda-extension | e2e-test-status (amd64)     

DataDog/datadog-lambda-extension | e2e-test-status (amd64, fips)     

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ffbb082 | Docs | Datadog PR Page | Give us feedback!}