Skip to content

Use the real Browser Debugger SDK in the runtime benchmark#440

Open
watson wants to merge 1 commit into
watson/DEBUG-5787/add-benchfrom
watson/DEBUG-5787/use-real-sdk-in-bench
Open

Use the real Browser Debugger SDK in the runtime benchmark#440
watson wants to merge 1 commit into
watson/DEBUG-5787/add-benchfrom
watson/DEBUG-5787/use-real-sdk-in-bench

Conversation

@watson

@watson watson commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

What and why?

The Live Debugger runtime benchmark measured dormant overhead against a hand-written stub of the SDK's instrumentation runtime ($dd_probes/$dd_entry/$dd_return/$dd_throw), which could silently drift from what actually ships. This PR replaces that stub with the real published Browser Debugger SDK, loaded but kept dormant, so the measured overhead reflects production. It also records per-run provenance — the SDK version and the CDN build fingerprint — so any benchmark result can be tied to the exact build it was measured against.

How?

  • Fetch the published datadog-debugger.js bundle from the CDN, inline it before the fixture runs, mock the probe-delivery poll same-origin to return an empty probe set, and call DD_DEBUGGER.init() after load so the SDK wires the real $dd_probes lookup while staying dormant (no active probes).
  • Assert the SDK loaded and is dormant, and read its version from DD_DEBUGGER.version, failing the run if it is missing.
  • Capture the CDN object's build fingerprint from the bundle-fetch response headers — publish date (Last-Modified) and S3 ETag (content hash, normalized from CloudFront's weak W/"…" form) — and assert both are present, since the /v7/ bundle reuses the same release-format version across many master builds and is therefore ambiguous on its own.
  • Add an optional onResponse hook to @dd/core's doRequest so callers can read response headers (it previously returned only the parsed body); the benchmark uses it to read Last-Modified and ETag.
  • Surface the version and fingerprint in the console summary, the PR comment (the hash labeled "S3 ETag" so it is not mistaken for a git commit SHA), and the results JSON, with unit tests covering the label formatting and weak-ETag normalization; update the live-debugger CONTRIBUTING.md.

watson commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jun 25, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 4e565bd | Docs | Datadog PR Page | Give us feedback!

@github-actions

github-actions Bot commented Jun 25, 2026

Copy link
Copy Markdown

Live Debugger Runtime Benchmark

SDK-loaded dormant-probe runtime overhead, measured against an uninstrumented bundle in the same browser session.

Browser Debugger SDK: 7.4.0 · built 2026-06-23 · S3 ETag 0766e1c8

Browser Workload Quality Per-call overhead upper
chrome Hot clean <= 4.84 ns
chrome Tiny clean <= 0.99 ns
firefox Hot clean <= 9.89 ns
firefox Tiny clean <= 0.00 ns
safari Hot clean <= 15.39 ns
safari Tiny clean <= 18.88 ns
Full diagnostics
browser  workload  quality  per-call overhead upper  overhead upper           95% CI        A/A diag         block CI  acf(1)   baseline  instrumented                         samples
-------  --------  -------  -----------------------  --------------  ---------------  --------------  ---------------  ------  ---------  ------------  ------------------------------
chrome   Hot       clean                 <= 4.84 ns       <= 13.35%    4.80..4.83 ns  -0.00..0.03 ns    4.80..4.84 ns   -0.10  55.647 ms     63.055 ms   102 (trim 20%, outliers 6.9%)
chrome   Tiny      clean                 <= 0.99 ns       <= 15.92%    0.74..0.95 ns  -0.01..0.02 ns    0.71..0.99 ns    0.03  48.073 ms     55.725 ms   102 (trim 20%, outliers 7.8%)
firefox  Hot       clean                 <= 9.89 ns       <= 38.02%    9.80..9.87 ns  -0.02..0.03 ns    9.79..9.89 ns   -0.00  39.900 ms     54.990 ms   102 (trim 20%, outliers 4.9%)
firefox  Tiny      clean                 <= 0.00 ns        <= 0.02%   -0.01..0.00 ns  -0.00..0.01 ns   -0.02..0.00 ns    0.25  97.980 ms     97.980 ms  102 (trim 20%, outliers 17.6%)
safari   Hot       clean                <= 15.39 ns       <= 61.66%  14.75..15.18 ns  -0.01..0.01 ns  14.64..15.39 ns    0.13  38.300 ms     61.300 ms   102 (trim 20%, outliers 2.9%)
safari   Tiny      clean                <= 18.88 ns      <= 724.80%  18.45..18.78 ns  -0.00..0.00 ns  18.33..18.88 ns    0.45   6.240 ms     50.870 ms   102 (trim 20%, outliers 3.9%)

Raw samples are in the live-debugger-runtime-bench-results artifact.

@watson watson force-pushed the watson/DEBUG-5787/use-real-sdk-in-bench branch from 1c55ebd to 8c50c4c Compare June 25, 2026 14:01
@watson watson force-pushed the watson/DEBUG-5787/add-bench branch from 1296572 to d80cc4f Compare June 25, 2026 14:01
@watson watson force-pushed the watson/DEBUG-5787/use-real-sdk-in-bench branch from 8c50c4c to 66f1360 Compare June 26, 2026 11:49
@watson watson force-pushed the watson/DEBUG-5787/add-bench branch from d80cc4f to ba9d257 Compare June 26, 2026 11:49
The benchmark previously installed a hand-written stub of the SDK's instrumentation
runtime ($dd_probes / $dd_entry / $dd_return / $dd_throw). That stub could silently drift
from what actually ships, so the measured dormant overhead was not guaranteed to reflect
production.

Fetch the published Browser Debugger SDK bundle from the CDN, inline it before the fixture
runs, and call DD_DEBUGGER.init() with its probe-delivery poll mocked same-origin to return
an empty set. This keeps the SDK dormant (no active probes) while wiring the real $dd_probes
lookup that the benchmark measures, so it cannot drift from the shipped runtime.

Record provenance for each run so a result can be tied to an exact build. The SDK version is
read from DD_DEBUGGER.version, and because the published /v7/ bundle reuses the same
release-format version across many master builds, the CDN object's build fingerprint (publish
date and S3 ETag content hash) is captured from the bundle-fetch response headers. The
version and both headers are asserted, so a missing one fails the run rather than reporting a
build that cannot be identified.

Add an optional onResponse hook to @dd/core's doRequest so callers can read response headers
(it previously returned only the body); the benchmark uses it to capture Last-Modified and
the ETag.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant