chore(deps): bump libdd-ffe-test-suite/ffe-system-test-data from d9d8020 to c017a15 - #2328
Conversation
Bumps [libdd-ffe-test-suite/ffe-system-test-data](https://github.com/DataDog/ffe-system-test-data) from `d9d8020` to `c017a15`. - [Commits](DataDog/ffe-system-test-data@d9d8020...c017a15) --- updated-dependencies: - dependency-name: libdd-ffe-test-suite/ffe-system-test-data dependency-version: c017a15e1a153c27f325486df771a08e2a344022 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
BenchmarksComparisonBenchmark execution time: 2026-08-06 22:54:49 Comparing candidate commit a2f0f83 in PR branch Found 0 performance improvements and 1 performance regressions! Performance is the same for 13 metrics, 0 unstable metrics.
|
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | a2f0f83 | 1786056259 | dependabot/submodules/libdd-ffe-test-suite/ffe-system-test-data-c017a15 |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| single_flag_killswitch/rules-based | execution_time | 156.623ns | 159.864ns ± 2.189ns | 159.681ns ± 1.255ns | 160.865ns | 163.516ns | 166.815ns | 169.329ns | 6.04% | 1.295 | 2.529 | 1.37% | 0.155ns | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| single_flag_killswitch/rules-based | execution_time | [159.561ns; 160.168ns] or [-0.190%; +0.190%] | None | None | None |
Group 2
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | a2f0f83 | 1786056259 | dependabot/submodules/libdd-ffe-test-suite/ffe-system-test-data-c017a15 |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | execution_time | 21.627ms | 21.897ms ± 0.089ms | 21.927ms ± 0.029ms | 21.955ms | 21.977ms | 21.987ms | 21.994ms | 0.31% | -1.609 | 1.432 | 0.41% | 0.006ms | 1 | 200 |
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | throughput | 113665.935op/s | 114170.583op/s ± 468.603op/s | 114016.358op/s ± 151.154op/s | 114193.796op/s | 115232.647op/s | 115531.613op/s | 115597.355op/s | 1.39% | 1.620 | 1.462 | 0.41% | 33.135op/s | 1 | 200 |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | execution_time | 95.764ms | 96.697ms ± 0.345ms | 96.705ms ± 0.136ms | 96.803ms | 97.125ms | 97.265ms | 100.026ms | 3.43% | 4.375 | 41.974 | 0.36% | 0.024ms | 1 | 200 |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | throughput | 9997.417op/s | 10341.763op/s ± 36.319op/s | 10340.741op/s ± 14.532op/s | 10359.052op/s | 10391.423op/s | 10402.729op/s | 10442.370op/s | 0.98% | -4.117 | 38.913 | 0.35% | 2.568op/s | 1 | 200 |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | execution_time | 600.015µs | 602.220µs ± 0.822µs | 602.195µs ± 0.569µs | 602.813µs | 603.533µs | 603.971µs | 604.362µs | 0.36% | -0.093 | -0.346 | 0.14% | 0.058µs | 1 | 200 |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | throughput | 165463.759op/s | 166052.676op/s ± 226.671op/s | 166059.056op/s ± 156.843op/s | 166208.445op/s | 166418.546op/s | 166551.949op/s | 166662.500op/s | 0.36% | 0.100 | -0.345 | 0.14% | 16.028op/s | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | execution_time | [21.885ms; 21.910ms] or [-0.057%; +0.057%] | None | None | None |
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | throughput | [114105.639op/s; 114235.527op/s] or [-0.057%; +0.057%] | None | None | None |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | execution_time | [96.649ms; 96.744ms] or [-0.049%; +0.049%] | None | None | None |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | throughput | [10336.729op/s; 10346.796op/s] or [-0.049%; +0.049%] | None | None | None |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | execution_time | [602.106µs; 602.334µs] or [-0.019%; +0.019%] | None | None | None |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | throughput | [166021.262op/s; 166084.090op/s] or [-0.019%; +0.019%] | None | None | None |
Group 3
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | a2f0f83 | 1786056259 | dependabot/submodules/libdd-ffe-test-suite/ffe-system-test-data-c017a15 |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | execution_time | 5.342ms | 5.359ms ± 0.008ms | 5.359ms ± 0.005ms | 5.363ms | 5.370ms | 5.375ms | 5.426ms | 1.26% | 2.809 | 19.716 | 0.16% | 0.001ms | 1 | 200 |
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | throughput | 460718.475op/s | 466512.547op/s ± 732.989op/s | 466543.816op/s ± 419.553op/s | 466960.632op/s | 467463.068op/s | 467735.715op/s | 468030.630op/s | 0.32% | -2.745 | 19.088 | 0.16% | 51.830op/s | 1 | 200 |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | execution_time | 7.943ms | 7.959ms ± 0.008ms | 7.957ms ± 0.003ms | 7.961ms | 7.970ms | 7.995ms | 8.002ms | 0.56% | 2.272 | 8.499 | 0.10% | 0.001ms | 1 | 200 |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | throughput | 124969.117op/s | 125650.934op/s ± 130.570op/s | 125669.149op/s ± 54.801op/s | 125718.979op/s | 125801.817op/s | 125884.416op/s | 125901.750op/s | 0.19% | -2.255 | 8.412 | 0.10% | 9.233op/s | 1 | 200 |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | execution_time | 180.333µs | 186.434µs ± 54.365µs | 182.566µs ± 0.163µs | 182.739µs | 183.447µs | 183.968µs | 951.404µs | 421.13% | 14.034 | 194.976 | 29.09% | 3.844µs | 1 | 200 |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | throughput | 105107.875op/s | 545465.483op/s ± 31325.569op/s | 547747.723op/s ± 490.303op/s | 548216.262op/s | 549523.814op/s | 551684.889op/s | 554530.598op/s | 1.24% | -13.994 | 194.220 | 5.73% | 2215.052op/s | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | execution_time | [5.358ms; 5.360ms] or [-0.022%; +0.022%] | None | None | None |
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | throughput | [466410.962op/s; 466614.133op/s] or [-0.022%; +0.022%] | None | None | None |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | execution_time | [7.957ms; 7.960ms] or [-0.014%; +0.014%] | None | None | None |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | throughput | [125632.838op/s; 125669.030op/s] or [-0.014%; +0.014%] | None | None | None |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | execution_time | [178.900µs; 193.969µs] or [-4.041%; +4.041%] | None | None | None |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | throughput | [541124.061op/s; 549806.906op/s] or [-0.796%; +0.796%] | None | None | None |
Group 4
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | a2f0f83 | 1786056259 | dependabot/submodules/libdd-ffe-test-suite/ffe-system-test-data-c017a15 |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sdk_test_data/rules-based | execution_time | 135.575µs | 138.673µs ± 5.759µs | 137.478µs ± 0.628µs | 138.254µs | 143.816µs | 161.154µs | 201.388µs | 46.49% | 7.813 | 73.963 | 4.14% | 0.407µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| sdk_test_data/rules-based | execution_time | [137.875µs; 139.471µs] or [-0.576%; +0.576%] | None | None | None |
Baseline
Baseline benchmark details
Group 1
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 05d785e | 1786044193 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| single_flag_killswitch/rules-based | execution_time | 156.475ns | 160.192ns ± 2.596ns | 159.788ns ± 1.615ns | 161.144ns | 165.433ns | 167.477ns | 169.465ns | 6.06% | 1.061 | 0.970 | 1.62% | 0.184ns | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| single_flag_killswitch/rules-based | execution_time | [159.832ns; 160.552ns] or [-0.225%; +0.225%] | None | None | None |
Group 2
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 05d785e | 1786044193 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | execution_time | 21.383ms | 21.496ms ± 0.047ms | 21.492ms ± 0.034ms | 21.528ms | 21.579ms | 21.607ms | 21.623ms | 0.61% | 0.248 | -0.271 | 0.22% | 0.003ms | 1 | 200 |
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | throughput | 115617.669op/s | 116300.755op/s ± 256.009op/s | 116322.937op/s ± 183.150op/s | 116492.363op/s | 116669.412op/s | 116822.077op/s | 116916.739op/s | 0.51% | -0.237 | -0.278 | 0.22% | 18.103op/s | 1 | 200 |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | execution_time | 96.196ms | 97.086ms ± 0.366ms | 97.075ms ± 0.219ms | 97.289ms | 97.567ms | 97.785ms | 100.169ms | 3.19% | 2.939 | 23.584 | 0.38% | 0.026ms | 1 | 200 |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | throughput | 9983.167op/s | 10300.311op/s ± 38.409op/s | 10301.288op/s ± 23.244op/s | 10326.080op/s | 10345.742op/s | 10369.651op/s | 10395.400op/s | 0.91% | -2.754 | 21.664 | 0.37% | 2.716op/s | 1 | 200 |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | execution_time | 592.806µs | 594.428µs ± 0.733µs | 594.438µs ± 0.477µs | 594.820µs | 595.585µs | 597.117µs | 597.215µs | 0.47% | 0.778 | 1.848 | 0.12% | 0.052µs | 1 | 200 |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | throughput | 167443.803op/s | 168229.297op/s ± 207.296op/s | 168226.202op/s ± 135.111op/s | 168381.886op/s | 168551.220op/s | 168649.239op/s | 168689.167op/s | 0.28% | -0.766 | 1.813 | 0.12% | 14.658op/s | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | execution_time | [21.490ms; 21.503ms] or [-0.031%; +0.031%] | None | None | None |
| flagevaluation_evp/payloads/scale/2500flags_500users_20fields | throughput | [116265.274op/s; 116336.235op/s] or [-0.031%; +0.031%] | None | None | None |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | execution_time | [97.035ms; 97.137ms] or [-0.052%; +0.052%] | None | None | None |
| flagevaluation_evp/payloads/stress/10flags_1000users_250fields | throughput | [10294.988op/s; 10305.634op/s] or [-0.052%; +0.052%] | None | None | None |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | execution_time | [594.326µs; 594.529µs] or [-0.017%; +0.017%] | None | None | None |
| flagevaluation_evp/payloads/typical/100flags_50users_10fields | throughput | [168200.568op/s; 168258.026op/s] or [-0.017%; +0.017%] | None | None | None |
Group 3
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 05d785e | 1786044193 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | execution_time | 5.395ms | 5.418ms ± 0.012ms | 5.415ms ± 0.006ms | 5.424ms | 5.440ms | 5.454ms | 5.463ms | 0.88% | 1.045 | 0.827 | 0.22% | 0.001ms | 1 | 200 |
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | throughput | 457632.037op/s | 461406.552op/s ± 1017.877op/s | 461651.883op/s ± 483.270op/s | 462114.891op/s | 462593.330op/s | 463024.416op/s | 463404.549op/s | 0.38% | -1.033 | 0.795 | 0.22% | 71.975op/s | 1 | 200 |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | execution_time | 7.937ms | 7.949ms ± 0.007ms | 7.949ms ± 0.003ms | 7.952ms | 7.956ms | 7.978ms | 7.994ms | 0.57% | 3.182 | 17.160 | 0.08% | 0.000ms | 1 | 200 |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | throughput | 125098.745op/s | 125800.838op/s ± 106.392op/s | 125809.585op/s ± 50.100op/s | 125860.974op/s | 125919.163op/s | 125975.673op/s | 125993.824op/s | 0.15% | -3.159 | 16.986 | 0.08% | 7.523op/s | 1 | 200 |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | execution_time | 171.751µs | 182.703µs ± 1.701µs | 183.052µs ± 0.162µs | 183.249µs | 183.479µs | 183.684µs | 183.963µs | 0.50% | -5.065 | 27.036 | 0.93% | 0.120µs | 1 | 200 |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | throughput | 543588.636op/s | 547385.890op/s ± 5351.165op/s | 546292.222op/s ± 482.644op/s | 546698.100op/s | 554015.695op/s | 581039.533op/s | 582239.088op/s | 6.58% | 5.161 | 27.968 | 0.98% | 378.385op/s | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | execution_time | [5.417ms; 5.420ms] or [-0.031%; +0.031%] | None | None | None |
| flagevaluation_evp/coalescer/scale/2500flags_500users_20fields | throughput | [461265.484op/s; 461547.620op/s] or [-0.031%; +0.031%] | None | None | None |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | execution_time | [7.948ms; 7.950ms] or [-0.012%; +0.012%] | None | None | None |
| flagevaluation_evp/coalescer/stress/10flags_1000users_250fields | throughput | [125786.093op/s; 125815.583op/s] or [-0.012%; +0.012%] | None | None | None |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | execution_time | [182.467µs; 182.939µs] or [-0.129%; +0.129%] | None | None | None |
| flagevaluation_evp/coalescer/typical/100flags_50users_10fields | throughput | [546644.270op/s; 548127.510op/s] or [-0.135%; +0.135%] | None | None | None |
Group 4
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 05d785e | 1786044193 | main |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sdk_test_data/rules-based | execution_time | 125.506µs | 128.366µs ± 5.493µs | 127.371µs ± 0.685µs | 128.145µs | 133.358µs | 147.202µs | 188.917µs | 48.32% | 8.008 | 77.411 | 4.27% | 0.388µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| sdk_test_data/rules-based | execution_time | [127.604µs; 129.127µs] or [-0.593%; +0.593%] | None | None | None |
📚 Documentation Check Results✅ No documentation warnings found! 📦
|
|
🔒 Cargo Deny Results📦
|
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
Bumps libdd-ffe-test-suite/ffe-system-test-data from
d9d8020toc017a15.Commits
c017a15Merge pull request #20 from greghuels/greg.huels/semver-comparison-fixtures7f3112cRefine SEMVER comparison boundaries1d7c325Add SEMVER comparison operator fixturesf3da9aeMerge pull request #18 from DataDog/leo.romanovsky/revert-regex-shard-bound-f...44092cefix: limit rollback to regex fixtures9864c01fix: restore static temporal allocation reasonscb09e08revert: remove regex and shard-bound fixturesa596df7Merge pull request #17 from DataDog/leo.romanovsky/regex-shard-bound-fixturesce45493Add regex and shard-bound fixtures6c7f63bMerge pull request #16 from DataDog/leo.romanovsky/malformed-split-shards-fix...Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)