Repository navigation
fix(grafana): stop the tenant stat panels double-counting every total - #606
Merged
Merged
Conversation
The four "Total" stat panels on the metrics dashboard were showing exactly
twice the real number. Measured against live prod Prometheus:
Total Users 4996 -> 2498
Total Groups 312 -> 156
Active Users (30d) 418 -> 209
Total Events 30824 -> 15412
MetricsService exports each gauge once per tenant AND once more with
tenant="all" as a pre-aggregated total (metrics.service.ts sets the per-tenant
label in the collect loop and then sets tenant="all" from the accumulated
totals). So the `tenant` label space is {all, <tenant1>, <tenant2>, ...} and
sum(max by (tenant) (users_total{...}))
adds the "all" rollup to the per-tenant series it is already the sum of --
double-counting by construction, and it would have kept reading exactly 2x no
matter how many tenants existed.
Excluding the rollup from the sum fixes it. Each corrected expression now
equals max(...{tenant="all"}) exactly, verified for all four metrics against
prod. Keeping the sum-over-tenants form rather than switching to
max(...{tenant="all"}) preserves the $tenant template variable: selecting a
single tenant still shows that tenant's number, which a bare "all" lookup would
break.
openmeet-performance.json was already correct -- it uses
max(...{tenant="all"}) -- which is why the two dashboards disagreed by 2x on
the same metric.
Unrelated to the cron duplication fixed in b4212d2. That changed how OFTEN the
rollup ran, not the values, and these panels have read double since the
tenant="all" series was introduced.
NOTE: these dashboards are not provisioned -- no ConfigMap in the monitoring
namespace contains these expressions, so Grafana holds a manually imported
copy. This commit fixes the file in git; the running dashboard keeps showing
2x until someone re-imports it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(grafana): stop the tenant stat panels double-counting every total
The four "Total" stat panels on the metrics dashboard were showing exactly
twice the real number. Measured against live prod Prometheus:
Total Users 4996 -> 2498
Total Groups 312 -> 156
Active Users (30d) 418 -> 209
Total Events 30824 -> 15412
MetricsService exports each gauge once per tenant AND once more with
tenant="all" as a pre-aggregated total (metrics.service.ts sets the per-tenant
label in the collect loop and then sets tenant="all" from the accumulated
totals). So the
tenantlabel space is {all, , , ...} andadds the "all" rollup to the per-tenant series it is already the sum of --
double-counting by construction, and it would have kept reading exactly 2x no
matter how many tenants existed.
Excluding the rollup from the sum fixes it. Each corrected expression now
equals max(...{tenant="all"}) exactly, verified for all four metrics against
prod. Keeping the sum-over-tenants form rather than switching to
max(...{tenant="all"}) preserves the $tenant template variable: selecting a
single tenant still shows that tenant's number, which a bare "all" lookup would
break.
openmeet-performance.json was already correct -- it uses
max(...{tenant="all"}) -- which is why the two dashboards disagreed by 2x on
the same metric.
Unrelated to the cron duplication fixed in b4212d2. That changed how OFTEN the
rollup ran, not the values, and these panels have read double since the
tenant="all" series was introduced.
NOTE: these dashboards are not provisioned -- no ConfigMap in the monitoring
namespace contains these expressions, so Grafana holds a manually imported
copy. This commit fixes the file in git; the running dashboard keeps showing
2x until someone re-imports it.
Verification
All four corrected expressions checked against live prod Prometheus and each now equals
max(...{tenant="all"})exactly:jq -e .confirms the file is still valid JSON. Diff is 4 lines.Follow-up needed (not code)
These dashboards are not provisioned — no ConfigMap in
monitoringcontains these expressions, so the running Grafana holds a manually imported copy. It will keep showing 2x until the dashboard is re-imported.