Skip to content

fix(grafana): stop the tenant stat panels double-counting every total - #606

Merged
tompscanlan merged 1 commit into
mainfrom
fix/dashboard-tenant-double-count
Sep 5, 2026
Merged

tompscanlan merged 1 commit into
mainfrom
fix/dashboard-tenant-double-count

Conversation

@tompscanlan

Copy link
Copy Markdown
Contributor

fix(grafana): stop the tenant stat panels double-counting every total

The four "Total" stat panels on the metrics dashboard were showing exactly
twice the real number. Measured against live prod Prometheus:

Total Users 4996 -> 2498
Total Groups 312 -> 156
Active Users (30d) 418 -> 209
Total Events 30824 -> 15412

MetricsService exports each gauge once per tenant AND once more with
tenant="all" as a pre-aggregated total (metrics.service.ts sets the per-tenant
label in the collect loop and then sets tenant="all" from the accumulated
totals). So the tenant label space is {all, , , ...} and

sum(max by (tenant) (users_total{...}))

adds the "all" rollup to the per-tenant series it is already the sum of --
double-counting by construction, and it would have kept reading exactly 2x no
matter how many tenants existed.

Excluding the rollup from the sum fixes it. Each corrected expression now
equals max(...{tenant="all"}) exactly, verified for all four metrics against
prod. Keeping the sum-over-tenants form rather than switching to
max(...{tenant="all"}) preserves the $tenant template variable: selecting a
single tenant still shows that tenant's number, which a bare "all" lookup would
break.

openmeet-performance.json was already correct -- it uses
max(...{tenant="all"}) -- which is why the two dashboards disagreed by 2x on
the same metric.

Unrelated to the cron duplication fixed in b4212d2. That changed how OFTEN the
rollup ran, not the values, and these panels have read double since the
tenant="all" series was introduced.

NOTE: these dashboards are not provisioned -- no ConfigMap in the monitoring
namespace contains these expressions, so Grafana holds a manually imported
copy. This commit fixes the file in git; the running dashboard keeps showing
2x until someone re-imports it.

Verification

All four corrected expressions checked against live prod Prometheus and each now equals max(...{tenant="all"}) exactly:

panel before after truth
Total Users 4996 2498 2498
Total Groups 312 156 156
Active Users (30d) 418 209 209
Total Events 30824 15412 15412

jq -e . confirms the file is still valid JSON. Diff is 4 lines.

Follow-up needed (not code)

These dashboards are not provisioned — no ConfigMap in monitoring contains these expressions, so the running Grafana holds a manually imported copy. It will keep showing 2x until the dashboard is re-imported.

The four "Total" stat panels on the metrics dashboard were showing exactly
twice the real number. Measured against live prod Prometheus:

  Total Users        4996  -> 2498
  Total Groups        312  ->  156
  Active Users (30d)  418  ->  209
  Total Events      30824  -> 15412

MetricsService exports each gauge once per tenant AND once more with
tenant="all" as a pre-aggregated total (metrics.service.ts sets the per-tenant
label in the collect loop and then sets tenant="all" from the accumulated
totals). So the `tenant` label space is {all, <tenant1>, <tenant2>, ...} and

    sum(max by (tenant) (users_total{...}))

adds the "all" rollup to the per-tenant series it is already the sum of --
double-counting by construction, and it would have kept reading exactly 2x no
matter how many tenants existed.

Excluding the rollup from the sum fixes it. Each corrected expression now
equals max(...{tenant="all"}) exactly, verified for all four metrics against
prod. Keeping the sum-over-tenants form rather than switching to
max(...{tenant="all"}) preserves the $tenant template variable: selecting a
single tenant still shows that tenant's number, which a bare "all" lookup would
break.

openmeet-performance.json was already correct -- it uses
max(...{tenant="all"}) -- which is why the two dashboards disagreed by 2x on
the same metric.

Unrelated to the cron duplication fixed in b4212d2. That changed how OFTEN the
rollup ran, not the values, and these panels have read double since the
tenant="all" series was introduced.

NOTE: these dashboards are not provisioned -- no ConfigMap in the monitoring
namespace contains these expressions, so Grafana holds a manually imported
copy. This commit fixes the file in git; the running dashboard keeps showing
2x until someone re-imports it.
@tompscanlan
tompscanlan merged commit 8fd54d9 into main Sep 5, 2026
4 checks passed
@tompscanlan
tompscanlan deleted the fix/dashboard-tenant-double-count branch September 5, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant