From c6a4b27c3f61b54f5f94f3c6ad83746657f91c19 Mon Sep 17 00:00:00 2001
From: Tom Scanlan <tompscanlan@gmail.com>
Date: Sat, 5 Sep 2026 13:27:24 -0400
Subject: [PATCH] fix(grafana): stop the tenant stat panels double-counting
 every total

The four "Total" stat panels on the metrics dashboard were showing exactly
twice the real number. Measured against live prod Prometheus:

  Total Users        4996  -> 2498
  Total Groups        312  ->  156
  Active Users (30d)  418  ->  209
  Total Events      30824  -> 15412

MetricsService exports each gauge once per tenant AND once more with
tenant="all" as a pre-aggregated total (metrics.service.ts sets the per-tenant
label in the collect loop and then sets tenant="all" from the accumulated
totals). So the `tenant` label space is {all, <tenant1>, <tenant2>, ...} and

    sum(max by (tenant) (users_total{...}))

adds the "all" rollup to the per-tenant series it is already the sum of --
double-counting by construction, and it would have kept reading exactly 2x no
matter how many tenants existed.

Excluding the rollup from the sum fixes it. Each corrected expression now
equals max(...{tenant="all"}) exactly, verified for all four metrics against
prod. Keeping the sum-over-tenants form rather than switching to
max(...{tenant="all"}) preserves the $tenant template variable: selecting a
single tenant still shows that tenant's number, which a bare "all" lookup would
break.

openmeet-performance.json was already correct -- it uses
max(...{tenant="all"}) -- which is why the two dashboards disagreed by 2x on
the same metric.

Unrelated to the cron duplication fixed in b4212d2. That changed how OFTEN the
rollup ran, not the values, and these panels have read double since the
tenant="all" series was introduced.

NOTE: these dashboards are not provisioned -- no ConfigMap in the monitoring
namespace contains these expressions, so Grafana holds a manually imported
copy. This commit fixes the file in git; the running dashboard keeps showing
2x until someone re-imports it.
---
 grafana/openmeet-metrics-dashboard.json | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/grafana/openmeet-metrics-dashboard.json b/grafana/openmeet-metrics-dashboard.json
index a6585522..28d82152 100644
--- a/grafana/openmeet-metrics-dashboard.json
+++ b/grafana/openmeet-metrics-dashboard.json
@@ -1189,7 +1189,7 @@
             "uid": "${DS_PROMETHEUS}"
           },
           "editorMode": "code",
-          "expr": "sum(max by (tenant) (users_total{namespace=~\"$namespace\", tenant=~\"$tenant\"}))",
+          "expr": "sum(max by (tenant) (users_total{namespace=~\"$namespace\", tenant=~\"$tenant\", tenant!=\"all\"}))",
           "legendFormat": "__auto",
           "range": true,
           "refId": "A"
@@ -1249,7 +1249,7 @@
             "uid": "${DS_PROMETHEUS}"
           },
           "editorMode": "code",
-          "expr": "sum(max by (tenant) (groups_total{namespace=~\"$namespace\", tenant=~\"$tenant\"}))",
+          "expr": "sum(max by (tenant) (groups_total{namespace=~\"$namespace\", tenant=~\"$tenant\", tenant!=\"all\"}))",
           "legendFormat": "__auto",
           "range": true,
           "refId": "A"
@@ -1501,7 +1501,7 @@
             "uid": "${DS_PROMETHEUS}"
           },
           "editorMode": "code",
-          "expr": "sum(max by (tenant) (active_users_30d{namespace=~\"$namespace\", tenant=~\"$tenant\"}))",
+          "expr": "sum(max by (tenant) (active_users_30d{namespace=~\"$namespace\", tenant=~\"$tenant\", tenant!=\"all\"}))",
           "legendFormat": "__auto",
           "range": true,
           "refId": "A"
@@ -1561,7 +1561,7 @@
             "uid": "${DS_PROMETHEUS}"
           },
           "editorMode": "code",
-          "expr": "sum(max by (tenant) (events_total{namespace=~\"$namespace\", tenant=~\"$tenant\"}))",
+          "expr": "sum(max by (tenant) (events_total{namespace=~\"$namespace\", tenant=~\"$tenant\", tenant!=\"all\"}))",
           "legendFormat": "__auto",
           "range": true,
           "refId": "A"