From 8a2516d7996b444585d2183c95549871230a5b3c Mon Sep 17 00:00:00 2001 From: CChislett Date: Tue, 20 Oct 2026 13:14:11 +0100 Subject: [PATCH 01/10] Add the Altinity ClickHouse and MongoDB integration pages Document two of the new Data-category integrations, each metrics-via- Grafana-Alloy: - Altinity ClickHouse - operator exporter + Keeper metrics in-cluster. - MongoDB - built-in Alloy exporter (clusterMonitor user) plus log collection, with the password/API-key file handling and the keep/drop relabel notes. Both reproduce the install configs verbatim and note the Dashboard data link. Added to the All integrations nav (Data). Co-Authored-By: Claude Opus 4.8 --- .../Integrations/Data/altinity-clickhouse.md | 191 +++++++++++++++ .../Features/Integrations/Data/mongodb.md | 222 ++++++++++++++++++ mkdocs.yml | 2 + 3 files changed, 415 insertions(+) create mode 100644 docs/Data-insights/Features/Integrations/Data/altinity-clickhouse.md create mode 100644 docs/Data-insights/Features/Integrations/Data/mongodb.md diff --git a/docs/Data-insights/Features/Integrations/Data/altinity-clickhouse.md b/docs/Data-insights/Features/Integrations/Data/altinity-clickhouse.md new file mode 100644 index 0000000..82e0636 --- /dev/null +++ b/docs/Data-insights/Features/Integrations/Data/altinity-clickhouse.md @@ -0,0 +1,191 @@ +# Altinity ClickHouse + +Monitor ClickHouse clusters managed by the Altinity Kubernetes operator - queries, inserts and merges, replication, tables, and Keeper. + +Your ClickHouse installations show up alongside the rest of your telemetry, on dashboards that are already built. You can see how each cluster is doing - query, insert, and merge rates, replication, memory and disk, every table's size and parts, and ClickHouse Keeper - and get alerted when a host stops reporting, replicas go read-only or fall behind, inserts are delayed or refused, a mutation never finishes, or Keeper loses its leader. + +!!! info "What this integration does" + It provisions **dashboards** and **alert rules**. It does not collect anything itself - the Altinity operator's metrics exporter already measures every ClickHouse host it runs, and **Grafana Alloy**, running in the same Kubernetes cluster, collects from that exporter and from ClickHouse Keeper and sends the metrics to OpsPilot. Until Alloy is collecting, the dashboards are empty. + +Navigate to **Integrations** from the left-hand sidebar, then select **Altinity ClickHouse**. + +--- + +## Permission tiers + +Altinity ClickHouse runs **Read-only**, the default. It requires permission to install **Grafana Alloy** in the Kubernetes cluster, and an **OpsPilot API key**. + +Its **capabilities** are **Dashboards** and **Alerts**. + +For background, the [Altinity ClickHouse operator](https://github.com/Altinity/clickhouse-operator) documents how the exporter works, and the [Grafana Alloy Helm chart](https://github.com/grafana/helm-charts) covers running Alloy another way - the config below works as it is. + +--- + +## 1. Turn on Keeper's metrics + +The exporter covers ClickHouse, not **ClickHouse Keeper**, which serves its own metrics once told to. Skip this if your installations use ZooKeeper, or no coordination at all. + +In each `ClickHouseKeeperInstallation`, add the settings and name the port in its pod template `metrics`: + +```yaml +spec: + configuration: + settings: + prometheus/endpoint: "/metrics" + prometheus/port: "7000" + prometheus/metrics: "true" + prometheus/events: "true" + prometheus/asynchronous_metrics: "true" + templates: + podTemplates: + - name: + spec: + containers: + - name: clickhouse-keeper + ports: + - name: metrics + containerPort: 7000 +``` + +The operator restarts the Keeper nodes one at a time to apply it. + +--- + +## 2. Install Alloy + +Create a secret with your OpsPilot API key - from **Data and Licenses > [API Keys](https://app.opspilot.com/organization/api-keys)** - in the namespace Alloy will run in: + +```bash +kubectl create namespace monitoring +kubectl -n monitoring create secret generic opspilot --from-literal=api-key='' +``` + +Save this as `clickhouse.alloy`: + +```river +discovery.kubernetes "clickhouse" { + role = "pod" +} + +// The Altinity operator's metrics exporter: one target per operator, which +// queries every ClickHouse host the operator manages on each scrape. +discovery.relabel "clickhouse_exporter" { + targets = discovery.kubernetes.clickhouse.targets + + rule { + source_labels = ["__meta_kubernetes_pod_label_app", "__meta_kubernetes_pod_container_port_name"] + regex = "clickhouse-operator;ch-metrics" + action = "keep" + } + rule { + source_labels = ["__meta_kubernetes_namespace"] + target_label = "operator_namespace" + } + rule { + source_labels = ["__meta_kubernetes_pod_name"] + target_label = "instance" + } +} + +// ClickHouse Keeper's own Prometheus endpoint, on the container port named +// metrics. +discovery.relabel "clickhouse_keeper" { + targets = discovery.kubernetes.clickhouse.targets + + rule { + source_labels = ["__meta_kubernetes_pod_label_clickhouse_keeper_altinity_com_chk", "__meta_kubernetes_pod_container_port_name"] + regex = ".+;metrics" + action = "keep" + } + rule { + source_labels = ["__meta_kubernetes_pod_label_clickhouse_keeper_altinity_com_chk"] + target_label = "chk" + } + rule { + source_labels = ["__meta_kubernetes_namespace"] + target_label = "namespace" + } + rule { + source_labels = ["__meta_kubernetes_pod_name"] + target_label = "instance" + } +} + +prometheus.scrape "clickhouse" { + targets = array.concat(discovery.relabel.clickhouse_exporter.output, discovery.relabel.clickhouse_keeper.output) + job_name = "integrations/altinity-clickhouse" + scrape_interval = "30s" + scrape_timeout = "20s" + forward_to = [prometheus.relabel.clickhouse.receiver] +} + +prometheus.relabel "clickhouse" { + forward_to = [prometheus.remote_write.opspilot.receiver] + + rule { + source_labels = ["__name__"] + regex = "up|chi_clickhouse_(table_.+|metric_(fetch_errors|Uptime|Query|LongestRunningQuery|MaxPartCountForPartition|PartsActive|Merge|BackgroundMergesAndMutationsPoolTask|ReadonlyReplica|ReplicasMaxAbsoluteDelay|ReplicasSumQueueSize|ZooKeeperSession|MemoryTracking|CGroupMemoryTotal|DiskFreeBytes|DiskTotalBytes|TCPConnection|HTTPConnection|DetachedParts)|event_(Query|SelectQuery|InsertQuery|FailedQuery|QueryTimeMicroseconds|InsertedRows|InsertedBytes|DelayedInserts|RejectedInserts|MergedRows|ZooKeeperHardwareExceptions))|ClickHouse(AsyncMetrics_Keeper.+|Metrics_Keeper(AliveConnections|OutstandingRequests)|ProfileEvents_Keeper(Commits|CommitsFailed))" + action = "keep" + } + rule { + source_labels = ["hostname"] + regex = "([^.]+)\\..*" + target_label = "hostname" + } +} + +prometheus.remote_write "opspilot" { + endpoint { + url = "https://api.opspilot.com/v1/metrics" + + headers = { + "authorization" = sys.env("OPSPILOT_API_KEY"), + } + } +} +``` + +and install Alloy with it: + +```bash +helm install alloy alloy --repo https://grafana.github.io/helm-charts -n monitoring \ + --set-file alloy.configMap.content=clickhouse.alloy \ + --set 'alloy.extraEnv[0].name=OPSPILOT_API_KEY' \ + --set 'alloy.extraEnv[0].valueFrom.secretKeyRef.name=opspilot' \ + --set 'alloy.extraEnv[0].valueFrom.secretKeyRef.key=api-key' +``` + +A few things worth understanding: + +- **One target covers every installation.** The exporter queries all the hosts the operator manages each time Alloy scrapes it, waiting up to nine seconds for slow ones - hence the 20-second timeout. Scrape it from **one Alloy only**: every scrape is a round of queries against every host. +- **The keep rule sends only what the dashboards read.** The exporter reports about a thousand series per host, and Keeper its whole metric set; the rule keeps about 35 per host plus five for each table holding data - ClickHouse's own system log tables, twenty or so, included. +- **`namespace` is the installation's.** The exporter labels each series with its `ClickHouseInstallation` (`chi`) and that installation's namespace, so the operator pod's own namespace goes in `operator_namespace`. Host names are shortened to the pod's name. + +Already running Alloy in the cluster - the Kubernetes Monitoring chart, for instance? Add the blocks above to its config instead, with `forward_to` naming its existing `prometheus.remote_write`. + +--- + +## 3. Confirm the data arrived + +In OpsPilot, open **Explore**, select the **Metrics** data source, and run: + +```promql +chi_clickhouse_metric_Uptime{job="integrations/altinity-clickhouse"} +``` + +You should get one series per ClickHouse host. For Keeper, run `ClickHouseAsyncMetrics_KeeperIsLeader{job="integrations/altinity-clickhouse"}`. + +Once data is flowing, the **Dashboard data** column on **Your installation** (for example *3 of 3 with data*) links straight to the provisioned dashboards. + +--- + +## Troubleshooting + +**Nothing comes back.** Open the Alloy UI (`kubectl -n monitoring port-forward ds/alloy 12345`) - it shows whether Alloy found the operator's `ch-metrics` port. + +**A host is missing.** A host listed in `chi_clickhouse_metric_fetch_errors{fetch_type="system_metrics"} == 1` is one the exporter cannot query. + +--- + +!!! question "Need more help?" + Contact support in the chat bubble and let us know how we can assist. diff --git a/docs/Data-insights/Features/Integrations/Data/mongodb.md b/docs/Data-insights/Features/Integrations/Data/mongodb.md new file mode 100644 index 0000000..3b62742 --- /dev/null +++ b/docs/Data-insights/Features/Integrations/Data/mongodb.md @@ -0,0 +1,222 @@ +# MongoDB + +Monitor MongoDB - operations, latency, replication, sharding, the WiredTiger cache, and the log. + +Your MongoDB servers show up alongside the rest of your telemetry, on dashboards that are already built. You can see how hard each server is working and where it is struggling - latency, connections, queues, and the WiredTiger cache - across your replica sets and sharded clusters, with the servers' logs, and get alerted when a server stops reporting, a member goes down, a secondary falls behind, or operations start to queue. + +!!! info "What this integration does" + It provisions **dashboards** and **alert rules**. It does not collect anything itself - **Grafana Alloy** has a MongoDB exporter built in, so collecting metrics needs a user on the server and a few lines of Alloy config, and the log needs a few more. This covers MongoDB 7.0 and later - standalone servers, replica sets, and sharded clusters. Until Alloy is collecting, the dashboards are empty. + +Navigate to **Integrations** from the left-hand sidebar, then select **MongoDB**. + +--- + +## Permission tiers + +MongoDB runs **Read-only**, the default. It requires a MongoDB user with the **`clusterMonitor`** role, read access to the MongoDB log for Alloy, and an **OpsPilot API key**. + +Its **capabilities** are **Dashboards** and **Alerts**. + +Grafana's own docs cover the exporter in full: see [`prometheus.exporter.mongodb`](https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.exporter.mongodb/) for every option, and [install Alloy](https://grafana.com/docs/alloy/latest/set-up/install/) on each MongoDB host if you want the log (Alloy reads it from disk), or anywhere that can reach each server's port for metrics alone. + +--- + +## 1. Create a monitoring user + +Alloy logs in to read the server's status. Give it a user of its own with MongoDB's built-in read-only **`clusterMonitor`** role, rather than an administrator or an application's user. In `mongosh`: + +```javascript +db.getSiblingDB("admin").createUser({ + user: "opspilot_exporter", + pwd: passwordPrompt(), + roles: [{ role: "clusterMonitor", db: "admin" }], +}) +``` + +Run it once on each replica set's primary; it reaches the other members by replication. In a sharded cluster, run it through a `mongos`, and once more on each shard's primary - a connection straight to a shard member checks the shard's own users. + +--- + +## 2. Configure Alloy + +Alloy's config file is in a different place on each system: + +| OS | Config file | After a change | +|---|---|---| +| **Linux** | `/etc/alloy/config.alloy` | `sudo systemctl restart alloy` | +| **Windows** | `C:\Program Files\GrafanaLabs\Alloy\config.alloy` | restart the Alloy service in `services.msc` | +| **macOS** | `$(brew --prefix)/etc/alloy/config.alloy` | `brew services restart grafana/grafana/alloy` | + +Keep the password in a file of its own beside it, readable only by Alloy. **URL-encode it**, because it goes inside a `mongodb://` URI - a raw `@` fails the login, and a raw `:`, `#`, `/`, `?`, or `%` stops the exporter starting. + +On **Linux**, this asks for the password and writes it encoded: + +```bash +python3 -c 'import getpass, urllib.parse; print(urllib.parse.quote(getpass.getpass(), safe=""), end="")' | sudo tee /etc/alloy/mongodb-password >/dev/null +sudo chown root:alloy /etc/alloy/mongodb-password +sudo chmod 640 /etc/alloy/mongodb-password +``` + +On **Windows**, in an administrator PowerShell: + +```powershell +Set-Content -NoNewline -Path "$env:ProgramFiles\GrafanaLabs\Alloy\mongodb-password" -Value ([uri]::EscapeDataString((Read-Host "Password"))) +``` + +On **macOS**, the first line alone, writing to `$(brew --prefix)/etc/alloy/mongodb-password` without `sudo`. + +!!! warning + The file must hold the encoded password and nothing else: a trailing newline, which most editors add, makes the login fail. + +Then add to the config file, with one exporter per `mongod` and `mongos`: + +```river +local.file "mongodb_password" { + // Windows: "C:/Program Files/GrafanaLabs/Alloy/mongodb-password" + filename = "/etc/alloy/mongodb-password" + is_secret = true +} + +prometheus.exporter.mongodb "db" { + mongodb_uri = "mongodb://opspilot_exporter:" + local.file.mongodb_password.content + "@dbhost:27017/admin" + direct_connect = true + collect_all = false + enable_diagnostic_data = true + enable_db_stats = true +} + +prometheus.scrape "db" { + targets = prometheus.exporter.mongodb.db.targets + scrape_interval = "30s" + forward_to = [prometheus.relabel.mongodb.receiver] +} + +prometheus.relabel "mongodb" { + forward_to = [prometheus.remote_write.opspilot.receiver] + + rule { + source_labels = ["__name__"] + regex = "mongodb_(ss|sys|oplog_stats|dbstats|config|local|admin)_.*" + action = "drop" + } +} +``` + +A few things matter here: + +- **`collect_all = false`.** Left at its default, the exporter also collects statistics for every collection and index, with the collection's name in the metric's name - thousands of series per server, and more with every collection you add. The two collectors enabled here are what the dashboards read. +- **The relabel rule drops raw duplicates.** The diagnostic data arrives twice: as the named metrics the dashboards read, and as some 5,500 raw series per server that nothing here uses. Dropping them leaves 200 to 350 per server. +- **`direct_connect = true` for each server.** Without it, pointed at one replica set member, the exporter finds the primary and reports the primary's numbers under that member's name. +- **Leave `job` alone.** Alloy labels these metrics `job="integrations/mongodb"` whatever you call the blocks, and the dashboards and alerts select on that, so add no `job_name` and no relabel rule for `job`. Each server appears as the `host:port` in its URI. +- **One exporter per server.** For each further `mongod` or `mongos`, add another exporter and scrape with different names, all forwarding to the same relabel block. + +--- + +## 3. Send data to OpsPilot + +You need an OpsPilot API key, from **Data and Licenses > [API Keys](https://app.opspilot.com/organization/api-keys)**. Put it in a file the same way, as `opspilot-api-key` beside the config file, and add: + +```river +local.file "opspilot_api_key" { + // Windows: "C:/Program Files/GrafanaLabs/Alloy/opspilot-api-key" + filename = "/etc/alloy/opspilot-api-key" +} + +prometheus.remote_write "opspilot" { + endpoint { + url = "https://api.opspilot.com/v1/metrics" + + headers = { + "authorization" = string.trim_space(local.file.opspilot_api_key.content), + } + } +} +``` + +!!! note "Using environment variables instead" + Environment variables work in place of files if you already manage Alloy's settings that way: write `sys.env("MONGODB_EXPORTER_PASSWORD")` in place of `local.file.mongodb_password.content`, and likewise for the key. Alloy reads its environment from `/etc/default/alloy` (Debian, Ubuntu), `/etc/sysconfig/alloy` (RHEL, SUSE), `$(brew --prefix)/etc/alloy/config.env` (macOS), or on Windows the `Environment` value of the registry key `HKEY_LOCAL_MACHINE\SOFTWARE\GrafanaLabs\Alloy`. + +--- + +## 4. Collect the log + +This needs Alloy on the same machine as the server, since it reads the file. The log's path is `systemLog.path` in the server's config file (`/etc/mongod.conf` on Linux, `mongod.cfg` beside `mongod.exe` on Windows); the packages use `/var/log/mongodb/mongod.log`. Use it here, with the same `instance` the metrics have - `dbhost:27017` - so the dashboards line up: + +```river +loki.source.file "mongodb_log" { + targets = [{ + // Windows: "C:/Program Files/MongoDB/Server/8.0/log/mongod.log" + __path__ = "/var/log/mongodb/mongod.log", + job = "integrations/mongodb", + instance = "dbhost:27017", + }] + forward_to = [loki.process.mongodb_log.receiver] +} + +loki.process "mongodb_log" { + stage.json { + expressions = { ts = "t.\"$date\"", s = "s", component = "c", id = "id" } + } + stage.drop { + source = "id" + expression = "^(22943|22944|51800|5286306|6788604|6788700|10483900)$" + } + stage.timestamp { + source = "ts" + format = "RFC3339" + } + stage.drop { + older_than = "24h" + } + stage.template { + source = "level" + template = "{{ $s := .s }}{{ if eq $s \"F\" }}fatal{{ else if eq $s \"E\" }}error{{ else if eq $s \"W\" }}warning{{ else if eq $s \"I\" }}info{{ else }}debug{{ end }}" + } + stage.labels { + values = { level = "", component = "" } + } + forward_to = [loki.write.opspilot.receiver] +} + +loki.write "opspilot" { + endpoint { + url = "https://api.opspilot.com/logs/v1/" + + headers = { + "authorization" = string.trim_space(local.file.opspilot_api_key.content), + } + } +} +``` + +Keep the stages as they are. MongoDB writes its log as JSON; the stages take each entry's time from it, turn its severity and component into the labels the log dashboard filters on, and drop connection churn - every client connecting, authenticating, and leaving, the exporter's own scrapes included - which is most of the log by volume. Failed logins are kept. The last 24 hours are sent when Alloy first starts, and nothing older. + +On **Linux**, let Alloy read the file. MongoDB creates its log readable by its own user only. Add this to `/etc/mongod.conf` and restart MongoDB, so the log is readable by its group: + +```yaml +setParameter: + processUmask: "027" +``` + +Then add Alloy to that group and restart Alloy: `sudo usermod -aG mongodb alloy` on Debian and Ubuntu, `sudo usermod -aG mongod alloy` on RHEL and its relatives. On **Windows**, Alloy runs as LocalSystem and can read the log as it is; write the path with forward slashes, as in the comment. + +--- + +## 5. Confirm the data arrived + +In OpsPilot, open **Explore**, select the **Metrics** data source, and run: + +```promql +mongodb_up +``` + +You should get one series per server, reading `1`. `0` means Alloy cannot log in, and no result at all that it cannot reach the server - either way the Alloy log for `prometheus.exporter.mongodb` says why. + +For the log, select the **Logs** data source and run `{job="integrations/mongodb"}`. If nothing comes back, the Alloy log for `loki.source.file` says whether it could open the file. + +Once data is flowing, the **Dashboard data** column on **Your installation** (for example *4 of 4 with data*) links straight to the provisioned dashboards. + +--- + +!!! question "Need more help?" + Contact support in the chat bubble and let us know how we can assist. diff --git a/mkdocs.yml b/mkdocs.yml index 90f4c8f..13e98a1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -90,6 +90,8 @@ nav: - Cloud: - AWS: Data-insights/Features/Integrations/Cloud/aws.md - Data: + - Altinity ClickHouse: Data-insights/Features/Integrations/Data/altinity-clickhouse.md + - MongoDB: Data-insights/Features/Integrations/Data/mongodb.md - OpenTrace: Data-insights/Features/Integrations/Data/opentrace.md - PostgreSQL: Data-insights/Features/Integrations/Data/postgres.md - Infrastructure: From 5dfb17fc5660a75eb1a3a5b5065b7e365556791b Mon Sep 17 00:00:00 2001 From: CChislett Date: Tue, 20 Oct 2026 13:27:55 +0100 Subject: [PATCH 02/10] Add the TigerData (TimescaleDB) integration page Document the TigerData integration (Data category): Grafana Alloy runs TimescaleDB information-view queries via the Postgres exporter with a custom-queries file, relabeled to its own job. Covers the no-privilege monitoring user, the queries YAML, the Alloy config with allowlist/ relabel notes, sending to OpsPilot, and confirming. Cross-links the PostgreSQL integration for the server itself. Co-Authored-By: Claude Opus 4.8 --- .../Features/Integrations/Data/tigerdata.md | 208 ++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 209 insertions(+) create mode 100644 docs/Data-insights/Features/Integrations/Data/tigerdata.md diff --git a/docs/Data-insights/Features/Integrations/Data/tigerdata.md b/docs/Data-insights/Features/Integrations/Data/tigerdata.md new file mode 100644 index 0000000..0504414 --- /dev/null +++ b/docs/Data-insights/Features/Integrations/Data/tigerdata.md @@ -0,0 +1,208 @@ +# TigerData + +Monitor TigerData (TimescaleDB) hypertables, the columnstore, and background jobs and policies. + +Your TimescaleDB databases show up alongside the rest of your telemetry, on dashboards that are already built. You can see how your hypertables are growing and how much the columnstore saves, and keep watch on the background jobs that compress, retain, and aggregate your data - with alerts when one fails, crashes, or falls behind, or when a server stops reporting. Works for self-hosted TimescaleDB and Tiger Cloud. + +!!! info "What this integration does" + It provisions **dashboards** and **alert rules**. It does not collect anything itself - **Grafana Alloy** gathers what TimescaleDB knows about its hypertables, columnstore, and background jobs by running a few queries against each database where TimescaleDB is installed. This works for self-hosted TimescaleDB 2.17 and later and for Tiger Cloud services. For the PostgreSQL server itself - connections, cache, locks - install the [PostgreSQL integration](postgres.md) as well. Until Alloy is collecting, the dashboards are empty. + +Navigate to **Integrations** from the left-hand sidebar, then select **TigerData**. + +--- + +## Permission tiers + +TigerData runs **Read-only**, the default. It requires a database user Alloy can log in as - it needs no privileges - and an **OpsPilot API key**. + +Its **capabilities** are **Dashboards** and **Alerts**. + +Grafana's own docs cover the exporter in full: see [`prometheus.exporter.postgres`](https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.exporter.postgres/) for every option, and [install Alloy](https://grafana.com/docs/alloy/latest/set-up/install/) anywhere that can reach the database. + +--- + +## 1. Create a monitoring user + +Connect as an administrator - on Tiger Cloud, `tsdbadmin` - and create a user for Alloy: + +```sql +CREATE USER opspilot_monitor WITH PASSWORD ''; +``` + +It needs no other privilege: TimescaleDB's information views and size functions are readable by any user who can connect, and show sizes and job status, never data. + +--- + +## 2. Save the queries + +Save this as `timescaledb-queries.yaml` beside Alloy's config file: + +```yaml +timescaledb_extension: + query: | + SELECT current_database() AS datname, extversion AS version, 1 AS info + FROM pg_extension WHERE extname = 'timescaledb' + metrics: + - datname: {usage: LABEL, description: "Database"} + - version: {usage: LABEL, description: "TimescaleDB version"} + - info: {usage: GAUGE, description: "TimescaleDB is installed in this database"} + +timescaledb_hypertable: + cache_seconds: 300 + query: | + SELECT current_database() AS datname, h.hypertable_schema AS schema, h.hypertable_name AS hypertable, + h.num_chunks AS chunks, + h.compression_enabled::int AS columnstore_enabled, + hypertable_size(format('%I.%I', h.hypertable_schema, h.hypertable_name)::regclass) AS size_bytes, + coalesce(cs.number_compressed_chunks, 0) AS columnstore_chunks, + coalesce(cs.before_compression_total_bytes, 0) AS columnstore_before_bytes, + coalesce(cs.after_compression_total_bytes, 0) AS columnstore_after_bytes + FROM timescaledb_information.hypertables h + LEFT JOIN LATERAL hypertable_compression_stats(format('%I.%I', h.hypertable_schema, h.hypertable_name)::regclass) cs ON true + metrics: + - datname: {usage: LABEL, description: "Database"} + - schema: {usage: LABEL, description: "Hypertable schema"} + - hypertable: {usage: LABEL, description: "Hypertable"} + - chunks: {usage: GAUGE, description: "Chunks"} + - columnstore_enabled: {usage: GAUGE, description: "1 if the columnstore (compression) is enabled"} + - size_bytes: {usage: GAUGE, description: "Total size, tables, indexes and TOAST"} + - columnstore_chunks: {usage: GAUGE, description: "Chunks in the columnstore (compressed)"} + - columnstore_before_bytes: {usage: GAUGE, description: "Size of the columnstore chunks before conversion"} + - columnstore_after_bytes: {usage: GAUGE, description: "Size of the columnstore chunks after conversion"} + +timescaledb_job: + query: | + SELECT current_database() AS datname, j.job_id::text AS job_id, j.proc_name AS proc, + coalesce(ca.view_name, j.hypertable_name, '') AS relation, + j.scheduled::int AS scheduled, + extract(epoch FROM j.schedule_interval) AS schedule_interval_seconds, + coalesce((s.last_run_status = 'Failed')::int, 0) AS last_run_failed, + coalesce((s.job_status = 'Running')::int, 0) AS running, + s.total_runs AS runs_total, s.total_failures AS failures_total, + coalesce(st.consecutive_failures, 0) AS consecutive_failures, + coalesce(st.total_crashes, 0) AS crashes_total, + coalesce(st.consecutive_crashes, 0) AS consecutive_crashes, + CASE WHEN s.last_successful_finish > '-infinity' THEN extract(epoch FROM s.last_successful_finish) END AS last_success_timestamp_seconds + FROM timescaledb_information.jobs j + JOIN timescaledb_information.job_stats s USING (job_id) + LEFT JOIN _timescaledb_internal.bgw_job_stat st USING (job_id) + LEFT JOIN timescaledb_information.continuous_aggregates ca + ON ca.materialization_hypertable_schema = j.hypertable_schema AND ca.materialization_hypertable_name = j.hypertable_name + metrics: + - datname: {usage: LABEL, description: "Database"} + - job_id: {usage: LABEL, description: "Job id"} + - proc: {usage: LABEL, description: "The job's procedure: a policy, or a user action"} + - relation: {usage: LABEL, description: "The hypertable or continuous aggregate the job works on"} + - scheduled: {usage: GAUGE, description: "1 if the job is scheduled, 0 if paused"} + - schedule_interval_seconds: {usage: GAUGE, description: "How often the job runs"} + - last_run_failed: {usage: GAUGE, description: "1 if the job's last run failed"} + - running: {usage: GAUGE, description: "1 while the job is running"} + - runs_total: {usage: COUNTER, description: "Runs"} + - failures_total: {usage: COUNTER, description: "Failed runs"} + - consecutive_failures: {usage: GAUGE, description: "Failed runs since the last success"} + - crashes_total: {usage: COUNTER, description: "Runs whose background worker crashed"} + - consecutive_crashes: {usage: GAUGE, description: "Crashed runs since the last success"} + - last_success_timestamp_seconds: {usage: GAUGE, description: "When the job last finished successfully"} +``` + +Sizes are re-read every five minutes rather than every scrape, since they walk every chunk. Alloy reads this file only when it starts. + +--- + +## 3. Configure Alloy + +Alloy's config file is in a different place on each system: + +| OS | Config file | After a change | +|---|---|---| +| **Linux** | `/etc/alloy/config.alloy` | `sudo systemctl restart alloy` | +| **Windows** | `C:\Program Files\GrafanaLabs\Alloy\config.alloy` | restart the Alloy service in `services.msc` | +| **macOS** | `$(brew --prefix)/etc/alloy/config.alloy` | `brew services restart grafana/grafana/alloy` | + +Put the monitoring user's password in a file beside the config, readable only by Alloy, written without a trailing newline, and add: + +```river +local.file "timescaledb_password" { + filename = "/etc/alloy/timescaledb-password" + is_secret = true +} + +prometheus.exporter.postgres "timescaledb" { + data_source_names = ["postgresql://opspilot_monitor:" + local.file.timescaledb_password.content + "@db.example.com:5432/tsdb?sslmode=require"] + disable_default_metrics = true + disable_settings_metrics = true + custom_queries_config_path = "/etc/alloy/timescaledb-queries.yaml" + + autodiscovery { + enabled = true + database_allowlist = ["tsdb"] + } +} + +discovery.relabel "timescaledb" { + targets = prometheus.exporter.postgres.timescaledb.targets + + rule { + target_label = "job" + replacement = "integrations/timescaledb" + } +} + +prometheus.scrape "timescaledb" { + targets = discovery.relabel.timescaledb.output + scrape_interval = "60s" + forward_to = [prometheus.remote_write.opspilot.receiver] +} +``` + +A few things worth understanding: + +- **List every database that has TimescaleDB in `database_allowlist`, and connect to one of them.** The queries run in each; in a database without TimescaleDB they fail on every scrape. On Tiger Cloud the database is `tsdb` and the host and port are the service's, from the Console. +- **URL-encode the password** if it contains `@`, `:`, `/`, `?`, `#`, or `%` - it is part of a URL. +- **Keep the relabel rule.** Alloy gives every PostgreSQL exporter the same `job`, so without it these series would land beside the PostgreSQL integration's, and the dashboards select on the `job` it sets. +- **For more than one server,** repeat the exporter for each under a name of its own (letters, digits, and underscores), and list their targets together: `targets = array.concat(prometheus.exporter.postgres.one.targets, prometheus.exporter.postgres.two.targets)`. + +--- + +## 4. Send data to OpsPilot + +You need an OpsPilot API key, from **Data and Licenses > [API Keys](https://app.opspilot.com/organization/api-keys)**. Put it in a file the same way, as `opspilot-api-key` beside the config file, and add: + +```river +local.file "opspilot_api_key" { + // Windows: "C:/Program Files/GrafanaLabs/Alloy/opspilot-api-key" + filename = "/etc/alloy/opspilot-api-key" +} + +prometheus.remote_write "opspilot" { + endpoint { + url = "https://api.opspilot.com/v1/metrics" + + headers = { + "authorization" = string.trim_space(local.file.opspilot_api_key.content), + } + } +} +``` + +!!! note "Using environment variables instead" + Environment variables work in place of files if you already manage Alloy's settings that way: write `sys.env("TIMESCALEDB_PASSWORD")` in place of `local.file.timescaledb_password.content`, and `sys.env("OPSPILOT_API_KEY")` for the key. Alloy reads its environment from `/etc/default/alloy` (Debian, Ubuntu), `/etc/sysconfig/alloy` (RHEL, SUSE), `$(brew --prefix)/etc/alloy/config.env` (macOS), or on Windows the `Environment` value of the registry key `HKEY_LOCAL_MACHINE\SOFTWARE\GrafanaLabs\Alloy`. + +--- + +## 5. Confirm the data arrived + +In OpsPilot, open **Explore**, select the **Metrics** data source, and run: + +```promql +timescaledb_extension_info +``` + +You should get one series per database, carrying its TimescaleDB version. If nothing comes back, run `pg_up{job="integrations/timescaledb"}`: `0` means Alloy cannot connect or log in, and the Alloy log for `prometheus.exporter.postgres` says why; no result at all means Alloy is not sending. + +Once data is flowing, the **Dashboard data** column on **Your installation** links straight to the provisioned dashboards. + +--- + +!!! question "Need more help?" + Contact support in the chat bubble and let us know how we can assist. diff --git a/mkdocs.yml b/mkdocs.yml index 13e98a1..c522270 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -94,6 +94,7 @@ nav: - MongoDB: Data-insights/Features/Integrations/Data/mongodb.md - OpenTrace: Data-insights/Features/Integrations/Data/opentrace.md - PostgreSQL: Data-insights/Features/Integrations/Data/postgres.md + - TigerData: Data-insights/Features/Integrations/Data/tigerdata.md - Infrastructure: - Docker: Data-insights/Features/Integrations/Infrastructure/docker.md - iDRAC: Data-insights/Features/Integrations/Infrastructure/idrac.md From d838a81cad71387152d1d5bb8c60ae97388a2435 Mon Sep 17 00:00:00 2001 From: CChislett Date: Tue, 20 Oct 2026 13:30:33 +0100 Subject: [PATCH 03/10] Add the Microsoft SQL Server integration page Document the Microsoft SQL Server integration (Data category): built-in Alloy mssql exporter (login with VIEW SERVER STATE + VIEW ANY DEFINITION) plus error-log collection. Covers the password URL-encoding, the Alloy config, the GODEBUG x509negativeserial certificate workaround, the log stages, and confirming with mssql_local_time_seconds. Co-Authored-By: Claude Opus 4.8 --- .../Features/Integrations/Data/mssql.md | 219 ++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 220 insertions(+) create mode 100644 docs/Data-insights/Features/Integrations/Data/mssql.md diff --git a/docs/Data-insights/Features/Integrations/Data/mssql.md b/docs/Data-insights/Features/Integrations/Data/mssql.md new file mode 100644 index 0000000..8118bd1 --- /dev/null +++ b/docs/Data-insights/Features/Integrations/Data/mssql.md @@ -0,0 +1,219 @@ +# Microsoft SQL Server + +Monitor Microsoft SQL Server - workload, deadlocks, I/O stall, memory, and the error log. + +Your SQL Server instances show up alongside the rest of your telemetry, on dashboards that are already built. You can see how hard each instance is working and where it is struggling - deadlocks, severe errors, I/O waits, transaction log growth, and memory - alongside their error logs, with alerts when one stops reporting, comes under pressure, or is refusing logins. + +!!! info "What this integration does" + It provisions **dashboards** and **alert rules**. It does not collect anything itself - **Grafana Alloy** has a SQL Server exporter built in, so collecting metrics needs a login on the server and a few lines of Alloy config, and the error log needs a few more. Until Alloy is collecting, the dashboards are empty. + +Navigate to **Integrations** from the left-hand sidebar, then select **Microsoft SQL Server**. + +--- + +## Permission tiers + +Microsoft SQL Server runs **Read-only**, the default. It requires a SQL Server login with **`VIEW SERVER STATE`** and **`VIEW ANY DEFINITION`**, read access to its error log for Alloy, and an **OpsPilot API key**. + +Its **capabilities** are **Dashboards** and **Alerts**. + +Grafana's own docs cover the exporter in full: see [`prometheus.exporter.mssql`](https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.exporter.mssql/) for every option, including Windows authentication, and [install Alloy](https://grafana.com/docs/alloy/latest/set-up/install/) on the SQL Server host if you want the error log (Alloy reads it from disk), or anywhere that can reach SQL Server's port for metrics alone. + +--- + +## 1. Create a monitoring login + +Alloy logs in to SQL Server to read its statistics. Give it a login of its own that can read those statistics and nothing else, rather than `sa` or an application's login. On each instance: + +```sql +CREATE LOGIN opspilot_exporter WITH PASSWORD = ''; +GRANT VIEW SERVER STATE TO opspilot_exporter; +GRANT VIEW ANY DEFINITION TO opspilot_exporter; +``` + +Both grants are read-only: they show server state and object definitions, never table data. Leave out `VIEW ANY DEFINITION` and the I/O stall panels are empty, with no error anywhere to say why. + +--- + +## 2. Configure Alloy + +Alloy's config file is in a different place on each system: + +| OS | Config file | After a change | +|---|---|---| +| **Linux** | `/etc/alloy/config.alloy` | `sudo systemctl restart alloy` | +| **Windows** | `C:\Program Files\GrafanaLabs\Alloy\config.alloy` | restart the Alloy service in `services.msc` | +| **macOS** | `$(brew --prefix)/etc/alloy/config.alloy` | `brew services restart grafana/grafana/alloy` | + +Keep the password in a file of its own beside it, readable only by Alloy. **URL-encode it first**, because it goes inside a URL: write `#` as `%23`, `/` as `%2F`, `?` as `%3F`, and `%` as `%25`. A raw `#`, `/`, or `?` stops Alloy loading its config at all, not just this block. On **Linux**: + +```bash +read -rs PW && printf '%s' "$PW" | sudo tee /etc/alloy/mssql-password >/dev/null; unset PW +sudo chown root:alloy /etc/alloy/mssql-password +sudo chmod 640 /etc/alloy/mssql-password +``` + +On **Windows**, in an administrator PowerShell: + +```powershell +Set-Content -NoNewline -Path "$env:ProgramFiles\GrafanaLabs\Alloy\mssql-password" -Value (Read-Host "Password") +``` + +On **macOS**, the first line alone, writing to `$(brew --prefix)/etc/alloy/mssql-password` without `sudo`. + +!!! warning + The file must hold the encoded password and nothing else: a trailing newline, which most editors add, makes the login fail. + +Then add to the config file: + +```river +local.file "mssql_password" { + // Windows: "C:/Program Files/GrafanaLabs/Alloy/mssql-password" + filename = "/etc/alloy/mssql-password" + is_secret = true +} + +prometheus.exporter.mssql "db" { + connection_string = "sqlserver://opspilot_exporter:" + local.file.mssql_password.content + "@sqlhost:1433?encrypt=true&trustservercertificate=true" +} + +prometheus.scrape "db" { + targets = prometheus.exporter.mssql.db.targets + scrape_interval = "30s" + forward_to = [prometheus.remote_write.opspilot.receiver] +} +``` + +- **Leave `job` alone.** Alloy labels these metrics `job="integrations/mssql"` whatever you call the block, and the dashboard and alerts select on that, so add no `job_name` and no relabel rule for `job`. +- **One exporter per instance.** For a second server, add a second pair of blocks with different names. Each instance appears on the dashboard as the `host:port` from its connection string. + +--- + +## 3. Send data to OpsPilot + +You need an OpsPilot API key, from **Data and Licenses > [API Keys](https://app.opspilot.com/organization/api-keys)**. Put it in a file the same way, as `opspilot-api-key` beside the config file, and add: + +```river +local.file "opspilot_api_key" { + // Windows: "C:/Program Files/GrafanaLabs/Alloy/opspilot-api-key" + filename = "/etc/alloy/opspilot-api-key" +} + +prometheus.remote_write "opspilot" { + endpoint { + url = "https://api.opspilot.com/v1/metrics" + + headers = { + "authorization" = string.trim_space(local.file.opspilot_api_key.content), + } + } +} +``` + +Alloy also needs one environment variable, **`GODEBUG=x509negativeserial=1`** (explained in [The self-signed certificate](#the-self-signed-certificate) below) - set it now rather than after the first failure. It goes in `/etc/default/alloy` (Debian, Ubuntu), `/etc/sysconfig/alloy` (RHEL, SUSE), `$(brew --prefix)/etc/alloy/config.env` (macOS), or on Windows the `Environment` value of the registry key `HKEY_LOCAL_MACHINE\SOFTWARE\GrafanaLabs\Alloy`, then restart Alloy. + +!!! note "Using environment variables instead" + Environment variables work in place of files, too, if you already manage Alloy's settings that way: write `sys.env("MSSQL_EXPORTER_PASSWORD")` in place of `local.file.mssql_password.content`, and likewise for the key, and set them in the same place as `GODEBUG`. + +--- + +## 4. Collect the error log + +This needs Alloy on the same machine as SQL Server, since it reads the file. Ask SQL Server where its log is: + +```sql +SELECT SERVERPROPERTY('ErrorLogFileName'); +``` + +and use that path here, with the same `instance` the metrics have - the `host:port` from the connection string - so the two dashboards line up: + +```river +loki.source.file "mssql_errorlog" { + targets = [{ + // Windows, SQL Server 2022 default instance: + // "C:/Program Files/Microsoft SQL Server/MSSQL16.MSSQLSERVER/MSSQL/Log/ERRORLOG" + __path__ = "/var/opt/mssql/log/errorlog", + job = "integrations/mssql", + instance = "sqlhost:1433", + }] + forward_to = [loki.process.mssql_errorlog.receiver] +} + +loki.process "mssql_errorlog" { + stage.multiline { + firstline = "^\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{2}" + max_wait_time = "3s" + } + stage.regex { + expression = "^(?P\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{2}) +(?P\\S+)" + } + stage.regex { + source = "source" + expression = "^(?Pspid)\\d+s?$" + } + stage.timestamp { + source = "ts" + format = "2006-01-02 15:04:05.00" + location = "Local" + } + stage.drop { + older_than = "24h" + } + stage.labels { + values = { source = "" } + } + forward_to = [loki.write.opspilot.receiver] +} + +loki.write "opspilot" { + endpoint { + url = "https://api.opspilot.com/logs/v1/" + + headers = { + "authorization" = string.trim_space(local.file.opspilot_api_key.content), + } + } +} +``` + +On **Linux**, let Alloy read the file. It is readable only by SQL Server's group, so `sudo usermod -aG mssql alloy` and restart Alloy. + +On **Windows**, write the path with forward slashes, as in the comment, or double every backslash. `MSSQL16.MSSQLSERVER` is the version and instance name, so a named instance or another version has its own folder, and an in-place upgrade moves the log to a new one: update the path after upgrading. The file is UTF-16; Alloy reads that as it is, with no encoding setting. + +Keep the stages as they are. They join multi-line entries, take each entry's time from the log itself, and turn `spid51`, `spid62s`, and the rest into one `source` label, which is what the logs dashboard filters on. The last 24 hours are sent when Alloy first starts, and nothing older. + +--- + +## The self-signed certificate + +With no certificate configured, SQL Server encrypts connections with one it generates itself. SQL Server 2017 and 2022 generate a new one every time they start, and about half of those carry a negative serial number, which Alloy refuses. So collection can work today and stop after the next SQL Server restart, with nothing failing visibly - the Alloy log just repeats: + +```text +TLS Handshake failed: tls: failed to parse certificate from server: x509: negative serial number +``` + +The fix is to give SQL Server a proper certificate. Until then, the **`GODEBUG=x509negativeserial=1`** set above makes Alloy accept the certificate and keeps the connection encrypted. + +!!! warning + Don't reach for `encrypt=disable` instead. It also makes the error go away, but it does so by sending the login, password included, **unencrypted**. + +--- + +## Confirm the data arrived + +In OpsPilot, open **Explore**, select the **Metrics** data source, and run: + +```promql +mssql_local_time_seconds +``` + +You should get one series per instance. Don't check `up` instead: it reads `1` whenever Alloy reaches the exporter, even when the exporter cannot log in to SQL Server and is sending nothing else. If this query comes back empty, the Alloy log for `prometheus.exporter.mssql` says why - the login, the permissions, and the certificate above are the usual three. + +For the error log, select the **Logs** data source and run `{job="integrations/mssql"}`. If nothing comes back, the Alloy log for `loki.source.file` says whether it could open the file. + +Once data is flowing, the **Dashboard data** column on **Your installation** links straight to the provisioned dashboards. + +--- + +!!! question "Need more help?" + Contact support in the chat bubble and let us know how we can assist. diff --git a/mkdocs.yml b/mkdocs.yml index c522270..4f2a6f0 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -91,6 +91,7 @@ nav: - AWS: Data-insights/Features/Integrations/Cloud/aws.md - Data: - Altinity ClickHouse: Data-insights/Features/Integrations/Data/altinity-clickhouse.md + - Microsoft SQL Server: Data-insights/Features/Integrations/Data/mssql.md - MongoDB: Data-insights/Features/Integrations/Data/mongodb.md - OpenTrace: Data-insights/Features/Integrations/Data/opentrace.md - PostgreSQL: Data-insights/Features/Integrations/Data/postgres.md From 55f408a264b8b33be31175b750518a2b01b84321 Mon Sep 17 00:00:00 2001 From: CChislett Date: Tue, 20 Oct 2026 13:32:33 +0100 Subject: [PATCH 04/10] Add the MySQL integration page Document the MySQL/MariaDB integration (Data category): built-in Alloy mysql exporter (PROCESS + REPLICATION CLIENT user, SLAVE MONITOR on MariaDB) plus error-log collection handling both log formats. Covers the exporter config with the instance relabel and deadlock collector, and the per-distro log locations incl. the MariaDB-on-Debian journal fix. Co-Authored-By: Claude Opus 4.8 --- .../Features/Integrations/Data/mysql.md | 242 ++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 243 insertions(+) create mode 100644 docs/Data-insights/Features/Integrations/Data/mysql.md diff --git a/docs/Data-insights/Features/Integrations/Data/mysql.md b/docs/Data-insights/Features/Integrations/Data/mysql.md new file mode 100644 index 0000000..4cf836b --- /dev/null +++ b/docs/Data-insights/Features/Integrations/Data/mysql.md @@ -0,0 +1,242 @@ +# MySQL + +Monitor MySQL and MariaDB - queries, connections, replication, InnoDB, and the error log. + +Your MySQL and MariaDB servers show up alongside the rest of your telemetry, on dashboards that are already built. You can see how hard each server is working and where it is struggling - connections, slow queries, deadlocks, InnoDB, and replication - alongside their error logs, with alerts when one stops reporting, a replica stops or falls behind, or a server runs short of connections. + +!!! info "What this integration does" + It provisions **dashboards** and **alert rules**. It does not collect anything itself - **Grafana Alloy** has a MySQL exporter built in, so collecting metrics needs a user on the server and a few lines of Alloy config, and the error log needs a few more. This covers MySQL 8.0 and later and MariaDB 10.11 and later. Until Alloy is collecting, the dashboards are empty. + +Navigate to **Integrations** from the left-hand sidebar, then select **MySQL**. + +--- + +## Permission tiers + +MySQL runs **Read-only**, the default. It requires a MySQL or MariaDB user that can read server status (**`PROCESS`** and **`REPLICATION CLIENT`**), read access to the error log for Alloy, and an **OpsPilot API key**. + +Its **capabilities** are **Dashboards** and **Alerts**. + +Grafana's own docs cover the exporter in full: see [`prometheus.exporter.mysql`](https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.exporter.mysql/) for every option, and [install Alloy](https://grafana.com/docs/alloy/latest/set-up/install/) on the database host if you want the error log (Alloy reads it from disk), or anywhere that can reach the server's port for metrics alone. + +--- + +## 1. Create a monitoring user + +Alloy logs in to read the server's status. Give it a user of its own that can read that status and nothing else, rather than `root` or an application's user. On each server: + +```sql +CREATE USER 'opspilot_exporter'@'%' IDENTIFIED BY '' WITH MAX_USER_CONNECTIONS 3; +GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'opspilot_exporter'@'%'; +``` + +On MariaDB, also: + +```sql +GRANT SLAVE MONITOR ON *.* TO 'opspilot_exporter'@'%'; +``` + +Neither grant reads table data. `mysqld_exporter`'s required grants also include `SELECT ON *.*`, which lets the user read every table; the collectors below do not need it, so leave it out. Replace `%` with the address Alloy connects from to allow only that host. `MAX_USER_CONNECTIONS 3` stops monitoring adding to the load on a server that is already struggling. The password cannot contain `?`; any other character works as it is. + +--- + +## 2. Configure Alloy + +Alloy's config file is in a different place on each system: + +| OS | Config file | After a change | +|---|---|---| +| **Linux** | `/etc/alloy/config.alloy` | `sudo systemctl restart alloy` | +| **Windows** | `C:\Program Files\GrafanaLabs\Alloy\config.alloy` | restart the Alloy service in `services.msc` | +| **macOS** | `$(brew --prefix)/etc/alloy/config.alloy` | `brew services restart grafana/grafana/alloy` | + +Keep the password in a file of its own beside it, readable only by Alloy. On **Linux**: + +```bash +read -rs PW && printf '%s' "$PW" | sudo tee /etc/alloy/mysql-password >/dev/null; unset PW +sudo chown root:alloy /etc/alloy/mysql-password +sudo chmod 640 /etc/alloy/mysql-password +``` + +On **Windows**, in an administrator PowerShell: + +```powershell +Set-Content -NoNewline -Path "$env:ProgramFiles\GrafanaLabs\Alloy\mysql-password" -Value (Read-Host "Password") +``` + +On **macOS**, the first line alone, writing to `$(brew --prefix)/etc/alloy/mysql-password` without `sudo`. + +!!! warning + The file must hold the password and nothing else: a trailing newline, which most editors add, makes the login fail with `Access denied`. + +Then add to the config file: + +```river +local.file "mysql_password" { + // Windows: "C:/Program Files/GrafanaLabs/Alloy/mysql-password" + filename = "/etc/alloy/mysql-password" + is_secret = true +} + +prometheus.exporter.mysql "db" { + data_source_name = "opspilot_exporter:" + local.file.mysql_password.content + "@tcp(dbhost:3306)/" + enable_collectors = ["info_schema.innodb_metrics"] +} + +discovery.relabel "db" { + targets = prometheus.exporter.mysql.db.targets + + rule { + source_labels = ["instance"] + regex = "[^(]*\\((.*)\\)/.*" + target_label = "instance" + } +} + +prometheus.scrape "db" { + targets = discovery.relabel.db.output + scrape_interval = "30s" + forward_to = [prometheus.remote_write.opspilot.receiver] +} +``` + +- **`enable_collectors`** is what counts deadlocks on MySQL. MariaDB reports them without it, so leave the line out there. +- **The relabel rule names each server `dbhost:3306`.** Without it the dashboards show `tcp(dbhost:3306)/`. +- **Leave `job` alone.** Alloy labels these metrics `job="integrations/mysql"` whatever you call the blocks, and the dashboards and alerts select on that, so add no `job_name` and no relabel rule for `job`. +- **One exporter per server.** For a second server, add a second set of blocks with different names. + +--- + +## 3. Send data to OpsPilot + +You need an OpsPilot API key, from **Data and Licenses > [API Keys](https://app.opspilot.com/organization/api-keys)**. Put it in a file the same way, as `opspilot-api-key` beside the config file, and add: + +```river +local.file "opspilot_api_key" { + // Windows: "C:/Program Files/GrafanaLabs/Alloy/opspilot-api-key" + filename = "/etc/alloy/opspilot-api-key" +} + +prometheus.remote_write "opspilot" { + endpoint { + url = "https://api.opspilot.com/v1/metrics" + + headers = { + "authorization" = string.trim_space(local.file.opspilot_api_key.content), + } + } +} +``` + +!!! note "Using environment variables instead" + Environment variables work in place of files if you already manage Alloy's settings that way: write `sys.env("MYSQL_EXPORTER_PASSWORD")` in place of `local.file.mysql_password.content`, and likewise for the key. Alloy reads its environment from `/etc/default/alloy` (Debian, Ubuntu), `/etc/sysconfig/alloy` (RHEL, SUSE), `$(brew --prefix)/etc/alloy/config.env` (macOS), or on Windows the `Environment` value of the registry key `HKEY_LOCAL_MACHINE\SOFTWARE\GrafanaLabs\Alloy`. + +--- + +## 4. Collect the error log + +This needs Alloy on the same machine as the server, since it reads the file. Ask the server where its log is: + +```sql +SELECT @@log_error, @@datadir; +``` + +A relative path is inside the data directory. Use it here, with the same `instance` the metrics have - `dbhost:3306` - so the dashboards line up: + +```river +loki.source.file "mysql_errorlog" { + targets = [{ + // Windows, MySQL: "C:/ProgramData/MySQL/MySQL Server 8.4/Data/.err" + // Windows, MariaDB: "C:/Program Files/MariaDB 11.8/data/.err" + __path__ = "/var/log/mysql/error.log", + job = "integrations/mysql", + instance = "dbhost:3306", + }] + forward_to = [loki.process.mysql_errorlog.receiver] +} + +loki.process "mysql_errorlog" { + stage.multiline { + firstline = "^\\d{4}-\\d{2}-\\d{2}[T ]\\d{2}:\\d{2}:\\d{2}" + max_wait_time = "3s" + } + stage.regex { + expression = "^(?P\\d{4}-\\d{2}-\\d{2}T\\S+) +\\d+ \\[(?P[A-Za-z]+)\\]" + } + stage.regex { + expression = "^(?P\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}) +\\d+ \\[(?P[A-Za-z]+)\\]" + } + stage.timestamp { + source = "ts_mysql" + format = "RFC3339Nano" + } + stage.timestamp { + source = "ts_mariadb" + format = "2006-01-02 15:04:05" + location = "Local" + } + stage.drop { + older_than = "24h" + } + stage.template { + source = "level" + template = "{{ ToLower .Value }}" + } + stage.labels { + values = { level = "" } + } + forward_to = [loki.write.opspilot.receiver] +} + +loki.write "opspilot" { + endpoint { + url = "https://api.opspilot.com/logs/v1/" + + headers = { + "authorization" = string.trim_space(local.file.opspilot_api_key.content), + } + } +} +``` + +Keep the stages as they are. They handle both MySQL's and MariaDB's formats, join multi-line entries, take each entry's time from the log itself, and turn the level into the label the dashboard filters on. The last 24 hours are sent when Alloy first starts, and nothing older. + +On **Windows**, write the path with forward slashes, as in the comment, or double every backslash. Alloy runs as LocalSystem there and can read the file as it is. + +On **Linux**, let Alloy read the file, then restart Alloy: + +| Server | Log | Run | +|---|---|---| +| MySQL, Debian or Ubuntu package | `/var/log/mysql/error.log` | `sudo usermod -aG adm alloy` | +| MySQL, RHEL family | `/var/log/mysql/mysqld.log` | `sudo usermod -aG mysql alloy` | +| MariaDB, RHEL family | `/var/log/mariadb/mariadb.log` | `sudo usermod -aG mysql alloy` | + +MariaDB on Debian and Ubuntu writes no log file - `@@log_error` comes back empty and the log goes to the system journal. Turn the file on: + +```bash +sudo install -d -o mysql -g mysql -m 750 /var/log/mysql +sudo sed -i 's|^#log_error = /var/log/mysql/error.log|log_error = /var/log/mysql/error.log|' /etc/mysql/mariadb.conf.d/50-server.cnf +sudo systemctl restart mariadb +sudo usermod -aG mysql alloy +``` + +--- + +## 5. Confirm the data arrived + +In OpsPilot, open **Explore**, select the **Metrics** data source, and run: + +```promql +mysql_up +``` + +You should get one series per server, reading `1`. `0` means Alloy cannot log in, and the Alloy log for `prometheus.exporter.mysql` says why - usually the password, or `%` replaced with an address that is not the one Alloy connects from. + +For the error log, select the **Logs** data source and run `{job="integrations/mysql"}`. If nothing comes back, the Alloy log for `loki.source.file` says whether it could open the file. + +Once data is flowing, the **Dashboard data** column on **Your installation** links straight to the provisioned dashboards. + +--- + +!!! question "Need more help?" + Contact support in the chat bubble and let us know how we can assist. diff --git a/mkdocs.yml b/mkdocs.yml index 4f2a6f0..ea3a0e7 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -93,6 +93,7 @@ nav: - Altinity ClickHouse: Data-insights/Features/Integrations/Data/altinity-clickhouse.md - Microsoft SQL Server: Data-insights/Features/Integrations/Data/mssql.md - MongoDB: Data-insights/Features/Integrations/Data/mongodb.md + - MySQL: Data-insights/Features/Integrations/Data/mysql.md - OpenTrace: Data-insights/Features/Integrations/Data/opentrace.md - PostgreSQL: Data-insights/Features/Integrations/Data/postgres.md - TigerData: Data-insights/Features/Integrations/Data/tigerdata.md From f40373ba69aaba88a9721703c96bdb241644e2ee Mon Sep 17 00:00:00 2001 From: CChislett Date: Tue, 20 Oct 2026 13:34:36 +0100 Subject: [PATCH 05/10] Add the Redis integration page Document the Redis/Valkey integration (Data category): built-in Alloy redis exporter with a narrow ACL user (no key access, config_command disabled) plus log collection parsing Redis's level/role format. Covers keeping credentials out of redis_addr and the per-package log locations. Co-Authored-By: Claude Opus 4.8 --- .../Features/Integrations/Data/redis.md | 199 ++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 200 insertions(+) create mode 100644 docs/Data-insights/Features/Integrations/Data/redis.md diff --git a/docs/Data-insights/Features/Integrations/Data/redis.md b/docs/Data-insights/Features/Integrations/Data/redis.md new file mode 100644 index 0000000..69d1c6f --- /dev/null +++ b/docs/Data-insights/Features/Integrations/Data/redis.md @@ -0,0 +1,199 @@ +# Redis + +Monitor Redis and Valkey - commands, latency, memory, clients, replication, persistence, and the log. + +Your Redis and Valkey servers show up alongside the rest of your telemetry, on dashboards that are already built. You can see how hard each server is working and where it is struggling - command latency, hit ratio, memory, clients, and persistence - with a view of replication and the servers' logs, and alerts when one stops reporting, runs out of memory, refuses connections, loses its primary, or stops saving to disk. + +!!! info "What this integration does" + It provisions **dashboards** and **alert rules**. It does not collect anything itself - **Grafana Alloy** has a Redis exporter built in, so collecting metrics needs a user on the server and a few lines of Alloy config, and the log needs a few more. This covers Redis 7.2 and later and Valkey 8 and later. Until Alloy is collecting, the dashboards are empty. + +Navigate to **Integrations** from the left-hand sidebar, then select **Redis**. + +--- + +## Permission tiers + +Redis runs **Read-only**, the default. It requires a Redis user allowed `INFO` and a few read-only diagnostic commands (no access to keys), read access to the Redis log for Alloy, and an **OpsPilot API key**. + +Its **capabilities** are **Dashboards** and **Alerts**. + +Grafana's own docs cover the exporter in full: see [`prometheus.exporter.redis`](https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.exporter.redis/) for every option, and [install Alloy](https://grafana.com/docs/alloy/latest/set-up/install/) on the Redis host if you want the log (Alloy reads it from disk), or anywhere that can reach the server's port for metrics alone. + +--- + +## 1. Create a monitoring user + +Alloy logs in to read the server's status. Give it a user of its own that can read that status and nothing else, rather than the default user or an application's. In `redis-cli` (or `valkey-cli`), on each server - users are not replicated: + +```text +ACL SETUSER opspilot_exporter on > -@all +info +ping +client|setname +slowlog|get +slowlog|len +latency|latest +latency|histogram +``` + +then `ACL SAVE` if the server keeps its users in an ACL file, or `CONFIG REWRITE` if not, so the user survives a restart. + +That user can read no keys. The exporter's own ACL is wider: it adds key reads (`+get`, `+scan`, `+eval`), which only its optional key-checking features use, and `+config|get`, which would let the monitoring user read `requirepass` and `masterauth` - Alloy is set below not to call `CONFIG` at all. That ACL also names a command that only exists from Redis 8.8, so on earlier Redis and on Valkey the whole command is refused. + +--- + +## 2. Configure Alloy + +Alloy's config file is in a different place on each system: + +| OS | Config file | After a change | +|---|---|---| +| **Linux** | `/etc/alloy/config.alloy` | `sudo systemctl restart alloy` | +| **Windows** | `C:\Program Files\GrafanaLabs\Alloy\config.alloy` | restart the Alloy service in `services.msc` | +| **macOS** | `$(brew --prefix)/etc/alloy/config.alloy` | `brew services restart grafana/grafana/alloy` | + +Keep the password in a file of its own beside it, readable only by Alloy. On **Linux**: + +```bash +read -rs PW && printf '%s' "$PW" | sudo tee /etc/alloy/redis-password >/dev/null; unset PW +sudo chown root:alloy /etc/alloy/redis-password +sudo chmod 640 /etc/alloy/redis-password +``` + +On **Windows**, in an administrator PowerShell: + +```powershell +Set-Content -NoNewline -Path "$env:ProgramFiles\GrafanaLabs\Alloy\redis-password" -Value (Read-Host "Password") +``` + +On **macOS**, the first line alone, writing to `$(brew --prefix)/etc/alloy/redis-password` without `sudo`. + +Then add to the config file: + +```river +prometheus.exporter.redis "db" { + redis_addr = "dbhost:6379" + redis_user = "opspilot_exporter" + // Windows: "C:/Program Files/GrafanaLabs/Alloy/redis-password" + redis_password_file = "/etc/alloy/redis-password" + config_command = "-" +} + +prometheus.scrape "db" { + targets = prometheus.exporter.redis.db.targets + scrape_interval = "30s" + forward_to = [prometheus.remote_write.opspilot.receiver] +} +``` + +- **`config_command = "-"` keeps the user narrow.** It stops the exporter running `CONFIG GET *`; the figures that call would give - `maxclients` and `maxmemory` - come from `INFO` as well. +- **Keep the credentials out of `redis_addr`.** Each server appears on the dashboards as its `redis_addr`, so a `redis://user:password@host` address would put the password in every series' labels. Give the address alone, as above. +- **Leave `job` alone.** Alloy labels these metrics `job="integrations/redis"` whatever you call the blocks, and the dashboards and alerts select on that, so add no `job_name` and no relabel rule for `job`. +- **One exporter per server.** For a replica or a second server, add another exporter and scrape with different names. + +--- + +## 3. Send data to OpsPilot + +You need an OpsPilot API key, from **Data and Licenses > [API Keys](https://app.opspilot.com/organization/api-keys)**. Put it in a file the same way, as `opspilot-api-key` beside the config file, and add: + +```river +local.file "opspilot_api_key" { + // Windows: "C:/Program Files/GrafanaLabs/Alloy/opspilot-api-key" + filename = "/etc/alloy/opspilot-api-key" +} + +prometheus.remote_write "opspilot" { + endpoint { + url = "https://api.opspilot.com/v1/metrics" + + headers = { + "authorization" = string.trim_space(local.file.opspilot_api_key.content), + } + } +} +``` + +!!! note "Using environment variables instead" + Environment variables work in place of files if you already manage Alloy's settings that way: write `redis_password = sys.env("REDIS_EXPORTER_PASSWORD")` in place of `redis_password_file`, and `sys.env("OPSPILOT_API_KEY")` for the key. Alloy reads its environment from `/etc/default/alloy` (Debian, Ubuntu), `/etc/sysconfig/alloy` (RHEL, SUSE), `$(brew --prefix)/etc/alloy/config.env` (macOS), or on Windows the `Environment` value of the registry key `HKEY_LOCAL_MACHINE\SOFTWARE\GrafanaLabs\Alloy`. + +--- + +## 4. Collect the log + +This needs Alloy on the same machine as the server, since it reads the file. `CONFIG GET logfile` (as an administrator) gives its path; an empty value means the server logs to standard output - as the official container images do - and `logfile` needs setting in its config first. Use the path here, with the same `instance` the metrics have - `dbhost:6379` - so the dashboards line up: + +```river +loki.source.file "redis_log" { + targets = [{ + __path__ = "/var/log/redis/redis-server.log", + job = "integrations/redis", + instance = "dbhost:6379", + }] + forward_to = [loki.process.redis_log.receiver] +} + +loki.process "redis_log" { + stage.multiline { + firstline = "^\\d+:[XCSM] " + max_wait_time = "3s" + } + stage.regex { + expression = "^\\d+:(?P[XCSM]) (?P\\d{1,2} \\w{3} \\d{4} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}) (?P[.*#-]) " + } + stage.timestamp { + source = "ts" + format = "2 Jan 2006 15:04:05.000" + location = "Local" + } + stage.drop { + older_than = "24h" + } + stage.template { + source = "level" + template = "{{ $l := .lvl }}{{ if eq $l \"#\" }}warning{{ else if eq $l \"*\" }}notice{{ else if eq $l \"-\" }}verbose{{ else if eq $l \".\" }}debug{{ end }}" + } + stage.template { + source = "role" + template = "{{ $r := .role }}{{ if eq $r \"M\" }}primary{{ else if eq $r \"S\" }}replica{{ else if eq $r \"C\" }}child{{ else if eq $r \"X\" }}sentinel{{ end }}" + } + stage.labels { + values = { level = "", role = "" } + } + forward_to = [loki.write.opspilot.receiver] +} + +loki.write "opspilot" { + endpoint { + url = "https://api.opspilot.com/logs/v1/" + + headers = { + "authorization" = string.trim_space(local.file.opspilot_api_key.content), + } + } +} +``` + +Keep the stages as they are. They join the multi-line blocks Redis writes at startup and on a crash, take each entry's time from the log itself, and turn its level character and role character into the labels the log dashboard filters on. The last 24 hours are sent when Alloy first starts, and nothing older. + +Let Alloy read the file, then restart Alloy: + +| Package | Log | Run | +|---|---|---| +| Redis, Debian or Ubuntu | `/var/log/redis/redis-server.log` | `sudo usermod -aG adm alloy` | +| Valkey, Debian or Ubuntu | `/var/log/valkey/valkey-server.log` | `sudo usermod -aG adm alloy` | +| Redis, RHEL family | `/var/log/redis/redis.log` | `sudo usermod -aG redis alloy` | + +--- + +## 5. Confirm the data arrived + +In OpsPilot, open **Explore**, select the **Metrics** data source, and run: + +```promql +redis_up +``` + +You should get one series per server, reading `1`. `0` means Alloy cannot reach the server or log in, and the Alloy log for `prometheus.exporter.redis` says which. + +For the log, select the **Logs** data source and run `{job="integrations/redis"}`. If nothing comes back, the Alloy log for `loki.source.file` says whether it could open the file. + +Once data is flowing, the **Dashboard data** column on **Your installation** links straight to the provisioned dashboards. + +--- + +!!! question "Need more help?" + Contact support in the chat bubble and let us know how we can assist. diff --git a/mkdocs.yml b/mkdocs.yml index ea3a0e7..92d62b2 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -96,6 +96,7 @@ nav: - MySQL: Data-insights/Features/Integrations/Data/mysql.md - OpenTrace: Data-insights/Features/Integrations/Data/opentrace.md - PostgreSQL: Data-insights/Features/Integrations/Data/postgres.md + - Redis: Data-insights/Features/Integrations/Data/redis.md - TigerData: Data-insights/Features/Integrations/Data/tigerdata.md - Infrastructure: - Docker: Data-insights/Features/Integrations/Infrastructure/docker.md From 1b88370bc4d1a826012d7ffab20a5d72557fced7 Mon Sep 17 00:00:00 2001 From: CChislett Date: Tue, 20 Oct 2026 13:43:14 +0100 Subject: [PATCH 06/10] Add the Strimzi Kafka integration page Document the Strimzi Kafka integration (Data category): Grafana Alloy discovers Strimzi's Kafka nodes, Kafka Exporters, and operators in the cluster by pod labels and scrapes their existing metrics endpoints. Covers turning on Strimzi's JMX metrics (ConfigMap + Kafka resource), the Alloy discovery/relabel config, and confirming. Completes the seven new Data integrations. Co-Authored-By: Claude Opus 4.8 --- .../Integrations/Data/strimzi-kafka.md | 198 ++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 199 insertions(+) create mode 100644 docs/Data-insights/Features/Integrations/Data/strimzi-kafka.md diff --git a/docs/Data-insights/Features/Integrations/Data/strimzi-kafka.md b/docs/Data-insights/Features/Integrations/Data/strimzi-kafka.md new file mode 100644 index 0000000..14cc153 --- /dev/null +++ b/docs/Data-insights/Features/Integrations/Data/strimzi-kafka.md @@ -0,0 +1,198 @@ +# Strimzi Kafka + +Monitor Kafka clusters managed by the Strimzi Kubernetes operator - brokers, the controller, topics, consumer lag, and the operators. + +Your Strimzi-managed Kafka clusters show up alongside the rest of your telemetry, on dashboards that are already built. You can see how each cluster is doing - brokers, the controller quorum, replication and throughput, every topic and consumer group, and the operators reconciling them - with alerts when a broker drops out, partitions go offline or under-replicated, a consumer group falls behind, a change stops being applied, or a certificate is about to expire. + +!!! info "What this integration does" + It provisions **dashboards** and **alert rules**. It does not collect anything itself - Strimzi already serves the metrics these dashboards read (each Kafka node's JMX metrics, its Kafka Exporter's topic and consumer metrics, and the operators' own), and **Grafana Alloy**, running in the same Kubernetes cluster, finds those pods and sends their metrics to OpsPilot. Until Alloy is collecting, the dashboards are empty. + +Navigate to **Integrations** from the left-hand sidebar, then select **Strimzi Kafka**. + +--- + +## Permission tiers + +Strimzi Kafka runs **Read-only**, the default. It requires Strimzi's metrics turned on for each Kafka cluster, permission to install **Grafana Alloy** in the Kubernetes cluster, and an **OpsPilot API key**. + +Its **capabilities** are **Dashboards** and **Alerts**. + +For background, see [Strimzi's metrics guide](https://strimzi.io/docs/) for every option, and the [Grafana Alloy Helm chart](https://github.com/grafana/helm-charts) if you run Alloy in the cluster another way - the config below works as it is. + +--- + +## 1. Turn on Strimzi's metrics + +Strimzi publishes the JMX exporter rules its dashboards use as a ConfigMap in its examples. Create it in each namespace holding a Kafka cluster: + +```bash +curl -fsSL https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/1.3.0/examples/metrics/kafka-metrics.yaml \ + | sed -n '/^kind: ConfigMap$/,$p' | kubectl -n apply -f - +``` + +The example file also defines a Kafka cluster; the `sed` keeps only the ConfigMap at its end. + +Then point each `Kafka` resource at it, and turn on the Kafka Exporter: + +```yaml +spec: + kafka: + metricsConfig: + type: jmxPrometheusExporter + valueFrom: + configMapKeyRef: + name: kafka-metrics + key: kafka-metrics-config.yml + kafkaExporter: + topicRegex: ".*" + groupRegex: ".*" +``` + +Strimzi rolls the Kafka nodes one at a time to apply it. The cluster operator serves its metrics without any setting. + +!!! note "Use these rules, not the Strimzi Metrics Reporter" + Strimzi also offers `type: strimziMetricsReporter`, whose metric names differ; these dashboards read the JMX exporter's. + +--- + +## 2. Install Alloy + +Create a secret with your OpsPilot API key - from **Data and Licenses > [API Keys](https://app.opspilot.com/organization/api-keys)** - in the namespace Alloy will run in: + +```bash +kubectl create namespace monitoring +kubectl -n monitoring create secret generic opspilot --from-literal=api-key='' +``` + +Save this as `strimzi.alloy`: + +```river +discovery.kubernetes "strimzi" { + role = "pod" +} + +discovery.relabel "strimzi_kafka" { + targets = discovery.kubernetes.strimzi.targets + + rule { + source_labels = ["__meta_kubernetes_pod_label_strimzi_io_kind", "__meta_kubernetes_pod_container_port_name"] + regex = "Kafka;tcp-prometheus" + action = "keep" + } + rule { + source_labels = ["__meta_kubernetes_pod_label_strimzi_io_cluster"] + target_label = "strimzi_io_cluster" + } + rule { + source_labels = ["__meta_kubernetes_pod_label_strimzi_io_component_type"] + target_label = "strimzi_io_component_type" + } + rule { + source_labels = ["__meta_kubernetes_namespace"] + target_label = "namespace" + } + rule { + source_labels = ["__meta_kubernetes_pod_name"] + target_label = "pod" + } + rule { + source_labels = ["__meta_kubernetes_pod_name"] + target_label = "instance" + } +} + +discovery.relabel "strimzi_operators" { + targets = discovery.kubernetes.strimzi.targets + + rule { + source_labels = ["__meta_kubernetes_pod_label_strimzi_io_kind", "__meta_kubernetes_pod_label_app_kubernetes_io_name", "__meta_kubernetes_pod_container_port_name"] + regex = "cluster-operator;.*;http|.*;entity-operator;healthcheck(-to|-uo)?" + action = "keep" + } + rule { + source_labels = ["__meta_kubernetes_pod_label_strimzi_io_cluster"] + target_label = "strimzi_io_cluster" + } + rule { + source_labels = ["__meta_kubernetes_pod_container_name"] + target_label = "container" + } + rule { + source_labels = ["__meta_kubernetes_namespace"] + target_label = "operator_namespace" + } + rule { + source_labels = ["__meta_kubernetes_pod_name"] + target_label = "pod" + } + rule { + source_labels = ["__meta_kubernetes_pod_name", "__meta_kubernetes_pod_container_name"] + separator = "/" + target_label = "instance" + } +} + +prometheus.scrape "strimzi" { + targets = array.concat(discovery.relabel.strimzi_kafka.output, discovery.relabel.strimzi_operators.output) + job_name = "integrations/strimzi" + scrape_interval = "30s" + forward_to = [prometheus.relabel.strimzi.receiver] +} + +prometheus.relabel "strimzi" { + forward_to = [prometheus.remote_write.opspilot.receiver] + + rule { + source_labels = ["__name__"] + regex = "kafka_(log_log|cluster_partition)_.+" + action = "drop" + } +} + +prometheus.remote_write "opspilot" { + endpoint { + url = "https://api.opspilot.com/v1/metrics" + + headers = { + "authorization" = sys.env("OPSPILOT_API_KEY"), + } + } +} +``` + +and install Alloy with it: + +```bash +helm install alloy alloy --repo https://grafana.github.io/helm-charts -n monitoring \ + --set-file alloy.configMap.content=strimzi.alloy \ + --set 'alloy.extraEnv[0].name=OPSPILOT_API_KEY' \ + --set 'alloy.extraEnv[0].valueFrom.secretKeyRef.name=opspilot' \ + --set 'alloy.extraEnv[0].valueFrom.secretKeyRef.key=api-key' +``` + +A few things worth understanding: + +- **It finds every Strimzi cluster in the Kubernetes cluster** - Kafka nodes, Kafka Exporters, Cruise Control, and the operators - by Strimzi's own pod labels and port names, so a new cluster is collected as soon as it starts. +- **The relabel rule drops the per-partition series** Strimzi's rules export: eleven for every partition replica a node holds, which the dashboards take from the Kafka Exporter instead. A Kafka node then sends about 400 series, however many partitions it has. +- **Keep `job_name` as it is.** Most of these metric names are the same as plain Kafka's, and the dashboards and alerts select on it. + +Already running Alloy in the cluster - the Kubernetes Monitoring chart, for instance? Add the blocks above to its config instead, with `forward_to` naming its existing `prometheus.remote_write`. + +--- + +## 3. Confirm the data arrived + +In OpsPilot, open **Explore**, select the **Metrics** data source, and run: + +```promql +kafka_server_kafkaserver_brokerstate{job="integrations/strimzi"} +``` + +You should get one series per Kafka node, labeled with its namespace and cluster. If nothing comes back, check the `Kafka` resource has `metricsConfig` and its pods have rolled. Then run `kafka_brokers{job="integrations/strimzi"}` (the Kafka Exporter) and `strimzi_resources` (the cluster operator). The Alloy UI (`kubectl -n monitoring port-forward ds/alloy 12345`) shows each target and any scrape error. + +Once data is flowing, the **Dashboard data** column on **Your installation** links straight to the provisioned dashboards. + +--- + +!!! question "Need more help?" + Contact support in the chat bubble and let us know how we can assist. diff --git a/mkdocs.yml b/mkdocs.yml index 92d62b2..fb6ea90 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -97,6 +97,7 @@ nav: - OpenTrace: Data-insights/Features/Integrations/Data/opentrace.md - PostgreSQL: Data-insights/Features/Integrations/Data/postgres.md - Redis: Data-insights/Features/Integrations/Data/redis.md + - Strimzi Kafka: Data-insights/Features/Integrations/Data/strimzi-kafka.md - TigerData: Data-insights/Features/Integrations/Data/tigerdata.md - Infrastructure: - Docker: Data-insights/Features/Integrations/Infrastructure/docker.md From aabaa7f067d1e0f733ae22dd1013caf20d1dfc32 Mon Sep 17 00:00:00 2001 From: CChislett Date: Tue, 20 Oct 2026 13:44:55 +0100 Subject: [PATCH 07/10] Drop shipped Data integrations from the Coming soon list Altinity ClickHouse, MongoDB, MySQL, Redis, Strimzi Kafka, and TigerData have shipped and now have pages, so remove them from the overview's Coming soon list. Kafka (non-Strimzi) and RabbitMQ remain. Co-Authored-By: Claude Opus 4.8 --- docs/Data-insights/Features/integrations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Data-insights/Features/integrations.md b/docs/Data-insights/Features/integrations.md index c91dd08..76a4e0c 100644 --- a/docs/Data-insights/Features/integrations.md +++ b/docs/Data-insights/Features/integrations.md @@ -261,7 +261,7 @@ The [FusionReactor agent](Integrations/SDKs/fusionreactor.md) is **managed autom === "Data" - Altinity ClickHouse Operator · Kafka · MongoDB · MySQL · RabbitMQ · Redis · Strimzi Kafka · TigerData + Kafka · RabbitMQ === "Infrastructure" From e38fac26cc56644d75523e031ecc96ba042d884d Mon Sep 17 00:00:00 2001 From: CChislett Date: Tue, 20 Oct 2026 14:03:36 +0100 Subject: [PATCH 08/10] Add the Kubernetes integration page Document the Kubernetes catalog integration (Infrastructure, v1.1.0), which uses the Grafana Kubernetes Monitoring Helm chart (distinct from the existing kube-prometheus-stack monitoring guide). Covers the values file with the required metricsTuning lists, installing the chart, confirming the jobs, resuming the paused alerts, troubleshooting, and removal. Adds it to the Infrastructure nav and drops it from Coming soon. Co-Authored-By: Claude Opus 4.8 --- .../Integrations/Infrastructure/kubernetes.md | 166 ++++++++++++++++++ docs/Data-insights/Features/integrations.md | 2 +- mkdocs.yml | 1 + 3 files changed, 168 insertions(+), 1 deletion(-) create mode 100644 docs/Data-insights/Features/Integrations/Infrastructure/kubernetes.md diff --git a/docs/Data-insights/Features/Integrations/Infrastructure/kubernetes.md b/docs/Data-insights/Features/Integrations/Infrastructure/kubernetes.md new file mode 100644 index 0000000..bef8941 --- /dev/null +++ b/docs/Data-insights/Features/Integrations/Infrastructure/kubernetes.md @@ -0,0 +1,166 @@ +# Kubernetes + +Monitor Kubernetes clusters - nodes, workloads, and control-plane health. + +Your clusters show up alongside the rest of your telemetry, on dashboards that are already built. You can see how each cluster is using what it has, from the whole cluster down to a single pod, with alerts when workloads fail, nodes come under pressure, or storage runs out. + +!!! info "What this integration does" + It provisions **dashboards**, **recording rules**, and **alert rules**. It does not collect anything itself - metrics come from the **Grafana Kubernetes Monitoring Helm chart**, which you run once in every cluster you want to see. Until your cluster is sending metrics, the dashboards are empty. + +Navigate to **Integrations** from the left-hand sidebar, then select **Kubernetes**. + +--- + +## Permission tiers + +Kubernetes runs **Read-only**, the default. It requires an **OpsPilot API key** for the collector in your cluster. + +Its **capabilities** are **Dashboards**, **Recording Rules**, and **Alerts**. + +--- + +## 1. Write the values file + +You need an OpsPilot API key, from **Data and Licenses > [API Keys](https://app.opspilot.com/organization/api-keys)**. + +Save this as `opspilot-values.yaml`, setting `cluster.name`: + +```yaml +cluster: + name: + +destinations: + opspilot: + type: prometheus + url: https://api.opspilot.com/v1/metrics + extraHeaders: + authorization: + +clusterMetrics: + enabled: true + apiServer: + enabled: true + kube-state-metrics: + metricsTuning: + includeMetrics: + - kube_pod_status_ready + - kube_poddisruptionbudget_status_current_healthy + - kube_poddisruptionbudget_status_desired_healthy + kubelet: + metricsTuning: + includeMetrics: + - kubelet_evictions + - kubelet_runtime_operations_duration_seconds_bucket + - rest_client_request_duration_seconds_bucket + - storage_operation_duration_seconds_bucket + +hostMetrics: + enabled: true + linuxHosts: + enabled: true + metricsTuning: + includeMetrics: + - node_bonding_active + - node_bonding_slaves + - node_disk_io_time_seconds_total + - node_disk_io_time_weighted_seconds_total + - node_disk_read_bytes_total + - node_disk_written_bytes_total + - node_filefd_allocated + - node_filefd_maximum + - node_load1 + - node_load5 + - node_load15 + - node_md_disks + - node_md_disks_required + - node_netstat_Tcp_OutSegs + - node_netstat_Tcp_RetransSegs + - node_netstat_TcpExt_TCPSynRetrans + - node_network_receive_errs_total + - node_network_receive_packets_total + - node_network_transmit_errs_total + - node_network_transmit_packets_total + - node_nf_conntrack_entries + - node_nf_conntrack_entries_limit + - node_textfile_scrape_error + - node_time_seconds + - node_timex_maxerror_seconds + - node_timex_offset_seconds + - node_timex_sync_status + - node_uname_info + - node_vmstat_pgmajfault + +collectors: + alloy-metrics: + presets: [clustered, statefulset] + +telemetryServices: + kube-state-metrics: + deploy: true + node-exporter: + deploy: true +``` + +- **`cluster.name` fills the dashboards' cluster selector.** Give every cluster a different one, or their series merge. Use only letters, digits, `-`, and `_`: Grafana escapes a quote or backslash in the name, and then every panel is empty. +- **The `includeMetrics` lists are needed.** The chart keeps a reduced set of metrics by default, and some dashboard panels and alerts read metrics outside it. Without these lines those panels are empty and those alerts never fire. If the chart already runs in a cluster, update the lists and run step 2 again; the Nodes dashboards stay empty until you do. + +--- + +## 2. Install the chart + +```bash +helm repo add grafana https://grafana.github.io/helm-charts +helm repo update grafana +helm upgrade --install k8s-monitoring grafana/k8s-monitoring \ + --version ^4 --namespace monitoring --create-namespace \ + --values opspilot-values.yaml +``` + +--- + +## 3. Confirm the data arrived + +After a couple of minutes, open **Explore**, select the **Metrics** data source, and run: + +```promql +count by (cluster, job) (up) +``` + +Each cluster should show `integrations/kubernetes/kube-state-metrics`, `integrations/kubernetes/kubelet`, `integrations/kubernetes/cadvisor`, `integrations/kubernetes/kube-apiserver`, and `integrations/node_exporter`. + +The Compute Resources and the Nodes USE Method dashboards read series that the installed recording rules derive, so they fill in a minute or two after the rest. + +Once data is flowing, the **Dashboard data** column on **Your installation** links straight to the provisioned dashboards. + +--- + +## 4. Turn on the alerts you want + +The alert rules are installed **paused**. Open **Alerting → Alert rules**, find the **Kubernetes** folder, and resume the ones you want. Each alert's runbook link explains what it means and what to do. + +--- + +## Troubleshooting + +**No series in Explore.** Check the Alloy pods with `kubectl -n monitoring logs statefulset/k8s-monitoring-alloy-metrics`. A 401 or 403 on remote write means the API key is wrong. + +**The API server row is empty on a managed cluster.** Some providers do not let a workload scrape the API server's metrics. The rest of the dashboards are unaffected. + +--- + +## Removing it + +Uninstalling the integration removes the dashboards and rules, not the chart. In each cluster, run: + +```bash +helm uninstall k8s-monitoring --namespace monitoring +kubectl delete crd alloys.collectors.grafana.com +kubectl delete namespace monitoring +``` + +Helm leaves the CRD and the namespace behind. Keep the CRD if another Grafana Alloy Operator runs in the cluster, and the namespace if anything else runs in `monitoring`. + +--- + +!!! question "Need more help?" + Contact support in the chat bubble and let us know how we can assist. diff --git a/docs/Data-insights/Features/integrations.md b/docs/Data-insights/Features/integrations.md index 76a4e0c..1120f62 100644 --- a/docs/Data-insights/Features/integrations.md +++ b/docs/Data-insights/Features/integrations.md @@ -265,7 +265,7 @@ The [FusionReactor agent](Integrations/SDKs/fusionreactor.md) is **managed autom === "Infrastructure" - ArgoCD · Host Metrics · KEDA · Kubernetes · Terraform · TrueNAS SCALE + ArgoCD · Host Metrics · KEDA · Terraform · TrueNAS SCALE === "Networking" diff --git a/mkdocs.yml b/mkdocs.yml index fb6ea90..bb3a2fb 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -102,6 +102,7 @@ nav: - Infrastructure: - Docker: Data-insights/Features/Integrations/Infrastructure/docker.md - iDRAC: Data-insights/Features/Integrations/Infrastructure/idrac.md + - Kubernetes: Data-insights/Features/Integrations/Infrastructure/kubernetes.md - Proxmox VE: Data-insights/Features/Integrations/Infrastructure/proxmox-ve.md - Unix: Data-insights/Features/Integrations/Infrastructure/unix.md - Windows: Data-insights/Features/Integrations/Infrastructure/windows.md From f6200080a03d2add4d6d52b9be878aa3a5013554 Mon Sep 17 00:00:00 2001 From: CChislett Date: Tue, 20 Oct 2026 14:08:31 +0100 Subject: [PATCH 09/10] Cross-link the two Kubernetes monitoring approaches The Kubernetes integration (Grafana Kubernetes Monitoring chart) and the Kube-Prometheus-Stack shipping guide are both valid. Add a note on each pointing at the other and explaining which chart it uses, so readers can pick the one that fits what they already run. Co-Authored-By: Claude Opus 4.8 --- .../Features/Integrations/Infrastructure/kubernetes.md | 3 +++ docs/Monitor-your-data/Kubernetes-monitoring/overview.md | 3 +++ 2 files changed, 6 insertions(+) diff --git a/docs/Data-insights/Features/Integrations/Infrastructure/kubernetes.md b/docs/Data-insights/Features/Integrations/Infrastructure/kubernetes.md index bef8941..83a6515 100644 --- a/docs/Data-insights/Features/Integrations/Infrastructure/kubernetes.md +++ b/docs/Data-insights/Features/Integrations/Infrastructure/kubernetes.md @@ -9,6 +9,9 @@ Your clusters show up alongside the rest of your telemetry, on dashboards that a Navigate to **Integrations** from the left-hand sidebar, then select **Kubernetes**. +!!! note "Already running Kube-Prometheus-Stack?" + This integration ships metrics with the **Grafana Kubernetes Monitoring** chart and is the simplest way to get the OpsPilot Kubernetes dashboards, recording rules, and alerts. If you already run - or prefer - the **Kube-Prometheus-Stack** chart, you can ship cluster metrics that way instead: see [Kubernetes monitoring](../../../../Monitor-your-data/Kubernetes-monitoring/overview.md). + --- ## Permission tiers diff --git a/docs/Monitor-your-data/Kubernetes-monitoring/overview.md b/docs/Monitor-your-data/Kubernetes-monitoring/overview.md index 3e09245..bfc8d42 100644 --- a/docs/Monitor-your-data/Kubernetes-monitoring/overview.md +++ b/docs/Monitor-your-data/Kubernetes-monitoring/overview.md @@ -14,6 +14,9 @@ OpsPilot brings AI-powered Kubernetes monitoring giving SREs, application develo !!! info "Learn more" [Shipping K8s data to OpsPilot](Shipping.md) +!!! note "Kubernetes integration" + These pages cover shipping cluster metrics with the **Kube-Prometheus-Stack** chart. OpsPilot also offers a **[Kubernetes integration](../../Data-insights/Features/Integrations/Infrastructure/kubernetes.md)** that installs the dashboards, recording rules, and alerts and collects metrics with the Grafana Kubernetes Monitoring chart - the simplest path for most clusters. Use whichever chart you already run or prefer. + ## Benefits 1. **Real time alerts**. Monitoring Kubernetes clusters in real-time can detect errors early and send alerts to you immediately. This saves your brand's reputation and reduces service restoration time. From f20275990b6faa23501da4da9476c35f9e00da88 Mon Sep 17 00:00:00 2001 From: CChislett Date: Tue, 20 Oct 2026 14:10:36 +0100 Subject: [PATCH 10/10] Update the Windows integration page for v1.3.0 Reconcile against v1.3.0: document the second dashboard (Windows Hosts - Detail), add the six new collectors (tcp, scheduled_task, pagefile, time, thermalzone, cpu_info) with their notes, and correct the job-label behavior - it is now set to integrations/windows automatically, not forced via job_name, so the confirm query and troubleshooting drop the job selector. Co-Authored-By: Claude Opus 4.8 --- .../Integrations/Infrastructure/windows.md | 54 +++++++++++++++---- 1 file changed, 43 insertions(+), 11 deletions(-) diff --git a/docs/Data-insights/Features/Integrations/Infrastructure/windows.md b/docs/Data-insights/Features/Integrations/Infrastructure/windows.md index 0601e64..8f7e1c1 100644 --- a/docs/Data-insights/Features/Integrations/Infrastructure/windows.md +++ b/docs/Data-insights/Features/Integrations/Infrastructure/windows.md @@ -2,10 +2,10 @@ Monitor Windows hosts - CPU, memory, disks, and network. -Your Windows hosts show up alongside the rest of your telemetry on a dashboard that is already built, including which services set to start automatically are not running. When an application slows down or stops responding, you can see whether the machine or a service underneath it is the reason. +Your Windows hosts show up alongside the rest of your telemetry, on dashboards that are already built. When an application slows down or stops responding, you can see whether the machine or a service underneath it is the reason. !!! info "What this integration does" - It provisions a **dashboard**. It does not collect anything itself and never connects to your hosts - the metrics come from **Grafana Alloy**, which has a windows_exporter collector built in. Until Alloy is collecting them, the dashboard is empty. + It provisions **dashboards**. It does not collect anything itself and never connects to your hosts - the metrics come from **Grafana Alloy**, which has a `windows_exporter` collector built in. Until Alloy is collecting them, the dashboards are empty. Navigate to **Integrations** from the left-hand sidebar, then select **Windows**. @@ -15,17 +15,32 @@ Navigate to **Integrations** from the left-hand sidebar, then select **Windows** Windows runs **Read-only**, which is the default and requires nothing - the metrics endpoint and signing key come from the service environment. +Its one **capability** is **Dashboards**. + +--- + +## The dashboards + +Two dashboards are provisioned: + +- **Windows Hosts** opens with a table of every host reporting, so you can see the estate before picking a machine; below that it covers CPU, memory, disk, network, and services for whichever one you choose. +- **Windows Hosts - Detail** is for working out *why* a machine is struggling rather than *whether* it is: memory commit and kernel pool, page file activity, disk latency and queue depth, per-core CPU, the processor queue, network errors, and TCP health. + +Until Alloy is collecting them, both will be empty. + --- ## 1. Install Alloy on the host -Alloy runs as a Windows service. Download the Windows installer from [Grafana's releases](https://github.com/grafana/alloy/releases) and run it, or install with winget: +Alloy runs as a Windows service. The quickest way to install it is winget, from an administrator PowerShell: ```powershell winget install Grafana.Alloy ``` -The installer puts the configuration at `C:\Program Files\GrafanaLabs\Alloy\config.alloy`. +If winget is not available, download the installer instead. On Grafana's [Alloy install guide](https://grafana.com/docs/alloy/latest/set-up/install/) follow the Windows page, which links to the current release; on that release, open **Assets** at the bottom and take `alloy-installer-windows-amd64.exe.zip`. Unzip it and run the installer. + +Either way the installer registers Alloy as a service and puts its configuration at `C:\Program Files\GrafanaLabs\Alloy\config.alloy`. That is the file you edit in the next step. --- @@ -43,6 +58,12 @@ prometheus.exporter.windows "host" { "os", "system", "service", + "tcp", + "scheduled_task", + "pagefile", + "time", + "thermalzone", + "cpu_info", ] } @@ -105,9 +126,9 @@ Restart-Service Alloy ### Four things that matter more than they look -**`memory` is not a default collector.** Alloy's defaults are `cpu`, `logical_disk`, `net`, `os`, `service` and `system` - no memory. Leave it out and the memory panels are permanently empty with nothing to explain why, so it is listed explicitly above. +**`memory` is not a default collector.** Alloy's defaults are `cpu`, `logical_disk`, `net`, `os`, `service`, and `system` - no memory. Leave it out and the memory panels are permanently empty with nothing to explain why, so it is listed explicitly above. -**`job_name = "windows"` is required.** Every panel filters on it, because your metrics land in a store shared with everything else you send to OpsPilot. Alloy's default job label is the component's own id, which would change if you renamed the component, so it is set explicitly. Change it and the dashboard goes blank. +**The `job` label is set for you, and not by `job_name`.** Alloy's built-in windows exporter labels its own metrics `job="integrations/windows"`, and `job_name` on the scrape does not override it. The label follows the exporter type rather than the name you give the component, so calling the component something else does not change it. The dashboard reads that label, and also accepts `windows` for anyone scraping a standalone `windows_exporter`. **windows_exporter 0.25 or newer.** The memory collector's `windows_memory_physical_total_bytes` and `windows_memory_physical_free_bytes` are relatively recent, and `windows_os_info` - which the host selector is built from - has to be present, or the dashboard shows nothing at all rather than partially working. Any Alloy release from 2024 onwards embeds a new enough exporter. @@ -131,12 +152,19 @@ prometheus.exporter.windows "host" { That is cheaper again, at the cost of the dashboard only knowing about the services you named. +**The last four collectors are optional.** `pagefile`, `time`, `thermalzone`, and `cpu_info` each fill one panel in the detail dashboard's bottom row, which says so where it is empty. Drop any of them and nothing else is affected. + +**`tcp` and `scheduled_task` are for the detail dashboard.** The overview needs neither. Leave them out and the overview is unaffected, while the detail dashboard's TCP panels and its scheduled-task table are empty. + +!!! note + `enabled_collectors` replaces Alloy's defaults rather than adding to them, which is why collectors documented as on-by-default have to appear in the list too. + !!! warning "Keep the collector list short" - Several collectors fail, or take the whole Alloy process down, when the thing they measure is not installed - `mscluster`, `vmware`, `hyperv`, `ad`, `dns`, `msmq` and `nps` among them. The list above is what this dashboard reads and nothing more. Add others deliberately, one at a time, and check Alloy is still running afterwards. + Several collectors fail, or take the whole Alloy process down, when the thing they measure is not installed - `mscluster`, `vmware`, `hyperv`, `ad`, `dns`, `msmq`, and `nps` among them. The list above is what these dashboards read and nothing more. Add others deliberately, one at a time, and check Alloy is still running afterwards. Each host appears in the dashboard's host selector under its `instance` label, which Alloy sets from the scrape target - the machine's own hostname. -If you already run windows_exporter as a standalone service and would rather point Alloy at that than enable the built-in collector, that works: scrape `http://localhost:9182/metrics` instead. The only difference is that `instance` then reads `HOSTNAME:9182` rather than the bare hostname, so that is what the host selector and every legend will show. +If you already run `windows_exporter` as a standalone service and would rather point Alloy at that than enable the built-in collector, that works: scrape `http://localhost:9182/metrics` instead. The only difference is that `instance` then reads `HOSTNAME:9182` rather than the bare hostname, so that is what the host selector and every legend will show. --- @@ -145,19 +173,23 @@ If you already run windows_exporter as a standalone service and would rather poi In OpsPilot, open **Explore**, select the **Metrics** data source, and run: ```promql -windows_os_info{job="windows"} +windows_os_info ``` -You should get one series per host. If that returns data, the dashboard will too. +You should get one series per host. If that returns data, the dashboards will too. + +Once data is flowing, the **Dashboard data** column on **Your installation** links straight to the provisioned dashboards. --- ## Troubleshooting -**Nothing in Explore.** Check the Alloy UI at `http://localhost:12345` on the host first - a failing component shows there with the error. If the scrape is healthy, check the `job` label really is `windows` by querying `windows_os_info` with no selector and reading the labels back. +**Nothing in Explore.** Check the Alloy UI at `http://localhost:12345` on the host first - a failing component shows there with the error. If the scrape is healthy, query `windows_os_info` with no selector and read the labels back - whatever the `job` label says, the dashboard accepts both `windows` and `integrations/windows`. **The memory panels are empty but everything else works.** The memory collector is not enabled. See step 2. +**The TCP panels or the scheduled-task table are empty.** The `tcp` and `scheduled_task` collectors are not enabled. Both are only read by the detail dashboard, so the overview will look fine without them. + **The service panels are empty.** The service collector is not enabled, or the Alloy service lacks the rights to enumerate services. It runs as LocalSystem by default, which has them. **Stopped services shows a count but the table below is empty.** The table also filters on start mode, so it lists only services set to start automatically - a stopped service set to Manual is stopped by design and is left out deliberately.