From e4af7c1ae1f6b8fbec008c901fee748685c2759b Mon Sep 17 00:00:00 2001 From: Luke Kim <80174+lukekim@users.noreply.github.com> Date: Wed, 28 Oct 2026 10:30:20 -0700 Subject: [PATCH] docs: restrict access to the metrics endpoint, which API keys do not cover --- website/docs/api/auth/index.md | 2 ++ website/docs/features/observability/index.md | 14 +++++++++++++- .../versioned_docs/version-2.4.x/api/auth/index.md | 2 ++ .../version-2.4.x/features/observability/index.md | 14 +++++++++++++- 4 files changed, 30 insertions(+), 2 deletions(-) diff --git a/website/docs/api/auth/index.md b/website/docs/api/auth/index.md index add3d34f4..65a5be31e 100644 --- a/website/docs/api/auth/index.md +++ b/website/docs/api/auth/index.md @@ -97,6 +97,8 @@ Or using `Authorization: Bearer`: The `/health` and `/v1/ready` endpoints are not protected and can be accessed without an API key. +API keys do not protect the Prometheus metrics endpoint. When `spiced` runs with `--metrics`, `/metrics` is served on its own listener without an API key check, and its labels include dataset names and HTTP route templates. Bind the metrics endpoint to a loopback or private interface, as described in [Restrict Access to the Metrics Endpoint](../../features/observability/index.md#restrict-access-to-the-metrics-endpoint). + ## Flight SQL For the Flight SQL endpoint, the API key is expected to be included in the `Authorization` header as a Bearer token, i.e. `Authorization: Bearer ${ api_key }`. diff --git a/website/docs/features/observability/index.md b/website/docs/features/observability/index.md index 237d44aa6..eca62dd6c 100644 --- a/website/docs/features/observability/index.md +++ b/website/docs/features/observability/index.md @@ -31,7 +31,7 @@ Query metrics on this endpoint — `query_duration_ms`, `query_executions`, `que ### Default Configuration -The metrics endpoint listens on port `9090` by default. The endpoint address is logged at startup: +The metrics endpoint is disabled by default. When `spiced` starts with `--metrics`, the endpoint address is logged at startup: ```bash 2024-11-28T19:48:10.942003Z INFO runtime::metrics_server: Spice Runtime Metrics listening on 127.0.0.1:9090 @@ -54,6 +54,18 @@ CMD ["--metrics", "0.0.0.0:9090"] EXPOSE 9090 ``` +### Restrict Access to the Metrics Endpoint + +The metrics endpoint does not check [API keys](../../api/auth/index.md). It also does not require a client certificate when [mTLS](../../api/tls/index.md#probe-and-metrics-access) is configured. Any caller that can reach the `--metrics` address can read every metric, and metric labels include dataset names and HTTP route templates. When [TLS](../../api/tls/index.md) is enabled, the endpoint is served over TLS, which encrypts the connection but does not restrict who can scrape it. + +Bind the metrics endpoint to a loopback or private interface that only the metrics scraper can reach. When the scraper runs on the same host, bind to loopback: + +```bash +spiced --metrics 127.0.0.1:9090 +``` + +In containers and Kubernetes, the endpoint must listen on `0.0.0.0` inside the container for a scraper in another pod or on another host to reach it, and the Spice Helm chart starts `spiced` with `--metrics 0.0.0.0:9090`. Keep the metrics port off public load balancers and ingress, and restrict which pods can connect to it, for example with a Kubernetes [NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/). Do not publish the metrics port to the host or the internet unless the network in front of it limits access to the scraper. + ### Verifying the Endpoint Verify the metrics endpoint is working with a GET request: diff --git a/website/versioned_docs/version-2.4.x/api/auth/index.md b/website/versioned_docs/version-2.4.x/api/auth/index.md index bb1faf825..5d7483f12 100644 --- a/website/versioned_docs/version-2.4.x/api/auth/index.md +++ b/website/versioned_docs/version-2.4.x/api/auth/index.md @@ -58,6 +58,8 @@ Or using `Authorization: Bearer`: The `/health` and `/v1/ready` endpoints are not protected and can be accessed without an API key. +API keys do not protect the Prometheus metrics endpoint. When `spiced` runs with `--metrics`, `/metrics` is served on its own listener without an API key check, and its labels include dataset names and HTTP route templates. Bind the metrics endpoint to a loopback or private interface, as described in [Restrict Access to the Metrics Endpoint](../../features/observability/index.md#restrict-access-to-the-metrics-endpoint). + ## Flight SQL For the Flight SQL endpoint, the API key is expected to be included in the `Authorization` header as a Bearer token, i.e. `Authorization: Bearer ${ api_key }`. diff --git a/website/versioned_docs/version-2.4.x/features/observability/index.md b/website/versioned_docs/version-2.4.x/features/observability/index.md index 68410969a..cccc25b7e 100644 --- a/website/versioned_docs/version-2.4.x/features/observability/index.md +++ b/website/versioned_docs/version-2.4.x/features/observability/index.md @@ -31,7 +31,7 @@ Query metrics on this endpoint — `query_duration_ms`, `query_executions`, `que ### Default Configuration -The metrics endpoint listens on port `9090` by default. The endpoint address is logged at startup: +The metrics endpoint is disabled by default. When `spiced` starts with `--metrics`, the endpoint address is logged at startup: ```bash 2024-11-28T19:48:10.942003Z INFO runtime::metrics_server: Spice Runtime Metrics listening on 127.0.0.1:9090 @@ -54,6 +54,18 @@ CMD ["--metrics", "0.0.0.0:9090"] EXPOSE 9090 ``` +### Restrict Access to the Metrics Endpoint + +The metrics endpoint does not check [API keys](../../api/auth/index.md). It also does not require a client certificate when [mTLS](../../api/tls/index.md#probe-and-metrics-access) is configured. Any caller that can reach the `--metrics` address can read every metric, and metric labels include dataset names and HTTP route templates. When [TLS](../../api/tls/index.md) is enabled, the endpoint is served over TLS, which encrypts the connection but does not restrict who can scrape it. + +Bind the metrics endpoint to a loopback or private interface that only the metrics scraper can reach. When the scraper runs on the same host, bind to loopback: + +```bash +spiced --metrics 127.0.0.1:9090 +``` + +In containers and Kubernetes, the endpoint must listen on `0.0.0.0` inside the container for a scraper in another pod or on another host to reach it, and the Spice Helm chart starts `spiced` with `--metrics 0.0.0.0:9090`. Keep the metrics port off public load balancers and ingress, and restrict which pods can connect to it, for example with a Kubernetes [NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/). Do not publish the metrics port to the host or the internet unless the network in front of it limits access to the scraper. + ### Verifying the Endpoint Verify the metrics endpoint is working with a GET request: