docs: production tests

_data/production-test-metrics/autoscaling-container.csv

@@ -0,0 +1,5 @@
+family,unit,description,suffixes
+active-nodes,count,Maximum number of active container nodes in the cluster during the test window.,-container
+non-active-node-fraction,fraction,"Peak fraction of non-active (mid-transition) nodes in the container cluster during the test window, non-zero while a resize is in progress.",-container
+query-growth-rate-vs-3h-average,ratio,"Ratio of the container cluster's query rate during the test window to its trailing 3-hour average, a proxy for the autoscaler's growth-headroom signal (values above 1.0 indicate rising traffic).",
+query-vs-write-rate-cpu-mix-driver,operations/second,Average query rate across the container cluster during the test window; one side of the query/write mix that shapes the autoscaler's ideal CPU target.,"-query, -write"

_data/production-test-metrics/autoscaling-content.csv

@@ -0,0 +1,9 @@
+family,unit,description,suffixes
+active-nodes-count,count,Maximum number of active nodes in the content cluster during the test window.,
+active-nodes-non-active-fraction,fraction (0-1),"Fraction of content-cluster nodes that are not yet active, indicating an in-progress resize or data redistribution.",
+autoscaling-actions,count,Number of autoscaler rescale actions taken for the content cluster during the test window.,-sum
+cpu-load-adjustment,"ratio (peak / ideal, 1.0 = ideally scaled)",Ratio of peak to ideal CPU load for the content cluster; above 1.0 signals scale-up pressure and below 1.0 signals the cluster is over-provisioned for CPU.,-ratio
+cpu-load-peak-vs-ideal,percentage (0-100),"The autoscaler's ideal CPU load target for the content cluster, as a percentage (already includes growth and failover headroom, so normally below 100).","-ideal, -peak"
+query-growth-rate-vs-3h-average-content,ratio (test-window average / trailing 3h average),"Ratio of the content cluster's query rate during the test window to its own trailing 3-hour average, a proxy for the autoscaler's internal query-growth headroom factor.",
+query-vs-write-rate-cpu-mix-driver-query-content,operations/second,"Query rate for the content cluster, one half of the query/write mix that determines the autoscaler's ideal CPU target.",
+query-vs-write-rate-cpu-mix-driver-write-content,operations/second,"Write rate (put + update + remove operations per second) for the content cluster, the other half of the query/write mix that determines the autoscaler's ideal CPU target.",

_data/production-test-metrics/content-node.csv

@@ -0,0 +1,16 @@
+family,unit,description,suffixes
+attribute-feeding-blocked,percentage (0-100),Percentage of the time window during which feeding was blocked due to attribute resource usage limits on any content node.,-pct
+bucket-move-pending,count,Maximum number of buckets pending a move operation (e.g. during cluster resizing or redistribution) on any content node.,-max
+document-store-cache-hit-rate,percentage (0-100),"Cache hit rate for the ready document store's summary-field cache, as a percentage; low values on quiescent clusters often show up as elevated CPU IOWait from disk-backed docsum reads.",-max
+document-store-disk-usage,bytes,Maximum disk space used by the ready document store on any content node.,-max
+documents-count,count,"Number of active/searchable document copies, summed across content clusters after taking the max across replica groups within each cluster to avoid double counting.","-active, -ready, -removed, -total"
+field-writer-saturation,percentage (0-100),Maximum observed saturation of Proton's field writer executor thread pool; high saturation indicates a write bottleneck.,-max
+field-writer-utilization,percentage (0-100),Average utilization of Proton's field writer executor thread pool across content nodes.,-avg
+filestor-average-queue-wait,milliseconds,Average time operations spent waiting in the filestor queue across content nodes.,-avg
+filestor-queue-size,count,Maximum filestor queue size observed on any content node; a growing queue indicates storage cannot keep up with incoming operations.,-max
+maintenance-job-activity,percentage (0-100),"Activity level of the attribute-flush maintenance job (0% = idle, 100% = fully active), maximum across content nodes.","-attr-flush, -disk-idx-fusion, -docstore-compact, -lid-compact, -mem-idx-flush"
+proton-disk-usage,percentage (0-100),"Relative disk usage of the content node's Proton process, as a percentage of the available disk; 100% means feeding is blocked.",-avg
+proton-executor-utilization,percentage (0-100),Average utilization of Proton's own executor thread pool across content nodes.,"-avg, -max"
+proton-memory-usage,percentage (0-100),"Relative memory usage of the content node's Proton process, as a percentage of the available memory.",-avg
+shared-executor-queue-size,count,Maximum shared executor queue size observed on any content node during the time window.,-max
+shared-executor-utilization,percentage (0-100),Average utilization of Proton's shared executor thread pool across content nodes.,"-avg, -max"

_data/production-test-metrics/feed.csv

@@ -0,0 +1,35 @@
+family,unit,description,suffixes
+container-feed-latency,milliseconds,"Maximum feed request latency observed at the container, across all operation types and clusters.",-max
+container-feed-operations,operations/second,"Total feed operation rate (puts, updates, removes) handled by container clusters, summed across operation types.",-sum
+content-commit-latency,milliseconds,Average commit latency (time to commit fed changes to the index/store) on content nodes.,-avg
+content-commit-operations,operations/second,Rate of commit operations (batched writes applied to the index/store) on content nodes.,-sum
+content-storage-feed-rates,operations/second,"Rate of PUT operations processed by the storage/persistence-engine layer on content nodes, counted per document replica.","-put, -remove, -update"
+content-storage-put-latency,milliseconds,Maximum PUT operation latency observed at the storage/persistence-engine layer on content nodes.,-max
+content-storage-update-latency,milliseconds,Average UPDATE operation latency at the storage/persistence-engine layer on content nodes.,"-avg, -max"
+distributor-latency-gets,milliseconds,Maximum get operation latency observed at the distributor level.,-max
+distributor-latency-puts,milliseconds,Maximum put operation latency observed at the distributor level.,-max
+distributor-latency-removes,milliseconds,Maximum remove operation latency observed at the distributor level.,-max
+distributor-latency-updates,milliseconds,Maximum update operation latency observed at the distributor level.,-max
+distributor-latency-visitor,milliseconds,Maximum visitor operation latency observed at the distributor level.,-max
+distributor-operation-gets-failures,operations/second,Rate of failed get operations at the distributor level; non-zero values indicate issues with writes.,-sum
+distributor-operation-puts-failures,operations/second,Rate of failed put operations at the distributor level; non-zero values indicate issues with writes.,-sum
+distributor-operation-rates-gets-ok,operations/second,Rate of successfully completed (OK) get operations at the distributor level.,-sum
+distributor-operation-rates-puts-ok,operations/second,Rate of successfully completed (OK) put operations at the distributor level.,-sum
+distributor-operation-rates-removes-ok,operations/second,Rate of successfully completed (OK) remove operations at the distributor level.,-sum
+distributor-operation-rates-updates-ok,operations/second,Rate of successfully completed (OK) update operations at the distributor level.,-sum
+distributor-operation-rates-visitor-ok,operations/second,Rate of successfully completed (OK) visitor operations at the distributor level.,-sum
+distributor-operation-removes-failures,operations/second,Rate of failed remove operations at the distributor level; non-zero values indicate issues with writes.,-sum
+distributor-operation-updates-failures,operations/second,Rate of failed update operations at the distributor level; non-zero values indicate issues with writes.,-sum
+distributor-operation-visitor-failures,operations/second,Rate of failed visitor operations at the distributor level; non-zero values indicate issues with writes.,-sum
+document-processing-latency,milliseconds,Average document processing (docproc chain) latency.,-avg
+document-processing-rate,operations/second,"Rate of documents processed through document processing (docproc) chains, summed across all chains and clusters.",-sum
+feed-blocked-nodes-above-resource-limit,count,"Number of content nodes currently exceeding a resource usage limit; when greater than zero, feeding is blocked for that cluster.",
+feed-latency-avg-ms,milliseconds,"Average feed latency, in milliseconds, for successful feed operations.",
+http-api-feed-rates,operations/second,Rate of PUT operations received via the Document/HTTP API.,"-put, -remove, -update"
+http-api-latency,milliseconds,"Average latency of Document/HTTP API requests, distinct from the generic jdisc HTTP layer latency.",-avg
+http-api-pending-requests,count,"Maximum number of Document/HTTP API requests pending (in-flight, not yet completed) on the container.",-max
+http-api-success-vs-failures,operations/second,Rate of failed Document/HTTP API operations.,"-failed, -parse-error, -success"
+http-write-latency,milliseconds,Average end-to-end HTTP write request latency at the container's HTTP layer.,"-avg, -p95, -p99"
+memory-index-docs-in-memory,count,"Maximum number of documents held in the in-memory index on content nodes, summed per content group (to combine per-node/stripe values) and maxed across groups (since each group holds a full replica) to avoid double-counting.",-max
+persistence-engine-input-queue,count,Average depth of the per-node persistence engine's input operation queue (operations waiting to be dispatched to Proton) over the time window.,"-avg, -max"
+persistence-engine-throttle-saturation,count,Maximum number of concurrently active (in-flight) operations in the content node's persistence engine.,"-active, -throttle"

_data/production-test-metrics/health.csv

@@ -0,0 +1,19 @@
+family,unit,description,suffixes
+cluster-state-changes,count,Number of cluster state changes observed in the given time window. Frequent changes are most often caused by Vespa upgrades or required node restarts.,-sum
+core-dumps-processed,count,Number of core dumps processed across all nodes in the given time window; should be 0.,-max
+deactivated-containers,count,"Average number of deactivated container instances per sample, over the given time window. Brief spikes during deployments are normal; persistently high values may indicate resource leaks.",
+estimated-time-to-in-sync,seconds,"Rough estimate, from the recent (fixed 30-minute) convergence rate, of the time remaining until all buckets are back in sync; capped at 7 days.",
+failed-component-graphs,operations/second,"Rate of failed application component-graph reconfigurations, averaged over the given time window; nonzero after a deployment may indicate a configuration or component issue.",
+http-requests-prematurely-closed,operations/second,"Rate of HTTP requests closed by the client before the server finished responding, averaged over the given time window.",
+merge-bucket-pending,count,"Number of pending bucket merge operations across all distributors, summed per application.",
+node-count,count,Total number of nodes currently in the active state for the application.,-max
+node-events,count,Number of node events reported by the cluster controller in the given time window.,-sum
+node-state-down,count,Number of content nodes in the 'down' state per cluster (nodes considered unavailable by the cluster controller).,-max
+node-state-initializing,count,Number of content nodes in the 'initializing' state per cluster (nodes currently starting up and not yet serving).,-max
+node-state-maintenance,count,Number of content nodes in the 'maintenance' state per cluster (nodes intentionally taken down for maintenance).,-max
+node-state-retired,count,Number of content nodes in the 'retired' state per cluster (nodes being decommissioned).,-max
+node-state-up,count,Number of content nodes in the 'up' state per cluster.,-max
+re-indexing-remaining,percentage (0-100),"Worst-case document reindexing progress remaining across document types, as a percentage (100 = not started, 0 = complete or no reindex running).",
+resource-usage-attribute-address-space,percentage (0-100),Worst-case attribute address-space utilization across content nodes; feed is blocked when this approaches 100%.,
+resource-usage-max-disk-utilization-vs-limit,percentage (0-100),Maximum disk utilization vs the configured limit; feed is blocked when utilization exceeds the limit.,
+service-restarts,count,"Total service restarts across all nodes, as tracked by the config sentinel. Increases indicate services crashing or being restarted.",-total

_data/production-test-metrics/nearest-neighbor-search.csv

@@ -0,0 +1,21 @@
+family,unit,description,suffixes
+ann-timeout-rate,requests/second,Rate of queries per second that terminated an approximate NNS search early after hitting the ANN timeout.,-sum
+approximate-nns-distances-computed,operations/second,Rate of distance computations performed during approximate NNS across the content cluster.,-rate
+approximate-nns-nodes-visited,operations/second,Rate of HNSW graph nodes visited during approximate NNS across the content cluster.,-rate
+approximate-nns-query-rate,requests/second,Rate of incoming queries per second that perform an approximate nearest-neighbor search.,
+approximate-nns-time,milliseconds,"Average time spent performing approximate NNS, measured only across queries that actually perform it.","-avg, -max"
+approximate-nns-visit-efficiency,ratio (dimensionless),Ratio of HNSW graph nodes visited to distances computed during approximate NNS; values above 1.0 indicate the filter-first (Acorn-1) heuristic is active.,
+buckets-pending-merge,count,"Number of buckets currently pending merge across distributors, which rises during data redistribution or node recovery.",-sum
+documents-active-total,count,Total number of active documents (primary copies) across all document types and content nodes in the cluster.,
+documents-ready-total,count,Total number of ready (searchable) documents across all document types and content nodes in the cluster.,
+exact-nns-distance-ratio,percentage (0-100),Share of NNS distance computations done via exact (brute-force) search rather than approximate search; high values may indicate the approximate-threshold is set too high.,-pct
+exact-nns-distances-computed,operations/second,Rate of distance computations performed during exact (brute-force) NNS across the content cluster.,-rate
+nns-query-latency,milliseconds,"Average total time (matching plus ranking) spent per query on the content nodes, specific to the NNS/matching pipeline.","-avg, -max"
+non-approximate-nns-query-rate,requests/second,"Rate of incoming queries per second that do not perform approximate NNS, including queries that fall back to exact search.",
+query-matching-time,milliseconds,"Average time spent on the core query matching process, computed as query latency minus query setup time; includes exact NNS.","-avg, -max"
+query-rate,requests/second,Rate of incoming queries per second against the content cluster's document matching pipeline (equal to approximate-nns-query-rate plus non-approximate-nns-query-rate).,
+query-setup-time,milliseconds,"Average time spent setting up and tearing down queries, including filter computation and approximate NNS overhead.","-avg, -max"
+query-setup-time-excl-ann,milliseconds,"Average time spent on query setup and teardown excluding approximate NNS time, isolating filter computation and other overhead.",-avg
+soft-timeout-rate,requests/second,"Rate of queries per second terminated early due to hitting the soft timeout, aggregated across rank profiles.",-sum
+thread-pool-match-utilization,percentage (0-100),Average utilization of the match thread pool across content nodes.,"-avg, -max"
+total-distances-computed,operations/second,"Total NNS distance computations per second, combining exact and approximate search, as an overall indicator of NNS-related CPU cost.",-sum

_data/production-test-metrics/overview.csv

@@ -0,0 +1,26 @@
+family,unit,description,suffixes
+buckets-out-of-sync,percentage (0-100),"Percentage of buckets currently out of sync, fleet-wide.",
+container-nodes-down,percentage (0-100),"Percentage of active container nodes with at least one service down, fleet-wide.",
+container-thread-saturation-document-api-only,percentage (0-100),"Worst-case container thread-pool utilization, fleet-wide, restricted to deployments running document-api only (no search).",
+container-thread-saturation-search-document-api,percentage (0-100),"Worst-case container thread-pool utilization, fleet-wide, restricted to deployments running both search and document-api.",
+container-thread-saturation-search-only,percentage (0-100),"Worst-case container thread-pool utilization, fleet-wide, restricted to deployments running search only (no document-api).",
+content-executor-saturation,percentage (0-100),"Worst-case content-node executor utilization/saturation across the match, docsum, and field-writer executors, fleet-wide.",
+content-groups-nodes-down,count,Fleet-wide count of cluster groups currently reporting at least one down node.,
+core-dumps,count,Total number of core dumps processed across content clusters during the time window.,
+disk-utilization,percentage (0-100),"Maximum disk utilization across nodes in container clusters (logs, working files).","-container, -content"
+documents-per-content-cluster,count,"Maximum number of unique documents stored, aggregated across content clusters.",-max
+feed-blocked,count,"Total number of nodes currently above the feed-block resource limit, across all content clusters.",
+feed-latency,milliseconds,Maximum feed latency observed (OK status only).,-max
+headroom-to-feed-block-per-content-cluster,percentage (0-100),"Smallest remaining headroom before the feed-block limit, across memory and disk, fleet-wide.",
+http-2xx-responses,requests/second,Rate of successful (2xx) HTTP responses across the container cluster.,-sum
+http-4xx-responses,requests/second,Rate of client error (4xx) HTTP responses across the container cluster.,-sum
+http-5xx-responses,requests/second,Rate of server error (5xx) HTTP responses across the container cluster.,-sum
+http-read-latency,milliseconds,"Average HTTP-level read latency, including time to receive the request and send the response.","-avg, -max, -p95, -p99"
+http-write-latency,milliseconds,Maximum HTTP-level write latency observed.,-max
+jvm-heap-pressure,percentage (0-100),Fleet-wide average JVM heap utilization (heap used / heap capacity) across container clusters.,
+memory-utilization-node,percentage (0-100),Maximum OS-level memory utilization across nodes in container clusters.,"-container, -content"
+nodes-alive,boolean (0 or 1),"Whether every node in the deployment is currently reporting a heartbeat (1 if all nodes are alive, 0 if at least one is down or not reporting).",-min
+qos-read-success,percentage (0-100),Percentage of read requests that did not return a 5xx server error.,
+qos-write-success,percentage (0-100),Percentage of write requests that did not return a 5xx server error.,
+query-latency,milliseconds,Average query latency across the container cluster (total latency divided by request count).,"-avg, -max"
+restarts,count,Total number of Vespa service restarts across the fleet during the time window.,