SCHED-2156: GB300 Dashboards

docs/slurm-exporter.md

@@ -94,7 +94,7 @@ Boolean state flag label convention:
 
 | Metric Name & Type | Description & Labels |
 |-------------------|---------------------|
-| **slurm_node_info**<br>*Gauge* | Provides detailed information about SLURM nodes<br><br>**Labels:**<br>• `node_name` - Name of the SLURM node<br>• `instance_id` - Kubernetes instance identifier<br>• `state_base` - Base node state (IDLE, ALLOCATED, DOWN, ERROR, MIXED, UNKNOWN)<br>• `state_is_drain` - Whether node is in drain state ("true"/"false")<br>• `state_is_maintenance` - Whether node is in maintenance state ("true"/"false")<br>• `state_is_reserved` - Whether node is in reserved state ("true"/"false")<br>• `state_is_completing` - Whether node is in completing state ("true" or empty)<br>• `state_is_fail` - Whether node is in fail state ("true" or empty)<br>• `state_is_planned` - Whether node is in planned state ("true" or empty)<br>• `state_is_not_responding` - Whether the node is marked as not responding ("true" or empty)<br>• `state_is_invalid` - Whether the node state is considered invalid by SLURM ("true" or empty)<br>• `state_is_cloud` - Whether the node is a cloud node powered on/off dynamically via Slurm power saving ("true" or empty)<br>• `state_is_power_down` - Whether the node is pending power down ("true" or empty)<br>• `state_is_power_drain` - Whether the node is draining as part of power down ("true" or empty)<br>• `state_is_powered_down` - Whether the node is currently powered down ("true" or empty)<br>• `state_is_powering_down` - Whether the node is transitioning to powered down ("true" or empty)<br>• `state_is_powering_up` - Whether the node is transitioning to powered up ("true" or empty)<br>• `state_is_power_up` - Whether the node is pending power up ("true" or empty)<br>• `is_unavailable` - Computed by the exporter: "true" when the node is considered unavailable (DOWN+* or IDLE+DRAIN+*), empty string otherwise. Power-managed nodes (powering up/down or powered down) are never reported as unavailable<br>• `reservation_name` - Reservation that currently includes the node (trimmed to 50 characters)<br>• `address` - IP address of the node<br>• `reason` - Reason for current node state (empty string if node has no reason set)<br>• `comment` - Comment set on the node (e.g., by active checks when GPU health check fails) |
+| **slurm_node_info**<br>*Gauge* | Provides detailed information about SLURM nodes<br><br>**Labels:**<br>• `node_name` - Name of the SLURM node<br>• `instance_id` - Kubernetes instance identifier<br>• `nodeset_name` - Name of the Soperator NodeSet from the worker Pod's `slurm.nebius.ai/nodeset` label, including for pending Pods; empty when the worker Pod is unavailable<br>• `state_base` - Base node state (IDLE, ALLOCATED, DOWN, ERROR, MIXED, UNKNOWN)<br>• `state_is_drain` - Whether node is in drain state ("true"/"false")<br>• `state_is_maintenance` - Whether node is in maintenance state ("true"/"false")<br>• `state_is_reserved` - Whether node is in reserved state ("true"/"false")<br>• `state_is_completing` - Whether node is in completing state ("true" or empty)<br>• `state_is_fail` - Whether node is in fail state ("true" or empty)<br>• `state_is_planned` - Whether node is in planned state ("true" or empty)<br>• `state_is_not_responding` - Whether the node is marked as not responding ("true" or empty)<br>• `state_is_invalid` - Whether the node state is considered invalid by SLURM ("true" or empty)<br>• `state_is_cloud` - Whether the node is a cloud node powered on/off dynamically via Slurm power saving ("true" or empty)<br>• `state_is_power_down` - Whether the node is pending power down ("true" or empty)<br>• `state_is_power_drain` - Whether the node is draining as part of power down ("true" or empty)<br>• `state_is_powered_down` - Whether the node is currently powered down ("true" or empty)<br>• `state_is_powering_down` - Whether the node is transitioning to powered down ("true" or empty)<br>• `state_is_powering_up` - Whether the node is transitioning to powered up ("true" or empty)<br>• `state_is_power_up` - Whether the node is pending power up ("true" or empty)<br>• `is_unavailable` - Computed by the exporter: "true" when the node is considered unavailable (DOWN+* or IDLE+DRAIN+*), empty string otherwise. Power-managed nodes (powering up/down or powered down) are never reported as unavailable<br>• `reservation_name` - Reservation that currently includes the node (trimmed to 50 characters)<br>• `address` - IP address of the node<br>• `reason` - Reason for current node state (empty string if node has no reason set)<br>• `comment` - Comment set on the node (e.g., by active checks when GPU health check fails) |
 | **slurm_node_gpu_seconds_total**<br>*Counter* | Total GPU seconds accumulated on SLURM nodes<br><br>**Labels:**<br>• `node_name` - Name of the SLURM node<br>• `state_base` - Base node state<br>• `state_is_drain` - Drain state flag<br>• `state_is_maintenance` - Maintenance state flag<br>• `state_is_reserved` - Reserved state flag |
 | **slurm_node_fails_total**<br>*Counter* | Total number of node state transitions to failed states (DOWN/DRAIN). Power-managed nodes (powering up/down or powered down) are excluded, since their DOWN/DRAIN states are part of the normal cloud lifecycle<br><br>**Labels:**<br>• `node_name` - Name of the SLURM node<br>• `state_base` - Base node state at time of failure<br>• `state_is_drain` - Drain state flag<br>• `state_is_maintenance` - Maintenance state flag<br>• `state_is_reserved` - Reserved state flag<br>• `reason` - Reason for the node failure |
 | **slurm_node_unavailability_duration_seconds**<br>*Histogram* | Duration of completed node unavailability events (DOWN+* or IDLE+DRAIN+*). Power-managed nodes (powering up/down or powered down) are excluded, since their DOWN/DRAIN states are part of the normal cloud lifecycle<br><br>**Labels:**<br>• `node_name` - Name of the SLURM node<br><br>**Note:** Observations are recorded when unavailability events complete. Duration tracking is reset on exporter restarts, which may affect accuracy |
@@ -108,9 +108,9 @@ Boolean state flag label convention:
 | **slurm_node_memory_free_bytes**<br>*Gauge* | Free memory on the node in bytes<br><br>**Labels:**<br>• `node_name` - Name of the SLURM node |
 | **slurm_node_memory_effective_bytes**<br>*Gauge* | Effective memory on the node in bytes (total minus specialized memory reserved for system daemons)<br><br>**Labels:**<br>• `node_name` - Name of the SLURM node |
 | **slurm_node_partition**<br>*Gauge* | Maps nodes to their partitions, enabling partition-level aggregation via PromQL joins<br><br>**Labels:**<br>• `node_name` - Name of the SLURM node<br>• `partition` - Name of the SLURM partition |
-| **slurm_node_nvlink_instance_group**<br>*Gauge* | Maps Slurm nodes to their NVLink instance group. Emitted only for nodes with the `topology.nebius.com/nvl-instance-group-id` Kubernetes label.<br><br>**Labels:**<br>• `node_name` - Name of the Slurm node and worker Pod<br>• `instance_id` - Compute instance identifier reported by Slurm<br>• `nvlink_instance_group` - Value of the Kubernetes Node's `topology.nebius.com/nvl-instance-group-id` label<br>• `nodeset_name` - Value of the Kubernetes Node's `slurm.nebius.ai/nodeset-name` label |
+| **slurm_node_nvlink_instance_group**<br>*Gauge* | Maps Slurm nodes to their NVLink instance group. Emitted only for nodes with the `topology.nebius.com/nvl-instance-group-id` Kubernetes label.<br><br>**Labels:**<br>• `node_name` - Name of the Slurm node and worker Pod<br>• `instance_id` - Compute instance identifier reported by Slurm<br>• `nvlink_instance_group` - Value of the Kubernetes Node's `topology.nebius.com/nvl-instance-group-id` label<br>• `nodeset_name` - Name of the Soperator NodeSet from the worker Pod's `slurm.nebius.ai/nodeset` label |
 | **slurm_job_info**<br>*Gauge* | Detailed information about SLURM jobs<br><br>**Labels:**<br>• `job_id` - SLURM job identifier<br>• `job_state` - Current job state (PENDING, RUNNING, COMPLETED, FAILED, etc.)<br>• `job_state_reason` - Reason for current job state<br>• `slurm_partition` - SLURM partition name<br>• `job_name` - User-defined job name<br>• `user_name` - Username who submitted the job<br>• `user_id` - Numeric user ID who submitted the job<br>• `standard_error` - Path to stderr file<br>• `standard_output` - Path to stdout file<br>• `array_job_id` - Array job ID (if applicable)<br>• `array_task_id` - Array task identity. Either a single task index (e.g. `3`) for an exploded task or a range expression (e.g. `1-5`, `1,3,5-9:2`) for a collapsed array master record on the accounting path. Empty for non-array jobs.<br>• `submit_time` - When the job was submitted (Unix timestamp seconds, empty if not available or zero)<br>• `start_time` - When the job started execution (Unix timestamp seconds, empty if not available or zero)<br>• `end_time` - When the job completed (Unix timestamp seconds, empty if not available or zero). **Warning:** For non-terminal states like RUNNING, this may contain a future timestamp representing the forecasted end time based on the job's time limit<br>• `finished_time` - When the job actually finished for terminal states only (Unix timestamp seconds, empty for non-terminal states or if end_time is zero). Unlike `end_time`, this field only contains actual completion times, never forecasted values |
-| **slurm_node_job**<br>*Gauge* | Mapping between jobs and the nodes they're running on<br><br>**Labels:**<br>• `job_id` - SLURM job identifier<br>• `node_name` - Name of the node running the job<br>• `nvlink_instance_group` - NVLink instance group of the node, or empty when unavailable<br>• `nodeset_name` - Slurm NodeSet name of the node, or empty when unavailable |
+| **slurm_node_job**<br>*Gauge* | Mapping between jobs and the nodes they're running on<br><br>**Labels:**<br>• `job_id` - SLURM job identifier<br>• `node_name` - Name of the node running the job<br>• `nvlink_instance_group` - NVLink instance group of the node, or empty when unavailable<br>• `nodeset_name` - Name of the Soperator NodeSet from the worker Pod's `slurm.nebius.ai/nodeset` label, or empty when unavailable |
 | **slurm_job_duration_seconds**<br>*Gauge* | Job duration in seconds. For running jobs, this is the time elapsed since the job started. For completed jobs, this is the total execution time.<br><br>**Labels:**<br>• `job_id` - SLURM job identifier<br><br>**Notes:**<br>• Only exported for jobs with a valid start time<br>• For non-terminal states (RUNNING, etc.): duration = current_time - start_time<br>• For terminal states (COMPLETED, FAILED, etc.): duration = end_time - start_time (only if end_time is valid) |
 | **slurm_job_cpus**<br>*Gauge* | Number of CPUs allocated to the job<br><br>**Labels:**<br>• `job_id` - SLURM job identifier |
 | **slurm_job_memory_bytes**<br>*Gauge* | Memory allocated to the job in bytes<br><br>**Labels:**<br>• `job_id` - SLURM job identifier |

helm/soperator-fluxcd/values.yaml

@@ -470,6 +470,8 @@ observability:
           requests:
             cpu: 100m
       grafana:
+        image:
+          tag: "11.6.14-security-04"
         nodeSelector:
           slurm.nebius.ai/nodeset: system
         affinity: