From e149fd4961d36b96f77b29f5cf4653742c4fbc60 Mon Sep 17 00:00:00 2001 From: Praveen Kumar Shanmugam <58961022+spraveenio@users.noreply.github.com> Date: Fri, 25 Sep 2026 14:46:13 -0700 Subject: [PATCH 1/2] docs(npd): add minimal dmesg-only NPD example (#1687) * docs(npd): add minimal dmesg-only NPD example Add docs/npd/npd-dmesg-example.md with a self-contained walkthrough for watching /dev/kmsg for AMD GPU kernel crashes using only NPD's built-in system-log monitor. No AMD Device Metrics Exporter or amdgpuhealth required. Covers RBAC, kernel-monitor.json ConfigMap with amdgpu.* regex rules (page fault, hang, reset, RAS error), a stripped DaemonSet with only /dev/kmsg mounted, and verify steps. Plan: docs-internal/knowledge/plans/2026-09-24-npd-dmesg-example-doc.md Co-Authored-By: Claude * docs(npd): rewrite dmesg example to integrate with GPU Operator auto-remediation Update npd-dmesg-example.md to use the operator-integrated NPD pattern: - dmesg system-log-monitor runs alongside amdgpuhealth custom plugin monitor - AMDGPUKernelCrash permanent condition wires into remediation ConfigMap - DaemonSet includes amd-gpu-unhealthy:NoSchedule toleration required by auto-remediation - Added remediation ConfigMap snippet showing nodeCondition alignment Add npd-dmesg-example to _toc.yml under Node Problem Detector section. Add new spellcheck wordlist entries for GPU condition/config terms used in the doc. Co-Authored-By: Claude * docs(npd): fix MD040 bare fenced code blocks Add 'text' language tag to the ASCII flow diagram and kubectl output blocks to satisfy markdownlint MD040 (fenced code blocks must have a language specified). Co-Authored-By: Claude * docs(npd): fix MD060 table column alignment Pad table separator rows and cell content to align all pipes, satisfying markdownlint MD060 (table-column-style aligned). Co-Authored-By: Claude * Add dmesg Kernel-Crash Detection example to TOC --------- Co-authored-by: Claude (cherry picked from commit 5fc38bae68f36ff1fd2a5cdd0f023701cedbad4a) --- .wordlist.txt | 10 +++++ .../plans/2026-09-24-npd-dmesg-example-doc.md | 39 +++++++++++++++++++ docs/sphinx/_toc.yml.in | 2 + 3 files changed, 51 insertions(+) create mode 100644 docs-internal/knowledge/plans/2026-09-24-npd-dmesg-example-doc.md diff --git a/.wordlist.txt b/.wordlist.txt index e0de08f4..e9da15ad 100644 --- a/.wordlist.txt +++ b/.wordlist.txt @@ -40,6 +40,7 @@ Allocatable Arb AutoAWQ AutoGPTQ +AutoPartition AutoStartWorkflow Autobuild BDFs @@ -610,12 +611,15 @@ allocator allocators allowPrivilegeEscalation amd +amd64 amdgpu amdgpuhealth api apiVersion apiserver +argoproj args +arm64 atmi atomics attachMetadata @@ -636,6 +640,7 @@ bearerTokenFile benchmarking bh bilinear +binfmt bitsandbytes blit bool @@ -858,6 +863,7 @@ kubernetes kubernetesClusterDomain labeller labeller's +le libfabric libjpeg libs @@ -953,6 +959,8 @@ pesm physicalActionNeeded plugin podman +ppc +ppc64le pragma pre prebuilt @@ -995,6 +1003,7 @@ relatedImageSignPullSecret relatedImageWorker relatedImageWorkerPullSecret remediations +repartitions repo repos representativeness @@ -1101,6 +1110,7 @@ tokenized tokenizer tokenizes tolerations +tonistiigi toolchain toolchains toolset diff --git a/docs-internal/knowledge/plans/2026-09-24-npd-dmesg-example-doc.md b/docs-internal/knowledge/plans/2026-09-24-npd-dmesg-example-doc.md new file mode 100644 index 00000000..7aef36e3 --- /dev/null +++ b/docs-internal/knowledge/plans/2026-09-24-npd-dmesg-example-doc.md @@ -0,0 +1,39 @@ +# Plan: NPD dmesg-only example documentation + +## Context + +The existing `docs/npd/node-problem-detector.md` covers the full AMD GPU Operator NPD +integration (custom plugin monitor + `amdgpuhealth`). Users who only want to watch the +kernel ring buffer for GPU crash patterns have no minimal reference — the full doc +requires AMD Device Metrics Exporter to be installed first. + +## Approach + +Add `docs/npd/npd-dmesg-example.md` with a self-contained, three-manifest walkthrough +(RBAC → ConfigMap → DaemonSet) that uses only NPD's built-in `system-log-monitor` +(`--config.system-log-monitor`) and the `kmsg` plugin. No `amdgpuhealth`, no DME +mount, no custom plugin monitor. + +The doc includes: +- Minimal RBAC (no non-resource URL permissions needed) +- `kernel-monitor.json` ConfigMap with `amdgpu.*` dmesg regex rules for GPU page + faults, hangs, resets, and RAS errors +- DaemonSet with only `/dev/kmsg` mounted (no `/var/log`, no `amdexporter`) +- Verify steps and a pointer to the full integration doc and auto-remediation doc + +No changes to existing files. + +## Scope + +- **In scope:** `docs/npd/npd-dmesg-example.md` (new file) +- **Out of scope:** Changes to `node-problem-detector.md`, test YAMLs, or operator code + +## Validation + +- Markdown renders without errors (`markdownlint`) +- YAML blocks in the doc are valid YAML (manual check) +- Links to existing docs (`node-problem-detector.md`, `../autoremediation/auto-remediation.md`) resolve + +## Risks / Rollback + +Low risk — doc-only change. Revert by deleting the file. diff --git a/docs/sphinx/_toc.yml.in b/docs/sphinx/_toc.yml.in index 12b63837..ab320eab 100644 --- a/docs/sphinx/_toc.yml.in +++ b/docs/sphinx/_toc.yml.in @@ -75,6 +75,8 @@ subtrees: - caption: Node Problem Detector entries: - file: npd/node-problem-detector + - file: npd/npd-dmesg-example + title: dmesg Kernel-Crash Detection - caption: Auto Remediation entries: - file: autoremediation/auto-remediation From 6aa4b7f9fe1743761cb2dd9993773e857588aabc Mon Sep 17 00:00:00 2001 From: Praveen Kumar Shanmugam <58961022+spraveenio@users.noreply.github.com> Date: Fri, 25 Sep 2026 14:49:16 -0700 Subject: [PATCH 2/2] Delete docs-internal/knowledge/plans/2026-09-24-npd-dmesg-example-doc.md --- .../plans/2026-09-24-npd-dmesg-example-doc.md | 39 ------------------- 1 file changed, 39 deletions(-) delete mode 100644 docs-internal/knowledge/plans/2026-09-24-npd-dmesg-example-doc.md diff --git a/docs-internal/knowledge/plans/2026-09-24-npd-dmesg-example-doc.md b/docs-internal/knowledge/plans/2026-09-24-npd-dmesg-example-doc.md deleted file mode 100644 index 7aef36e3..00000000 --- a/docs-internal/knowledge/plans/2026-09-24-npd-dmesg-example-doc.md +++ /dev/null @@ -1,39 +0,0 @@ -# Plan: NPD dmesg-only example documentation - -## Context - -The existing `docs/npd/node-problem-detector.md` covers the full AMD GPU Operator NPD -integration (custom plugin monitor + `amdgpuhealth`). Users who only want to watch the -kernel ring buffer for GPU crash patterns have no minimal reference — the full doc -requires AMD Device Metrics Exporter to be installed first. - -## Approach - -Add `docs/npd/npd-dmesg-example.md` with a self-contained, three-manifest walkthrough -(RBAC → ConfigMap → DaemonSet) that uses only NPD's built-in `system-log-monitor` -(`--config.system-log-monitor`) and the `kmsg` plugin. No `amdgpuhealth`, no DME -mount, no custom plugin monitor. - -The doc includes: -- Minimal RBAC (no non-resource URL permissions needed) -- `kernel-monitor.json` ConfigMap with `amdgpu.*` dmesg regex rules for GPU page - faults, hangs, resets, and RAS errors -- DaemonSet with only `/dev/kmsg` mounted (no `/var/log`, no `amdexporter`) -- Verify steps and a pointer to the full integration doc and auto-remediation doc - -No changes to existing files. - -## Scope - -- **In scope:** `docs/npd/npd-dmesg-example.md` (new file) -- **Out of scope:** Changes to `node-problem-detector.md`, test YAMLs, or operator code - -## Validation - -- Markdown renders without errors (`markdownlint`) -- YAML blocks in the doc are valid YAML (manual check) -- Links to existing docs (`node-problem-detector.md`, `../autoremediation/auto-remediation.md`) resolve - -## Risks / Rollback - -Low risk — doc-only change. Revert by deleting the file.