Skip to content

feat: NV_ERR_NO_MEMORY on GPU panel (slow journal scan) - #48

Open
MiaAI-Lab wants to merge 1 commit into
mainfrom
feat/nv-err-no-memory
Open

feat: NV_ERR_NO_MEMORY on GPU panel (slow journal scan)#48
MiaAI-Lab wants to merge 1 commit into
mainfrom
feat/nv-err-no-memory

Conversation

@MiaAI-Lab

Copy link
Copy Markdown
Owner

Why this replaces #40

#40 put an NV_ERR_NO_MEMORY counter on CpuPanel and ran journalctl -k | grep -c on the unified-memory poll (~2s). CpuPanel is gone, and a full kernel-journal scan every 2s is not a metric.

What this does

  • Count NVRM NV_ERR_NO_MEMORY lines in the kernel journal since boot
  • Attach it to GPU metrics and show it on the GPU panel when the count is > 0
  • Scan at most once per POLL_INTERVAL_NVERR (default 60s), cached — not on the 2s GPU/memory loop
  • Local Docker uses host nsenter (same as nvidia-smi); remotes use SSH
  • No per-browser localStorage reset; the number is journal-since-boot
  • Parser unit tests only (no --experimental-test-module-mocks)

Test plan

  • node --test server/collectors/__tests__/SystemCollector.nvErr.test.js
  • npm run typecheck
  • On a Spark/host with no NVRM errors: GPU panel unchanged (row hidden)
  • After NV_ERR_NO_MEMORY in journalctl -k: row appears on GPU panel, not a CPU panel

Kernel journal scan is cached off the 2s GPU loop. Count is since boot; the row is hidden at zero. No CPU panel.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant