Skip to content

GPU count includes amd-smi warning lines, inflating MAD_SYSTEM_NGPUS #191

Description

@mkuznet1

rocm_tool_manager.py:281 counts GPUs with amd-smi list --csv | tail -n +3 | wc -l. Without the render/video groups amd-smi prints one WARNING per GPU on stdout, and those lines get counted:

WARNING: User is missing the following required groups: render, video.   (x9)
$ amd-smi list --csv | tail -n +3 | wc -l   # node has 8 GPUs
17

Fails tests/integration/test_gpu_management.py:492 (test_renderD_count_matches_gpu_count): assert 8 == 17.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions