Problem Description
I get a randomly in our CI the error Memory access fault by GPU node-4, this is in very simple application with alpaka3. Currently I do not have a native HIP reproducer because I am not able to trigger the issue locally. It is most likely triggered because 8 runners using the smae GPU in parallel. Neve rthe less this should not happen.
Operating System
Ubuntu 24.04.4 LTS
CPU
AMD EPYC 7351 16-Core Processor
GPU
AMD Radeon PRO W6800 - gfx1030
ROCm Version
7.0.0 (but also 6.4+ up to 7.2 randomly)
ROCm Component
HIP
Steps to Reproduce
Let me post here for now the alpak code, this should be understandable and with the ROCM VERBOSE lock easy to see that it is crashing after the stream is created and before the memory allocation happen.
TEMPLATE_LIST_TEST_CASE("memory using std::vector", "[docs]", docs::test::TestBackends)
{
auto computeDevSpec = alpaka::onHost::DeviceSpec{alpaka::api::hip, alpaka::deviceKind::amdGpu};
auto computeDevSelector = alpaka::onHost::makeDeviceSelector(computeDevSpec);
if(!computeDevSelector.isAvailable())
return;
onHost::Device computeDev = computeDevSelector.makeDevice(0);
onHost::Queue asyncComputeQueue = computeDev.makeQueue();
onHost::SharedBuffer computeBuffer = onHost::alloc<int>(computeDev, 10);
// BEGIN-TUTORIAL-stdVector
// use std::vector instead of an alpaka view
std::vector stdVec = std::vector<int>(10, 0);
onHost::fill(asyncComputeQueue, computeBuffer, 42);
onHost::memcpy(asyncComputeQueue, stdVec, computeBuffer);
// END-TUTORIAL-stdVector
onHost::wait(asyncComputeQueue);
// check that the data is valid
for(auto const& v : stdVec)
CHECK(v == 42);
}log
:3:rocdevice.cpp :480 : 277859969612 us: [pid:20423 tid: 0x7cd396980f80] Initalizing runtime stack, Enumerated GPU agents = 1
:3:rocdevice.cpp :233 : 277859969683 us: [pid:20423 tid: 0x7cd396980f80] Numa selects cpu agent[3]=0x27900830(fine=0x27900a40,coarse=0x27901580) for gpu agent=0x27935e10 CPU<->GPU XGMI=0
:3:rocsettings.cpp :282 : 277859969701 us: [pid:20423 tid: 0x7cd396980f80] Using dev kernel arg wa = 0
:3:comgrctx.cpp :127 : 277859977901 us: [pid:20423 tid: 0x7cd396980f80] Loaded COMGR library version 3.0.
:3:rocdevice.cpp :1677: 277859978468 us: [pid:20423 tid: 0x7cd396980f80] Gfx Major/Minor/Stepping: 10/3/0
:3:rocdevice.cpp :1679: 277859978475 us: [pid:20423 tid: 0x7cd396980f80] HMM support: 1, XNACK: 0, Direct host access: 0
:3:rocdevice.cpp :1681: 277859978480 us: [pid:20423 tid: 0x7cd396980f80] Max SDMA Read Mask: 0x3, Max SDMA Write Mask: 0x3
:4:rocdevice.cpp :2113: 277859979466 us: [pid:20423 tid: 0x7cd396980f80] Allocate hsa host memory 0x7cd294e00000, size 0x400000, numa_node = 3, mem_seg = 0
:4:rocdevice.cpp :2113: 277859979625 us: [pid:20423 tid: 0x7cd396980f80] Allocate hsa host memory 0x7cd396975000, size 0x38, numa_node = 3, mem_seg = 1
:3:runtime.cpp :82 : 277859979655 us: [pid:20423 tid: 0x7cd396980f80] ROCclr version: a3e329ad8
:3:hip_context.cpp :56 : 277859979668 us: [pid:20423 tid: 0x7cd396980f80] HIP Version: 7.0.51831.a3e329ad8, Direct Dispatch: 1
:3:os_posix.cpp :966 : 277859979679 us: [pid:20423 tid: 0x7cd396980f80] HIP Library Path: /opt/rocm-7.0.0/lib/libamdhip64.so.7
:3:hip_device_runtime.cpp :702 : 277859979732 us: [pid:20423 tid: 0x7cd396980f80] hipGetDeviceCount ( 0x7ffd482e5898 )
:3:hip_device_runtime.cpp :704 : 277859979747 us: [pid:20423 tid: 0x7cd396980f80] hipGetDeviceCount: Returned hipSuccess :
:3:hip_device_runtime.cpp :702 : 277859979754 us: [pid:20423 tid: 0x7cd396980f80] hipGetDeviceCount ( 0x7ffd482e5658 )
:3:hip_device_runtime.cpp :704 : 277859979759 us: [pid:20423 tid: 0x7cd396980f80] hipGetDeviceCount: Returned hipSuccess :
:3:hip_error.cpp :36 : 277859979765 us: [pid:20423 tid: 0x7cd396980f80] hipGetLastError ( )
:3:hip_device.cpp :656 : 277859979772 us: [pid:20423 tid: 0x7cd396980f80] hipGetDevicePropertiesR0600 ( 0x7ffd482e5000, 0 )
:3:hip_device.cpp :658 : 277859979779 us: [pid:20423 tid: 0x7cd396980f80] hipGetDevicePropertiesR0600: Returned hipSuccess :
:3:hip_error.cpp :36 : 277859979785 us: [pid:20423 tid: 0x7cd396980f80] hipGetLastError ( )
:3:hip_device_runtime.cpp :717 : 277859979790 us: [pid:20423 tid: 0x7cd396980f80] hipSetDevice ( 0 )
:3:hip_device_runtime.cpp :721 : 277859979794 us: [pid:20423 tid: 0x7cd396980f80] hipSetDevice: Returned hipSuccess :
:3:hip_error.cpp :36 : 277859979803 us: [pid:20423 tid: 0x7cd396980f80] hipGetLastError ( )
:3:hip_device_runtime.cpp :717 : 277859979813 us: [pid:20423 tid: 0x7cd396980f80] hipSetDevice ( 0 )
:3:hip_device_runtime.cpp :721 : 277859979816 us: [pid:20423 tid: 0x7cd396980f80] hipSetDevice: Returned hipSuccess :
:3:hip_error.cpp :36 : 277859979820 us: [pid:20423 tid: 0x7cd396980f80] hipGetLastError ( )
:3:hip_stream.cpp :271 : 277859979833 us: [pid:20423 tid: 0x7cd396980f80] hipStreamCreateWithFlags ( 0x279eee90, 1 )
:3:rocdevice.cpp :2967: 277859979846 us: [pid:20423 tid: 0x7cd396980f80] Number of allocated hardware queues with low priority: 0, with normal priority: 0, with high priority: 0, maximum per priority is: 4
:3:rocdevice.cpp :3048: 277859996147 us: [pid:20423 tid: 0x7cd396980f80] Created SWq=0x7cd39695c000 to map on HWq=0x7cd294400000 with size 16384 with priority 1, cooperative: 0
:3:rocdevice.cpp :3141: 277859996170 us: [pid:20423 tid: 0x7cd396980f80] acquireQueue refCount: 0x7cd294400000 (1)
:4:rocdevice.cpp :2113: 277859996703 us: [pid:20423 tid: 0x7cd396980f80] Allocate hsa host memory 0x7cd294200000, size 0x100000, numa_node = 3, mem_seg = 2
:1:rocdevice.cpp :3434: 277860058315 us: [pid:20423 tid: 0x7cd3965ff6c0] Memory Fault Error
Memory access fault by GPU node-4 (Agent handle: 0x27935e10) on address 0x76bbced3c000. Reason: Page not present or supervisor privilege.
:3:devprogram.cpp :2621: 277860368910 us: [pid:20423 tid: 0x7cd396980f80] Using Code Object V5.
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
rocminfo --support output
/opt/rocm/bin/rocminfo
ROCk module version 6.19.14.31400000 is loaded
=====================
HSA System Attributes
=====================
Runtime Version: 1.18
Runtime Ext Version: 1.11
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
Mwaitx: DISABLED
XNACK enabled: NO
DMAbuf Support: YES
VMM Support: YES
==========
HSA Agents
==========
*******
Agent 1
*******
Name: AMD EPYC 7351 16-Core Processor
Uuid: CPU-XX
Marketing Name: AMD EPYC 7351 16-Core Processor
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 2400
BDFID: 0
Internal Node ID: 0
Compute Unit: 8
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Memory Properties:
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 16298844(0xf8b35c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 16298844(0xf8b35c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 16298844(0xf8b35c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 4
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 16298844(0xf8b35c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:
*******
Agent 2
*******
Name: AMD EPYC 7351 16-Core Processor
Uuid: CPU-XX
Marketing Name: AMD EPYC 7351 16-Core Processor
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 1
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 2400
BDFID: 0
Internal Node ID: 1
Compute Unit: 8
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Memory Properties:
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 16506444(0xfbde4c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 16506444(0xfbde4c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 16506444(0xfbde4c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 4
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 16506444(0xfbde4c) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:
*******
Agent 3
*******
Name: AMD EPYC 7351 16-Core Processor
Uuid: CPU-XX
Marketing Name: AMD EPYC 7351 16-Core Processor
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 2
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 2400
BDFID: 0
Internal Node ID: 2
Compute Unit: 8
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Memory Properties:
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 16506448(0xfbde50) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 16506448(0xfbde50) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 16506448(0xfbde50) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 4
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 16506448(0xfbde50) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:
*******
Agent 4
*******
Name: AMD EPYC 7351 16-Core Processor
Uuid: CPU-XX
Marketing Name: AMD EPYC 7351 16-Core Processor
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 3
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 2400
BDFID: 0
Internal Node ID: 3
Compute Unit: 8
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Memory Properties:
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 16500024(0xfbc538) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 16500024(0xfbc538) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 16500024(0xfbc538) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 4
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 16500024(0xfbc538) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:
*******
Agent 5
*******
Name: gfx1030
Uuid: GPU-77981301ef789fe2
Marketing Name: AMD Radeon PRO W6800
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 4
Device Type: GPU
Cache Info:
L1: 16(0x10) KB
L2: 4096(0x1000) KB
L3: 131072(0x20000) KB
Chip ID: 29603(0x73a3)
ASIC Revision: 1(0x1)
Cacheline Size: 128(0x80)
Max Clock Freq. (MHz): 2555
BDFID: 25344
Internal Node ID: 4
Compute Unit: 60
SIMDs per CU: 2
Shader Engines: 4
Shader Arrs. per Eng.: 2
WatchPts on Addr. Ranges:4
Coherent Host Access: FALSE
Memory Properties:
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 32(0x20)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 2147483647(0x7fffffff)
y 65535(0xffff)
z 65535(0xffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 134
SDMA engine uCode:: 88
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 31440896(0x1dfc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 31440896(0x1dfc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Recommended Granule:0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx1030
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 2147483647(0x7fffffff)
y 65535(0xffff)
z 65535(0xffff)
FBarrier Max Size: 32
ISA 2
Name: amdgcn-amd-amdhsa--gfx10-3-generic
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 2147483647(0x7fffffff)
y 65535(0xffff)
z 65535(0xffff)
FBarrier Max Size: 32
Additional Information
The job whcih crashed where I posted the log from use dthe ubuntu 24.04 container and installed ROCM via apt.
Using Docker executor with image ubuntu:24.04 ...
Using effective pull policy of [always] for container ubuntu:24.04
Pulling docker image ubuntu:24.04 ...
Using docker image sha256:ef91e4b15da8323a1523adb2b371998dcd3063dae8553cc2744c178ccc065bc4 for ubuntu:24.04 with
sudo DEBIAN_FRONTEND=noninteractive apt install --no-install-recommends -y rocm-llvm7.0.0 hip-runtime-amd7.0.0 rocm-dev7.0.0 rocm-utils7.0.0 rocrand-dev7.0.0 rocminfo7.0.0 rocm-cmake7.0.0 rocm-device-libs7.0.0 rocm-core7.0.0 rocm-smi-lib7.0.0
Problem Description
I get a randomly in our CI the error
Memory access fault by GPU node-4, this is in very simple application with alpaka3. Currently I do not have a native HIP reproducer because I am not able to trigger the issue locally. It is most likely triggered because 8 runners using the smae GPU in parallel. Neve rthe less this should not happen.Operating System
Ubuntu 24.04.4 LTS
CPU
AMD EPYC 7351 16-Core Processor
GPU
AMD Radeon PRO W6800 - gfx1030
ROCm Version
7.0.0 (but also 6.4+ up to 7.2 randomly)
ROCm Component
HIP
Steps to Reproduce
Let me post here for now the alpak code, this should be understandable and with the ROCM VERBOSE lock easy to see that it is crashing after the stream is created and before the memory allocation happen.
log
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
rocminfo --support output
Additional Information
The job whcih crashed where I posted the log from use dthe ubuntu 24.04 container and installed ROCM via apt.