Repository navigation
Build and test with CUDA 13.4.3 - #788
Conversation
jameslamb
added
non-breaking
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
/ok to test 3964091 |
| script: ci/build_python.sh | ||
| # Build a conda package for each CUDA x ARCH x minimum supported Python version | ||
| matrix_filter: group_by({CUDA_VER, ARCH}) | map(min_by(.PY_VER | split(".") | map(tonumber))) | ||
| matrix_type: cuda-13-4-pr |
There was a problem hiding this comment.
Python builds (and only conda Python builds) w/ CUDA 13.4.3 are failing like this:
│ cuda-nvcc-dev_linux-aarch64 ┆ 13.4.92 ┆ h4310d6a_0 ┆ conda-forge ┆ 29.58 KiB │
│ cuda-nvcc-impl ┆ 13.4.92 ┆ h8f3c8d4_0 ┆ conda-forge ┆ 28.47 KiB │
│ cuda-nvcc-tools ┆ 13.4.92 ┆ h8f3c8d4_0 ┆ conda-forge ┆ 30.81 MiB │
│ cuda-nvcc_linux-aarch64 ┆ 13.4.92 ┆ hce25ccd_1 ┆ conda-forge ┆ 28.05 KiB │
│ cuda-nvvm-dev_linux-aarch64 ┆ 13.4.92 ┆ h579c4fd_0 ┆ conda-forge ┆ 28.56 KiB │
│ cuda-nvvm-impl ┆ 13.4.92 ┆ hbe86820_0 ┆ conda-forge ┆ 21.00 MiB │
│ cuda-nvvm-tools ┆ 13.4.92 ┆ hbe86820_0 ┆ conda-forge ┆ 28.22 MiB │
...
│ gcc_impl_linux-aarch64 ┆ 14.4.0 ┆ hcd8d054_7 ┆ conda-forge ┆ 66.60 MiB │
│ gcc_linux-aarch64 ┆ 14.4.0 ┆ hab84b30_2 ┆ conda-forge ┆ 29.42 KiB │
│ gxx_impl_linux-aarch64 ┆ 14.4.0 ┆ h66ee75d_7 ┆ conda-forge ┆ 13.72 MiB │
│ gxx_linux-aarch64 ┆ 14.4.0 ┆ hdf092f9_2 ┆ conda-forge ┆ 27.90 KiB │
...
+CC=aarch64-conda-linux-gnu-cc
+CC_FOR_BUILD=aarch64-conda-linux-gnu-cc
...
+CPP=aarch64-conda-linux-gnu-cpp
...
+GCC=aarch64-conda-linux-gnu-gcc
...
-- The C compiler identification is GNU 14.4.0
-- The CXX compiler identification is GNU 14.4.0
...
sccache: distributed compilation failed and local compile disabled
I downloaded the sccache.log from the URL in those CI logs
[rapids-upload-to-s3] Browse uploads (NVIDIA Employee VPN Required): https://downloads.rapids.ai/ci/ucxx/pull-request/788/3964091/cuda13_aarch64_py312.sccache.log
https://github.com/rapidsai/ucxx/actions/runs/40887650880/job/123670936707#step:17:32
There I see 1 error:
[2026-10-29T19:11:59Z ERROR sccache::server] [ucxx/_lib/CMakeFiles/libucxx.dir/libucxx.cxx.o]: distributed compilation failed: failed to enter toolchain in weak map
There was a problem hiding this comment.
@trxcllnt suggested that an sccache update in our CI images might help, just merged rapidsai/ci-imgs#469
Will re-run this when those are up.
There was a problem hiding this comment.
Turned out that this was a different issue.
Quoting @trxcllnt from our offline conversation:
This is the actual error:
failed to create file `/tmp/conda-bld-output/bld/rattler-build_ucxx/work/.cache/sccache-dist-client/client/weak_map.json`: No such file or directory (os error 2)[9:35 PM]The sccache server is being started by the ucxx recipe build, but it's not being stopped (or restarted) by the ucxx-tests recipe build. Once ucxx is finished, rattler deletes the rattler-build_ucxx/work/ directory, but the sccache daemon from that build is still alive and thinks it should write to that path.
I pushed a9e79a9 to resolve that.
Not sure why it's just showing up now, but the fix is small and low-risk, I think it's ok to pick up.
|
/ok to test a9e79a9 |
|
/merge |
Contributes to rapidsai/build-planning#335