tools, ci, hf: problems found on the side of the DGX Spark work, by severity #524

These are problems I ran into while doing other work on DGX Sparks and on an x86_64 machine. None of them was what I was working on at the time, so I collect them here instead of opening an issue for each one. Each item stands on its own and can be split out, fixed or closed by itself. I checked every item at fda5eca2, current main, and the line links below point at that commit.

How they are sorted. High would give people running bloomery a wrong answer or a refused load, or block a contributor step with no workaround; nothing here is at that level. Medium is a contributor tool or gate that stops or gives a wrong verdict every time under the conditions given, on a machine set up the way the contributor docs describe, with a workaround. Low is test coverage, failures that need a setup the contributor docs do not ask for or a loaded machine, and one-line fixes.

High

None.

Medium

• The t0 check in CI can die with SIGILL: target/ is cached under a key with no CPU in it, but built with -C target-cpu=native.

  tools/contrib-check.sh exports RUSTFLAGS='-C target-cpu=native' on x86_64 Linux when RUSTFLAGS is unset (contrib-check.sh:148-152). The PR workflow caches the whole target directory under pr-check-${{ runner.os }}-<hash of Cargo.lock and rust-toolchain.toml> (pr.yml:55-66). Cargo's fingerprint holds the RUSTFLAGS text, not the CPU that native resolved to, so an artifact built on a CPU with AVX-512 or GFNI counts as fresh on a CPU without them. The runner pool is mixed (the logs show EPYC 7763, 9V74 and 9V45, and Xeon 8370C, 8573C and 6973P).

  What I saw. The 146c1302 push run saved the new cache on an EPYC 9V74 (run 39344389990). The regex and gzip-fixture test paths (Target::parse, quant_config.rs:230-238, and the gzipped fixtures read by quant_config::tests::fixture) already existed before that commit. Every run since then restores that cache, and the bloomery-hf unit test binary dies with SIGILL, no panic text, on some runners and passes on others with the same merge code and the same restored bytes:

  run, attempt	head	runner CPU	bloomery-hf tests
  39350331912, 1 (gpu: refuse misaligned vector views at launch #520)	5f836760	EPYC 7763	SIGILL
  39350935430, 1 (gguf, gpu, placement: MXFP8 weights (E4M3 blocks of 32 with an E8M0 scale) in a gemv and the F32 tile, bloomery's own types named with rows #521)	0be9b34f	EPYC 9V74	SIGILL
  39350331912, 2 (gpu: refuse misaligned vector views at launch #520, rerun)	5f836760	EPYC 9V45	pass
  39350935430, 2 (gguf, gpu, placement: MXFP8 weights (E4M3 blocks of 32 with an E8M0 scale) in a gemv and the F32 tile, bloomery's own types named with rows #521, rerun)	0be9b34f	Xeon 8573C	pass

  Both PRs went red in t0 and green on a rerun. gpu: refuse misaligned vector views at launch #520 touches only crates/gpu and tools/unsafe-ratchet.txt, so neither PR is the cause. Both PRs passed on rerun without a head change. Cross-CPU cache reuse is a suspected shared cause; the 9V74 failure remains unexplained. Of the 57 earlier runs listed in the run history I checked, 55 restored the previous cache.

  Reproduction on an x86_64 machine with a Ryzen 5 PRO 6650H (Zen 3+, no AVX-512, no GFNI), at 146c1302: build cargo test -p bloomery-hf --lib --no-run with RUSTFLAGS='-C target-cpu=native', but compile flate2, miniz_oxide, adler2, simd_adler32 and crc32fast with -C target-cpu=znver4 (what a 9V74 runner's native is), and run the test binary. With --test-threads=4 it completes the same 34 test names as the CI logs before SIGILL (the default thread count completes 26), and gdb stops in miniz_oxide::inflate::core::init_tree at vgf2p8affineqb (GFNI), called from quant_config::tests::fixture. Compiling regex, regex_automata, regex_syntax, aho_corasick and memchr for znver4 instead stops in regex::builders::Builder::default at vmovdqu64 on a zmm register (AVX-512), called from Target::parse. Built natively on the same machine, 146c1302 and gpu: refuse misaligned vector views at launch #520's head passed 180 of 180 runs at the default thread count, 4 threads and 1 thread. No failures were observed in the 180 native runs. Neither the run on the 7763 nor the one on the 9V74 could be replayed here, because cache contents cannot be downloaded. The gguf, gpu, placement: MXFP8 weights (E4M3 blocks of 32 with an E8M0 scale) in a gemv and the F32 tile, bloomery's own types named with rows #521 crash on a 9V74 is not explained by the CPU model, since that model has the features. Either that VM exposes fewer CPUID bits, or the faulting object comes from the older cache. The logs print only the CPU model, so I could not tell which. The fix below covers both.

  Where it reaches. Without --target, RUSTFLAGS also applies to build scripts and proc macros. A cached native build script or proc-macro library from a wider CPU can stop the build itself. In the cross-CPU build above, generic-array's build_script_build died with SIGILL.

  Fix direction. The simplest is to make CI artifacts CPU-independent: set a fixed level such as -C target-cpu=x86-64-v3 on the contrib-check step in pr.yml (the script keeps a preset RUSTFLAGS, contrib-check.sh:148-149), and put the same value into the cache key and the restore-keys prefix. Without the key change, an exact hit on the old key restores the native-built target, cargo rebuilds everything because the flags text changed, and nothing is saved again until Cargo.lock changes. Every CPU in the pool above supports x86-64-v3. The alternative is to keep native and put a hash of the CPU flags (the flags line of /proc/cpuinfo) into the key, at the cost of one cache of about 1.3 GB per CPU type, which is close to the repository's cache limit with six types. Printing the CPU feature flags in the contrib-check header would let the next failure of this kind be read from the log. Until the key changes, deleting the cache pr-check-Linux-08a9a91c... with gh cache delete helps only until the next save on a wide CPU.

  docs/BUILD.md ("The CPU flag") and the comment at contrib-check.sh:26-28 already describe an illegal instruction on another CPU for built binaries. This item is the same class in the CI cache.

• drive_dev in tools/ref/drive-read.sh reads the wrong /sys/block path inside a container, so the depth and nvtier runners stop before they time anything.

  drive_dev takes findmnt -n -o SOURCE --target <dir> as the device (drive-read.sh:13). When the model directory is a bind mount, findmnt prints the bind root after the device, for example /dev/nvme0n1p2[/models]. lsblk -n -o PKNAME fails on that string, the fallback basename gives models] (drive-read.sh:14-15), and the next line refuses with no counters at /sys/block/models]/stat. Both callers exit 2 on it: depth-qwen3moe.sh:2847 and nvtier-read.sh:219.

  Reproduction, inside a container on a DGX Spark where the model folder is a bind mount of a directory on the internal NVMe drive: source tools/ref/drive-read.sh; drive_dev "$BLOOMERY_REF_MODEL". It printed the error above on two separate measurement setups, before any timing started (no counters at /sys/block/models]/stat with mount source /dev/nvme0n1p2[/home/<user>/bloomery-port/models], and the same message for a fixtures directory). No timing result was affected. A task-local wrapper that calls findmnt --nofsroot and then reads KNAME and PKNAME (here partition nvme0n1p2, disk nvme0n1) let those runs continue. I did not rerun the unmodified script at fda5eca2; the error comes from the earlier runs, and the line references are from reading fda5eca2.

  Fixed in tools: resolve bind-mounted model drives through the shared reader #604: one drive_source reads the mount with findmnt --nofsroot, and drive_dev resolves PKNAME, then KNAME.

• gate_nvtier loads its all-RAM arm while the paged arm still holds its RAM arena, so on a 128 GB host the all-RAM arm is refused although it fits there alone.

  The gate loads the paged arm first and the all-RAM arm after it (gate_nvtier.rs:1999-2013). The paged arm's tier keeps its arena (23,494,107,136 B for Qwen3.8 UD-Q4_K_XL) because later clauses read it, so when GpuModel::load_placed reads the host's room for the all-RAM arm, those bytes are already gone. The placed load itself counts its need once, before any upload; the order is what costs the room. The order came in with 7713836 (nvtier R2b), and the ds41 and glm5next nvtier gates already load the all-RAM arm first. The same gate's row in tools/gate-batch-resources.tsv:39 declares 92,054,154,240 B, under the 105,818,045,440 B the all-RAM arm needs with its churn pool.

  Reproduction, on an RTX 5090 machine with 128 GB of RAM and nothing else running: BLOOMERY_BOX_LOCAL=1 just gate-gpu-nvtier stops at the all-RAM arm with the host has 102243987456 B available ... under the 105818045440 B this load needs. With the all-RAM arm loaded first, the same command ends with gate_nvtier: every clause passed.

  Fixed in gate_nvtier: load the all-RAM arm before the paged arm, and declare its real host need #633: the all-RAM arm loads first, and the resources row states the arm's real need.

Low

• A compressed-tensors class-name target other than Linear is read as a module name, and Linear matches every module.

  Target::parse makes re: a regex, the exact string Linear Target::Linear, and everything else Target::Name, which matches a module only when the names are equal (quant_config.rs:230-246). Target::Linear returns true for every module (quant_config.rs:243). The compressed-tensors format names a target by module name, re: pattern or module class name, and this reader has no class information.

  Source inspection at fda5eca2 shows what a synthetic plan input would do with a packed INT4 tensor set for model.embed_tokens and a pack-quantized config. With targets: ["Embedding"], no target matches the module, so the planner takes the refusal path that says the config leaves model.embed_tokens unquantized (quant_config.rs:653). With targets: ["Linear"], every module matches, so the embedding would be treated as a linear weight. I did not run this; it is a reading of the code, not a test of any supported model.

  So a class name would not load a weight silently, but the refusal says the config leaves a tensor unquantized when the config declares it, and Linear is wider than its name. The only test with a compressed-tensors Linear target uses tensors that are all linear (quant_config.rs:1250). Packed non-linear weights under a Linear target are unusual in practice, which is why this is low.

  Fixed in hf: resolve compressed-tensors targets by checkpoint module class #605: Linear and Embedding resolve through the checkpoint's module classes, and a class name the reader does not know is refused by name.

Hardware

• The CI results are from GitHub Actions runs on ubuntu-latest; the CPUs are the ones printed in the run logs.
• The SIGILL reproduction: an x86_64 Linux machine with a Ryzen 5 PRO 6650H (Zen 3+, no AVX-512, no GFNI), nightly-2026-08-28 as pinned.
• The drive_dev failure: one DGX Spark (GB10) inside its container, model folder on the internal NVMe drive through a bind mount.
• The class-target item comes from reading the source at fda5eca2; no hardware needed.

AI help

AI 코딩 에이전트가 문제를 조사하고 재현해 이슈 초안을 썼습니다. 이슈 초안 리뷰는 제가 직접 했습니다.