bazel: switch to hermetic_cc_toolchain

[mgeisler]

I've seen spurious out of disk space build failures in CI and I believe they are caused by the huge (5+ GB) LLVM toolchain I introduced in #3204. We've already disabled the Bazel repository cache (#3210), but builds are still failing.

It turns out that Uber maintains a 40 MB toolchain for Bazel. It's based on Zig, which apparently knows how to compile C/C++ as well.

I'm torn on this since it feels a bit experimental to start using Zig to compile C/C++ code... but from a quick test, it seems to work, so maybe I'm overthinking this?

Originally, the LLVM toolchain was introduced to fix

warning: the gold linker is deprecated and has known bugs with Rust

warnings emitted during the build. We can also fix those warnigns by telling rustc to use its own
linker (rust-lld)).

However, I believe this would not be a complete solution as but there are crates in our dependency tree which links with C code. One example is aws-lc-sys:

% cargo tree -i -p cc
cc v1.2.51
└── cmake v0.1.57
    [build-dependencies]
    └── aws-lc-sys v0.35.0
        └── aws-lc-rs v1.15.2
            └── rustls v0.23.31
                └── hyper-rustls v0.27.7
                    └── reqwest v0.13.4
                        └── sync-exercises v0.1.0 (.../src/concurrency/sync-exercises)

Hence, I figured it was better to configure a CC toolchain for the entire Bazel build instead of just telling rustc to do the linking.

Using the toolchain here would continue this path, but with a much smaller toolchain. Since I don't think we want to rely on any particular behavior of our C/C++ compilers, I think this should be fine for us going forward.

Googlers will of course have to let me know if this is broken internally... and/or if there is a better way to go about this!

[mgeisler]

Oh well, this seems incompatible with the macOS build:

[389 / 1,026] Compiling Rust (without process_wrapper) bin process_wrapper (6 files) [for tool]; 8s disk-cache, darwin-sandbox
ERROR: /Users/runner/.bazel/external/rules_rust+/util/process_wrapper/BUILD.bazel:5:36: Compiling Rust (without process_wrapper) bin process_wrapper (6 files) [for tool] failed: (Exit 1): bootstrap_process_wrapper.sh failed: error executing Rustc command (from rust_binary_without_process_wrapper rule target @@rules_rust+//util/process_wrapper:process_wrapper) bazel-out/darwin_arm64-opt-exec/bin/external/rules_rust+/util/process_wrapper/bootstrap_process_wrapper.sh -- ... (remaining 29 arguments skipped)
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
error: linking with `external/hermetic_cc_toolchain++toolchains+zig_config/tools/aarch64-macos-none/c++` failed: exit status: 1
  |
  = note:  "external/hermetic_cc_toolchain++toolchains+zig_config/tools/aarch64-macos-none/c++" "/Users/runner/.bazel/sandbox/darwin-sandbox/2/execroot/_main/bazel-out/darwin_arm64-opt-exec/bin/external/rules_rust+/util/process_wrapper/rustcGuwCHp/symbols.o" "<17 object files omitted>" "/Users/runner/.bazel/execroot/_main/bazel-out/darwin_arm64-opt-exec/bin/external/rules_rust++i+rules_rust_tinyjson/libtinyjson-1317710085.rlib" "<sysroot>/lib/rustlib/aarch64-apple-darwin/lib/{libstd-*,libpanic_unwind-*,libobject-*,libmemchr-*,libaddr2line-*,libgimli-*,libcfg_if-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libminiz_oxide-*,libadler2-*,libunwind-*,liblibc-*,librustc_std_workspace_core-*,liballoc-*,libcore-*,libcompiler_builtins-*}.rlib" "-lSystem" "-lc" "-lm" "-arch" "arm64" "-mmacosx-version-min=11.0.0" "-L" "<sysroot>/lib/rustlib/aarch64-apple-darwin/lib" "-o" "bazel-out/darwin_arm64-opt-exec/bin/external/rules_rust+/util/process_wrapper/process_wrapper" "-Wl,-dead_strip
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: error: unsupported linker arg: -oso_prefix
          
error: aborting due to 1 previous error

So this specific toolchain is probably a non-starter.