queue: tflite-runtime 2.14.0 - green on PR #2154; skills: gotchas 494-497

Both riscv64 legs built and tested on the first CI cycle, and the publish job
dry-ran clean.

The four gotchas are the reusable half of that port: a hermetic Python that
predates riscv64 supplies both the Python and the numpy headers, so two
repositories have to be stood in (494); a Bazel port's loading phase, including
those overrides, rehearses on x86_64 in minutes even behind blocked egress
(495); gotcha 420's XNNPACK fp16 define does not apply to a 2023 pin whose
riscv64 production microkernels are scalar-only (496); and an upstream build
script's own env hooks take the whole riscv64 delta, with the later flag
cancelling one the script hardcodes (497).

Author: luhenry

.queue.yml

@@ -3797,7 +3797,7 @@ packages:
   repo: https://github.com/tensorflow/tensorflow
   status: in-review
   pr: https://github.com/riseproject-dev/python-wheels/pull/2154
-  notes: 'PR #2154 open as a draft, first CI run started. 12 Linux wheels upstream (abi: cp310,cp311,cp38,cp39); no riscv64 on PyPI or pypi.riseproject.dev. Feasible and much smaller than tensorflow itself (gotcha 426): 2.14.0 is the last tflite-runtime release and predates the LiteRT split, so upstream is tensorflow/tensorflow at tag v2.14.0 (verified: PyPI lists only www.tensorflow.org/lite, and tensorflow/tools/pip_package/setup.py at that tag reads _VERSION = 2.14.0 while tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh builds the tflite_runtime distribution). The recipe builds ONE bazel target, //tensorflow/lite/python/interpreter_wrapper:_pywrap_tensorflow_interpreter_wrapper, with -c opt --config=monolithic --config=noaws --config=nogcp --config=nohdfs --config=nonccl, then copies it next to interpreter.py/metrics_portable.py and runs setup_with_binary.py - no TF core, no MLIR, and flex is behind a define we do not set, so this is comparable in scope to ai-edge-litert (PR #2135) rather than to a full TF build. FOUR riscv64 questions checked statically before the first cycle: (1) bazel: .bazelversion says 6.1.0 and no riscv64 binary exists for any bazel release, so 7.5.0 is bootstrapped as elsewhere in this repo - TF 2.14 has no MODULE.bazel, its only version gate is workspace2.bzl versions.check("1.0.0"), and every flag its .bazelrc sets (--experimental_cc_shared_library, --experimental_link_static_libraries_once, --incompatible_enforce_config_setting_visibility, --experimental_repo_remote_exec, configs monolithic/noaws/nogcp/nohdfs/nonccl) was verified to parse under a real bazel 7.5.0 locally; build:linux is auto-applied via --enable_platform_specific_config and already carries --cxxopt=-std=c++17. (2) XNNPACK at this TF commit is b9d4073a, where riscv64 PRODUCTION microkernels are scalar only (scalar-riscv amalgam / PROD_SCALAR_RISCV_MICROKERNEL_SRCS); ALL_RVV_MICROKERNEL_SRCS and the XNN_ENABLE_RISCV_VECTOR define appear only in bench_microkernels/test_microkernels, and the tree has no rvvfp16arith at all - so neither gotcha 420 (zvfh vs binutils 2.41) nor an RVV-intrinsics-vs-GCC-14 problem can arise, and no xnn_enable_* define is needed. cpuinfo 87d82345 has a linux_riscv64 srcs branch. (3) hermetic python IS the one real gap: TF 2.14 pins rules_python 0.23.1, whose PLATFORMS/TOOL_VERSIONS have no riscv64 CPython, and BOTH //third_party/python_runtime:headers (via @local_config_python -> @python//:python_headers) and //third_party/py/numpy:headers (@pypi_numpy//:numpy_headers) come from it; WORKSPACE also load()s @python//:defs.bzl, so the toolchain_aliases repo is fetched eagerly and get_host_platform() fails on riscv64. Solved with no patch by --override_repository=python=... and --override_repository=pypi_numpy=..., two repos written at run time from the container CPython include dir and numpy.get_include(); VALIDATED off-target: a real bazel 7.5.0 on x86_64 evaluated the whole WORKSPACE python/pip_parse block with those overrides and got as far as workspace3.bzl fetching tf_runtime, i.e. past every hermetic-python line, and pip_parse never ran pip. (4) numpy: nothing needs patching either - the interpreter_wrapper C++ uses only APIs numpy 2 still exports (PyArray_SimpleNewFromData/EMPTY/FromAny/Return/SetBaseObject/ENABLEFLAGS/GETITEM/Iter*, NPY_ARRAY_*, no descr->elsize), so it builds against the registry numpy (2.2.2 cp310 / 2.4.3 cp311) and that keeps the wheel loadable under numpy 1.x and 2.x alike. Matrix is cp310+cp311 by upstream constraint, not by choice: upstream ships cp38-cp311 only, pybind11 is pinned at 2.10.4 which predates CPython 3.12, TF_PYTHON_VERSION accepts 3.9-3.11, and the manylinux riscv64 image has no interpreter below 3.10; registry numpy covers both. Local validation could not go further than WORKSPACE evaluation: this sandbox egress blocks codeload.github.com, so every tf_http_archive 403s.'
+  notes: 'PR #2154 open as a draft and FULLY GREEN on the first CI cycle - both legs built and tested, publish dry run clean (cp310 53m, cp311 52m; bazel build 991 actions / 2739s, wheels 1.8MB, tflite_runtime/_pywrap_tensorflow_interpreter_wrapper.so 3.9MB with ELF e_machine 0xF3, tag cp3XX-cp3XX-manylinux_2_39_riscv64 after auditwheel retagged linux_riscv64, test installs numpy 2.4.3 from the registry and runs testdata/add.bin to 3x its input). 12 Linux wheels upstream (abi: cp310,cp311,cp38,cp39); no riscv64 on PyPI or pypi.riseproject.dev. Feasible and much smaller than tensorflow itself (gotcha 426): 2.14.0 is the last tflite-runtime release and predates the LiteRT split, so upstream is tensorflow/tensorflow at tag v2.14.0 (verified: PyPI lists only www.tensorflow.org/lite, and tensorflow/tools/pip_package/setup.py at that tag reads _VERSION = 2.14.0 while tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh builds the tflite_runtime distribution). The recipe builds ONE bazel target, //tensorflow/lite/python/interpreter_wrapper:_pywrap_tensorflow_interpreter_wrapper, with -c opt --config=monolithic --config=noaws --config=nogcp --config=nohdfs --config=nonccl, then copies it next to interpreter.py/metrics_portable.py and runs setup_with_binary.py - no TF core, no MLIR, and flex is behind a define we do not set, so this is comparable in scope to ai-edge-litert (PR #2135) rather than to a full TF build. FOUR riscv64 questions checked statically before the first cycle: (1) bazel: .bazelversion says 6.1.0 and no riscv64 binary exists for any bazel release, so 7.5.0 is bootstrapped as elsewhere in this repo - TF 2.14 has no MODULE.bazel, its only version gate is workspace2.bzl versions.check("1.0.0"), and every flag its .bazelrc sets (--experimental_cc_shared_library, --experimental_link_static_libraries_once, --incompatible_enforce_config_setting_visibility, --experimental_repo_remote_exec, configs monolithic/noaws/nogcp/nohdfs/nonccl) was verified to parse under a real bazel 7.5.0 locally; build:linux is auto-applied via --enable_platform_specific_config and already carries --cxxopt=-std=c++17. (2) XNNPACK at this TF commit is b9d4073a, where riscv64 PRODUCTION microkernels are scalar only (scalar-riscv amalgam / PROD_SCALAR_RISCV_MICROKERNEL_SRCS); ALL_RVV_MICROKERNEL_SRCS and the XNN_ENABLE_RISCV_VECTOR define appear only in bench_microkernels/test_microkernels, and the tree has no rvvfp16arith at all - so neither gotcha 420 (zvfh vs binutils 2.41) nor an RVV-intrinsics-vs-GCC-14 problem can arise, and no xnn_enable_* define is needed. cpuinfo 87d82345 has a linux_riscv64 srcs branch. (3) hermetic python IS the one real gap: TF 2.14 pins rules_python 0.23.1, whose PLATFORMS/TOOL_VERSIONS have no riscv64 CPython, and BOTH //third_party/python_runtime:headers (via @local_config_python -> @python//:python_headers) and //third_party/py/numpy:headers (@pypi_numpy//:numpy_headers) come from it; WORKSPACE also load()s @python//:defs.bzl, so the toolchain_aliases repo is fetched eagerly and get_host_platform() fails on riscv64. Solved with no patch by --override_repository=python=... and --override_repository=pypi_numpy=..., two repos written at run time from the container CPython include dir and numpy.get_include(); VALIDATED off-target: a real bazel 7.5.0 on x86_64 evaluated the whole WORKSPACE python/pip_parse block with those overrides and got as far as workspace3.bzl fetching tf_runtime, i.e. past every hermetic-python line, and pip_parse never ran pip. (4) numpy: nothing needs patching either - the interpreter_wrapper C++ uses only APIs numpy 2 still exports (PyArray_SimpleNewFromData/EMPTY/FromAny/Return/SetBaseObject/ENABLEFLAGS/GETITEM/Iter*, NPY_ARRAY_*, no descr->elsize), so it builds against the registry numpy (2.2.2 cp310 / 2.4.3 cp311) and that keeps the wheel loadable under numpy 1.x and 2.x alike. Matrix is cp310+cp311 by upstream constraint, not by choice: upstream ships cp38-cp311 only, pybind11 is pinned at 2.10.4 which predates CPython 3.12, TF_PYTHON_VERSION accepts 3.9-3.11, and the manylinux riscv64 image has no interpreter below 3.10; registry numpy covers both. Local validation could not go further than WORKSPACE evaluation: this sandbox egress blocks codeload.github.com, so every tf_http_archive 403s.'
 - pkg: cmeel-zlib
   version: 1.3.2
   home: https://github.com/cmake-wheel/cmeel-zlib

skills/python-project-porting/references/gotchas-index.md

@@ -1,6 +1,6 @@
 # Gotchas index — router for the themed gotcha files
 
-The porting gotchas (479 of them) live in [`references/gotchas/`](gotchas/), split by theme so only the relevant slice loads. Every gotcha keeps a **permanent number** cited elsewhere as "gotcha N" (and in workflow comments as "CLAUDE.md gotcha N"). Numbers are stable IDs — **not sequential**, and four are **reused** with different content (two each of 33, 55, 56, 57), disambiguated by theme below.
+The porting gotchas (483 of them) live in [`references/gotchas/`](gotchas/), split by theme so only the relevant slice loads. Every gotcha keeps a **permanent number** cited elsewhere as "gotcha N" (and in workflow comments as "CLAUDE.md gotcha N"). Numbers are stable IDs — **not sequential**, and four are **reused** with different content (two each of 33, 55, 56, 57), disambiguated by theme below.
 
 ## How to find the gotcha you need
 
@@ -558,6 +558,12 @@ The porting gotchas (479 of them) live in [`references/gotchas/`](gotchas/), spl
   child, so a cmeel distribution that delegates its whole build to one ships into
   `cmeel.prefix/lib64/` rather than `lib/`; read the released wheel's namelist, not the sibling
   workflow (the cmeel-zlib case, and the mechanism behind gotcha 492).
+- **494** — A hermetic Python predating riscv64 is a *two*-repository problem — Python headers
+  and numpy headers both come from it — and in a WORKSPACE tree `--override_repository` over two
+  run-time-written repos settles it with no patch (the tflite-runtime/TensorFlow 2.14 case).
+- **497** — Drive an upstream build script through the env hooks it already exposes
+  (`CUSTOM_BAZEL_FLAGS`, `BAZEL_STARTUP_OPTIONS`), and use the fact that the later flag wins to
+  cancel one it hardcodes, such as `-s`.
 
 ### The manylinux image & toolchain — [`gotchas/manylinux-image-and-toolchain.md`](gotchas/manylinux-image-and-toolchain.md)
 
@@ -658,6 +664,9 @@ The porting gotchas (479 of them) live in [`references/gotchas/`](gotchas/), spl
 - **485** — CMake's `find_package(Python3 COMPONENTS Development)` cannot configure in the
   manylinux image because PEP 513 forbids shipping `libpython`; the fix is a zero-byte file at
   the path FindPython validates, and upstream probably already carries it (the usd-core case).
+- **496** — Gotcha 420's XNNPACK fp16 define does not belong in an older tree: at a 2023 pin the
+  riscv64 *production* microkernels are scalar-only and every RVV gate sits in a bench/test
+  target, so attribute each `riscv` line to its target before adding a define.
 
 ### Native dependencies & linking — [`gotchas/native-deps-and-linking.md`](gotchas/native-deps-and-linking.md)
 
@@ -967,6 +976,9 @@ The porting gotchas (479 of them) live in [`references/gotchas/`](gotchas/), spl
   wheel hands you a byte-comparable feature oracle: configure once under QEMU and diff it
   against the released wheel's before compiling anything; also where an ECMWF binary-wrapper
   distribution's real build recipe lives when its wheel job is private (the eckitlib case).
+- **495** — A Bazel port's loading phase rehearses on x86_64 in minutes: check the project's
+  `.bazelrc` flags against the bazel you bootstrap in an empty workspace, then evaluate the real
+  WORKSPACE with the real overrides — blocked egress only stops it at the first archive fetch.
 
 ### PR, CI, triggers, publishing & maintainer signals — [`gotchas/pr-ci-and-maintainer.md`](gotchas/pr-ci-and-maintainer.md)
 

skills/python-project-porting/references/gotchas/local-validation-and-rehearsal.md

@@ -36,6 +36,8 @@ To pull up one entry: `grep -n '^N\. ' references/gotchas/local-validation-and-r
 - **444** — Verify a hand-edited `.patch` with `git apply --check`, never with `patch`:
   a wrong `@@` line count makes GNU `patch` silently swallow the *next* hunk and exit 0.
 - **490** — An ecbuild/CMake project that installs its generated config header into the
+- **495** — A Bazel port's loading phase rehearses on x86_64 in minutes, even in a sandbox
+  that cannot fetch the dependencies.
   wheel hands you a byte-comparable feature oracle
 
 ---
@@ -545,3 +547,26 @@ To pull up one entry: `grep -n '^N\. ' references/gotchas/local-validation-and-r
       be dropped (keeping it also means cython in the container). The per-interpreter wheel
       list is a mirage for the same reason — cp310..cp314 differ only in tag, nothing links the
       Python C API — so build one `py3-none` wheel, as `build-eccodeslib.yml` already does.
+
+495. **A Bazel port's loading phase rehearses on x86_64 in minutes, even in a sandbox that
+    cannot fetch the dependencies.** Two checks, both arch-independent, both cheaper than the
+    hours-long riscv64 cycle they replace:
+    - **The project's `.bazelrc` against the bazel version you actually bootstrap.** Copy it
+      into an empty workspace (`touch WORKSPACE`) and run `bazel build --nobuild` with the
+      same `--config`s the upstream script passes. An old tree pinned to bazel 6 may name
+      flags a newer bazel deleted, and an unknown one is a startup failure, not a warning —
+      `--experimental_cc_shared_library`, `--experimental_link_static_libraries_once` and
+      `--incompatible_enforce_config_setting_visibility` all still parse in 7.5.0, which is
+      one reason these trees want 7.x and not 8 (which also dropped WORKSPACE `bind()`).
+      With no targets bazel exits 0 on "requested an empty set of targets", so the run is
+      purely a flag check.
+    - **WORKSPACE evaluation on the real checkout, with the real repository overrides.**
+      Everything up to the first `http_archive` fetch is host-independent: it proves the
+      overrides resolve, every `load()` finds its symbols, and no repository rule shells out
+      to an interpreter that isn't there (gotcha 494's failure mode lands here). Where egress
+      blocks `codeload.github.com` the run dies on the first archive with a 403 — *after* that
+      whole block, which is the part worth testing; read the traceback's `WORKSPACE:<line>` to
+      confirm how far it got.
+    - Point `--output_user_root` at `.git/pw-scratch/<pkg>/` and delete it afterwards: a
+      bazel install base plus a shallow clone of a monorepo is ~700MB on a disk other agents
+      share.

skills/python-project-porting/references/gotchas/manylinux-image-and-toolchain.md

@@ -91,6 +91,8 @@ To pull up one entry: `grep -n '^N\. ' references/gotchas/manylinux-image-and-to
   nothing else in the tree declares: translate its `-dev` packages to Rocky names before the
   first run, or the image's missing header stops the compile (the vllm `numa.h` case).
 - **485** — CMake's `find_package(Python3 COMPONENTS Development)` fails inside the manylinux
+- **496** — Before copying gotcha 420's XNNPACK fp16 define into an older tree, check which
+  target the gate sits in — an old pin may compile no vector kernels at all.
   image because PEP 513 forbids shipping `libpython`, and the fix is a zero-byte file, not a
   build-flag change (the usd-core/OpenUSD case).
 ---
@@ -1522,3 +1524,23 @@ To pull up one entry: `grep -n '^N\. ' references/gotchas/manylinux-image-and-to
       linux/riscv64 $MANYLINUX_RISCV64_IMAGE` and print `os.path.exists(LIBDIR/LDLIBRARY)` for
       each interpreter you plan to build. `False` everywhere means this gotcha applies to any
       `Development`-requiring configure in that image.
+
+496. **Before copying gotcha 420's `--define=xnn_enable_riscv_fp16_vector=false` into an
+    older tree, check which XNNPACK *target* the gate sits in — an old pin may compile no
+    vector kernels at all.** At the commit TensorFlow 2.14 pins (`b9d4073a`, mid-2023) the
+    riscv64 production library is scalar-only: `riscv_srcs` is
+    `src/amalgam/gen/scalar-riscv.c` / `PROD_SCALAR_RISCV_MICROKERNEL_SRCS`, while
+    `ALL_RVV_MICROKERNEL_SRCS`, the `-march=rv64gcv` copts and the `XNN_ENABLE_RISCV_VECTOR`
+    define appear only in `bench_microkernels` and `test_microkernels`, and `rvvfp16arith`
+    is not in the tree at all. Neither the `zvfh`-vs-binutils-2.41 trap (420) nor the
+    pre-0.12 RVV intrinsic spellings GCC 14 no longer accepts can fire, so the define would
+    be a deviation with nothing behind it — the build was green without it.
+    - **Attribute each `riscv` hit to its target before concluding anything**:
+      `grep -n riscv BUILD.bazel` then, per line, `awk 'NR<=<line> && /    name = /'
+      BUILD.bazel | tail -1`. A gate inside a benchmark or test library never reaches a
+      wheel.
+    - **Read `:riscv_vector_enabled` twice**: its `explicit_false` branch resolves to the
+      *explicit_true* config_setting, which looks like a bug and is not — with
+      `--define xnn_enable_riscv_vector=false` that setting no longer matches, so the select
+      falls through to its default and RVV is off. The off-switch works; the indirection
+      just does not read like one.