build: pick the BTF kernel by capability, not by directory name - #193
Open
majianpeng wants to merge 1 commit into
Open
majianpeng wants to merge 1 commit into
majianpeng wants to merge 1 commit into
Conversation
GEN-BTF decided which kernel to pull ceph.ko's BTF from with:
kver=$(find /lib/modules -mindepth 1 -maxdepth 1 -type d -printf '%f\n' \
| sort -V | tail -1)
i.e. "the highest-sorting directory directly under /lib/modules is the newest
kernel". That assumes every such directory is named after a kernel release.
Distributions are not obliged to comply.
Kylin Linux Advanced Server V11 (Swan25) ships /lib/modules/ksaf: a
kernel-release-independent directory holding Kylin's own LSM modules
(ksaf_main.ko from ksaf-main-module, kysec_exectl.ko from
kysec2-exectl-module). It is deliberately kept outside /lib/modules/<release>/
so that the dracut module 98ksaf_main can copy one fixed path into the
initramfs for every kernel, and the security modules get loaded as early as
possible. It contains no ceph.ko.
GNU `sort -V` ranks a run of letters ABOVE a run of digits, so any name that
starts with a letter sorts above every real version number:
$ find /lib/modules -mindepth 1 -maxdepth 1 -type d -printf '%f\n' | sort -V
5.4.241-24.0017.26
6.6.0-32.12.v2505.ky11.x86_64
6.6.0-32.21.v2505.ky11.x86_64
6.6.0-32.22.v2505.ky11.x86_64
6.6.0-32.24.v2505.ky11.x86_64
ksaf
Hence `tail -1` returns ksaf, the recipe looks for ceph.ko under
/lib/modules/ksaf, finds nothing, and aborts the build:
GEN-BTF .../src/ceph_btf_local.h
No ceph.ko* found under /lib/modules/ksaf
make: *** [Makefile:164: .../src/ceph_btf_local.h] Error 1
ceph_btf_local.h is a prerequisite of the .bpf.o targets, so the build dies
before a single object is compiled, which makes the error look unrelated to
kernel discovery. The failure is also conditional: when the generated header is
already present, make skips GEN-BTF and the bug stays hidden until a
`make clean`, `git clean`, or a fresh checkout removes it.
Fix the selection to match on capability instead of on the directory name: a
candidate must actually contain a ceph.ko*, which excludes any non-kernel
directory no matter what it is called. Prefer the running kernel when it is
installed and ships ceph.ko*, because
- tracing attaches to the live kernel, and
- the base BTF fallback /sys/kernel/btf/vmlinux is only valid for the running
kernel, so falling back to a different (e.g. newer, not yet rebooted)
kernel would otherwise fail later with "No usable vmlinux/base BTF found".
Verified on Kylin V11, kernel 6.6.0-32.24.v2505.ky11.x86_64:
Using installed kernel: 6.6.0-32.24.v2505.ky11.x86_64
Found ceph kernel module: /lib/modules/6.6.0-32.24.v2505.ky11.x86_64/kernel/fs/ceph/ceph.ko.xz
Using base BTF source: /sys/kernel/btf/vmlinux
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GEN-BTF decided which kernel to pull ceph.ko's BTF from with:
i.e. "the highest-sorting directory directly under /lib/modules is the newest kernel". That assumes every such directory is named after a kernel release. Distributions are not obliged to comply.
Kylin Linux Advanced Server V11 (Swan25) ships /lib/modules/ksaf: a kernel-release-independent directory holding Kylin's own LSM modules (ksaf_main.ko from ksaf-main-module, kysec_exectl.ko from kysec2-exectl-module). It is deliberately kept outside /lib/modules// so that the dracut module 98ksaf_main can copy one fixed path into the initramfs for every kernel, and the security modules get loaded as early as possible. It contains no ceph.ko.
GNU
sort -Vranks a run of letters ABOVE a run of digits, so any name that starts with a letter sorts above every real version number:Hence
tail -1returns ksaf, the recipe looks for ceph.ko under /lib/modules/ksaf, finds nothing, and aborts the build:ceph_btf_local.h is a prerequisite of the .bpf.o targets, so the build dies before a single object is compiled, which makes the error look unrelated to kernel discovery. The failure is also conditional: when the generated header is already present, make skips GEN-BTF and the bug stays hidden until a
make clean,git clean, or a fresh checkout removes it.Fix the selection to match on capability instead of on the directory name: a candidate must actually contain a ceph.ko*, which excludes any non-kernel directory no matter what it is called. Prefer the running kernel when it is installed and ships ceph.ko*, because
Verified on Kylin V11, kernel 6.6.0-32.24.v2505.ky11.x86_64: