Skip to content

build: pick the BTF kernel by capability, not by directory name - #193

Open
majianpeng wants to merge 1 commit into
taodd:mainfrom
majianpeng:fix/gen-btf-kernel-selection
Open

majianpeng wants to merge 1 commit into
taodd:mainfrom
majianpeng:fix/gen-btf-kernel-selection

Conversation

@majianpeng

Copy link
Copy Markdown

GEN-BTF decided which kernel to pull ceph.ko's BTF from with:

kver=$(find /lib/modules -mindepth 1 -maxdepth 1 -type d -printf '%f\n' \
       | sort -V | tail -1)

i.e. "the highest-sorting directory directly under /lib/modules is the newest kernel". That assumes every such directory is named after a kernel release. Distributions are not obliged to comply.

Kylin Linux Advanced Server V11 (Swan25) ships /lib/modules/ksaf: a kernel-release-independent directory holding Kylin's own LSM modules (ksaf_main.ko from ksaf-main-module, kysec_exectl.ko from kysec2-exectl-module). It is deliberately kept outside /lib/modules// so that the dracut module 98ksaf_main can copy one fixed path into the initramfs for every kernel, and the security modules get loaded as early as possible. It contains no ceph.ko.

GNU sort -V ranks a run of letters ABOVE a run of digits, so any name that starts with a letter sorts above every real version number:

$ find /lib/modules -mindepth 1 -maxdepth 1 -type d -printf '%f\n' | sort -V
5.4.241-24.0017.26
6.6.0-32.12.v2505.ky11.x86_64
6.6.0-32.21.v2505.ky11.x86_64
6.6.0-32.22.v2505.ky11.x86_64
6.6.0-32.24.v2505.ky11.x86_64
ksaf

Hence tail -1 returns ksaf, the recipe looks for ceph.ko under /lib/modules/ksaf, finds nothing, and aborts the build:

GEN-BTF  .../src/ceph_btf_local.h
No ceph.ko* found under /lib/modules/ksaf
make: *** [Makefile:164: .../src/ceph_btf_local.h] Error 1

ceph_btf_local.h is a prerequisite of the .bpf.o targets, so the build dies before a single object is compiled, which makes the error look unrelated to kernel discovery. The failure is also conditional: when the generated header is already present, make skips GEN-BTF and the bug stays hidden until a make clean, git clean, or a fresh checkout removes it.

Fix the selection to match on capability instead of on the directory name: a candidate must actually contain a ceph.ko*, which excludes any non-kernel directory no matter what it is called. Prefer the running kernel when it is installed and ships ceph.ko*, because

  • tracing attaches to the live kernel, and
  • the base BTF fallback /sys/kernel/btf/vmlinux is only valid for the running kernel, so falling back to a different (e.g. newer, not yet rebooted) kernel would otherwise fail later with "No usable vmlinux/base BTF found".

Verified on Kylin V11, kernel 6.6.0-32.24.v2505.ky11.x86_64:

Using installed kernel: 6.6.0-32.24.v2505.ky11.x86_64
Found ceph kernel module: /lib/modules/6.6.0-32.24.v2505.ky11.x86_64/kernel/fs/ceph/ceph.ko.xz
Using base BTF source: /sys/kernel/btf/vmlinux

GEN-BTF decided which kernel to pull ceph.ko's BTF from with:

    kver=$(find /lib/modules -mindepth 1 -maxdepth 1 -type d -printf '%f\n' \
           | sort -V | tail -1)

i.e. "the highest-sorting directory directly under /lib/modules is the newest
kernel". That assumes every such directory is named after a kernel release.
Distributions are not obliged to comply.

Kylin Linux Advanced Server V11 (Swan25) ships /lib/modules/ksaf: a
kernel-release-independent directory holding Kylin's own LSM modules
(ksaf_main.ko from ksaf-main-module, kysec_exectl.ko from
kysec2-exectl-module). It is deliberately kept outside /lib/modules/<release>/
so that the dracut module 98ksaf_main can copy one fixed path into the
initramfs for every kernel, and the security modules get loaded as early as
possible. It contains no ceph.ko.

GNU `sort -V` ranks a run of letters ABOVE a run of digits, so any name that
starts with a letter sorts above every real version number:

    $ find /lib/modules -mindepth 1 -maxdepth 1 -type d -printf '%f\n' | sort -V
    5.4.241-24.0017.26
    6.6.0-32.12.v2505.ky11.x86_64
    6.6.0-32.21.v2505.ky11.x86_64
    6.6.0-32.22.v2505.ky11.x86_64
    6.6.0-32.24.v2505.ky11.x86_64
    ksaf

Hence `tail -1` returns ksaf, the recipe looks for ceph.ko under
/lib/modules/ksaf, finds nothing, and aborts the build:

    GEN-BTF  .../src/ceph_btf_local.h
    No ceph.ko* found under /lib/modules/ksaf
    make: *** [Makefile:164: .../src/ceph_btf_local.h] Error 1

ceph_btf_local.h is a prerequisite of the .bpf.o targets, so the build dies
before a single object is compiled, which makes the error look unrelated to
kernel discovery. The failure is also conditional: when the generated header is
already present, make skips GEN-BTF and the bug stays hidden until a
`make clean`, `git clean`, or a fresh checkout removes it.

Fix the selection to match on capability instead of on the directory name: a
candidate must actually contain a ceph.ko*, which excludes any non-kernel
directory no matter what it is called. Prefer the running kernel when it is
installed and ships ceph.ko*, because

  - tracing attaches to the live kernel, and
  - the base BTF fallback /sys/kernel/btf/vmlinux is only valid for the running
    kernel, so falling back to a different (e.g. newer, not yet rebooted)
    kernel would otherwise fail later with "No usable vmlinux/base BTF found".

Verified on Kylin V11, kernel 6.6.0-32.24.v2505.ky11.x86_64:

    Using installed kernel: 6.6.0-32.24.v2505.ky11.x86_64
    Found ceph kernel module: /lib/modules/6.6.0-32.24.v2505.ky11.x86_64/kernel/fs/ceph/ceph.ko.xz
    Using base BTF source: /sys/kernel/btf/vmlinux
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant