Repository navigation
Conversation
There was a problem hiding this comment.
🚨 Changes Requested - Found 1 issue(s) that must be addressed
The rename is complete. In the base tree I grepped for nhp_ebpf_xdp and every hit is in a file this PR changes: the Makefile, both workflows, ebpfegine.go, the source-shape and load tests, docs, RUNBOOK, CLAUDE.md/AGENTS.md and comments. Nothing else ships the object under its old name: Dockerfiles, userdata and the Makefile cp ./ac/main/etc/*.toml only copy .toml files. For the demo, deploy-ac uploads the new name before the restart, so the deploy pipeline order is fine.
1. Breaking change for any existing FilterMode = 1 deployment, and nothing marks it as one (endpoints/ac/ebpf/ebpfegine.go:36)
The loaded object name is hardcoded relative to the working directory (etc/nhp_ac_xdp.o). As deploy/config-templates/ac/config.toml says, nhp-acd refuses to start when either object is missing. So an operator outside the demo pipeline upgrades nhp-acd but keeps their existing etc/, which only has nhp_ebpf_xdp.o. Their AC then fails to start after the upgrade, and the AC fails closed, so the protected resource becomes unreachable. Examples: a package or tarball upgrade, a binary swap, or their own CI that copies etc/*.o by name.
The commits are typed fix: / refactor: / docs: / ci: with no ! and no BREAKING CHANGE: footer. release-please therefore writes nothing in the changelog that would warn them.
Do one of these:
- Preferred: for one or two releases, fall back to the legacy file name when
etc/nhp_ac_xdp.odoes not exist, and log aWarningasking the operator to rename it. Keep the fallback limited to which file name is opened, so it cannot load anything other than the AC object. For example:prog := filepath.Join(bpfDir, "nhp_ac_xdp.o") if _, err := os.Stat(prog); errors.Is(err, fs.ErrNotExist) { legacy := filepath.Join(bpfDir, "nhp_ebpf_xdp.o") if _, lerr := os.Stat(legacy); lerr == nil { log.Warning("etc/nhp_ebpf_xdp.o is deprecated; rename it to etc/nhp_ac_xdp.o") prog = legacy } }
- Or: make it an explicit breaking change (
refactor(xdp)!:plus aBREAKING CHANGE:footer naming the file) so the release notes tell operators to rename the object before they upgrade.
Minor / informational
- Demo host leftover:
deploy-acuploadsnhp_ac_xdp.obut never removes the old$REMOTE_DIR/etc/nhp_ebpf_xdp.o. That file is harmless, and it actually keeps a binary rollback to the previous release working. Please say that in a comment or the PR body, so nobody later "cleans it up" and breaks the rollback path. - "Self-tested" is unchecked in the PR body. The
ebpf-verifierjob and the source-shape tests (acxdp_source_test.go,xdpfilter_test.go,ebpf_test.go,cexpr_test.go,engine_linux_test.go) read the new path, so please confirm they are green on this head. A path typo would make themt.Fatalf, not skip, which is the behaviour we want, but it should show up as green CI before merge.
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## main #1895 +/- ##
==========================================
+ Coverage 41.62% 41.64% +0.02%
==========================================
Files 142 142
Lines 22553 22562 +9
==========================================
+ Hits 9388 9397 +9
Misses 12262 12262
Partials 903 903
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
🚨 Changes Requested - Found 1 issue that must be addressed
The rename itself is complete. Every nhp_ebpf_xdp reference on main (27 files, including the workflows, the Makefile, ebpfegine.go, the tests, the docs, RUNBOOK, CLAUDE.md/AGENTS.md and the Terraform comment) is updated in this diff. The C source is a 100%-similarity rename, and make acd only copies *.toml into release/nhp-ac/etc/, so the build itself can't pick up the wrong object. My concern is what this does to hosts that are already running.
1. Breaking on-disk contract for FilterMode = 1 ACs, and the commit type doesn't say so (category 3)
endpoints/ac/ebpf/ebpfegine.go:36 now hard-codes etc/nhp_ac_xdp.o. If an operator upgrades the nhp-acd binary and keeps the existing etc/ directory, which still holds only nhp_ebpf_xdp.o, loadAcEngine (nhp/utils/ebpf/engine_linux.go:610-612) fails the os.Stat. It logs eBPF object file not found without the path and returns, so nhp-acd won't start. The AC fails closed, which means an outage of the protected resource rather than a quiet degradation. The demo pipeline is fine because deploy-ac uploads the new object every time. Other deployments get no warning:
- The PR is titled
refactor(xdp): …, so release-please won't mark it breaking or put it in the changelog as a required action. - The error doesn't name the file it looked for, so an operator has no clue that the file was renamed.
Requested (any of these is enough; the first two together are best):
- Make the not-found error actionable, for example
log.Error("eBPF object file not found: %s", specPath). When the legacyetc/nhp_ebpf_xdp.ois present, also say that it was renamed tonhp_ac_xdp.oand must be rebuilt/redeployed. Please don't silently fall back to loading the legacy file:setAcConstantsalready documents that a stale object next to a new binary is the failure mode to refuse. - Mark the change as breaking (
refactor(xdp)!:or aBREAKING CHANGE:footer) so the release notes tell operators to ship the new object together with the binary. - Optionally, have
deploy-acremove the stale$REMOTE_DIR/etc/nhp_ebpf_xdp.oafter uploading the new one, so the demo AC doesn't keep a dead object that a later rollback could pair with the wrong binary.
Minor
endpoints/ac/main/etc/nhp_ac_xdp.ois a different blob from the deletednhp_ebpf_xdp.o(0e3186b5b→ceb4bd772), so it was rebuilt rather than moved. That's probably only the BTF source path changing, but reviewers can't verify a binary. Please note in the PR how and from which commit it was built, or confirm thatmake ebpffrom this tree reproduces it.- The PR checklist ("Self-tested") is unticked. Please confirm the
ebpf-verifierjob and thenhp/utils/ebpfsource-contract tests (which now readnhp_ac_xdp.c) are green.
Superseded by Claude review for 7fff9b7.
Update the remaining comment and documentation references to the AC eBPF/XDP filter from nhp_ebpf_xdp to nhp_ac_xdp, matching the nhp-server's nhp_server_xdp naming and the AC's existing nhp_ac_xdp log prefix.
Rename the AC eBPF/XDP filter source and object from nhp_ebpf_xdp.* to nhp_ac_xdp.* to match the nhp-server's nhp_server_xdp.* convention and the AC's existing nhp_ac_xdp telemetry-log prefix.
The rename moved the blob but not its bytes: the committed object still came from the pre-rename source, so its DWARF/BTF named nhp_ebpf_xdp.c and `bpftool btf dump file` printed the old name. It was also older than that: compiled as of eaacb6d, before the AC filter's later fixes, and missing the maps and the .rodata constants the current loader requires (nhp_events, nhp_action_stats, nhp_port_hits, nhp_src_hits, nhp_ac_filter_ipv6 and the DHCP/NTP source ranges), so LoadAndAssign fails on it. Rebuild it from nhp/ebpf/xdp/nhp_ac_xdp.c with the Makefile's flags, and normalize the build path the way the Go binaries' -trimpath normalizes theirs.
7fff9b7 to
76fc54a
Compare
There was a problem hiding this comment.
🚨 Changes Requested - Found 1 issue that must be addressed
The rename itself is complete: every reference to nhp_ebpf_xdp in the tree is updated, including the paths that load the object at runtime (endpoints/ac/ebpf/ebpfegine.go:36), the Makefile build targets, both workflows, the source-reading tests and the docs. There is one compatibility problem and one small cleanup.
1. Breaking change: existing AC installs stop starting in FilterMode = 1 (category 3)
endpoints/ac/ebpf/ebpfegine.go:36 changes the object nhp-acd loads from etc/nhp_ebpf_xdp.o to etc/nhp_ac_xdp.o. The demo pipeline is handled because deploy-demo-v2 uploads the new name before it restarts. Any other install is not:
- An operator who replaces only the
nhp-acdbinary and keeps theiretc/directory (the usual upgrade for a config directory) hasetc/nhp_ebpf_xdp.obut noetc/nhp_ac_xdp.o. - On restart,
nhp-acdfails with "eBPF object file not found" (see the comment atdeploy-demo-v2.yml~L2712). - The AC's failure direction is closed, so the protected resource is unreachable until someone works out that a file needs renaming.
The PR does not flag this anywhere. The commits are fix: / ci: / docs:, so release-please will publish it as a patch release with no breaking-change note. Please do one of the following:
- Preferred: when
nhp_ac_xdp.ois missing andnhp_ebpf_xdp.oexists, load the legacy name and log aWarningsaying to rename it. Remove the fallback in a later release. - Or: mark it as breaking so it reaches the changelog and the version bump. Use a
!in the type (e.g.refactor(ac)!:) or aBREAKING CHANGE:footer, and add an upgrade note todocs/ebpf.md.
2. Nit: the old object stays on the demo AC host
.github/workflows/deploy-demo-v2.yml (~L2720) now uploads nhp_ac_xdp.o, but nothing removes the old $REMOTE_DIR/etc/nhp_ebpf_xdp.o. It does no harm, but anyone looking at the host or following RUNBOOK.md will find two objects and not know which one is live. Consider adding ssh nhp-ac "rm -f $REMOTE_DIR/etc/nhp_ebpf_xdp.o" next to the upload.
Other notes
- The "Self-tested" box in the PR description is unchecked. Since this changes the filename the daemon loads, please confirm the
ebpf-verifierjob is green. It loadsrelease/nhp-ac/etc/nhp_ac_xdp.othroughnhp-ebpf-verify, and the AC load tests resolve the new name throughacObjPath.
Superseded by Claude review for 76fc54a.
The AC's XDP object is tracked in the source tree (endpoints/ac/main/etc/nhp_ac_xdp.o), because a run started from there loads it from there -- the loader's object paths are relative to the working directory -- and not from the release directory `make ebpf-objs` fills. It is a build artifact of a toolchain the repository does not pin, and no gate had ever asked the kernel about it: the ebpf-verifier job only loaded the objects it builds itself, and the Go load tests resolve the release directory only. An object nobody loads is an object nobody knows -- a refusal of it would surface only on the machine that runs from the source tree, where the fail-closed backstop then keeps tcp/443 shut. The PR gate now loads the tracked file into the runner's kernel as well, through nhp-ebpf-verify, the same tool and the same loader the daemons use. A new test keeps it equal to what the current source builds -- the same maps, the same programs, the same constants -- so a copy left behind by a source change is caught instead of enforcing the old policy on the machine that runs from the source tree.
There was a problem hiding this comment.
🚨 Changes Requested - Found 1 issue that must be addressed (plus 2 minor suggestions)
The rename is thorough. At the base commit, git grep nhp_ebpf_xdp finds 27 files, and this PR touches every one of them. The workflows, the Makefile, the loader, the tests and the docs all agree on nhp_ac_xdp.o. The new ebpf-verifier step and acxdp_tracked_test.go are a good addition, because nothing in CI loaded the tracked object before. One upgrade-path problem needs handling before merge.
1. Breaking change: an AC upgraded in place with FilterMode = 1 will not start (fails closed)
endpoints/ac/ebpf/ebpfegine.go:36 now asks only for etc/nhp_ac_xdp.o. loadAcEngine (nhp/utils/ebpf/engine_linux.go:609-612) returns an error when that file is missing, and udpac.go passes that error up, so nhp-acd does not start.
The demo pipeline is fine, because deploy-ac now uploads the new name. But on any other deployment, upgrading the binary while keeping the existing etc/ directory (the usual way, since etc/ holds the host's config and keys) leaves etc/nhp_ebpf_xdp.o beside a binary that looks for etc/nhp_ac_xdp.o. In FilterMode = 1 the AC then refuses to start, and with the backstop installed the protected resource stays shut. That is an outage caused only by a file rename.
It is also hard to diagnose: the log line is "eBPF object file not found " and does not include the path, so an operator cannot see which name the daemon wanted.
Please do at least one of these:
- Fall back to
etc/nhp_ebpf_xdp.owhennhp_ac_xdp.ois missing, and log aWarningthat names both paths and asks the operator to rename the file. This could be removed in a later release. - Or mark the change as breaking. The PR title is
refactor(xdp), which release-please will not surface. Userefactor(xdp)!:or aBREAKING CHANGE:footer and add an upgrade note telling operators to rename or replaceetc/nhp_ebpf_xdp.o.
Either way, please include specPath / tcSpecPath in the two "not found" log lines (engine_linux.go:610,614).
Related housekeeping: existing demo AC hosts will keep a stale $REMOTE_DIR/etc/nhp_ebpf_xdp.o forever. It is harmless, but an rm -f in deploy-ac after the upload would stop it from confusing someone during an incident.
2. Minor: backticks inside a double-quoted string run a command (.github/workflows/ubuntu-build.yml, new step "Ask the kernel about the tracked AC object")
echo "::error::$verify is missing; the step above builds it with `make ebpf-verify`"Inside double quotes, `make ebpf-verify` is command substitution. On this error path the step would run make ebpf-verify and print its output (or nothing) in place of the hint. Use single quotes around that part, or escape the backticks.
3. Minor: the staleness test can skip silently
TestTrackedAcObjectMatchesTheBuiltOne calls t.Skip when release/nhp-ac/etc/nhp_ac_xdp.o has not been built. It actually guards something only in a job that ran make ebpf-objs first. Please confirm the eBPF test step in the ebpf-verifier job runs after make ebpf-objs in the same job, so the test never skips there. Otherwise CLAUDE.md and docs/ebpf.md promise more than CI enforces ("a stale copy cannot be committed unnoticed"). Making it fail instead of skip when CI is set would close that gap.
I did not compile the new test against cilium/ebpf v0.22.0 locally. CI (build / ebpf-verifier, still pending as I write this) will confirm it compiles.
Superseded by Claude review for 23eb7cf.
The rename from nhp_ebpf_xdp.o to nhp_ac_xdp.o broke in-place upgrades: a host keeping its existing etc/ directory has etc/nhp_ebpf_xdp.o beside a binary that now asks for etc/nhp_ac_xdp.o, so FilterMode = 1 refuses to start and the fail-closed backstop holds the protected resource shut. Fall back to etc/nhp_ebpf_xdp.o when the new name is missing, log a warning naming both paths, include the path in the loader's "not found" messages, and drop the stale object in deploy-ac after the upload.
There was a problem hiding this comment.
🚨 Changes Requested - Found 1 issue(s) that must be addressed
This rename looks complete. Outside .git, every reference to nhp_ebpf_xdp (Makefile, workflows, loader, tests, comments, docs) is updated, and the acProgObjPath fallback plus its test is a good way to protect hosts upgraded in place. One problem in the deploy pipeline can still take the AC down, and there are two smaller notes.
1. [Bug / outage] A config-only deploy deletes the object the running binary still needs
.github/workflows/deploy-demo-v2.yml (deploy-ac, after scp ./artifact/ac/etc/nhp_ac_xdp.o …):
ssh nhp-ac "rm -f $REMOTE_DIR/etc/nhp_ebpf_xdp.o"The nhp-acd upload just above is gated on DEPLOY_BINARIES = yes. The eBPF objects and this rm are not gated; the comment above them says they are uploaded unconditionally on purpose. So on the first deploy_binaries=no run after this PR merges:
- nhp-acd is stopped.
- The binary on the host is still the old one. It asks for
etc/nhp_ebpf_xdp.oby its fixed name and has no fallback. nhp_ac_xdp.ois uploaded, which the old binary never reads, andnhp_ebpf_xdp.ois deleted.- With
FilterMode = 1, nhp-acd fails at startup with "eBPF object file not found". The fail-closed backstop then holds tcp/443 shut. The protected resource is down until someone does a binary deploy.
The fallback in acProgObjPath only helps a new binary next to an old object. This step creates the reverse case: an old binary with only the new object.
Suggested fix, either of these:
- Run the
rmonly whenDEPLOY_BINARIES = yes, i.e. only when the binary that knows the new name has actually been installed. - Or keep both names on the host for one release cycle:
scpthe same object to bothnhp_ac_xdp.oandnhp_ebpf_xdp.o, and delete the legacy name in a later PR.
2. [Minor bug] Backticks in a double-quoted echo run a command
.github/workflows/ubuntu-build.yml, step "Ask the kernel about the tracked AC object":
echo "::error::$verify is missing; the step above builds it with `make ebpf-verify`"Inside double quotes, bash treats the backticks as command substitution. On this error path the step actually runs make ebpf-verify and puts its output into the message. Use single quotes around that part, or escape the backticks.
3. [Nit] PR description is out of date
The description lists 28 files with 60/60 lines changed, a pure rename. The diff also adds a new CI step, acxdp_tracked_test.go, ebpfegine_test.go, the loader fallback, a host-side rm in the deploy job, and a new CLAUDE.md paragraph. Please update the description so reviewers and the release notes show these changes in behaviour, the deploy-job change in particular.
Looked at, no change needed
acProgObjPath: aStaterror other than ENOENT falls through to the new path, so the loader's error names the file it actually wants. Good.acxdp_tracked_test.go: comparing interfaces (programs, maps with the data-section maps excluded, and.rodatavariables) rather than instructions is reasonable for an artifact whose toolchain the repo does not pin.- The
engine_linux.golog messages now include the missing path. Nice improvement.
Superseded by Claude review for 3148335.
There was a problem hiding this comment.
💬 Comment - Found 2 suggestion(s) for improvement
I checked the whole rename against main with git grep nhp_ebpf_xdp. Every remaining reference to the old name (Makefile, both workflows, Go loaders and tests, C comments, docs, RUNBOOK, terraform comment, backstop script, config template) is updated in this PR. The upgrade path also holds up:
acProgObjPath(endpoints/ac/ebpf/ebpfegine.go) falls back tonhp_ebpf_xdp.oonly when the new name is missing (IsNotExist) and the legacy file exists. It logs a Warning that names both paths. In every other case it returns the new name, so the loader's "not found" error, which now includes the path, points at the file the daemon really wants.ebpfegine_test.gocovers all three cases.- The
rm -f $REMOTE_DIR/etc/nhp_ebpf_xdp.oindeploy-demo-v2.ymlonly runs whenDEPLOY_BINARIES = yes. Without that gate, a config-only deploy would leave the old binary (which has no fallback) with onlynhp_ac_xdp.oon disk, and nhp-acd would fail closed. Good catch. - The tracked object's blob changed (
0e3186b→ceb4bd7), so it was rebuilt.acxdp_tracked_test.gocompares it withrelease/nhp-ac/etc/nhp_ac_xdp.oby interface (programs, maps,.rodatavariables). That test really runs inebpf-verifier, aftermake ebpf-objs. In the plain unit-test job it skips.
Suggestions
-
.github/workflows/ubuntu-build.yml, new step "Ask the kernel about the tracked AC object" (bug, error path only). Inecho "::error::$verify is missing; the step above builds it withmake ebpf-verify", the backticks sit inside double quotes, so bash treats them as command substitution. If this branch is ever reached, the step runsmake ebpf-verifyand pastes its output into the annotation, instead of printing the command name. Use single quotes around that part, or escape the backticks. -
Make the tracked-object tests required in
ebpf-verifier. The step that runs the tests checks a fixed list of test names for--- PASS.TestTrackedAcObjectMatchesTheBuiltOneis not on that list, and it skips itself when the built object is missing. If that path or the build step changes later, the "stale copy cannot be committed unnoticed" guarantee in CLAUDE.md anddocs/ebpf.mdwould quietly stop holding. AddTestTrackedAcObjectIsTheAcFilterandTestTrackedAcObjectMatchesTheBuiltOneto the list.
Nit
- The PR description is out of date. It shows a 60/60 rename-only diffstat, but the PR also adds the legacy-name fallback, the deploy cleanup, a new CI step and
acxdp_tracked_test.go(+429/−62). Please update it so reviewers and release notes see the behaviour change (fallback plus Warning on hosts that still have the old file).
I have not verified whether CI passes on this head: build and ebpf-verifier were still pending when this review was written.
Superseded by Claude review for eac2fd3.
There was a problem hiding this comment.
🚨 Changes Requested - Found 2 issue(s) that must be addressed
The rename itself is complete. A grep of the PR head finds no remaining nhp_ebpf_xdp reference outside the intentional legacy handling, and acProgObjPath with its test is a sensible fallback for hosts upgraded in place. Two problems need fixing before merge. Both are in the parts the PR description does not mention: the description lists only a rename, but the diff also adds a new CI step, a new test file (acxdp_tracked_test.go), a rebuilt tracked .o blob, a runtime fallback and a deploy-time rm. Please update the description so reviewers can see what changed.
1. Bug (CI): backticks inside a double-quoted echo run make ebpf-verify
.github/workflows/ubuntu-build.yml:325 (new step Ask the kernel about the tracked AC object):
echo "::error::$verify is missing; the step above builds it with `make ebpf-verify`"Inside double quotes, backticks are command substitution. When this error path is hit, the shell runs make ebpf-verify and pastes its stdout into the message. The message is garbled, a build runs on a path that is supposed to be a plain failure, and under set -e that build's own failure becomes the step's exit reason. Use single quotes, or escape the backticks:
echo "::error::$verify is missing; the step above builds it with 'make ebpf-verify'"2. Deploy hazard: on a config-only deploy the verify gate and the TC object no longer match the XDP object the daemon loads
.github/workflows/deploy-demo-v2.yml:2670 and :2720-2731.
Before this PR, every deploy uploaded nhp_ebpf_xdp.o and tc_egress.o together, so the pair the running binary loads was always the pair the verifier had just accepted. After it, on a deploy with DEPLOY_BINARIES != yes against a host still running the pre-rename nhp-acd:
- the verify gate loads the new
nhp_ac_xdp.o+ newtc_egress.o, - the new
tc_egress.ooverwrites the old one inetc/, - the old binary has no fallback, so it keeps loading the stale
etc/nhp_ebpf_xdp.o. The newrmis correctly gated off here, but nothing replaces the file either.
The daemon is then restarted with a mixed pair: new TC object, old XDP object. The verifier never saw that pair. The two objects share pinned maps by name, and nhp_maps.h / RUNBOOK eBPF map pin mismatch at startup says the loader rejects any shape disagreement. So any change to a shared map shipped through a config-only deploy during the transition causes an AC start failure: an outage, with the backstop holding tcp/443 shut. That is exactly what the verify gate exists to prevent. The comment above the upload also promises that a config-only deploy that flips FilterMode to 1 works without a binary deploy. That promise no longer holds for a host running the old binary that has no legacy file.
Suggested fix: for the transition, on non-binary deploys also upload the new object under the legacy name, so whatever binary is running gets the object the gate verified:
scp ./artifact/ac/etc/nhp_ac_xdp.o nhp-ac:$REMOTE_DIR/etc/
if [ "$DEPLOY_BINARIES" = "yes" ]; then
ssh nhp-ac "rm -f $REMOTE_DIR/etc/nhp_ebpf_xdp.o"
else
# the running binary may predate the rename and load the legacy name
scp ./artifact/ac/etc/nhp_ac_xdp.o nhp-ac:$REMOTE_DIR/etc/nhp_ebpf_xdp.o
fiThe else copy is harmless for a new binary, because acProgObjPath prefers nhp_ac_xdp.o when both exist. Alternatively, require deploy_binaries=yes for the first deploy after this change and say so in RUNBOOK.md.
Minor / suggestions
endpoints/ac/ebpf/ebpfegine.goacProgObjPath: falling back only onos.IsNotExistis right; an EACCES on the new file correctly surfaces as a load error rather than silently using the legacy one. Consider adding a removal target (version or date) to the comment, so the fallback does not stay forever.nhp/utils/ebpf/acxdp_tracked_test.go:TestTrackedAcObjectMatchesTheBuiltOneskips whenrelease/nhp-ac/etc/nhp_ac_xdp.ois missing, so it only takes effect in theebpf-verifierjob, aftermake ebpf-objs. Please confirm that job'sgo testinvocation covers./utils/ebpf/...without a-runfilter that would exclude this test. Otherwise the staleness check never runs.- The tracked blob changed (
0e3186b5b→ceb4bd772), so this is a rebuild, not a pure rename. Please note the clang/libbpf version used, so the next person can reproduce it. engine_linux.go: adding the path to the "object file not found" errors is a good improvement.
What changed
Notes