DOWNSTREAM: carry: OCPBUGS-103516: Replace bbolt with patched fork to remove MADV_RANDOM - #395
Conversation
openshift-ci-robot
added
jira/severity-critical
|
@hasbro17: This pull request references Jira Issue OCPBUGS-103516, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe pull request updates dependency versions in four Go modules, adds a pinned ChangesDependency alignment
Estimated code review effort: 1 (Trivial) | ~5 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
openshift-ci
Bot
added
the
approved
|
/jira refresh |
openshift-ci-robot
added
jira/valid-bug
|
@hasbro17: This pull request references Jira Issue OCPBUGS-103516, which is valid. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@hasbro17: This pull request references Jira Issue OCPBUGS-103516, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/pj-rehearse pull-ci-openshift-etcd-main-perfscale-control-plane-6nodes |
|
/test perfscale-control-plane-6nodes |
|
/payload-job periodic-ci-openshift-eng-ocp-perfscale-main-aws-5.0-nightly-x86-control-plane-fips-24nodes |
|
@mcornea: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/45e4cef0-90a6-11f1-84e2-697c00fba475-0 |
…to remove MADV_RANDOM Since Linux 6.4, MADV_RANDOM causes the kernel to aggressively evict mmap'd pages, resulting in major page faults on bbolt's database file and order-of-magnitude compaction duration increases. The RHEL 10 kernel will not revert this behavior change, so the fix must be in bbolt. This adds a go.mod replace directive pointing to a patched bbolt fork with the MADV_RANDOM madvise hint removed. The replace should be dropped once the fix is merged upstream (etcd-io/bbolt#939). Ref: etcd-io/bbolt#939 Ref: https://redhat.atlassian.net/browse/OCPBUGS-103516 Assisted-by: Claude Code (Opus 4.6)
76de772 to
b74556a
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
|
@hasbro17: This pull request references Jira Issue OCPBUGS-103516, which is valid. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@REBASE.openshift.md`:
- Around line 78-82: Update step 3 of the release instructions to require
regenerating the complete openshift/bbolt pseudo-version, including both its
commit timestamp and commit hash, then apply that full version consistently in
the root, server, etcdutl, and tests go.mod files.
- Around line 88-89: Keep the carry patch until the corresponding openshift/etcd
branch adopts a released bbolt version containing the MADV_RANDOM fix; do not
remove it solely because the upstream change has been merged. Update the
patch-removal note to make the released-version requirement explicit.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 72c941b4-468c-47c7-8563-db186d74be3b
📒 Files selected for processing (1)
REBASE.openshift.md
c902411 to
ad0b11f
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@REBASE.openshift.md`:
- Around line 78-80: Update Step 2 in the bbolt release instructions to make
cherry-picking the MADV_RANDOM removal patch conditional: maintainers should
apply it only when the selected base bbolt tag does not already contain the fix,
and skip it otherwise.
- Line 85: Update the fenced command block near the closing fence in
REBASE.openshift.md to include a shell language identifier, using sh or bash, so
the block satisfies MD040.
- Around line 81-87: Update the module-update instructions to run the bbolt
replacement and go mod tidy for each module: root, server, etcdutl, and tests.
Use an explicit per-directory command or loop so all four go.mod files resolve
the same commit revision.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 114b4170-2986-401e-bea2-55339c94c527
⛔ Files ignored due to path filters (4)
etcdutl/go.sumis excluded by!**/*.sumgo.sumis excluded by!**/*.sumserver/go.sumis excluded by!**/*.sumtests/go.sumis excluded by!**/*.sum
📒 Files selected for processing (5)
REBASE.openshift.mdetcdutl/go.modgo.modserver/go.modtests/go.mod
🚧 Files skipped from review as they are similar to previous changes (4)
- tests/go.mod
- etcdutl/go.mod
- server/go.mod
- go.mod
…in REBASE.openshift.md Assisted-by: Claude Code (Opus 4.6)
ad0b11f to
e66403a
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
|
/payload-job periodic-ci-openshift-eng-ocp-perfscale-main-aws-5.0-nightly-x86-control-plane-fips-24nodes |
|
@hasbro17: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/3c881a00-915b-11f1-9ca0-358c3187473d-0 |
|
@hasbro17: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dusk125, hasbro17 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/retest-required |
|
/payload 5.0 nightly informing |
|
@hasbro17: trigger 67 job(s) of type informing for the nightly release of OCP 5.0
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/6e3da420-91ce-11f1-9129-a0a8a2976de3-0 |
|
/payload 5.0 nightly blocking |
|
@hasbro17: trigger 14 job(s) of type blocking for the nightly release of OCP 5.0
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/78c80160-91ce-11f1-8bfc-e73edb50fc97-0 |
|
/payload 5.0 nightly blocking |
|
@hasbro17: trigger 14 job(s) of type blocking for the nightly release of OCP 5.0
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/85dcc950-91ea-11f1-83f3-e1c21fa25ec8-0 |
|
/payload 5.0 nightly blocking |
|
@hasbro17: trigger 14 job(s) of type blocking for the nightly release of OCP 5.0
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/28126020-9283-11f1-8940-4aa44e08f917-0 |
|
The blocking payload jobs look good. |
|
/verified by ci |
openshift-ci-robot
added
the
verified
|
@hasbro17: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
609b11e
into
openshift:main
|
@hasbro17: Jira Issue OCPBUGS-103516: Some pull requests linked via external trackers have merged: The following pull request, linked via external tracker, has not merged:
All associated pull requests must be merged or unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with Jira Issue OCPBUGS-103516 has not been moved to the MODIFIED state. This PR is marked as verified. If the remaining PRs listed above are marked as verified before merging, the issue will automatically be moved to VERIFIED after all of the changes from the PRs are available in an accepted nightly payload. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
5 participants
Summary
This replaces the bbolt dependency with the patched
openshift/bboltrelease-5.0branch (v1.4.3 + MADV_RANDOM removal) that removes themadvise(MADV_RANDOM)call from bbolt's mmap path.Updates REBASE.openshift.md with instructions for maintaining the openshift/bbolt fork across etcd rebases, including when to drop it once upstream ships the fix.
Background
Since Linux 6.4 (torvalds/linux@8788f678),
MADV_RANDOMcauses the kernel to aggressively evict mmap'd pages byshort-circuiting the second-chance LRU mechanism. This results in high major page fault rates on bbolt's database file, particularly during compaction which sees order-of-magnitude duration increases. On OCP
5.0 FIPS clusters (RHCOS10, kernel 6.12), this manifests as significant etcd compaction duration increases and elevated major page faults under perfscale workloads.
This is the same root cause as OCPBUGS-50521 (OCP 4.19), which was fixed kernel-side by reverting the behavior change in RHEL 9.6. The RHEL kernel team has stated this behavior will not be reverted in RHEL 10,
so the fix must be in bbolt.
Next steps
openshift/bboltto upstream v1.4.3 and apply the patch thereopenshift/bbolt(or open a new PR)etcd-io/bbolt(#939)References
Summary by CodeRabbit
Chores