Repository navigation
fix(infrastructure): rpk, Confluent and librdkafka from vendor repos - #104
Merged
Merged
Conversation
rpk now comes from Redpanda's package repo as redpanda-rpk, the CLI alone, instead of a GitHub zip in /usr/local/bin that nothing ever updated. The old binary is removed once the package is in. The Confluent CLI moves to its own unversioned repo. The Confluent Platform repo it came from was picked by a lexical sort of version strings, and it carries a confluent-cli numbered by the platform release (8.3.2) that wraps an older CLI (4.72) and outranks the real one on version. The old repo and key are removed, and a confluent-cli the CLI repo does not carry is replaced. librdkafka for the rdkafka crate comes from Confluent's clients repo on Ubuntu and Fedora, matching what scalo-rs images ship, instead of the distro builds (2.3.0 on noble). An apt pin keeps Confluent's build ahead of Ubuntu's same-named packages, and on Fedora a dnf swap replaces the differently named distro package in one transaction. Every new repo key is downloaded to an untrusted path, checked to hold exactly one primary key with the pinned fingerprint, and only then trusted, with failures recorded as deploy warnings. The ClickHouse repo on Fedora now checks its signed metadata; its rpms are unsigned, so package gpgcheck stays off.
The Fedora librdkafka swap no longer passes --allowerasing, which silently removed python3-confluent-kafka (it requires librdkafka by name, which Confluent's librdkafka1 does not provide) along with its python dependants. A host with such a package now keeps Fedora's librdkafka, the Confluent clients repo stays off it so dnf upgrade does not hit the file conflict, and the run records a warning naming the package. Later converges repeat the same check without changing anything. A confluent-cli from the Confluent Platform repo is now replaced in one transaction, a downgrade to the CLI repo's newest build, instead of a remove then install that left no CLI when the download failed. On Fedora the repo query passes -y so the new repo's key is imported on first use. The Platform repo definition is removed, and on Ubuntu its key; on Fedora its key stays in the rpm database, because it is the same key that signs the Confluent clients repo librdkafka now comes from. When a pinned fingerprint check fails, that vendor repo is removed so the apt module's cache refresh does not fail for the rest of the run, and the previously trusted key is left in place. The Ubuntu repo is removed as a file because deb822_repository state=absent also deletes the keyring of the same name. In check mode on a fresh host the key read, verify and install steps and the package installs are skipped instead of failing into a deploy warning. The install docs list the Confluent CLI in the data group and librdkafka under the Confluent clients repo. The data_tools.yml header is ASCII and the stale macOS rpk fallback note is gone.
Contributor
|
Released in v2.24.13 -- https://github.com/hyperi-io/hyperi-developer/releases/tag/v2.24.13 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rpk, the Confluent CLI and librdkafka now come from their vendors' signed repos, so a normal apt or dnf upgrade keeps them current.
redpanda-rpkfrom Redpanda's repo (rpk alone, not the broker). The old GitHub zip copy in /usr/local/bin goes once the package is in.librdkafkaby name (python3-confluent-kafka) keeps Fedora's build and gets a warning, rather than having that package erased.Tested in ubuntu:24.04, ubuntu:26.04, fedora:43 and fedora:44: fresh installs, hosts migrated from the old setup (one copy of each tool afterwards), second converges changed=0, wrong fingerprints fail closed, and check mode on a fresh host raises no spurious warnings. All six pins were re-derived from signed metadata.
Done when rpk, confluent and librdkafka update with the host and nothing else installs them.