fix(infrastructure): rpk, Confluent and librdkafka from vendor repos

README.md

@@ -77,7 +77,7 @@ flowchart TD
 | `developer` | Generic CLI dev base (the default: git, docker, shell utilities) |
 | `developer-gui` | VS Code, Ghostty, DBeaver. Privacy + AI-upsell de-nag profile for VSCode/VSCodium/Cursor off unless `-e vscode_privacy_enabled=true` |
 | `developer-rust` / `-go` / `-python` / `-node` / `-typescript` / `-c` | Language toolchains |
-| `infrastructure` | OpenTofu, OpenBao, AWS CLI, helm, terraform-docs, `k8s` (kubectl, kubectx, kubens, k9s, kind, argocd, dive, kustomize, kubeconform, kube-linter), `data` (clickhouse-client, rpk, valkey-cli, vector), `cloudflare` (flarectl, wrangler) |
+| `infrastructure` | OpenTofu, OpenBao, AWS CLI, helm, terraform-docs, `k8s` (kubectl, kubectx, kubens, k9s, kind, argocd, dive, kustomize, kubeconform, kube-linter), `data` (clickhouse-client, confluent-cli, rpk, valkey-cli, vector), `cloudflare` (flarectl, wrangler) |
 | `contributor` | hyperi-ci + its check tools (semgrep, alint), gitleaks, trivy, hadolint, pip-audit, yamllint, ansible-lint, pre-commit, act, git-scrub, macbash |
 | `soe` / `soe-gui` | HyperI org policy (opt-in) |
 | `--full-stack` / `--infra` / `--languages [list]` | Persona bundles (see `--help`) |
@@ -185,7 +185,7 @@ digest. Read that before changing a role or adding a tool.
 - `developer-gui`: VS Code, Ghostty (Solarized theme), DBeaver
 - `vscode-privacy` (off by default): strips the Copilot/AI upsell UI and the telemetry that stock VSCode ships enabled, across VSCode, VSCodium and Cursor. Enable with `-e vscode_privacy_enabled=true`. It merges one marked block into `settings.json` and never touches a comment or a key it does not manage, backs the file up before its first write, and `-e vscode_privacy_uninstall=true` takes only its own keys back out. Where you have set one of those keys yourself further down the file, yours wins and the run tells you which ones -- so it cannot look applied while changing nothing
 - Languages: Rust, Go, Python, C/C++, Node.js, TypeScript (the Astral suite -- uv, ruff, ty -- ships in the base, as does Node.js: it is core tooling that semantic-release and CI need)
-- `infrastructure`: OpenTofu + OpenBao (the OSS forks, no HashiCorp BUSL tools), AWS CLI v2, checkov, and terraform-docs for generating IaC module reference docs (engine-agnostic -- it reads `.tf` whichever binary runs it, and OpenTofu has no native `tofu docs`). Under `k8s`: kubectl + kubectx + kubens + k9s + kind + argocd + dive + kustomize + kubeconform + kube-linter. helm is NOT in that group -- it sits in `cloud`, so plain `--tags infrastructure` gets it whether or not you select `k8s`. The `data` group: clickhouse-client, rpk, valkey-cli, vector. The `cloudflare` group: flarectl + wrangler (flarectl builds from source on both platforms -- Cloudflare ships no binary -- so Linux needs `developer-go`)
+- `infrastructure`: OpenTofu + OpenBao (the OSS forks, no HashiCorp BUSL tools), AWS CLI v2, checkov, and terraform-docs for generating IaC module reference docs (engine-agnostic -- it reads `.tf` whichever binary runs it, and OpenTofu has no native `tofu docs`). Under `k8s`: kubectl + kubectx + kubens + k9s + kind + argocd + dive + kustomize + kubeconform + kube-linter. helm is NOT in that group -- it sits in `cloud`, so plain `--tags infrastructure` gets it whether or not you select `k8s`. The `data` group: clickhouse-client, confluent-cli, rpk, valkey-cli, vector. The `cloudflare` group: flarectl + wrangler (flarectl builds from source on both platforms -- Cloudflare ships no binary -- so Linux needs `developer-go`)
 - `contributor`: hyperi-ci and the tools its checks drive (semgrep, alint), gitleaks, trivy, hadolint, pip-audit, ansible-lint, pre-commit, act, and git-scrub for rewriting AI residue or a leaked secret out of git history -- gitleaks scans FULL history, so a secret removed from HEAD still fails the gate. macbash is here too: it flags the GNU-only bash constructs that break on macOS, and CONTRIBUTING.md asks for it on every shell change
 - `soe` / `soe-gui`: HyperI org policy: VPN clients, Claude Code, Slack, LibreOffice, RDP client, telemetry-disable, auto-updates, GNOME taskbar
 - `developer-ai` (off by default): the OpenAI Codex CLI as a second opinion alongside Claude Code rather than a replacement for it, plus OpenAI's Codex plugin FOR Claude Code, so `/codex:review` and `/codex:adversarial-review` are things Claude asks Codex for. The plugin is skipped -- with a warning naming the tag that fixes it -- unless claude, codex and a new enough node are all present for that user, because it installs happily without them and then throws on every invocation. Sign-in stays the person's: `codex login --device-auth` on a box with no browser

ansible/roles/developer-rust/defaults/main.yml

@@ -169,3 +169,20 @@ rust_governor_cpu_weight: 30
 developer_rust_uid: "{{ actual_user_uid | default(ansible_facts['user_uid']) }}"
 developer_rust_user_env:
   XDG_RUNTIME_DIR: "/run/user/{{ developer_rust_uid }}"
+
+# ---------------------------------------------------------------------------
+# librdkafka from Confluent's clients repository
+# ---------------------------------------------------------------------------
+# Confluent publishes no fingerprint for this key. This pins the key served at
+# packages.confluent.io/clients/{deb,rpm}/archive.key, which signs that repo's
+# metadata -- the same pin scalo-rs's generated images assert.
+developer_rust_confluent_clients_key_fingerprint: CBBB821E8FAF364F79835C438B1DA6120C2BF624  # gitleaks:allow -- public key fingerprint
+
+# Clients-repo apt suites, oldest first; the last entry is the fallback for an
+# Ubuntu release Confluent has not published yet.
+developer_rust_confluent_clients_suites:
+  - noble
+  - resolute
+
+# Confluent builds the clients rpms for RHEL only; Fedora takes this EL release.
+developer_rust_confluent_clients_el_release: 10

ansible/roles/developer-rust/tasks/rust.yml

@@ -385,24 +385,240 @@
 # ============================================================
 # Not for the cargo tools above -- these build the developer's OWN Rust
 # services: protoc for prost/tonic (gRPC codegen at build time), and the
-# librdkafka headers the rdkafka crate links against. Distro packages, so the
-# host's normal updates keep them current.
+# librdkafka headers the rdkafka crate links against. Package repos either way,
+# so the host's normal updates keep them current.
 
-- name: Install gRPC/Kafka build dependencies (Ubuntu)
-  ansible.builtin.apt:
-    name:
-      - protobuf-compiler
-      - librdkafka-dev
+- name: Install the protobuf compiler (Linux)
+  ansible.builtin.package:
+    name: protobuf-compiler
     state: present
-  when: ansible_facts['distribution'] == 'Ubuntu'
+  when: ansible_facts['distribution'] in ['Fedora', 'Ubuntu']
 
-- name: Install gRPC/Kafka build dependencies (Fedora)
-  ansible.builtin.dnf:
-    name:
-      - protobuf-compiler
-      - librdkafka-devel
-    state: present
-  when: ansible_facts['distribution'] == 'Fedora'
+# librdkafka comes from Confluent's clients repo, the build scalo-rs images ship,
+# because the distro builds trail the Kafka protocol. `latest`, so a host that
+# already holds the distro build moves onto Confluent's.
+- name: Install librdkafka from Confluent's clients repository (Linux)
+  vars:
+    developer_rust_confluent_clients_key: >-
+      {{ '/etc/apt/keyrings/confluent-clients.asc' if ansible_facts['distribution'] == 'Ubuntu'
+         else '/etc/pki/rpm-gpg/RPM-GPG-KEY-confluent-clients' }}
+    developer_rust_confluent_clients_suite: >-
+      {{ ansible_facts['distribution_release']
+         if ansible_facts['distribution_release'] in developer_rust_confluent_clients_suites
+         else developer_rust_confluent_clients_suites | last }}
+    # Check mode adds no repo, so there is nothing to install from yet.
+    developer_rust_confluent_clients_repo_pending: >-
+      {{ ansible_check_mode and ((developer_rust_confluent_clients_deb_repo | default({})) is changed
+         or (developer_rust_confluent_clients_rpm_repo | default({})) is changed) }}
+  block:
+    # A minimal Ubuntu ships gpgv but not gpg, which the fingerprint read needs.
+    - name: Ensure gpg is present for the fingerprint check
+      ansible.builtin.package:
+        name: "{{ 'gpg' if ansible_facts['distribution'] == 'Ubuntu' else 'gnupg2' }}"
+        state: present
+
+    # Staged under a name nothing trusts and forced, so a rejected download is
+    # replaced on the next run instead of kept by an If-Modified-Since 304.
+    - name: Download the Confluent clients signing key
+      ansible.builtin.get_url:
+        url: >-
+          https://packages.confluent.io/clients/{{ 'deb' if ansible_facts['distribution'] == 'Ubuntu' else 'rpm' }}/archive.key
+        dest: "{{ developer_rust_confluent_clients_key }}.unverified"
+        mode: '0644'
+        force: true
+      register: developer_rust_confluent_clients_key_download
+
+    # Check mode downloads nothing, so there is no new key to read.
+    - name: Read the Confluent clients signing key fingerprint
+      ansible.builtin.command:
+        argv: [gpg, --show-keys, --with-colons, "{{ developer_rust_confluent_clients_key }}.unverified"]
+      register: developer_rust_confluent_clients_key_read
+      changed_when: false
+      check_mode: false
+      when: not (ansible_check_mode and developer_rust_confluent_clients_key_download is changed)
+
+    # Fails closed: one primary key, and the pinned one, or nothing is trusted.
+    - name: Verify the Confluent clients signing key fingerprint
+      ansible.builtin.assert:
+        that:
+          - developer_rust_key_primaries | int == 1
+          - developer_rust_key_fpr == developer_rust_confluent_clients_key_fingerprint
+        fail_msg: >-
+          Confluent clients key holds {{ developer_rust_key_primaries }} key(s), first
+          {{ developer_rust_key_fpr or 'missing' }}; expected only
+          {{ developer_rust_confluent_clients_key_fingerprint }}
+        quiet: true
+      when: not (ansible_check_mode and developer_rust_confluent_clients_key_download is changed)
+      vars:
+        developer_rust_key_primaries: >-
+          {{ developer_rust_confluent_clients_key_read.stdout_lines | select('match', '^pub:') | list | length }}
+        developer_rust_key_fpr: >-
+          {{ (developer_rust_confluent_clients_key_read.stdout_lines | select('match', '^fpr:')
+              | first | default('')).split(':')[9] | default('') }}
+
+    - name: Install the verified Confluent clients signing key
+      ansible.builtin.copy:
+        src: "{{ developer_rust_confluent_clients_key }}.unverified"
+        dest: "{{ developer_rust_confluent_clients_key }}"
+        remote_src: true
+        owner: root
+        group: root
+        mode: '0644'
+      when: not (ansible_check_mode and developer_rust_confluent_clients_key_download is changed)
+
+    - name: Add the Confluent clients APT repository (Ubuntu)
+      ansible.builtin.deb822_repository:
+        name: confluent-clients
+        types: deb
+        uris: https://packages.confluent.io/clients/deb
+        suites: "{{ developer_rust_confluent_clients_suite }}"
+        components: main
+        signed_by: "{{ developer_rust_confluent_clients_key }}"
+        state: present
+      register: developer_rust_confluent_clients_deb_repo
+      when: ansible_facts['distribution'] == 'Ubuntu'
+
+    # Ubuntu ships librdkafka under the same package names, so the Confluent
+    # build wins on priority rather than on whichever version is higher.
+    - name: Prefer Confluent's librdkafka packages (Ubuntu)
+      ansible.builtin.copy:
+        dest: /etc/apt/preferences.d/confluent-clients
+        owner: root
+        group: root
+        mode: '0644'
+        content: |
+          Package: librdkafka*
+          Pin: origin packages.confluent.io
+          Pin-Priority: 600
+      when: ansible_facts['distribution'] == 'Ubuntu'
+
+    - name: Install librdkafka (Ubuntu)
+      ansible.builtin.apt:
+        name:
+          - librdkafka1
+          - librdkafka-dev
+        state: latest
+        update_cache: true
+      when:
+        - ansible_facts['distribution'] == 'Ubuntu'
+        - not developer_rust_confluent_clients_repo_pending
+
+    - name: Check for Fedora's own librdkafka (Fedora)
+      ansible.builtin.command:
+        argv: [rpm, -q, librdkafka]
+      register: developer_rust_fedora_librdkafka
+      changed_when: false
+      failed_when: false
+      check_mode: false
+      when: ansible_facts['distribution'] == 'Fedora'
+
+    # librdkafka1 does not provide the name `librdkafka`, so a package that
+    # requires it by name (python3-confluent-kafka does) blocks the swap. Checked
+    # before the repo goes in, so a blocked host keeps a working dnf on every run.
+    - name: Find packages that require Fedora's librdkafka by name (Fedora)
+      ansible.builtin.command:
+        argv: [rpm, -q, --whatrequires, librdkafka, --qf, "%{NAME}\n"]
+      register: developer_rust_librdkafka_dependants
+      changed_when: false
+      failed_when: false
+      check_mode: false
+      when:
+        - ansible_facts['distribution'] == 'Fedora'
+        - developer_rust_fedora_librdkafka.rc == 0
+
+    - name: Keep Fedora's librdkafka while a package requires it by name (Fedora)
+      ansible.builtin.fail:
+        msg: >-
+          kept Fedora's librdkafka (required by name by:
+          {{ developer_rust_librdkafka_dependants.stdout_lines | join(', ') }});
+          Confluent's was not installed
+      when:
+        - ansible_facts['distribution'] == 'Fedora'
+        - developer_rust_fedora_librdkafka.rc == 0
+        - developer_rust_librdkafka_dependants.rc == 0
+
+    # Confluent publishes RHEL builds only; Fedora takes the newest EL one,
+    # which links against an older glibc than Fedora ships. The priority masks
+    # Fedora's librdkafka-devel, which shares the Confluent package's name.
+    - name: Add the Confluent clients repository (Fedora)
+      ansible.builtin.yum_repository:
+        name: confluent-clients
+        description: Confluent clients
+        baseurl: "https://packages.confluent.io/clients/rpm/centos/{{ developer_rust_confluent_clients_el_release }}/$basearch"
+        enabled: true
+        gpgcheck: true
+        repo_gpgcheck: true
+        gpgkey: "file://{{ developer_rust_confluent_clients_key }}"
+        priority: '10'
+      register: developer_rust_confluent_clients_rpm_repo
+      when: ansible_facts['distribution'] == 'Fedora'
+
+    # Fedora's librdkafka and Confluent's librdkafka1 ship the same library
+    # files under different names, so one has to replace the other in a single
+    # transaction; packages linked against librdkafka.so.1 stay satisfied.
+    # No --allowerasing: a package that needs Fedora's build by name fails the
+    # swap instead of being removed with it.
+    - name: Swap Fedora's librdkafka for Confluent's (Fedora)
+      ansible.builtin.command:
+        argv: [dnf, -y, swap, librdkafka, librdkafka1]
+      changed_when: true
+      when:
+        - ansible_facts['distribution'] == 'Fedora'
+        - developer_rust_fedora_librdkafka.rc == 0
+
+    - name: Install librdkafka (Fedora)
+      ansible.builtin.dnf:
+        name:
+          - librdkafka1
+          - librdkafka-devel
+        state: latest
+      when:
+        - ansible_facts['distribution'] == 'Fedora'
+        - not developer_rust_confluent_clients_repo_pending
+
+  rescue:
+    # After a key rotation the repo no longer verifies against the trusted
+    # key, and the apt module fails every later cache refresh in the run on it,
+    # where apt-get only warns. The previously trusted key stays.
+    # Deleted as a file: deb822_repository state=absent also deletes the
+    # keyring of the same name, which is the trusted key here.
+    - name: Remove the Confluent clients APT repository after a key mismatch (Ubuntu)
+      ansible.builtin.file:
+        path: /etc/apt/sources.list.d/confluent-clients.sources
+        state: absent
+      when:
+        - ansible_failed_task.action in ['assert', 'ansible.builtin.assert']
+        - ansible_facts['distribution'] == 'Ubuntu'
+
+    - name: Check whether Fedora's librdkafka is still installed (Fedora)
+      ansible.builtin.command:
+        argv: [rpm, -q, librdkafka]
+      register: developer_rust_fedora_librdkafka_kept
+      changed_when: false
+      failed_when: false
+      check_mode: false
+      when: ansible_facts['distribution'] == 'Fedora'
+
+    # Also covers a refused swap: with Fedora's librdkafka kept, the
+    # higher-priority Confluent librdkafka-devel would fail dnf upgrades on its
+    # file conflict.
+    - name: Remove the Confluent clients repository (Fedora)
+      ansible.builtin.yum_repository:
+        name: confluent-clients
+        state: absent
+      when:
+        - ansible_facts['distribution'] == 'Fedora'
+        - (ansible_failed_task.action in ['assert', 'ansible.builtin.assert'])
+          or developer_rust_fedora_librdkafka_kept.rc == 0
+
+    - name: Record that librdkafka did not install
+      # noqa: var-naming[no-role-prefix] -- deploy_warnings is the shared accumulator
+      ansible.builtin.set_fact:
+        deploy_warnings: >-
+          {{ deploy_warnings | default([])
+             + ['librdkafka: ' ~ (ansible_failed_result.msg | default('repo/install failed'))] }}
+
+  when: ansible_facts['distribution'] in ['Fedora', 'Ubuntu']
 
 - name: Install gRPC/Kafka build dependencies (macOS)
   community.general.homebrew:

ansible/roles/infrastructure/defaults/main.yml

@@ -9,6 +9,18 @@
 infrastructure_helm_apt_key_fingerprint: DDF78C3E6EBB2D2CC223C95C62BA89D07698DBC6  # gitleaks:allow -- public key fingerprint
 infrastructure_helm_apt_key_url: https://packages.buildkite.com/helm-linux/helm-debian/gpgkey
 
+# Redpanda, Confluent and ClickHouse publish no fingerprint for their repo keys,
+# so these pin the key each serves at the URL in data_tools.yml, checked to sign
+# that repo's current metadata. A rotation then fails the install loudly.
+#
+# Redpanda's repos sit on Google Artifact Registry: its key signs the apt
+# metadata, Redpanda's own key signs the rpms.
+infrastructure_redpanda_repo_key_fingerprint: 35BAA0B33E9EB396F59CA838C0BA5CE6DC6315A3  # gitleaks:allow -- public key fingerprint
+infrastructure_redpanda_rpm_key_fingerprint: 16C58F679A886A0A8468225BC45B532A7981C4D8  # gitleaks:allow -- public key fingerprint
+infrastructure_confluent_cli_deb_key_fingerprint: BF154723E8BB4B969BA7BAE1F00BDC5567B31D07  # gitleaks:allow -- public key fingerprint
+infrastructure_confluent_cli_rpm_key_fingerprint: 62B65702E8AA3A70B538C8D1E0ECCDAE9610976F  # gitleaks:allow -- public key fingerprint
+infrastructure_clickhouse_key_fingerprint: 3A9EA1193A97B548BE1457D48919F6BD2B48D754  # gitleaks:allow -- public key fingerprint
+
 # Microsoft publishes the Azure CLI repo per Ubuntu codename and has no resolute
 # suite, so a resolute host takes the noble build deliberately.
 # Tracks the supported window (min_ubuntu_version), oldest first; the last entry