From cf47275875e336396d5a8a66edfc6fdc6a075de0 Mon Sep 17 00:00:00 2001 From: Derek Date: Thu, 3 Sep 2026 08:51:17 +1000 Subject: [PATCH 1/2] fix(install): resolve every tool version at install time, and upgrade on re-run A version written into this repo is current on the day someone types it and stale every day after. Go was pinned to 1.26.7 while upstream was on 1.27.1, which is also why hyperi-ci had to bump golangci-lint -- a linter behind the toolchain cannot read the newer stdlib. The `--pinned` flag was never the thing biting: it defaults off, so a normal install already resolved latest for all twelve tools in `hyperi_versions`. What bound on every install was `go`, `rustup`, `fnm`, `openbao` and `flarectl`. Removed outright: - `hyperi_versions`, `hyperi_pinned`, `install.sh --pinned`, and `tools/check_version_pins.py` with its tests. Every task already carried an `else latest` branch, so deleting the map changed no default -- it removed the only path that installed something older. The drift check existed to keep a hand-copied mirror of hyperi-ci honest, and the mirror is what went. Resolved at install time instead: - go: `go.dev/dl/?mode=json` gives the current stable and a SHA256 per file, so the download stays verified. hyperi-update already used this endpoint. - rustup: the current `rustup-init` plus the `.sha256` published beside it. - fnm, openbao, flarectl: `/releases/latest`, or `@latest` for flarectl. Re-running a role now upgrades rather than skipping: - `cargo install-update --all --locked`, gated on `--list` reporting something outdated so a current box pays one registry poll instead of a rebuild sweep. - `rustup update stable`, gated on `rustup check`. - flarectl loses its `creates:` guard, since `go install` re-links regardless. Kept deliberately: `node_major` selects an LTS line rather than a release, and NodeSource's signed repo patches it in place. Rust needs no equivalent -- edition 2024 comes with whatever stable rustup installs. Also fixes a check-mode gap the live resolution exposed: unpacking Go had no `not ansible_check_mode` guard, which never showed while the pin matched what was already installed. run-tests.sh green, 42 tests. Full-repo ansible-lint goes 307 findings to 304. --- .../inventories/localhost/group_vars/all.yml | 71 +---- ansible/roles/contributor/tasks/git_scrub.yml | 14 +- ansible/roles/contributor/tasks/gitleaks.yml | 11 +- ansible/roles/contributor/tasks/hadolint.yml | 12 +- ansible/roles/contributor/tasks/hyperi_ci.yml | 22 +- ansible/roles/developer-go/defaults/main.yml | 9 +- ansible/roles/developer-go/tasks/go.yml | 84 ++--- .../roles/developer-rust/defaults/main.yml | 10 +- ansible/roles/developer-rust/tasks/rust.yml | 120 +++++-- ansible/roles/developer/defaults/main.yml | 2 +- .../files/update/hyperi-update-linux.sh | 7 +- ansible/roles/developer/tasks/nodejs.yml | 11 +- .../roles/infrastructure/defaults/main.yml | 22 +- ansible/roles/infrastructure/tasks/cloud.yml | 3 - .../roles/infrastructure/tasks/cloudflare.yml | 21 +- ansible/roles/infrastructure/tasks/k8s.yml | 21 +- ansible/roles/soe/tasks/arcane.yml | 8 +- docs/install-matrix.md | 292 +++++++++--------- install.sh | 22 +- tools/check_release_matrix.py | 2 +- tools/check_version_pins.py | 138 --------- tools/ci/run-tests.sh | 15 - tools/tests/test_check_version_pins.py | 98 ------ 23 files changed, 343 insertions(+), 672 deletions(-) delete mode 100644 tools/check_version_pins.py delete mode 100644 tools/tests/test_check_version_pins.py diff --git a/ansible/inventories/localhost/group_vars/all.yml b/ansible/inventories/localhost/group_vars/all.yml index 124dc69..6ade18a 100644 --- a/ansible/inventories/localhost/group_vars/all.yml +++ b/ansible/inventories/localhost/group_vars/all.yml @@ -1,21 +1,12 @@ --- -# --pinned mode SSoT. Default is LATEST: hyperi_pinned is false, so every tool -# installs the newest release. `install.sh --pinned` sets hyperi_pinned=true (a -# -e extra var, highest precedence), and each retrofitted manual-binary task -# then fetches the exact tag below instead of /releases/latest. +# Every tool resolves its version at install time, so a run in four months +# installs what is current in four months. # -# hyperi_versions MIRRORS hyperi-ci's config/versions.yaml `tools:` block, so a -# pinned local box matches CI exactly. `tools/ci/run-tests.sh` compares the two -# and fails on any disagreement, so this no longer relies on someone -# remembering; edit a version here and the test says which side is wrong. +# Manual binaries query `/releases/latest`. Cargo tools install unversioned. Go +# and rustup fetch the current version with the checksum published beside it. # -# Pinning is by TAG, exactly as hyperi-ci does today -- a fetched release ASSET -# cannot be force-moved by a tag rewrite, but it is not yet digest-verified. -# SHA256-at-download is the planned hardening (tracked upstream as hyperi-ci -# #66); when it lands, add a `sha256:` per tool here and a `checksum:` on the -# get_url/unarchive. Tools absent from this map fall back to latest even in -# pinned mode (they carry no CI-parity pin to mirror). -hyperi_pinned: false +# What remains below is a selection rather than a version -- a Node LTS line, +# which NodeSource's signed repo then keeps patched. # GitHub's anonymous release API allows 60 requests an hour PER IP, and a run # selecting infrastructure plus a couple of languages spends a good fraction of @@ -40,20 +31,6 @@ hyperi_github_headers: >- hyperi_github_env: >- {{ {'GITHUB_TOKEN': hyperi_github_token} if hyperi_github_token else {} }} -hyperi_versions: - alint: v0.14.1 - gitleaks: v8.30.1 - hadolint: v2.15.1 - kubeconform: v0.8.0 - kube-linter: v0.8.3 - osv-scanner: v2.4.0 - golangci-lint: v2.12.2 - gosec: v2.28.0 - govulncheck: v1.1.4 - cargo-audit: v0.22.2 - cargo-deny: 0.20.2 - cargo-chef: v0.1.77 - # ============================================================================ # Core component versions -- the ONE place to bump them # ============================================================================ @@ -75,11 +52,12 @@ hyperi_versions: # its own via `apt/dnf upgrade`, so only the MAJOR is pinned # here -- there is no patch version to chase, and routine OS # updates carry the tool forward for free. -# pinned artefact go, rustup, fnm. Upstream publishes no repo, so we pin an -# exact version (rung 3) and `hyperi-update` refreshes them. +# resolved artefact go, rustup, fnm. Upstream publishes no repo, so the role +# asks upstream for the current release when it runs and +# verifies the download against the checksum served with it. # -# Bumping a major in a year is editing a number here. Nothing below is repeated -# in a role. +# Only the Node majors remain, because a major is a choice rather than a version +# to chase. Bumping one in a year is editing a number here. hyperi_core_versions: # Node n and n-1. n is installed system-wide from the NodeSource repo, so # root, systemd units, semantic-release and CI all resolve the same node; @@ -88,30 +66,3 @@ hyperi_core_versions: # 26 and 24. node_major: 24 node_major_previous: 22 - - # Go publishes tarballs only -- no apt/dnf repo exists -- so this is the - # pinned-artefact rung: exact version plus a SHA256 we hold, per arch. This - # is the hardening the `hyperi_versions` note above calls planned; for these - # two tools it is done. - # Floor is 1.26.6, which fixed 10 CVEs -- among them a remote DoS against any - # Go TLS server (CVE-2026-56862) and two sumdb flaws (CVE-2026-56864/56865) - # letting a hostile GOPROXY smuggle unauthenticated modules into the cache. - go: "1.26.7" - go_sha256: - amd64: "ffb5f8de10c62550dfddab66b36b57030721e0a44a3218e9e1181d7b59f121ca" - arm64: "5a4ec883379d51ee9ce1040d5e87f8d35e20387574dd8c947feb01eabc3c1b37" - - # rustup-init only BOOTSTRAPS; the toolchain then tracks stable via `rustup - # update`, so this pin ages slowly and does not gate the Rust version. - # Upstream on both distros, not the distro package: Ubuntu's rustup is 1.26.0 - # and ships no rustup-init at all, so it never creates ~/.cargo/bin -- the - # layout every later task in the rust role assumes. - rustup: "1.29.0" - rustup_sha256: - x86_64: "4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10" - aarch64: "9732d6c5e2a098d3521fca8145d826ae0aaa067ef2385ead08e6feac88fa5792" - - # fnm installs the n-1 Node major per user. Pinned by TAG, on the same - # reasoning as hyperi_versions above: a fetched release asset cannot be - # force-moved by a tag rewrite. - fnm: "v1.39.0" diff --git a/ansible/roles/contributor/tasks/git_scrub.yml b/ansible/roles/contributor/tasks/git_scrub.yml index 7b21b3d..93275d0 100644 --- a/ansible/roles/contributor/tasks/git_scrub.yml +++ b/ansible/roles/contributor/tasks/git_scrub.yml @@ -17,12 +17,6 @@ - name: Install git-scrub (re-fetched GitHub release, Tier 3) block: - # --pinned takes the CI-exact tag from group_vars, latest otherwise, so a - # pinned install never depends on the GitHub API. - # - # hyperi-ci carries no git-scrub pin, so --pinned falls through to latest. - # Adding an entry here without the matching hyperi-ci pin turns the build - # red via check_version_pins.py. - name: Get latest git-scrub version from GitHub API ansible.builtin.uri: url: https://api.github.com/repos/hyperi-io/git-scrub/releases/latest @@ -30,14 +24,10 @@ headers: "{{ hyperi_github_headers }}" register: contributor_git_scrub_release check_mode: false - when: not (hyperi_pinned | default(false) and 'git-scrub' in (hyperi_versions | default({}))) - - name: Resolve the git-scrub tag (pinned or latest) + - name: Resolve the git-scrub tag ansible.builtin.set_fact: - contributor_git_scrub_ref: >- - {{ hyperi_versions['git-scrub'] - if (hyperi_pinned | default(false) and 'git-scrub' in (hyperi_versions | default({}))) - else contributor_git_scrub_release.json.tag_name }} + contributor_git_scrub_ref: "{{ contributor_git_scrub_release.json.tag_name }}" # hyperi_arch_deb already spells the architecture the way this release does. - name: Build the git-scrub asset name diff --git a/ansible/roles/contributor/tasks/gitleaks.yml b/ansible/roles/contributor/tasks/gitleaks.yml index d2af4c1..97135cd 100644 --- a/ansible/roles/contributor/tasks/gitleaks.yml +++ b/ansible/roles/contributor/tasks/gitleaks.yml @@ -4,8 +4,7 @@ # Fedora dnf (8.30.0 against upstream 8.30.1), Ubuntu release binary, macOS brew. # # Ubuntu universe carries 8.16.0 and freezes for the life of the release, which -# is fourteen minors adrift on a secret scanner. It also cannot honour the -# `hyperi_versions` pin, so a `--pinned` box ran a different scanner from CI. +# is fourteen minors adrift on a secret scanner. # # There is no better Linux channel to move to, and it was looked for. Upstream # documents only Homebrew, Docker, Go and the release binaries -- they run no @@ -67,14 +66,10 @@ headers: "{{ hyperi_github_headers }}" register: contributor_gitleaks_latest check_mode: false - when: not (hyperi_pinned | default(false) and 'gitleaks' in (hyperi_versions | default({}))) - - name: Resolve the Gitleaks tag (pinned or latest) + - name: Resolve the Gitleaks tag ansible.builtin.set_fact: - contributor_gitleaks_ref: >- - {{ hyperi_versions['gitleaks'] - if (hyperi_pinned | default(false) and 'gitleaks' in (hyperi_versions | default({}))) - else contributor_gitleaks_latest.json.tag_name }} + contributor_gitleaks_ref: "{{ contributor_gitleaks_latest.json.tag_name }}" # gitleaks calls amd64 "x64"; arm64 keeps its own name. - name: Map the architecture to the Gitleaks asset token diff --git a/ansible/roles/contributor/tasks/hadolint.yml b/ansible/roles/contributor/tasks/hadolint.yml index c9ece75..9e383f5 100644 --- a/ansible/roles/contributor/tasks/hadolint.yml +++ b/ansible/roles/contributor/tasks/hadolint.yml @@ -52,9 +52,7 @@ ansible.builtin.set_fact: contributor_hadolint_arch: "{{ 'arm64' if hyperi_arch_deb == 'arm64' else 'x86_64' }}" - # --pinned takes the CI-exact tag from group_vars; latest otherwise -- and - # only then do we hit the GitHub API, so a pinned install does not depend - # on it. Same shape as gitleaks.yml in this role. + # Same shape as gitleaks.yml in this role. - name: Get latest hadolint version ansible.builtin.uri: url: https://api.github.com/repos/hadolint/hadolint/releases/latest @@ -62,14 +60,10 @@ headers: "{{ hyperi_github_headers }}" register: contributor_hadolint_latest check_mode: false - when: not (hyperi_pinned | default(false) and 'hadolint' in (hyperi_versions | default({}))) - - name: Resolve the hadolint tag (pinned or latest) + - name: Resolve the hadolint tag ansible.builtin.set_fact: - contributor_hadolint_ref: >- - {{ hyperi_versions['hadolint'] - if (hyperi_pinned | default(false) and 'hadolint' in (hyperi_versions | default({}))) - else contributor_hadolint_latest.json.tag_name }} + contributor_hadolint_ref: "{{ contributor_hadolint_latest.json.tag_name }}" - name: Download hadolint binary (Ubuntu) ansible.builtin.get_url: diff --git a/ansible/roles/contributor/tasks/hyperi_ci.yml b/ansible/roles/contributor/tasks/hyperi_ci.yml index 66ede3f..a500383 100644 --- a/ansible/roles/contributor/tasks/hyperi_ci.yml +++ b/ansible/roles/contributor/tasks/hyperi_ci.yml @@ -25,7 +25,6 @@ # in place, not skipped when already present. `uv tool install --upgrade` # installs it if missing and pulls the latest PyPI release if present, so a # staff/contributor box that ran an earlier version does not stay pinned to it. -# (In --pinned mode this instead installs the version from versions.yml.) - name: Install or upgrade hyperi-ci to the latest release ansible.builtin.command: cmd: uv tool install --upgrade hyperi-ci @@ -104,14 +103,9 @@ # of THIS repo (see CONTRIBUTING), not something hyperi-ci calls. # # Not packaged anywhere, so cargo it is -- the bottom rung, and the only rung. -# --pinned takes the exact crates.io release from hyperi_versions -- latest otherwise. - name: Install alint (repository-structure linter, NOT ansible-lint) ansible.builtin.command: - cmd: >- - cargo install alint - {{ ('--version ' ~ (hyperi_versions['alint'] | regex_replace('^v', ''))) - if (hyperi_pinned | default(false) and 'alint' in (hyperi_versions | default({}))) - else '' }} + cmd: cargo install alint creates: "{{ user_home }}/.cargo/bin/alint" become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" @@ -149,7 +143,7 @@ # honoured and the confined build can read only $HOME. Repos under /projects are # invisible to it, and the resulting scan failure surfaces through hyperi-ci as # "issues found (non-blocking)" -- a gate failing open while reading like a -# result. It also ignores hyperi_versions and refreshes on snapd's schedule. +# result. It also refreshes on snapd's own schedule. - name: Install osv-scanner (macOS) community.general.homebrew: name: osv-scanner @@ -168,9 +162,7 @@ - name: Install osv-scanner (Linux -- re-fetched GitHub release, Tier 3) when: ansible_facts['distribution'] in ['Ubuntu', 'Fedora'] block: - # --pinned takes the CI-exact tag from group_vars; latest otherwise -- and - # only then do we hit the GitHub API, so a pinned install does not depend - # on it. Same shape as kubeconform in the infrastructure role. + # Same shape as kubeconform in the infrastructure role. - name: Get latest osv-scanner version ansible.builtin.uri: url: https://api.github.com/repos/google/osv-scanner/releases/latest @@ -178,14 +170,10 @@ headers: "{{ hyperi_github_headers }}" register: contributor_osv_latest check_mode: false - when: not (hyperi_pinned | default(false) and 'osv-scanner' in (hyperi_versions | default({}))) - - name: Resolve the osv-scanner tag (pinned or latest) + - name: Resolve the osv-scanner tag ansible.builtin.set_fact: - contributor_osv_ref: >- - {{ hyperi_versions['osv-scanner'] - if (hyperi_pinned | default(false) and 'osv-scanner' in (hyperi_versions | default({}))) - else contributor_osv_latest.json.tag_name }} + contributor_osv_ref: "{{ contributor_osv_latest.json.tag_name }}" - name: Download osv-scanner ansible.builtin.get_url: diff --git a/ansible/roles/developer-go/defaults/main.yml b/ansible/roles/developer-go/defaults/main.yml index fbdf54a..9eb3a16 100644 --- a/ansible/roles/developer-go/defaults/main.yml +++ b/ansible/roles/developer-go/defaults/main.yml @@ -1,12 +1,9 @@ --- # Developer-Go defaults. # -# The version lives in the SSoT -- hyperi_core_versions in -# inventories/localhost/group_vars/all.yml. This file only selects the checksum -# for the running architecture, so a Go bump stays a one-line edit there. - -go_version: "{{ hyperi_core_versions.go }}" -go_sha256: "{{ hyperi_core_versions.go_sha256[hyperi_arch_deb | default('amd64')] }}" +# go_version and go_sha256 are not set here. Both are resolved from +# go.dev/dl/?mode=json when the role runs, so the install tracks the current +# stable release and still verifies what it downloads. # Go is installed to /usr/local/go, so it is NOT on a non-login shell's PATH # during the run itself -- /etc/profile.d is only sourced by login shells, and diff --git a/ansible/roles/developer-go/tasks/go.yml b/ansible/roles/developer-go/tasks/go.yml index 0b5bfec..19a8a85 100644 --- a/ansible/roles/developer-go/tasks/go.yml +++ b/ansible/roles/developer-go/tasks/go.yml @@ -1,17 +1,15 @@ --- # Go toolchain -- installs Go + gopls + dlv. # -# Linux: the official tarball from go.dev, pinned by version with a SHA256 we -# hold (see hyperi_core_versions in group_vars/all.yml). Go publishes no apt or -# dnf repo, so this is the best rung of the trust ladder available for it. -# macOS: Homebrew, which tracks upstream. +# Linux: the official tarball from go.dev, resolved to the current stable +# release at install time and verified against the SHA256 go.dev publishes with +# it. Go offers no apt or dnf repo, so this is the best rung of the trust +# ladder available for it. macOS: Homebrew, which tracks upstream. # # NOT the distro package. Ubuntu 24.04 ships golang-go 1.22, which upstream # stopped supporting long ago -- Go maintains only the last two majors -- so a # box provisioned from the distro cannot build against a current toolchain and -# gets no security fixes for the one it has. The old comment here already -# admitted the version "lags" and pointed at the official tarball as the -# fallback; nothing implemented it, so the lag was simply shipped. +# gets no security fixes for the one it has. # # Installs to /usr/local/go, the layout go.dev documents and every Go tool # expects. delve stays a distro package: it is a debugger, not the toolchain, @@ -19,9 +17,31 @@ - name: Install the Go toolchain (Linux) block: - # `go version` prints e.g. "go version go1.26.5 linux/arm64". If the pinned - # version is already in place there is nothing to remove or unpack, so the - # run stays idempotent instead of re-downloading a ~70MB tarball each time. + # go.dev serves the current stable releases newest-first, each with a + # SHA256 per file. Taking both from the same response is what keeps the + # download verified while the version is resolved at install time. + - name: Look up the current Go release + ansible.builtin.uri: + url: https://go.dev/dl/?mode=json + return_content: true + register: developer_go_releases + check_mode: false + + - name: Select the Go version and its checksum + ansible.builtin.set_fact: + go_version: "{{ developer_go_release.version | regex_replace('^go', '') }}" + go_sha256: >- + {{ (developer_go_release.files + | selectattr('os', 'equalto', 'linux') + | selectattr('arch', 'equalto', hyperi_arch_deb) + | selectattr('kind', 'equalto', 'archive') + | first).sha256 }} + vars: + developer_go_release: "{{ developer_go_releases.json | first }}" + + # `go version` prints e.g. "go version go1.26.5 linux/arm64". If the + # current version is already in place there is nothing to remove or unpack, + # so the run stays idempotent instead of re-downloading a ~70MB tarball. - name: Check the installed Go version ansible.builtin.command: cmd: /usr/local/go/bin/go version @@ -29,7 +49,7 @@ changed_when: false failed_when: false - - name: Note whether the pinned Go is already installed + - name: Note whether the current Go is already installed ansible.builtin.set_fact: go_installed_current: >- {{ ('go' ~ go_version ~ ' ') in (developer_go_installed.stdout | default('')) }} @@ -50,6 +70,8 @@ state: absent when: not go_installed_current + # Check mode does not perform the download above, so the source would be + # missing and this would fail the run rather than preview it. - name: Unpack Go into /usr/local ansible.builtin.unarchive: src: "/tmp/go{{ go_version }}.linux-{{ hyperi_arch_deb }}.tar.gz" @@ -57,7 +79,9 @@ remote_src: true owner: root group: root - when: not go_installed_current + when: + - not go_installed_current + - not ansible_check_mode - name: Remove the Go tarball download ansible.builtin.file: @@ -122,16 +146,12 @@ # golangci-lint: GitHub release binary on Linux (Tier 3), brew on macOS. # -# `hyperi_versions` pins it for CI parity, and neither a snap nor a dnf package -# can honour a version pin -- so under `--pinned` the box ran whatever the -# channel happened to hold instead of the version CI runs. The release binary is -# the only Linux channel that takes the pin. -# -# It is also the current one: Fedora's dnf package trails two minors behind -# upstream, and the snap cannot install on Fedora at all because nothing here -# installs snapd. +# The release binary is the current one: Fedora's dnf package trails two minors +# behind upstream, and the snap cannot install on Fedora at all because nothing +# here installs snapd. A linter behind the Go toolchain cannot read the newer +# stdlib, so trailing is a real failure rather than a cosmetic one. # -# The asset drops the leading v (v2.12.2 -> golangci-lint-2.12.2-linux-amd64) +# The asset drops the leading v (v2.13.1 -> golangci-lint-2.13.1-linux-amd64) # and nests the binary one directory deep. - name: Install golangci-lint (macOS) community.general.homebrew: @@ -150,14 +170,10 @@ headers: "{{ hyperi_github_headers }}" register: golangci_latest check_mode: false - when: not (hyperi_pinned | default(false) and 'golangci-lint' in (hyperi_versions | default({}))) - - name: Resolve the golangci-lint tag (pinned or latest) + - name: Resolve the golangci-lint tag ansible.builtin.set_fact: - golangci_ref: >- - {{ hyperi_versions['golangci-lint'] - if (hyperi_pinned | default(false) and 'golangci-lint' in (hyperi_versions | default({}))) - else golangci_latest.json.tag_name }} + golangci_ref: "{{ golangci_latest.json.tag_name }}" - name: Download and extract golangci-lint ansible.builtin.unarchive: @@ -219,14 +235,9 @@ when: ansible_facts['distribution'] == 'Fedora' -# --pinned takes the exact module version from hyperi_versions -- latest otherwise. - name: Install gosec (Ubuntu — go install) ansible.builtin.command: - cmd: >- - go install github.com/securego/gosec/v2/cmd/gosec@{{ - hyperi_versions['gosec'] - if (hyperi_pinned | default(false) and 'gosec' in (hyperi_versions | default({}))) - else 'latest' }} + cmd: go install github.com/securego/gosec/v2/cmd/gosec@latest environment: "{{ go_bin_env }}" become: true become_user: "{{ actual_user }}" @@ -243,14 +254,9 @@ environment: "{{ homebrew_env }}" when: ansible_facts['distribution'] == 'MacOSX' -# --pinned takes the exact module version from hyperi_versions -- latest otherwise. - name: Install govulncheck (Linux — go install) ansible.builtin.command: - cmd: >- - go install golang.org/x/vuln/cmd/govulncheck@{{ - hyperi_versions['govulncheck'] - if (hyperi_pinned | default(false) and 'govulncheck' in (hyperi_versions | default({}))) - else 'latest' }} + cmd: go install golang.org/x/vuln/cmd/govulncheck@latest environment: "{{ go_bin_env }}" become: true become_user: "{{ actual_user }}" diff --git a/ansible/roles/developer-rust/defaults/main.yml b/ansible/roles/developer-rust/defaults/main.yml index d3023e2..fe73f33 100644 --- a/ansible/roles/developer-rust/defaults/main.yml +++ b/ansible/roles/developer-rust/defaults/main.yml @@ -1,13 +1,9 @@ --- # Developer-Rust defaults. # -# The versions themselves live in the SSoT -- hyperi_core_versions in -# inventories/localhost/group_vars/all.yml. This file only maps them onto the -# arch token rust-lang.org uses in its paths (the GNU triple spelling), so a -# bump is still a one-line edit in the SSoT and nothing here changes. - -rustup_version: "{{ hyperi_core_versions.rustup }}" -rustup_sha256: "{{ hyperi_core_versions.rustup_sha256[hyperi_arch_gnu | default('x86_64')] }}" +# No rustup version is set here. The role fetches static.rust-lang.org's +# current rustup-init together with the checksum published beside it, and +# rustup then tracks stable on its own. # --------------------------------------------------------------------------- # Build cache caps diff --git a/ansible/roles/developer-rust/tasks/rust.yml b/ansible/roles/developer-rust/tasks/rust.yml index c0f0285..3b5b285 100644 --- a/ansible/roles/developer-rust/tasks/rust.yml +++ b/ansible/roles/developer-rust/tasks/rust.yml @@ -1,11 +1,11 @@ --- # Rust toolchain and cargo tools installation # -# rustup comes from upstream (static.rust-lang.org), pinned by version with a -# SHA256 we hold -- see hyperi_core_versions in group_vars/all.yml. Not a -# `curl | sh`: the binary is fetched by get_url and its checksum is VERIFIED -# before it executes, which is the whole difference the install policy is -# about. And not the distro package either, on either distro: +# rustup comes from upstream (static.rust-lang.org), resolved to the current +# rustup-init at install time and checked against the SHA256 published beside +# it. Not a `curl | sh`: the binary is fetched by get_url and its checksum is +# verified before it executes, which is the whole difference the install policy +# is about. And not the distro package either, on either distro: # # Ubuntu's rustup is 1.26.0 and ships NO rustup-init -- only /usr/bin/rustup # plus proxies. `rustup default stable` there installs a toolchain but never @@ -16,9 +16,9 @@ # Fedora's rustup is current TODAY, but that is a coincidence of timing, not # a guarantee -- it freezes on its own schedule like every distro package. # -# rustup only bootstraps; the toolchain then tracks stable via `rustup update` -# (hyperi-update runs it), so the pin above ages slowly and never gates the -# Rust version a developer actually compiles with. +# rustup only bootstraps. The toolchain then tracks stable via `rustup update` +# (hyperi-update runs it), so what a developer compiles with is whatever stable +# is on the day, edition 2024 included. # ============================================================ # Rust Installation (Linux -- one path for both distros) @@ -60,15 +60,24 @@ - name: Bootstrap the Rust toolchain when: not developer_rustup_bin.stat.exists block: - # checksum: is the point of this task. A tag or a CDN object can move; - # a SHA256 we hold cannot be moved by anyone upstream. + # The checksum is fetched from the same publisher, immediately before + # the binary, so the download stays verified without holding a version + # here. The file reads ` *./rustup-init`. + - name: Look up the current rustup-init checksum + ansible.builtin.uri: + url: >- + https://static.rust-lang.org/rustup/dist/{{ hyperi_arch_gnu }}-unknown-linux-gnu/rustup-init.sha256 + return_content: true + register: developer_rustup_sha + check_mode: false + - name: Download rustup-init ansible.builtin.get_url: url: >- - https://static.rust-lang.org/rustup/archive/{{ rustup_version }}/{{ hyperi_arch_gnu }}-unknown-linux-gnu/rustup-init + https://static.rust-lang.org/rustup/dist/{{ hyperi_arch_gnu }}-unknown-linux-gnu/rustup-init dest: /tmp/rustup-init mode: '0755' - checksum: "sha256:{{ rustup_sha256 }}" + checksum: "sha256:{{ developer_rustup_sha.content.split() | first }}" # --no-modify-path: this role already manages the ~/.cargo/bin PATH # entry in .bashrc below, and letting rustup-init add its own leaves two. @@ -87,6 +96,39 @@ path: /tmp/rustup-init state: absent + # rustup-init above only runs on a box that has no rustup, so this is what + # moves an existing toolchain forward and makes a re-run land on current + # stable. `rustup check` reports without changing anything, and exits + # non-zero merely because a rustup self-update is offered. + - name: Check whether the Rust toolchain is current + ansible.builtin.command: + cmd: "{{ user_home }}/.cargo/bin/rustup check" + become: true + become_user: "{{ actual_user }}" + environment: + CARGO_HOME: "{{ user_home }}/.cargo" + RUSTUP_HOME: "{{ user_home }}/.rustup" + register: developer_rustup_check + changed_when: false + failed_when: false + check_mode: false + + # Matched on the stable line alone -- `rustup - update available` is rustup + # offering to replace itself, which `rustup self update` owns, not this. + - name: Update the Rust toolchain to current stable + ansible.builtin.command: + cmd: "{{ user_home }}/.cargo/bin/rustup update stable" + become: true + become_user: "{{ actual_user }}" + environment: + CARGO_HOME: "{{ user_home }}/.cargo" + RUSTUP_HOME: "{{ user_home }}/.rustup" + changed_when: true + when: >- + developer_rustup_check.stdout_lines + | select('search', '^stable-.*update available') + | list | length > 0 + - name: Install Rust components via rustup ansible.builtin.command: cmd: "{{ user_home }}/.cargo/bin/rustup component add rustfmt clippy" @@ -293,15 +335,10 @@ # tested and a semver-compatible upstream release can break one box and not # the next. These compile from source, so a cold box pays for it once. # -# --pinned resolves per-item from hyperi_versions keyed by item.crate -- only # cargo-deny carries a pin today, so every other entry falls through to latest. - name: Install the cargo tool set ansible.builtin.command: - cmd: >- - cargo install {{ item.crate }} --locked - {{ ('--version ' ~ (hyperi_versions[item.crate] | regex_replace('^v', ''))) - if (hyperi_pinned | default(false) and item.crate in (hyperi_versions | default({}))) - else '' }} + cmd: cargo install {{ item.crate }} --locked creates: "{{ user_home }}/.cargo/bin/{{ item.bin }}" loop: - {crate: bacon, bin: bacon} @@ -357,14 +394,9 @@ become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" -# --pinned takes the exact crates.io release from hyperi_versions -- latest otherwise. - name: Install cargo-chef (Docker layer caching) ansible.builtin.command: - cmd: >- - cargo install cargo-chef --locked - {{ ('--version ' ~ (hyperi_versions['cargo-chef'] | regex_replace('^v', ''))) - if (hyperi_pinned | default(false) and 'cargo-chef' in (hyperi_versions | default({}))) - else '' }} + cmd: cargo install cargo-chef --locked creates: "{{ user_home }}/.cargo/bin/cargo-chef" environment: "{{ cargo_env }}" become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" @@ -374,16 +406,9 @@ # cargo-audit scans deps against the RustSec advisory DB, cargo-hack runs the # feature-matrix build. Same cargo channel + `cargo install-update -a` refresh # path as the tools above. -# -# --pinned takes cargo-audit's exact release from hyperi_versions -- cargo-hack -# carries no pin, so it always tracks latest. - name: Install cargo-audit (RustSec advisory audit) ansible.builtin.command: - cmd: >- - cargo install cargo-audit --locked - {{ ('--version ' ~ (hyperi_versions['cargo-audit'] | regex_replace('^v', ''))) - if (hyperi_pinned | default(false) and 'cargo-audit' in (hyperi_versions | default({}))) - else '' }} + cmd: cargo install cargo-audit --locked creates: "{{ user_home }}/.cargo/bin/cargo-audit" environment: "{{ cargo_env }}" become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" @@ -410,6 +435,37 @@ become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" +# The installs above are guarded by `creates:`, so on a box that already has a +# tool they never look at its version. This is what carries them forward, and +# it is why re-running the role upgrades rather than only installing. +# +# Listed first so a run with nothing outdated costs one registry poll instead of +# a from-source rebuild sweep. The list prints a `Needs update` column per tool. +- name: Check which cargo tools are outdated + ansible.builtin.command: + cmd: cargo install-update --list + environment: "{{ cargo_env }}" + become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" + become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" + register: developer_cargo_outdated + changed_when: false + failed_when: false + check_mode: false + +# --locked matches how the installs above were done and how hyperi-update +# refreshes them. Without it, install-update rebuilds against freshly resolved +# dependencies rather than the lockfile the author published. +- name: Update the cargo-installed tools to latest + ansible.builtin.command: + cmd: cargo install-update --all --locked + environment: "{{ cargo_env }}" + become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" + become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" + changed_when: true + when: + - developer_cargo_outdated.rc | default(1) == 0 + - developer_cargo_outdated.stdout_lines | select('search', '\s+Yes\s*$') | list | length > 0 + # ============================================================ # Build acceleration: sccache + mold # ============================================================ diff --git a/ansible/roles/developer/defaults/main.yml b/ansible/roles/developer/defaults/main.yml index 3f60d0d..de589af 100644 --- a/ansible/roles/developer/defaults/main.yml +++ b/ansible/roles/developer/defaults/main.yml @@ -65,7 +65,7 @@ nodesource_deb_gpg_url: "{{ nodesource_deb_base_url }}/gpgkey/nodesource-repo.gp nodesource_rpm_gpg_url: "{{ nodesource_rpm_base_url }}/gpgkey/ns-operations-public.key" # fnm ships one asset per arch, and its arm64 asset is not named with the # Debian token, so it gets its own mapping rather than reusing hyperi_arch_deb. -fnm_version: "{{ hyperi_core_versions.fnm }}" +# The release tag is resolved at install time, so no version sits here. fnm_asset: "{{ 'fnm-arm64.zip' if hyperi_arch_deb | default('amd64') == 'arm64' else 'fnm-linux.zip' }}" # GitHub CLI -- GitHub's own signed repo on both distros. No version is pinned: diff --git a/ansible/roles/developer/files/update/hyperi-update-linux.sh b/ansible/roles/developer/files/update/hyperi-update-linux.sh index 7ec9258..33966ef 100644 --- a/ansible/roles/developer/files/update/hyperi-update-linux.sh +++ b/ansible/roles/developer/files/update/hyperi-update-linux.sh @@ -301,14 +301,13 @@ fi # --- Go toolchain ---------------------------------------------------------- # Go publishes no apt/dnf repo, so `apt/dnf upgrade` never moves it -- the -# playbook installs a pinned tarball into /usr/local/go and this section is what -# carries it forward. The pin in group_vars is the BOOTSTRAP floor, not the -# running version; same arrangement as rustup, where the pinned rustup-init +# playbook installs the current tarball into /usr/local/go and this section is +# what carries it forward. Same arrangement as rustup, where rustup-init # bootstraps and `rustup update` tracks stable after that. # # The checksum is taken from the same go.dev index that gives the URL, so it # guards against a corrupt or truncated download rather than against go.dev -# itself. The install-time pin in group_vars is the one we hold. +# itself. section "Go toolchain" if [[ ! -x /usr/local/go/bin/go ]]; then skip "Go toolchain not found in /usr/local/go" diff --git a/ansible/roles/developer/tasks/nodejs.yml b/ansible/roles/developer/tasks/nodejs.yml index 2361e73..99bdd3b 100644 --- a/ansible/roles/developer/tasks/nodejs.yml +++ b/ansible/roles/developer/tasks/nodejs.yml @@ -138,10 +138,19 @@ - name: Install fnm and the previous Node major (Linux) block: + - name: Get the latest fnm version + ansible.builtin.uri: + url: https://api.github.com/repos/Schniz/fnm/releases/latest + return_content: true + headers: "{{ hyperi_github_headers }}" + register: developer_fnm_latest + check_mode: false + - name: Download fnm ansible.builtin.get_url: url: >- - https://github.com/Schniz/fnm/releases/download/{{ fnm_version }}/{{ fnm_asset }} + https://github.com/Schniz/fnm/releases/download/{{ + developer_fnm_latest.json.tag_name }}/{{ fnm_asset }} dest: /tmp/fnm.zip mode: '0644' diff --git a/ansible/roles/infrastructure/defaults/main.yml b/ansible/roles/infrastructure/defaults/main.yml index 12d2739..6427037 100644 --- a/ansible/roles/infrastructure/defaults/main.yml +++ b/ansible/roles/infrastructure/defaults/main.yml @@ -1,26 +1,8 @@ --- # Infrastructure role defaults -# OpenBao publishes no APT repository, only GitHub release .debs, so Ubuntu -# needs an explicit version here. Fedora (dnf) and macOS (brew) ignore this and -# track their own repos. -# -# Pinned rather than `latest` so a run is reproducible and a bad upstream -# release cannot silently land on every workstation. Bump it deliberately: -# gh api repos/openbao/openbao/releases/latest --jq .tag_name -openbao_version: "2.6.0" - -# OpenBao's release assets are named linux_amd64 / linux_arm64. Do not hardcode -# amd64: arm64 Linux is real here (arm64 VMs, containers on Apple Silicon). -openbao_deb_arch: "{{ 'arm64' if ansible_facts['architecture'] == 'aarch64' else 'amd64' }}" - -# flarectl only exists on cloudflare-go's v0 branch, which carries no semver -# compatibility promise, so a tag is pinned rather than tracking @latest. It has -# no hyperi_versions entry because hyperi-ci does not use it, and that map is a -# mirror of hyperi-ci's pins. Bump it deliberately: -# gh api 'repos/cloudflare/cloudflare-go/tags?per_page=100' \ -# --jq '[.[].name | select(startswith("v0."))] | .[0]' -infrastructure_flarectl_version: "v0.117.0" +# OpenBao comes from its signed APT repo on Ubuntu, dnf on Fedora and brew on +# macOS, so no version is carried here -- each repo keeps the box current. # Microsoft publishes the Azure CLI repo per Ubuntu codename and has no resolute # suite, so a resolute host takes the noble build deliberately. diff --git a/ansible/roles/infrastructure/tasks/cloud.yml b/ansible/roles/infrastructure/tasks/cloud.yml index 3829359..1f218c2 100644 --- a/ansible/roles/infrastructure/tasks/cloud.yml +++ b/ansible/roles/infrastructure/tasks/cloud.yml @@ -358,9 +358,6 @@ # The repo is signed by the `[S]` subkey E617DCD4065C2AFC0B2CF7A7BA8BC08C0F691F94 # of the project key below -- verify that chain again before changing the URL. # Fedora keeps its native dnf package (a higher rung); macOS uses brew. -# -# (openbao_version / openbao_deb_arch in defaults/main.yml are now unused on -# Ubuntu -- kept for the opt-in --pinned path, which is out of scope here.) - name: Install OpenBao from the official repository (Ubuntu) block: - name: Add the OpenBao GPG key diff --git a/ansible/roles/infrastructure/tasks/cloudflare.yml b/ansible/roles/infrastructure/tasks/cloudflare.yml index c21bbdf..9f25412 100644 --- a/ansible/roles/infrastructure/tasks/cloudflare.yml +++ b/ansible/roles/infrastructure/tasks/cloudflare.yml @@ -66,14 +66,12 @@ # path below resolves only along the v0 line and `@latest` cannot jump to a # version without flarectl in it. # -# Pinned to a tag rather than @latest. v0.x carries no semver compatibility -# promise, so @latest can change behaviour on the next fresh box with nothing -# recording what the previous one had. The Go module proxy verifies the module -# against sum.golang.org, so the tag is the reproducibility gap, not integrity. +# The Go module proxy verifies the module against sum.golang.org, so integrity +# holds regardless of which version `@latest` resolves to. # -# `creates:` supplies the idempotency, at the cost of no auto-update on a later -# run: `go install` re-links unconditionally and there is no version to compare -# against. Bumping the pin means removing the binary first. +# The task always reports changed. `go install` re-links unconditionally and +# offers no version to compare against, so there is nothing to key idempotency +# off without losing the upgrade. - name: Install flarectl (Linux) when: ansible_facts['distribution'] in ['Fedora', 'Ubuntu'] @@ -98,12 +96,17 @@ ~ 'Cloudflare ships no flarectl binary, so Go is the only route on Linux.'] }} when: infrastructure_go_present.rc != 0 + # flarectl lives on cloudflare-go's v0 branch, which makes no semver + # promise, so `@latest` can bring a breaking CLI change. Taken deliberately + # -- it is a command-line tool, and a stale one is the worse failure. + # No `creates:` guard. `go install` re-links unconditionally, which is what + # carries an existing flarectl forward on a re-run. - name: Build and install flarectl ansible.builtin.command: cmd: >- go install - github.com/cloudflare/cloudflare-go/cmd/flarectl@{{ infrastructure_flarectl_version }} - creates: "{{ user_home }}/go/bin/flarectl" + github.com/cloudflare/cloudflare-go/cmd/flarectl@latest + changed_when: true become: true become_user: "{{ actual_user }}" environment: diff --git a/ansible/roles/infrastructure/tasks/k8s.yml b/ansible/roles/infrastructure/tasks/k8s.yml index 59dcd95..9d4e499 100644 --- a/ansible/roles/infrastructure/tasks/k8s.yml +++ b/ansible/roles/infrastructure/tasks/k8s.yml @@ -266,9 +266,7 @@ - ansible_facts['distribution'] == 'Ubuntu' - not ansible_check_mode - # kubeconform (CI parity: hyperi-ci runs it as a blocking gate). --pinned - # takes the CI-exact tag from group_vars; latest otherwise -- and only then - # do we hit the GitHub API, so a pinned install does not depend on it. + # kubeconform (CI parity: hyperi-ci runs it as a blocking gate). - name: Get latest kubeconform version ansible.builtin.uri: url: https://api.github.com/repos/yannh/kubeconform/releases/latest @@ -276,14 +274,10 @@ headers: "{{ hyperi_github_headers }}" register: kubeconform_latest check_mode: false - when: not (hyperi_pinned | default(false) and 'kubeconform' in (hyperi_versions | default({}))) - - name: Resolve the kubeconform tag (pinned or latest) + - name: Resolve the kubeconform tag ansible.builtin.set_fact: - kubeconform_ref: >- - {{ hyperi_versions['kubeconform'] - if (hyperi_pinned | default(false) and 'kubeconform' in (hyperi_versions | default({}))) - else kubeconform_latest.json.tag_name }} + kubeconform_ref: "{{ kubeconform_latest.json.tag_name }}" - name: Download and extract kubeconform ansible.builtin.unarchive: @@ -296,7 +290,6 @@ # kube-linter (CI parity). Asset naming is uneven: amd64 has no arch suffix # (kube-linter-linux.tar.gz), arm64 does (kube-linter-linux_arm64.tar.gz). - # --pinned takes the CI-exact tag from group_vars (see kubeconform above). - name: Get latest kube-linter version ansible.builtin.uri: url: https://api.github.com/repos/stackrox/kube-linter/releases/latest @@ -304,14 +297,10 @@ headers: "{{ hyperi_github_headers }}" register: kube_linter_latest check_mode: false - when: not (hyperi_pinned | default(false) and 'kube-linter' in (hyperi_versions | default({}))) - - name: Resolve the kube-linter tag (pinned or latest) + - name: Resolve the kube-linter tag ansible.builtin.set_fact: - kube_linter_ref: >- - {{ hyperi_versions['kube-linter'] - if (hyperi_pinned | default(false) and 'kube-linter' in (hyperi_versions | default({}))) - else kube_linter_latest.json.tag_name }} + kube_linter_ref: "{{ kube_linter_latest.json.tag_name }}" - name: Download and extract kube-linter ansible.builtin.unarchive: diff --git a/ansible/roles/soe/tasks/arcane.yml b/ansible/roles/soe/tasks/arcane.yml index 6c9cbb2..9eedc33 100644 --- a/ansible/roles/soe/tasks/arcane.yml +++ b/ansible/roles/soe/tasks/arcane.yml @@ -15,15 +15,11 @@ become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" become_user: "{{ actual_user }}" block: - # Absent from hyperi_versions, so this is `latest` unless someone pins it. # Adding a pin there is only correct once hyperi-ci carries the same one. - - name: Resolve the Arcane image tag (pinned or latest) + - name: Resolve the Arcane image tag # noqa: var-naming[no-role-prefix] -- soe_ IS the role prefix here ansible.builtin.set_fact: - soe_arcane_ref: >- - {{ hyperi_versions['arcane'] - if (hyperi_pinned | default(false) and 'arcane' in (hyperi_versions | default({}))) - else 'latest' }} + soe_arcane_ref: "{{ 'latest' }}" # macOS keeps its socket under $HOME rather than /var/run, and the SOE run # does not inherit the .zshenv that fixes that for interactive shells. diff --git a/docs/install-matrix.md b/docs/install-matrix.md index 2ddb56e..5b86af7 100644 --- a/docs/install-matrix.md +++ b/docs/install-matrix.md @@ -9,8 +9,8 @@ persona. Four levels, smallest to largest: **tool -> group -> role -> persona.** Every level is selectable the same way from Ansible and from `install.sh` (`--tags `). The bare `./install.sh` runs only the additive `developer` -base; everything else is opt-in. Two fetch modes: **latest** (default, no -maintenance) and **`--pinned`** (opt-in: exact versions mirroring hyperi-ci). +base; everything else is opt-in. Every tool is fetched at whatever version is +current when the installer runs. ```mermaid flowchart TD @@ -124,17 +124,15 @@ without the whole role. Per-app tags stay too (`--tags vscode`, `--tags slack`, ...): run `./install.sh --list-apps` for the full per-app list. -## Tool matrix - install method and pin policy per tool +## Tool matrix - install method per tool -`Method` is how the tool is fetched. **Default installs are always latest and -unpinned** - the `Pinned` column applies ONLY when you opt into `--pinned`, and -then says what that mode does: **SHA256** = manual binary; **version** = -repo/brew-signed, pin the version only; **n/a** = not pinnable/meta. +`Method` is how the tool is fetched. Every install resolves the current version +at run time, so there is no per-tool version policy to read here. -**SHA256 marks the rows that NEED a checksum, not the rows that have one.** -Pinning is by tag today across the board, and digest verification at download is -still the planned hardening (see Auto-update below, and hyperi-ci #66). A tag can -be force-moved, so those rows currently rest on HTTPS and the tag alone. +**Digest verification is not universal.** `go` and `rustup` fetch a checksum +from their publisher and verify it. The rest rest on HTTPS and the release tag, +which a publisher can in principle force-move. Extending checksum-at-download to +the other manual binaries is the outstanding hardening (hyperi-ci #66). Four fetches do verify a digest today, and they are the ones whose SHA is held in the repo or read from a published manifest: the Go toolchain, rustup-init, @@ -143,27 +141,27 @@ downloads.hyperi.io, so the fetch also re-pulls a republished binary). ### developer (base, additive - runs bare) -| Tool(s) | Platforms | Method | Pinned | -|---|---|---|---| -| astral suite: uv, ruff, ty (uv bundles `uv audit` + `uv check`) | all | Fedora dnf / macOS brew; Ubuntu has no apt package (see Auto-update) | version / SHA256 | -| CLI utils (jq, gron, bat, fzf, ripgrep, fd, git-delta, moreutils, miller, rsync, tmux, htop, wget, shellcheck, age, parallel, ...) | all | distro repo / brew | version | -| sd | all | distro (apt/dnf) / brew | version | -| yq (mikefarah; apt `yq` is kislyuk/yq, a different tool) | all | Fedora dnf / Ubuntu re-fetch (Tier 3) / brew | version / SHA256 | -| lazygit | all | Fedora COPR / Ubuntu apt (re-fetch on 24.04 LTS) / brew | version / SHA256 | -| docker (Engine on Linux, CLI-only on macOS) | all | vendor-repo / brew | version | -| git, git-lfs, git-filter-repo, gh | all | distro/PPA/brew | version | -| chrome, brave (opt-in `never`) | all | vendor-repo / cask | version | -| Homebrew (bootstrap) | macOS | vendor-script | n/a | +| Tool(s) | Platforms | Method | +|---|---|---| +| astral suite: uv, ruff, ty (uv bundles `uv audit` + `uv check`) | all | Fedora dnf / macOS brew; Ubuntu has no apt package (see Auto-update) | +| CLI utils (jq, gron, bat, fzf, ripgrep, fd, git-delta, moreutils, miller, rsync, tmux, htop, wget, shellcheck, age, parallel, ...) | all | distro repo / brew | +| sd | all | distro (apt/dnf) / brew | +| yq (mikefarah; apt `yq` is kislyuk/yq, a different tool) | all | Fedora dnf / Ubuntu re-fetch (Tier 3) / brew | +| lazygit | all | Fedora COPR / Ubuntu apt (re-fetch on 24.04 LTS) / brew | +| docker (Engine on Linux, CLI-only on macOS) | all | vendor-repo / brew | +| git, git-lfs, git-filter-repo, gh | all | distro/PPA/brew | +| chrome, brave (opt-in `never`) | all | vendor-repo / cask | +| Homebrew (bootstrap) | macOS | vendor-script | ### developer-gui -| Tool(s) | Platforms | Method | Pinned | -|---|---|---|---| -| VS Code | all | vendor-repo / cask | version | -| Ghostty | Fedora (COPR) / macOS (cask) | vendor-repo / cask | version | -| Ghostty | Ubuntu | github-binary (.deb) | SHA256 | -| DBeaver | all | Ubuntu vendor-repo (dbeaver.io/debs) / Fedora flatpak / cask | version | -| VS Code privacy profile (opt-in: `-e vscode_privacy_enabled=true`) | all | bundled script (`hyperi-vscode-privacy`) | n/a | +| Tool(s) | Platforms | Method | +|---|---|---| +| VS Code | all | vendor-repo / cask | +| Ghostty | Fedora (COPR) / macOS (cask) | vendor-repo / cask | +| Ghostty | Ubuntu | github-binary (.deb) | +| DBeaver | all | Ubuntu vendor-repo (dbeaver.io/debs) / Fedora flatpak / cask | +| VS Code privacy profile (opt-in: `-e vscode_privacy_enabled=true`) | all | bundled script (`hyperi-vscode-privacy`) | The privacy profile is off by default because it edits a personal `settings.json`. It merges one marked block of managed keys into VSCode, @@ -180,20 +178,20 @@ is the meta-role pulling them all. #### developer-rust -| Tool(s) | Platforms | Method | Pinned | -|---|---|---|---| -| rustup + stable toolchain, rustfmt, clippy | all | vendor-script + rustup | n/a | -| cargo-binstall | all | cargo | n/a | -| cargo-* (nextest, deny, chef, bacon, update) | all | cargo | n/a | -| cargo-audit, cargo-hack, cargo-pgo | all | cargo | n/a | -| cargo-machete (unused deps), cargo-semver-checks (API breaks) | all | cargo | n/a | -| cargo-llvm-cov + llvm-tools-preview | all | cargo / rustup | n/a | -| protobuf-compiler, librdkafka-dev | Linux | distro repo | version | -| mold, clang | Linux | distro repo | version | -| sccache | all | github-binary, latest each run (Tier 3) / brew | SHA256 verified | -| cargo-sweep | all | cargo-binstall / cargo | n/a | -| hyperi-rust-setup, hyperi-rust-cache-prune | all | role file -> `/usr/local/bin` | n/a | -| build governor (`hyperi-rust-govern` + `rust-build.slice`) | all | role file + user unit | n/a | +| Tool(s) | Platforms | Method | +|---|---|---| +| rustup + stable toolchain, rustfmt, clippy | all | vendor-script + rustup | +| cargo-binstall | all | cargo | +| cargo-* (nextest, deny, chef, bacon, update) | all | cargo | +| cargo-audit, cargo-hack, cargo-pgo | all | cargo | +| cargo-machete (unused deps), cargo-semver-checks (API breaks) | all | cargo | +| cargo-llvm-cov + llvm-tools-preview | all | cargo / rustup | +| protobuf-compiler, librdkafka-dev | Linux | distro repo | +| mold, clang | Linux | distro repo | +| sccache | all | github-binary, latest each run (Tier 3) / brew | +| cargo-sweep | all | cargo-binstall / cargo | +| hyperi-rust-setup, hyperi-rust-cache-prune | all | role file -> `/usr/local/bin` | +| build governor (`hyperi-rust-govern` + `rust-build.slice`) | all | role file + user unit | Every cargo tool is installed with `--locked`, and `hyperi-update` refreshes them with `cargo install-update -a --locked`. `cargo install` ignores the crate's @@ -234,63 +232,63 @@ That tag installs both tools, writes the caps, and schedules the prune #### developer-go -| Tool(s) | Platforms | Method | Pinned | -|---|---|---|---| -| go, delve | all | distro repo / brew | version | -| gopls | all | go install | n/a | -| golangci-lint* (Tier 3: re-fetch) | all | github-binary / brew | SHA256 | -| gosec, govulncheck | all | Fedora dnf / Ubuntu `go install` / brew | version | +| Tool(s) | Platforms | Method | +|---|---|---| +| go, delve | all | distro repo / brew | +| gopls | all | go install | +| golangci-lint* (Tier 3: re-fetch) | all | github-binary / brew | +| gosec, govulncheck | all | Fedora dnf / Ubuntu `go install` / brew | #### developer-python The base ships the Astral suite (uv, ruff, ty) and `uv` bundles `uv audit` / `uv check`, so this role adds only the opt-in legacy type checker. -| Tool(s) | Platforms | Method | Pinned | -|---|---|---|---| -| mypy (opt-in; `uv check`/`ty` is the default) | all | uv-tool | version | +| Tool(s) | Platforms | Method | +|---|---|---| +| mypy (opt-in; `uv check`/`ty` is the default) | all | uv-tool | #### developer-node -| Tool(s) | Platforms | Method | Pinned | -|---|---|---|---| -| node, npm | all | vendor-repo / brew | version | -| semantic-release (+ plugins) | all | npm global | n/a | -| pnpm, corepack | all | corepack (ships with node) | n/a | +| Tool(s) | Platforms | Method | +|---|---|---| +| node, npm | all | vendor-repo / brew | +| semantic-release (+ plugins) | all | npm global | +| pnpm, corepack | all | corepack (ships with node) | #### developer-typescript (dep: developer-node) -| Tool(s) | Platforms | Method | Pinned | -|---|---|---|---| -| typescript, tsx, ts-node | all | pnpm global | n/a | +| Tool(s) | Platforms | Method | +|---|---|---| +| typescript, tsx, ts-node | all | pnpm global | #### developer-c -| Tool(s) | Platforms | Method | Pinned | -|---|---|---|---| -| C build tools (gcc, make, cmake, pkg-config) | Linux distro; macOS CLT | distro / xcode-select | version | +| Tool(s) | Platforms | Method | +|---|---|---| +| C build tools (gcc, make, cmake, pkg-config) | Linux distro; macOS CLT | distro / xcode-select | ### infrastructure -| Tool(s) | Platforms | Method | Pinned | -|---|---|---|---| -| kubectl | all | vendor-repo (pkgs.k8s.io) / brew | version | -| helm | all | vendor-repo (baltocdn apt/dnf) / brew | version | -| kubectx, kubens | all | distro (apt universe / dnf) / brew | version | -| k9s | all | Fedora dnf / Ubuntu re-fetch (Tier 3) / brew | version / SHA256 | -| kind, argocd (Tier 3: re-fetch) | all | github-binary / brew | SHA256 | -| kustomize | all | Fedora dnf / Ubuntu re-fetch (Tier 3) / brew | version / SHA256 | -| checkov | all | uv-tool (Tier 2) / brew | version | -| terraform-docs (Tier 3: re-fetch) | all | github-binary / brew | SHA256 | -| dive (Tier 3: re-fetch) | all | github-binary / brew | SHA256 | -| aws-cli v2 | all | Fedora dnf (`awscli2`) / Ubuntu official snap / brew | version | -| aws-vault (Tier 3: re-fetch) | all | github-binary / brew | SHA256 | -| opentofu (`tofu`) | all | vendor-repo (packages.opentofu.org), apt AND dnf / brew | version | -| openbao | all | vendor-repo (pkgs.openbao.org) / Fedora dnf / brew | version | -| azure-cli, google-cloud-cli | all | vendor-repo / cask | version | -| clickhouse-client, rpk, valkey-cli, vector (the `data` group) | Linux; macOS partial | vendor-repo / distro | version | -| wrangler (the `cloudflare` group) | all | npm-global / brew | version | -| flarectl (the `cloudflare` group) | all | `go install` from source / brew | source tag | +| Tool(s) | Platforms | Method | +|---|---|---| +| kubectl | all | vendor-repo (pkgs.k8s.io) / brew | +| helm | all | vendor-repo (baltocdn apt/dnf) / brew | +| kubectx, kubens | all | distro (apt universe / dnf) / brew | +| k9s | all | Fedora dnf / Ubuntu re-fetch (Tier 3) / brew | +| kind, argocd (Tier 3: re-fetch) | all | github-binary / brew | +| kustomize | all | Fedora dnf / Ubuntu re-fetch (Tier 3) / brew | +| checkov | all | uv-tool (Tier 2) / brew | +| terraform-docs (Tier 3: re-fetch) | all | github-binary / brew | +| dive (Tier 3: re-fetch) | all | github-binary / brew | +| aws-cli v2 | all | Fedora dnf (`awscli2`) / Ubuntu official snap / brew | +| aws-vault (Tier 3: re-fetch) | all | github-binary / brew | +| opentofu (`tofu`) | all | vendor-repo (packages.opentofu.org), apt AND dnf / brew | +| openbao | all | vendor-repo (pkgs.openbao.org) / Fedora dnf / brew | +| azure-cli, google-cloud-cli | all | vendor-repo / cask | +| clickhouse-client, rpk, valkey-cli, vector (the `data` group) | Linux; macOS partial | vendor-repo / distro | +| wrangler (the `cloudflare` group) | all | npm-global / brew | +| flarectl (the `cloudflare` group) | all | `go install` from source / brew | Almost every macOS path resolves to brew or a cask. The language managers that remain there carry no formula at all: `alint` and `maid` have none, and @@ -309,24 +307,24 @@ warning and continues. `*` = blocking CI gate. -| Tool(s) | Platforms | Method | Pinned | -|---|---|---|---| -| hyperi-ci, semgrep | all | uv-tool (Tier 2) / brew | version | -| alint | all | cargo (Tier 2) / brew | version | -| osv-scanner | all | Linux re-fetch (Tier 3) / brew | version / SHA256 | -| gitleaks* | all | Fedora dnf / Ubuntu re-fetch (Tier 3) / brew | version / SHA256 | -| act | all | Fedora COPR / Ubuntu re-fetch (Tier 3) / brew | version / SHA256 | -| trivy | all | vendor-repo (official aquasecurity apt/dnf) / brew | version | -| hadolint* | all | Fedora dnf / Ubuntu re-fetch (Tier 3) / brew | version / SHA256 | -| pip-audit* | all | uv-tool (Tier 2) / brew | version | -| kubeconform*, kube-linter | all | github-binary (Tier 3: re-fetch) / brew | SHA256 | -| yamllint, ansible-lint, pre-commit | all | distro (apt universe / dnf) / brew | version | -| actionlint | all | Ubuntu snap / Fedora re-fetch (Tier 3) / brew | version / SHA256 | -| vulture | all | Ubuntu apt / Fedora uv-tool (Tier 2) / brew | version | -| typos | all | cargo (Tier 2) / brew | version | -| maid (mermaid validator, used by `/docs`) | all | npm global (Tier 2) | n/a | -| git-scrub (git-history scrubber) | all | github-binary (Tier 3: re-fetch) | version | -| macbash (macOS bash portability checker) | all | Linux downloads.hyperi.io binary (Tier 3, digest-verified) / brew tap | SHA256 | +| Tool(s) | Platforms | Method | +|---|---|---| +| hyperi-ci, semgrep | all | uv-tool (Tier 2) / brew | +| alint | all | cargo (Tier 2) / brew | +| osv-scanner | all | Linux re-fetch (Tier 3) / brew | +| gitleaks* | all | Fedora dnf / Ubuntu re-fetch (Tier 3) / brew | +| act | all | Fedora COPR / Ubuntu re-fetch (Tier 3) / brew | +| trivy | all | vendor-repo (official aquasecurity apt/dnf) / brew | +| hadolint* | all | Fedora dnf / Ubuntu re-fetch (Tier 3) / brew | +| pip-audit* | all | uv-tool (Tier 2) / brew | +| kubeconform*, kube-linter | all | github-binary (Tier 3: re-fetch) / brew | +| yamllint, ansible-lint, pre-commit | all | distro (apt universe / dnf) / brew | +| actionlint | all | Ubuntu snap / Fedora re-fetch (Tier 3) / brew | +| vulture | all | Ubuntu apt / Fedora uv-tool (Tier 2) / brew | +| typos | all | cargo (Tier 2) / brew | +| maid (mermaid validator, used by `/docs`) | all | npm global (Tier 2) | +| git-scrub (git-history scrubber) | all | github-binary (Tier 3: re-fetch) | +| macbash (macOS bash portability checker) | all | Linux downloads.hyperi.io binary (Tier 3, digest-verified) / brew tap | `hyperi-ci` is a Python tool from PyPI, installed via `uv tool` and refreshed to the latest release on every run (upgrade-if-present, not install-once). soe @@ -335,19 +333,19 @@ hyperi-ci. ### soe / soe-gui (HyperI org policy) -| Tool(s) | Platforms | Method | Pinned | -|---|---|---|---| -| Claude Code CLI | Linux github-binary (SHA-verified) / macOS cask | github-binary / cask | SHA256 | -| tea (Forgejo/Gitea CLI, `forgejo` / `codeberg` tags; `gh` is GitHub-only) | Linux github-binary / macOS brew | github-binary / brew | SHA256 | -| openvpn3 client (-> vpn-clients group) | Fedora COPR / Ubuntu vendor-repo / macOS brew | vendor-repo / brew | version | -| WireGuard, Tunnelblick (macOS) | all / macOS | distro / cask | version | -| Slack | all | vendor-repo / cask | version | -| LibreOffice (org office suite) | Linux | distro repo | version | -| Nemo, GNOME extensions (gext), fonts | Linux | distro / uv-tool / vendored | version | -| colima + Apple `container` (macOS only) | macOS | brew / github-binary | version / SHA256 | -| Arcane container UI (opt-in `soe_arcane_enabled`) | all | container image | tag | -| Local ClickHouse + Redpanda (opt-in `soe_local_services_enabled`) | all | container image | latest, always | -| removals / update_command / admin-scripts (opt-in `never`, on for soe) | Linux | tombstones + scripts | n/a | +| Tool(s) | Platforms | Method | +|---|---|---| +| Claude Code CLI | Linux github-binary (SHA-verified) / macOS cask | github-binary / cask | +| tea (Forgejo/Gitea CLI, `forgejo` / `codeberg` tags; `gh` is GitHub-only) | Linux github-binary / macOS brew | github-binary / brew | +| openvpn3 client (-> vpn-clients group) | Fedora COPR / Ubuntu vendor-repo / macOS brew | vendor-repo / brew | +| WireGuard, Tunnelblick (macOS) | all / macOS | distro / cask | +| Slack | all | vendor-repo / cask | +| LibreOffice (org office suite) | Linux | distro repo | +| Nemo, GNOME extensions (gext), fonts | Linux | distro / uv-tool / vendored | +| colima + Apple `container` (macOS only) | macOS | brew / github-binary | +| Arcane container UI (opt-in `soe_arcane_enabled`) | all | container image | +| Local ClickHouse + Redpanda (opt-in `soe_local_services_enabled`) | all | container image | +| removals / update_command / admin-scripts (opt-in `never`, on for soe) | Linux | tombstones + scripts | ### Targeted deployment roles (opt-in, not in any persona) @@ -442,40 +440,33 @@ The same trap sits in the molecule verify, which takes the desktop user from `MOLECULE_TARGET_DESKTOP_USER` -- asserting against the service account's empty home passes every user-scoped check on a host that was never fixed. -## Two install modes +## One install mode: latest, at the time you run it -**Latest is the default for everything. Pinning is never a default - `--pinned` -is an explicit opt-in** (for reproducibility or CI parity). Nothing is pinned -unless you ask for it; any version currently hardcoded in a task moves onto the -opt-in path. +**Every tool resolves its version when the installer runs.** A box built today +gets what is current today, and the same command in four months gets what is +current then. No release number for anything with an upstream to ask lives in +this repo, and there is no flag that changes it. -| Mode | Flag | Behaviour | -|---|---|---| -| latest | (default) | `/releases/latest`, distro `state: present`, brew latest. No maintenance, no pins. | -| pinned | `--pinned` (opt-in) | Exact versions (tags) from the `versions.yml` SSoT, mirroring hyperi-ci. | - -`--pinned`'s SSoT lives in `inventories/localhost/group_vars/all.yml` -(`hyperi_versions`) and **mirrors hyperi-ci's `config/versions.yaml`**, so a -pinned local box matches CI exactly (hadolint, cargo-audit, kubeconform, -kube-linter, golangci-lint, ...). A retrofitted manual-binary task branches on -`hyperi_pinned | default(false)`: pinned + pin-exists -> the exact tag (and it -skips the GitHub API entirely); else latest. Pinning is by TAG today, exactly as -hyperi-ci does; **SHA256 digest-verification at download is the planned -hardening** (tracked upstream as hyperi-ci #66) - when it lands, add `sha256:` -per tool and a `checksum:` on the fetch. The distro-first packaging ladder keeps -the manual-binary set small. - -That mirror is checked, not assumed. `tools/ci/run-tests.sh` reads hyperi-ci's -pins through its own interpreter and fails on any disagreement, in either -direction -- a missed bump upstream and a hand-edit here are the same defect. -A tool hyperi-ci pins that is absent here is fine and reported as a note: an -absent entry falls back to latest, which is the documented behaviour. The -reverse is not, because an entry with no counterpart pins a version CI never -runs. +| Source | Behaviour | +|---|---| +| distro or vendor repo | `state: present`, and the repo carries the box forward on `apt`/`dnf upgrade`. | +| manual binary | `/releases/latest` through the GitHub API, then the matching asset. | +| cargo / go tools | `cargo install X`, `go install X@latest`. | +| go, rustup | The publisher's current release, verified against the checksum it serves beside it. | + +Two things are deliberately not versions. **Node's major** (`node_major`) picks +an LTS line rather than a release, and NodeSource's signed repo patches it in +place. **Rust's edition** comes with whatever stable rustup installs, so edition +2024 needs no pin. + +`go` and `rustup` keep digest verification while tracking latest, by fetching +the checksum from the publisher at install time -- `go.dev/dl/?mode=json` carries +a SHA256 per file, and static.rust-lang.org serves `rustup-init.sha256` beside +the binary. ### GitHub rate limits on an unattended build -Latest-mode resolves versions through `api.github.com`, which allows **60 +Version resolution goes through `api.github.com`, which allows **60 requests an hour per IP** anonymously. A run selecting `infrastructure` plus a couple of languages spends a good fraction of that by itself, and every machine sharing an egress address draws on the same 60 -- so a fleet rollout or an image @@ -492,8 +483,7 @@ GITHUB_TOKEN=$(gh auth token) ansible-playbook ... ``` Without one the header is empty and every call stays anonymous, so a laptop -install needs no token and behaves exactly as before. `--pinned` sidesteps the -API for any tool carrying a pin, since a pinned task skips the lookup entirely. +install needs no token and behaves exactly as before. **Packaging ladder** (pick the highest that works): distro repo (auto-updates) > official vendor apt/dnf repo > official snap/flatpak > manual binary (last @@ -530,12 +520,10 @@ Ubuntu (no apt package and no vendor repo), and the DBeaver flatpak on Fedora stays only so the tombstones in `removals.yml` can clear ones an older revision left behind. -**A pinned tool cannot live on a snap or a distro package.** Neither channel -takes an arbitrary version, so a tool in `hyperi_versions` must come from a -release binary or `--pinned` silently installs whatever the channel holds. That -is why golangci-lint is a Tier 3 binary on Linux despite Fedora packaging it: -the dnf build trailed the CI pin, so a "pinned" box was running a different -linter from CI. +**A snap or distro package installs whatever its channel holds, which can +trail upstream badly.** That is why golangci-lint is a Tier 3 binary on Linux +despite Fedora packaging it: the dnf build trails upstream by two minors, and a +linter behind the Go toolchain cannot read the newer stdlib. **Moving a tool between channels leaves the old copy behind, and it wins.** `/usr/local/bin` and `/snap/bin` both precede `/usr/bin` on the default PATH, so @@ -573,8 +561,8 @@ release for these, and skips the ones the running distro installs from a repo so the binary cannot shadow the packaged copy. golangci-lint is here for a different reason: Fedora does package it, but the -package cannot honour the `hyperi_versions` pin and trailed it. A CI-parity tool -belongs on the only channel that takes a version. +build trails upstream, and a linter behind the Go toolchain cannot read the +newer stdlib. `hyperi-update` (the "update my system" command) runs all three tiers plus the OS / snap / flatpak sweep, so one command brings everything current. soe diff --git a/install.sh b/install.sh index 8532060..acaab66 100755 --- a/install.sh +++ b/install.sh @@ -16,7 +16,6 @@ # --tags-include TAGS Include specific tags to run (comma-separated) # --tags-exclude TAGS Exclude specific tags from running (comma-separated) # --region REGION Apply regional settings (e.g. au, en_AU.UTF-8) -# --pinned Pin manual binaries to CI-exact versions (default: latest) # --branch BRANCH Git branch to use (default: main) # Personas: --soe / --contributor / --full-stack / --infra / --languages [list] # --help Show this help message @@ -56,9 +55,6 @@ OPTIONS: --tags-exclude TAGS Exclude specific tags from running (comma-separated) --branch BRANCH Git branch to use (default: main) --region REGION Apply regional settings (e.g. au, en_AU.UTF-8) - --pinned Reproducible mode: pin the manual-binary tools to the - exact versions in group_vars (mirrors hyperi-ci) instead - of latest. Latest is the default. --soe Shortcut: HyperI staff workstation defaults (generic dev + CI toolchain + HyperI org policy; no IaC, no language toolchains) @@ -114,9 +110,6 @@ EXAMPLES: Rust + Go toolchains only: ./install.sh --languages rust,go - Reproducible (CI-exact) install: - ./install.sh --contributor --pinned - Install the RDP server (GNOME Remote Login) for inbound access: ./install.sh --tags rdp-server @@ -243,7 +236,7 @@ Composability examples: ./install.sh --tags data The data-tools group ./install.sh --tags vscode,ghostty VS Code + Ghostty only ./install.sh --soe --languages rust,go SOE + Rust + Go - ./install.sh --infra --pinned SRE box, CI-exact versions + ./install.sh --infra SRE box ./install.sh --tags power-profile Never sleep on mains power ./install.sh --tags power-profile -e power_profile=vm Never sleep at all (RDP guest) @@ -261,8 +254,8 @@ append_tags() { fi } -# Append one `key=value` Ansible extra var, so --pinned and --region compose -# instead of clobbering each other. +# Append one `key=value` Ansible extra var, so several of them compose instead +# of clobbering each other. append_extra_var() { if [[ -n "$ANSIBLE_EXTRA_VARS" ]]; then ANSIBLE_EXTRA_VARS="$ANSIBLE_EXTRA_VARS -e $1" @@ -397,13 +390,6 @@ while [[ $# -gt 0 ]]; do shift fi ;; - --pinned) - # Opt-in reproducible mode: pin the manual-binary tools to the exact - # versions in inventories/localhost/group_vars/all.yml (which mirrors - # hyperi-ci) instead of /releases/latest. Latest stays the default. - append_extra_var "hyperi_pinned=true" - shift - ;; --users) TARGET_USERS="$(printf '%s' "$2" | tr ',' ' ')" shift 2 @@ -451,7 +437,7 @@ if [[ -n "${REGION_ARG:-}" ]]; then # Add region tag append_tags "region" - # Pass desktop_region as extra var (append so --pinned survives) + # Pass desktop_region as an extra var append_extra_var "desktop_region=${DESKTOP_REGION}" print_info "Region: ${DESKTOP_REGION}" fi diff --git a/tools/check_release_matrix.py b/tools/check_release_matrix.py index a4441a2..5ca0de2 100644 --- a/tools/check_release_matrix.py +++ b/tools/check_release_matrix.py @@ -9,7 +9,7 @@ Molecule cannot include another YAML file, so `molecule/matrix/molecule.yml` must repeat the image list. This makes that repetition checked rather than -trusted, the same bargain check_version_pins.py makes with hyperi-ci. +trusted. The playbook's own OS gate is a third copy, hardcoded in a `when:` because pre_tasks run before role defaults are in scope. It is matched by pattern here diff --git a/tools/check_version_pins.py b/tools/check_version_pins.py deleted file mode 100644 index 6c42a25..0000000 --- a/tools/check_version_pins.py +++ /dev/null @@ -1,138 +0,0 @@ -#!/usr/bin/env python3 -"""Check that hyperi_versions still mirrors hyperi-ci's tool pins. - -`hyperi_versions` in the localhost group_vars exists so that a `--pinned` box -installs the same tool versions CI runs. Nothing enforced that: both files are -hand-edited, in two repos, and a bump on either side left the other behind -without saying so. Three of twelve had drifted before this existed. - -Renovate cannot cover it either -- they are plain YAML strings in an Ansible -group_vars file, not a manifest format it has a manager for. - -Reads hyperi-ci's pins through hyperi-ci's OWN interpreter rather than -importing them here: it is installed as an isolated uv tool, so `import -hyperi_ci` from any other Python fails. No network is involved. -""" - -from __future__ import annotations - -import argparse -import json -import os -import shutil -import subprocess -import sys -from pathlib import Path - -import yaml - -REPO_ROOT = Path(__file__).resolve().parent.parent -PINS_FILE = REPO_ROOT / "ansible/inventories/localhost/group_vars/all.yml" - -# Dumped through hyperi-ci's public accessors rather than reading its config -# file directly, so a change to that file's shape does not silently break this. -DUMP_PINS = ( - "import json; from hyperi_ci import versions; " - "print(json.dumps({n: versions.tool_version(n) for n in versions.tool_names()}))" -) - - -def hyperi_ci_python() -> Path | None: - """The interpreter of the installed hyperi-ci, or None if it is not there. - - A uv tool's entry point sits beside the venv's python, so resolving the - binary through any symlinks gives the interpreter that can import it. - """ - binary = shutil.which("hyperi-ci") - if not binary: - return None - candidate = Path(os.path.realpath(binary)).parent / "python" - return candidate if candidate.is_file() else None - - -def ci_pins(python: Path) -> dict[str, str]: - proc = subprocess.run( - [str(python), "-c", DUMP_PINS], - capture_output=True, - text=True, - encoding="utf-8", - errors="replace", - timeout=120, - check=False, - ) - if proc.returncode != 0: - raise RuntimeError(f"hyperi-ci pins unreadable: {proc.stderr.strip()}") - return json.loads(proc.stdout) - - -def local_pins(path: Path) -> dict[str, str]: - data = yaml.safe_load(path.read_text(encoding="utf-8")) or {} - return data.get("hyperi_versions") or {} - - -def compare(local: dict[str, str], ci: dict[str, str]) -> tuple[list[str], list[str]]: - """Return (problems, notes). - - A tool CI pins and we do not is fine -- an absent entry falls back to latest - even under --pinned, which is the documented behaviour. The reverse is not: - an entry with no counterpart pins a version CI never runs, which is the - opposite of what this map is for. - """ - problems = [] - for name in sorted(set(local) & set(ci)): - if local[name] != ci[name]: - problems.append(f"{name}: pinned {local[name]} here, {ci[name]} in hyperi-ci") - - problems += [ - f"{name}: pinned {local[name]} here, but hyperi-ci does not pin it at all" - for name in sorted(set(local) - set(ci)) - ] - notes = [ - f"{name}: hyperi-ci pins {ci[name]}, not mirrored here (falls back to latest)" - for name in sorted(set(ci) - set(local)) - ] - return problems, notes - - -def main(argv: list[str]) -> int: - parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) - parser.add_argument( - "--pins-file", - type=Path, - default=PINS_FILE, - help="group_vars file holding hyperi_versions (default: the repo's own)", - ) - args = parser.parse_args(argv) - - python = hyperi_ci_python() - if python is None: - # Same bargain the Ansible checks make: absent tooling skips rather - # than reddening a runner that was never going to have it. - print("--> hyperi-ci not installed, skipping the pin comparison") - return 0 - - try: - ci = ci_pins(python) - except (RuntimeError, json.JSONDecodeError) as exc: - print(f" FAILED: {exc}") - return 1 - - local = local_pins(args.pins_file) - problems, notes = compare(local, ci) - - for note in notes: - print(f" note: {note}") - for problem in problems: - print(f" DRIFT: {problem}") - - if problems: - print(f" {len(problems)} pin(s) disagree with hyperi-ci.") - print(" hyperi_versions must mirror hyperi-ci, or --pinned stops matching CI.") - return 1 - - print(f" ok ({len(set(local) & set(ci))} pin(s) match hyperi-ci)") - return 0 - - -if __name__ == "__main__": - sys.exit(main(sys.argv[1:])) diff --git a/tools/ci/run-tests.sh b/tools/ci/run-tests.sh index c78ccb5..f3983c6 100755 --- a/tools/ci/run-tests.sh +++ b/tools/ci/run-tests.sh @@ -81,21 +81,6 @@ else echo "--> pytest or PyYAML not installed, skipping tools tests" fi -# hyperi_versions only means anything if it still matches hyperi-ci. Both files -# are hand-edited in two repos, and a bump on either side used to leave the -# other behind silently -- three of twelve had drifted before this ran. -if have python3 && python3 -c "import yaml" >/dev/null 2>&1; then - ran=$((ran + 1)) - echo "==> version pins vs hyperi-ci" - if python3 tools/check_version_pins.py; then - : - else - failures=$((failures + 1)) - fi -else - echo "--> PyYAML not installed, skipping the pin comparison" -fi - # molecule/vars.yml is the SSoT for the supported releases, but molecule cannot # include it and the OS gate restates it. Both copies are checked here rather # than trusted to a comment. diff --git a/tools/tests/test_check_version_pins.py b/tools/tests/test_check_version_pins.py deleted file mode 100644 index d7237c8..0000000 --- a/tools/tests/test_check_version_pins.py +++ /dev/null @@ -1,98 +0,0 @@ -"""Tests for the hyperi_versions / hyperi-ci pin comparison.""" - -import importlib.util -from pathlib import Path - -import pytest - -MODULE = Path(__file__).resolve().parents[1] / "check_version_pins.py" - - -def load_module(): - spec = importlib.util.spec_from_file_location("check_version_pins", MODULE) - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - return module - - -@pytest.fixture(scope="module") -def check(): - return load_module() - - -def test_matching_pins_are_clean(check): - problems, notes = check.compare({"gosec": "v2.28.0"}, {"gosec": "v2.28.0"}) - assert problems == [] - assert notes == [] - - -def test_a_disagreeing_version_is_a_problem(check): - """The drift that actually happened: hyperi-ci moved, this repo did not.""" - problems, _ = check.compare({"gosec": "v2.27.1"}, {"gosec": "v2.28.0"}) - assert len(problems) == 1 - assert "v2.27.1" in problems[0] - assert "v2.28.0" in problems[0] - - -def test_drift_is_caught_in_either_direction(check): - """A hand-edit here is as wrong as a missed bump from there.""" - ahead, _ = check.compare({"gosec": "v9.9.9"}, {"gosec": "v2.28.0"}) - behind, _ = check.compare({"gosec": "v1.0.0"}, {"gosec": "v2.28.0"}) - assert ahead and behind - - -def test_pinning_a_tool_ci_does_not_pin_is_a_problem(check): - """An entry with no counterpart pins a version CI never runs.""" - problems, _ = check.compare({"ripgrep": "v14.0.0"}, {"gosec": "v2.28.0"}) - assert len(problems) == 1 - assert "does not pin it at all" in problems[0] - - -def test_a_tool_ci_pins_but_we_do_not_is_only_a_note(check): - """An absent entry falls back to latest, which is documented behaviour.""" - problems, notes = check.compare({}, {"gosec": "v2.28.0"}) - assert problems == [] - assert len(notes) == 1 - assert "falls back to latest" in notes[0] - - -def test_local_pins_reads_the_group_vars_shape(check, tmp_path): - pins = tmp_path / "all.yml" - pins.write_text( - "hyperi_pinned: false\nhyperi_versions:\n gosec: v2.28.0\n alint: v0.14.1\n", - encoding="utf-8", - ) - assert check.local_pins(pins) == {"gosec": "v2.28.0", "alint": "v0.14.1"} - - -def test_a_file_without_the_map_yields_nothing(check, tmp_path): - """Absent is empty, not a crash -- the caller decides what that means.""" - pins = tmp_path / "all.yml" - pins.write_text("hyperi_pinned: false\n", encoding="utf-8") - assert check.local_pins(pins) == {} - - -def test_missing_hyperi_ci_skips_rather_than_fails(check, tmp_path, monkeypatch, capsys): - """A runner without hyperi-ci must stay green, as the other checks do.""" - monkeypatch.setattr(check, "hyperi_ci_python", lambda: None) - assert check.main(["--pins-file", str(tmp_path / "absent.yml")]) == 0 - assert "skipping" in capsys.readouterr().out - - -def test_drift_exits_non_zero(check, tmp_path, monkeypatch, capsys): - pins = tmp_path / "all.yml" - pins.write_text("hyperi_versions:\n gosec: v2.27.1\n", encoding="utf-8") - monkeypatch.setattr(check, "hyperi_ci_python", lambda: Path("/nonexistent/python")) - monkeypatch.setattr(check, "ci_pins", lambda _python: {"gosec": "v2.28.0"}) - - assert check.main(["--pins-file", str(pins)]) == 1 - assert "DRIFT" in capsys.readouterr().out - - -def test_matching_pins_exit_zero(check, tmp_path, monkeypatch): - pins = tmp_path / "all.yml" - pins.write_text("hyperi_versions:\n gosec: v2.28.0\n", encoding="utf-8") - monkeypatch.setattr(check, "hyperi_ci_python", lambda: Path("/nonexistent/python")) - monkeypatch.setattr(check, "ci_pins", lambda _python: {"gosec": "v2.28.0"}) - - assert check.main(["--pins-file", str(pins)]) == 0 From d59e0f9d85259ec79499563cd02fca661c5cb4b2 Mon Sep 17 00:00:00 2001 From: Derek Date: Thu, 3 Sep 2026 09:22:37 +1000 Subject: [PATCH 2/2] fix(developer-rust): attempt the upgrade when the version check itself failed `rustup check` and `cargo install-update --list` both carry failed_when: false, so a broken toolchain returned empty output and the update tasks skipped as if everything were current. Unmeasurable is not the same as up to date. Found on desktop-derek, where ~/.rustup is a dangling symlink to a /cache volume that was rebuilt without it, so every rustup call exits non-zero. --- ansible/roles/developer-rust/tasks/rust.yml | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/ansible/roles/developer-rust/tasks/rust.yml b/ansible/roles/developer-rust/tasks/rust.yml index 3b5b285..5d25c9b 100644 --- a/ansible/roles/developer-rust/tasks/rust.yml +++ b/ansible/roles/developer-rust/tasks/rust.yml @@ -115,6 +115,10 @@ # Matched on the stable line alone -- `rustup - update available` is rustup # offering to replace itself, which `rustup self update` owns, not this. + # + # A check that did not run is not the same as a toolchain that is current, + # so a failed check falls through to attempting the update rather than + # silently deciding there was nothing to do. - name: Update the Rust toolchain to current stable ansible.builtin.command: cmd: "{{ user_home }}/.cargo/bin/rustup update stable" @@ -125,7 +129,8 @@ RUSTUP_HOME: "{{ user_home }}/.rustup" changed_when: true when: >- - developer_rustup_check.stdout_lines + developer_rustup_check.rc | default(1) != 0 + or developer_rustup_check.stdout_lines | select('search', '^stable-.*update available') | list | length > 0 @@ -462,9 +467,13 @@ become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" changed_when: true - when: - - developer_cargo_outdated.rc | default(1) == 0 - - developer_cargo_outdated.stdout_lines | select('search', '\s+Yes\s*$') | list | length > 0 + # Same reasoning as the rustup update above: a list that failed to run tells us + # nothing, and skipping on it would quietly stop upgrading. + when: >- + developer_cargo_outdated.rc | default(1) != 0 + or developer_cargo_outdated.stdout_lines + | select('search', '\s+Yes\s*$') + | list | length > 0 # ============================================================ # Build acceleration: sccache + mold