Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 28 additions & 2 deletions .github/workflows/mirror-tool.yml
Original file line number Diff line number Diff line change
@@ -1,18 +1,21 @@
name: Mirror Tool

# Tests for scripts/mirror-binaries, which is its own Go module and so is
# not covered by the Build & Test workflow.
# Tests for scripts/mirror-binaries and scripts/generate-manifests-from-r2,
# which are Go modules of their own and so are not covered by the Build &
# Test workflow.

on:
pull_request:
branches: [ main ]
paths:
- 'scripts/mirror-binaries/**'
- 'scripts/generate-manifests-from-r2/**'
- '.github/workflows/mirror-tool.yml'
push:
branches: [ main ]
paths:
- 'scripts/mirror-binaries/**'
- 'scripts/generate-manifests-from-r2/**'
- '.github/workflows/mirror-tool.yml'

permissions:
Expand Down Expand Up @@ -43,3 +46,26 @@ jobs:
# MIRROR_NETWORK_TESTS=1, so this needs neither network nor msitools.
- name: Test
run: go test -race ./...

test-generator:
name: Test manifest generator
runs-on: ubuntu-latest
defaults:
run:
working-directory: scripts/generate-manifests-from-r2

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup Go
uses: actions/setup-go@v5
with:
go-version-file: 'scripts/generate-manifests-from-r2/go.mod'
cache-dependency-path: 'scripts/generate-manifests-from-r2/go.sum'

- name: Vet
run: go vet ./...

- name: Test
run: go test -race ./...
6 changes: 4 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -327,9 +327,11 @@ cd src && go test -cover ./... # With coverage
| `deploy-manifests.yml` | Push to main (manifests/) | Deploy manifest files to R2 |
| `mirror-all.yml` | Manual dispatch | Mirror all runtime binaries to R2 |
| `mirror-sync.yml` | Scheduled | Sync new versions to R2 mirror (installs msitools for Python) |
| `mirror-tool.yml` | PR, push to main (`scripts/mirror-binaries/`) | Vet and test the mirror tool, a separate Go module |
| `mirror-tool.yml` | PR, push to main (`scripts/mirror-binaries/`, `scripts/generate-manifests-from-r2/`) | Vet and test the mirror tool and the manifest generator, separate Go modules |

**Windows Python builds in the mirror**: python-build-standalone is the first source. For versions it lacks, the mirror assembles full builds instead of using python.org's embeddable zips (`scripts/mirror-binaries/python_winbuild.go`): `windows-amd64` from python.org's per-component MSIs (verified against the pinned Windows signing key in `keys/`, unpacked with `msiextract`), `windows-arm64` from python.org's `pythonarm64` NuGet packages (verified against nuget.org's SHA-512). Both are packed as a `.tar.gz` with everything under `python/`. Builds that turn out not to be full installs are skipped, not published. `MIRROR_NETWORK_TESTS=1 go test -run Network ./...` (needs msitools) builds real archives.
**Windows Python builds in the mirror**: python-build-standalone is the first source. For versions it lacks, the mirror assembles full builds instead of using python.org's embeddable zips (`scripts/mirror-binaries/python_winbuild.go`): `windows-amd64` from python.org's per-component MSIs up to 3.13 (verified against the pinned Windows signing key in `keys/`, unpacked with `msiextract`) and from python.org's install-manager zips from 3.14 on, where the MSIs carry no OpenPGP signature (verified against the SHA-256 in python.org's `index-windows.json`, `python_installmanager.go`); `windows-arm64` from python.org's `pythonarm64` NuGet packages (verified against nuget.org's SHA-512). All are packed as a `.tar.gz` with everything under `python/`. Builds that turn out not to be full installs are skipped, not published. `MIRROR_NETWORK_TESTS=1 go test -run Network ./...` (needs msitools) builds real archives.

**Signed Python builds**: every python.org build is also published at `python/<version>/python.org/<platform>.tar.gz`, whatever the default build is, so `dtvem install python <version> --signed` can install python.org's Authenticode-signed build (`collectJobs` dedupes on the metadata key, so python-build-standalone's default never displaces it). The manifest generator labels each Python entry with `build` (`python-build-standalone` or `python.org`, from the metadata's source URL) and nests the python.org build under the default entry's `alternates`; released CLIs ignore both fields. The CLI's manifest cache has a `format` (2); older cache files are refetched. Each install records its build in `.dtvem-build.json` in the version's directory (`src/runtimes/python/build.go`); `dtvem list` shows it, and `dtvem list-all` marks the versions with a signed build.

### Reusable Workflows (from `dtvem/.github` repo)

Expand Down
13 changes: 13 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,19 @@ dtvem migrate python

See [Getting Started](https://dtvem.io/docs/user-guide/getting-started) for more examples.

### 🔏 Signed Python Builds (Windows)

On Windows, dtvem installs [python-build-standalone](https://github.com/astral-sh/python-build-standalone)'s build of Python by default, and its executables are not Authenticode-signed. To install python.org's own build instead, signed by the Python Software Foundation, add `--signed`:

```powershell
dtvem install python 3.12.10 --signed # python.org's build of 3.12.10
dtvem install python 3.12 --signed # the newest 3.12.x with a python.org build
dtvem list-all python # the Signed column shows which versions have one
dtvem list python # the Build column shows which build each install is
```

python.org ships no Windows binaries for security-only releases (3.12.11 onward, for example), so those versions have no signed build. dtvem says so and names the nearest version that has one, rather than installing an unsigned build. A signed build can therefore lack security fixes that newer releases on the same line carry, and dtvem warns when it does.

## 📚 Documentation

| Topic | Description |
Expand Down
31 changes: 29 additions & 2 deletions schemas/manifest.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,24 @@
"source": {
"type": "string",
"description": "Build source indicator: 'built-from-source' when the binary was compiled by dtvem rather than obtained from upstream"
},
"build": {
"type": "string",
"enum": ["python-build-standalone", "python.org"],
"description": "Whose build of the runtime this is, for runtimes with more than one. Python: 'python-build-standalone', or 'python.org' for python.org's own build (on Windows, Authenticode-signed by the Python Software Foundation)"
},
"alternates": {
"type": "array",
"description": "Other builds of the same version for the same platform, installed only when asked for (dtvem install python <version> --signed). Released CLIs that predate this field ignore it and install the entry's own url",
"items": {
"allOf": [
{ "$ref": "#/$defs/download" },
{
"required": ["build"],
"not": { "required": ["alternates"] }
}
]
}
}
}
}
Expand All @@ -78,9 +96,18 @@
"versions": {
"3.13.1": {
"windows-amd64": {
"url": "https://builds.dtvem.io/python/3.13.1/windows-amd64.zip",
"url": "https://builds.dtvem.io/python/3.13.1/windows-amd64.tar.gz",
"sha256": "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2",
"sha256_source": "upstream"
"sha256_source": "upstream",
"build": "python-build-standalone",
"alternates": [
{
"url": "https://builds.dtvem.io/python/3.13.1/python.org/windows-amd64.tar.gz",
"sha256": "c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4",
"sha256_source": "dtvem",
"build": "python.org"
}
]
},
"darwin-arm64": null,
"linux-amd64": {
Expand Down
Loading
Loading