diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index 10a71045f..ac962dac3 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -1028,7 +1028,7 @@ v5.0 replaces v4's per-purl reverts and whole-ledger reverse replay (`revert_rem * **vlt** — `vlt-lock.json`: slot [2] from the registry's `dist.integrity`, slot [3] per the lock's own convention (see the vlt hosted-mode contract); every hosted instance of the pin together. * **cargo** — `Cargo.lock` back on crates.io (source + the sparse index's checksum, `SOCKET_CRATES_INDEX`). When the lock already holds a crates.io block for the same `name@version` (a later dependent locked its own copy, #679), the Socket block merges into it instead: it is dropped (with a v1 lock's `[metadata]` line), and dependents' references take the spelling cargo writes, so no duplicate block is left that cargo cannot parse (#863), and no index lookup is needed; every `Cargo.toml` declaration loses its `registry = "socket-patch-"` pin (the shorthand the rewriter produced collapses back); every `[registries.socket-patch-]` block no manifest or lock still references leaves the project cargo config — including a superseded patch generation's block an earlier re-pin left behind (#864). A declaration it cannot unpin refuses. * **golang** — the hosted `replace` and the socket module's go.sum lines go; the upstream module's two go.sum lines come back, hashed from the module proxy (`SOCKET_GOPROXY`, else `GOPROXY` / `GONOPROXY` / `GOPRIVATE` as go reads them) and cross-checked against the checksum database (`SOCKET_GOSUMDB_URL`, else `sum.golang.org` unless `GOSUMDB=off` / `GONOSUMDB` / `GOPRIVATE` say go would not ask it). A `replace` the user had before the hosted run is not recorded anywhere, so the restore lands on the plain upstream module. - * **pypi** — `Pipfile.lock`, `requirements.txt` (+ in-root `-r` includes), Hatch PEP 508 direct references (`pyproject.toml` / `hatch.toml`), `poetry.lock`, `pdm.lock`, `uv.lock`, PEP 723 script locks and PEP 751 `pylock*.toml` (+ the paired `pyproject.toml` / script metadata): hashes re-derived from PyPI's JSON API (`SOCKET_PYPI_JSON_API`). A restored `requirements.txt` line gets `--hash` options only when the file is in pip's hash-checking mode. The mode is read off the file's other requirement lines (an `-e` / `--editable` line means unhashed). When every requirement is a hosted pin, it is read off the hosted line itself (`--hash` vs a `#sha256=` url fragment) (#410). Refused: a `pdm.lock` without `cross_platform`, or a uv / script / pylock lock, whose release has a wheel that is not pure Python 3 (which files the lock keeps is not re-derivable); a uv lock whose options filter files (`exclude-newer`, `no-binary`, `no-build`), or whose other registry packages name no registry, several, or one other than PyPI's simple index; a pylock whose other registry packages show neither an `index` nor (as `uv pip compile` writes them) only PyPI files with none, which restores the entry without an `index` too; uv 0.2 `[[distribution]]` locks. Restored artifact fields keep the spelling the lock's other entries show, including the `upload_time` that uv 0.6.15–0.6.17 write. A restored pylock entry's `upload-time`s are whole seconds, as uv writes them, unless the lock's other entries show fractions. Its artifacts come back in the TOML spelling the other entries use: uv's inline `wheels = [{ … }]`, or the standard tables `pip lock` writes (`[[packages.wheels]]` with a `[packages.wheels.hashes]` sub-table, `[packages.sdist]`). A `pip lock` file (`created-by = "pip"`) records only the artifact pip selected, so the entry is restored with only the release's wheel (its sdist when it has none), and a release with several wheels is refused. A transitive `override-dependencies` entry hosted mode added is removed (`upstream_uv_override_removed`). Hosted mode keeps no ledger, so it marks the entry it adds with a `# socket-patch hosted: …` comment line above it and removes only a marked entry; a user's own `==` override (which the hosted rewrite reuses rather than adding a second one) is kept, and the lock's `[manifest] overrides` record of it gets its specifier back. + * **pypi** — `Pipfile.lock`, `requirements.txt` (+ in-root `-r` includes), Hatch PEP 508 direct references (`pyproject.toml` / `hatch.toml`), `poetry.lock`, `pdm.lock`, `uv.lock`, PEP 723 script locks and PEP 751 `pylock*.toml` (+ the paired `pyproject.toml` / script metadata): hashes re-derived from PyPI's JSON API (`SOCKET_PYPI_JSON_API`). A restored `requirements.txt` line gets `--hash` options only when the file is in pip's hash-checking mode. The mode is read off the file's other requirement lines (an `-e` / `--editable` line means unhashed). When every requirement is a hosted pin, it is read off the hosted line itself (`--hash` vs a `#sha256=` url fragment) (#410). Refused: a `static_urls` `pdm.lock` whose project installs from an index other than PyPI (a `[[tool.pdm.source]]` in the sibling `pyproject.toml`, a project `pypi.url` (`.pdm.toml`, then `pdm.toml`), or, with neither, another package's file URL in the lock off PyPI's file host), since restoring PyPI's file URLs would make `pdm sync` bypass that index (#413); a `pdm.lock` without `cross_platform`, or a uv / script / pylock lock, whose release has a wheel that is not pure Python 3 (which files the lock keeps is not re-derivable); a uv lock whose options filter files (`exclude-newer`, `no-binary`, `no-build`), or whose other registry packages name no registry, several, or one other than PyPI's simple index; a pylock whose other registry packages show neither an `index` nor (as `uv pip compile` writes them) only PyPI files with none, which restores the entry without an `index` too; uv 0.2 `[[distribution]]` locks. Restored artifact fields keep the spelling the lock's other entries show, including the `upload_time` that uv 0.6.15–0.6.17 write. A restored pylock entry's `upload-time`s are whole seconds, as uv writes them, unless the lock's other entries show fractions. Its artifacts come back in the TOML spelling the other entries use: uv's inline `wheels = [{ … }]`, or the standard tables `pip lock` writes (`[[packages.wheels]]` with a `[packages.wheels.hashes]` sub-table, `[packages.sdist]`). A `pip lock` file (`created-by = "pip"`) records only the artifact pip selected, so the entry is restored with only the release's wheel (its sdist when it has none), and a release with several wheels is refused. A transitive `override-dependencies` entry hosted mode added is removed (`upstream_uv_override_removed`). Hosted mode keeps no ledger, so it marks the entry it adds with a `# socket-patch hosted: …` comment line above it and removes only a marked entry; a user's own `==` override (which the hosted rewrite reuses rather than adding a second one) is kept, and the lock's `[manifest] overrides` record of it gets its specifier back. * **gem** — `Gemfile.lock` / `gems.locked` + `Gemfile` / `gems.rb`: the spec moves back into the upstream `GEM` section (or the Socket remote leaves a merged section), the `source "" do … end` block is undone, the `CHECKSUMS` entry is re-pinned from the rubygems.org compact index (`SOCKET_RUBYGEMS_URL`) and the `DEPENDENCIES` pin loses its `!`, unless the declaration comes back inside the user's own `source "…" do` block (Bundler writes the `!` for any dependency declared in one, rubygems.org included), where the `!` stays. The declaration's original constraint is not recorded, so it comes back as the exact pin `gem "", ""`. A transitive gem (one the manifest never declared) gets an appended block with a blank line before it; the restore removes that block, its blank line and the `DEPENDENCIES` entry, so the pair comes back byte for byte. An appended block with no blank line before it (written by a release before this one) can't be told apart from an in-place rewrite, so it still comes back as the exact pin. Refused: an ambiguous upstream section, an upstream remote other than rubygems.org. The manifest pair can't make a committed bundler cache upstream: a `-.gem` left in Bundler's cache dir that isn't the upstream archive is named by `upstream_gem_stale_cache`, never deleted. * **composer** — `composer.lock`: `dist` and the deleted `source` block from packagist's composer v2 metadata (`SOCKET_PACKAGIST_URL`). Refused unless the entry is packagist-sourced and packagist still serves the lock's `dist.reference` for the version. * **maven** — `pom.xml` (the `-socket.` version suffix, the added `` / `` entry) and the `.mvn/maven.config` / `.mvn/checksums/checksums.sha256` lines hosted mode writes: **no network**, so it restores under `--offline` too. `.mvn` files holding anything else keep the resolver lines (`maven_trusted_checksums_left`). diff --git a/crates/socket-patch-cli/tests/in_process_redirect_pdm.rs b/crates/socket-patch-cli/tests/in_process_redirect_pdm.rs index 07dc9c557..fbf288438 100644 --- a/crates/socket-patch-cli/tests/in_process_redirect_pdm.rs +++ b/crates/socket-patch-cli/tests/in_process_redirect_pdm.rs @@ -464,6 +464,64 @@ async fn lock_only_pdm_project_redirects_attests_rescans_and_rolls_back() { assert_no_ledger(tmp.path()); } +/// #413: a `static_urls` lock on a project whose `[[tool.pdm.source]]` +/// named `pypi` replaces PyPI with a private mirror. Rollback cannot know +/// the mirror's file URLs, and restoring PyPI's would make `pdm sync` +/// bypass the mirror, so both the dry run and the real rollback refuse +/// (exit 1) and leave the hosted lock in place. +#[tokio::test] +#[serial] +async fn static_urls_lock_on_a_private_index_is_not_rolled_back_to_pypi() { + const MIRROR: &str = "http://127.0.0.1:18780"; + let server = MockServer::start().await; + mock_api(&server).await; + let tmp = tempfile::tempdir().unwrap(); + let lock = LOCK + .replace( + "strategy = [\"inherit_metadata\"]", + "strategy = [\"inherit_metadata\", \"static_urls\"]", + ) + .replace("{file = \"", &format!("{{url = \"{MIRROR}/files/")); + write_project(tmp.path(), &lock); + let pyproject = format!( + "{PYPROJECT}\n[[tool.pdm.source]]\nname = \"pypi\"\nurl = \"{MIRROR}/simple\"\nverify_ssl = false\n" + ); + std::fs::write(tmp.path().join("pyproject.toml"), &pyproject).unwrap(); + let lock_path = tmp.path().join("pdm.lock"); + + let code = run(hosted_args(tmp.path(), server.uri(), None)).await; + assert_eq!(code, 0, "hosted redirect must succeed"); + let redirected = read(&lock_path); + assert!(redirected.contains(HOSTED_URL), "{redirected}"); + + mock_pypi(&server).await; + std::env::set_var("SOCKET_PYPI_JSON_API", format!("{}/pypi", server.uri())); + for dry_run in [true, false] { + let code = rollback::run(RollbackArgs { + targets: Vec::new(), + common: GlobalArgs { + patch_server_url: Some(PATCH_SERVER.to_string()), + dry_run, + ..global(tmp.path(), server.uri()) + }, + preserve_state: false, + }) + .await; + assert_eq!(code, 1, "dry_run={dry_run}: the restore is refused"); + let after = read(&lock_path); + assert!( + !after.contains("files.pythonhosted.org"), + "dry_run={dry_run}: PyPI URLs must never replace the mirror's: {after}" + ); + assert_eq!( + after, redirected, + "dry_run={dry_run}: the lock is untouched" + ); + } + std::env::remove_var("SOCKET_PYPI_JSON_API"); + assert_eq!(read(&tmp.path().join("pyproject.toml")), pyproject); +} + /// A PDM project whose `pyproject.toml` names `hatchling` as its build /// backend is still a PDM project. The hatch rewriter claims every pypi uuid /// for such a project and then yields to the lock without confirming any, so diff --git a/crates/socket-patch-core/src/patch/redirect/upstream/pypi_locks.rs b/crates/socket-patch-core/src/patch/redirect/upstream/pypi_locks.rs index 6d14fdc94..280aed818 100644 --- a/crates/socket-patch-core/src/patch/redirect/upstream/pypi_locks.rs +++ b/crates/socket-patch-core/src/patch/redirect/upstream/pypi_locks.rs @@ -34,7 +34,8 @@ use toml_edit::{DocumentMut, Item}; use super::client::PypiFile; use super::pypi::{ - fetch_release_files, multiline_toml_array, pin_of, toml_quote, toml_value, universal_release, + fetch_release_files, is_pypi_simple, multiline_toml_array, pin_of, toml_quote, toml_value, + universal_release, }; use super::{by_uuid, read_or_refuse, refuse_all_in, Ctx, FormatResult, HostedPin, View}; use crate::crawlers::python_crawler::canonicalize_pypi_name; @@ -309,6 +310,120 @@ pub(crate) async fn restore_poetry( // ── pdm.lock ───────────────────────────────────────────────────────────────── +/// The `url`s of a `static_urls` lock's file entries outside `skip` (the +/// hosted packages): inline `files` and lock_version 2 `[metadata.files]`. +fn pdm_file_urls(doc: &DocumentMut, skip: &BTreeSet) -> Vec { + fn urls_in(item: Option<&Item>, out: &mut Vec) { + let Some(files) = item.and_then(Item::as_array) else { + return; + }; + for file in files.iter().filter_map(|f| f.as_inline_table()) { + if let Some(url) = file.get("url").and_then(|u| u.as_str()) { + out.push(url.to_string()); + } + } + } + let mut out = Vec::new(); + if let Some(packages) = doc.get("package").and_then(Item::as_array_of_tables) { + for (index, package) in packages.iter().enumerate() { + if !skip.contains(&index) { + urls_in(package.get("files"), &mut out); + } + } + } + out +} + +/// Whether `url` is one of PyPI's own file hosts. +fn is_pypi_file_url(url: &str) -> bool { + reqwest::Url::parse(url).is_ok_and(|u| { + u.scheme() == "https" + && matches!( + u.host_str(), + Some("files.pythonhosted.org" | "pypi.org" | "pypi.python.org") + ) + }) +} + +/// Why a `static_urls` `pdm.lock` cannot get PyPI's file URLs back (#413): +/// such a lock records where PDM downloaded every file, and the restore +/// only knows PyPI's. When the project installs from another index — a +/// `[[tool.pdm.source]]` in its `pyproject.toml` (one named `pypi` +/// replaces PyPI; any other may have served the package), a project +/// `pypi.url` (`.pdm.toml`, then `pdm.toml`), or, failing both, the index +/// the lock's other packages were downloaded from — writing PyPI's URLs +/// would make `pdm sync` bypass that mirror (or fail where only the mirror +/// is reachable). +/// Like the uv and Pipenv restores, refuse rather than guess. +async fn pdm_static_index_refusal( + view: &mut View<'_>, + rel: &str, + doc: &DocumentMut, + hits: &[LockHit], +) -> Option { + let dir = match rel.rfind('/') { + Some(i) => &rel[..=i], + None => "", + }; + let mut foreign: Vec<(String, String)> = Vec::new(); + let pyproject = format!("{dir}pyproject.toml"); + if let Ok(Some(text)) = view.read(&pyproject).await { + if let Ok(project) = text.parse::() { + let sources = project + .get("tool") + .and_then(|t| t.get("pdm")) + .and_then(|p| p.get("source")) + .and_then(Item::as_array_of_tables); + for source in sources.into_iter().flatten() { + let url = source.get("url").and_then(Item::as_str).unwrap_or(""); + if !is_pypi_simple(url) { + foreign.push(( + url.to_string(), + format!("`[[tool.pdm.source]]` in {pyproject}"), + )); + } + } + } + } + // PDM overlays the legacy `.pdm.toml` on `pdm.toml`, so the first of + // the two that sets `pypi.url` is the project's. + for name in [".pdm.toml", "pdm.toml"] { + let config_rel = format!("{dir}{name}"); + let Ok(Some(text)) = view.read(&config_rel).await else { + continue; + }; + let Ok(config) = text.parse::() else { + continue; + }; + let Some(url) = config + .get("pypi") + .and_then(|p| p.get("url")) + .and_then(Item::as_str) + else { + continue; + }; + if !is_pypi_simple(url) { + foreign.push((url.to_string(), format!("`pypi.url` in {config_rel}"))); + } + break; + } + if foreign.is_empty() { + let skip: BTreeSet = hits.iter().map(|h| h.index).collect(); + if let Some(url) = pdm_file_urls(doc, &skip) + .into_iter() + .find(|u| !is_pypi_file_url(u)) + { + foreign.push((url, format!("another package's file in {rel}"))); + } + } + let (url, origin) = foreign.into_iter().next()?; + Some(format!( + "{rel} is a static_urls lock and the project installs from {url} ({origin}), not \ + PyPI; the restore can only re-derive PyPI's file URLs, which would make `pdm sync` \ + bypass that index, so the original URLs are not derivable" + )) +} + pub(crate) async fn restore_pdm( view: &mut View<'_>, pins: &[&HostedPin], @@ -347,6 +462,14 @@ pub(crate) async fn restore_pdm( } }; let static_urls = flags.iter().any(|f| f == "static_urls"); + if static_urls { + if let Some(why) = pdm_static_index_refusal(view, rel, &doc, &hits).await { + for hit in &hits { + result.refuse(&hit.uuid, why.clone()); + } + continue; + } + } // lock_version 2 (PDM 0.x/1.x) always recorded every release file. let cross_platform = version == "2" || flags.iter().any(|f| f == "cross_platform"); let released = fetch_release_files(&wanted(&hits, &result), ctx, &mut result).await; diff --git a/crates/socket-patch-core/tests/upstream_restore_golden.rs b/crates/socket-patch-core/tests/upstream_restore_golden.rs index a6049c0b1..9681cdcc3 100644 --- a/crates/socket-patch-core/tests/upstream_restore_golden.rs +++ b/crates/socket-patch-core/tests/upstream_restore_golden.rs @@ -1494,6 +1494,138 @@ async fn pdm_static_urls_round_trip() { assert_pypi_round_trip("pdm static_urls", &input, &[urllib3_dep()], None).await; } +/// The 2.29.2 fixture as a `static_urls` lock whose files come from +/// `host` (PDM orders them by URL). +fn pdm_static_urls_lock(url_of: impl Fn(&str) -> String) -> String { + let lock = fixture("pdm-native/2.29.2.lock").replace( + "strategy = [\"inherit_metadata\"]", + "strategy = [\"inherit_metadata\", \"static_urls\"]", + ); + let mut files = [URLLIB3_WHEEL, URLLIB3_SDIST].map(|f| { + let sha = if f == URLLIB3_WHEEL { + URLLIB3_WHEEL_SHA + } else { + URLLIB3_SDIST_SHA + }; + format!( + " {{url = \"{}\", hash = \"sha256:{sha}\"}},\n", + url_of(f) + ) + }); + files.sort(); + let start = lock.find("files = [\n").unwrap() + "files = [\n".len(); + let end = start + lock[start..].find("]\n").unwrap(); + format!("{}{}{}", &lock[..start], files.concat(), &lock[end..]) +} + +/// #413: a `static_urls` lock records where every file was downloaded +/// from. When the project installs from a private index or mirror, the +/// restore cannot know that index's URLs, and writing PyPI's would make +/// `pdm sync` bypass the mirror (or fail behind a firewall). Refuse, like +/// the uv and Pipenv restores refuse a non-PyPI registry, and leave the +/// hosted lock as it is. +#[tokio::test] +#[serial] +async fn pdm_static_urls_private_index_restore_is_refused() { + let (_server, _env) = pypi_mock(&[urllib3_release()]).await; + const MIRROR: &str = "http://127.0.0.1:18780"; + let mirror_url = |f: &str| format!("{MIRROR}/files/{f}"); + let lock = pdm_static_urls_lock(mirror_url); + let pyproject = |sources: &str| { + format!( + "[project]\nname = \"proj\"\nversion = \"0.1.0\"\nrequires-python = \">=3.8\"\n\ + dependencies = [\"urllib3==1.26.18\"]\n\n[tool.pdm]\ndistribution = false\n{sources}" + ) + }; + let replaced = pyproject(&format!( + "\n[[tool.pdm.source]]\nname = \"pypi\"\nurl = \"{MIRROR}/simple\"\nverify_ssl = false\n" + )); + let extra = pyproject(&format!( + "\n[[tool.pdm.source]]\nname = \"private\"\nurl = \"{MIRROR}/simple\"\n" + )); + let sibling = lock.replace( + "\n[[package]]\nname = \"urllib3\"", + &format!( + "\n[[package]]\nname = \"idna\"\nversion = \"3.4\"\nsummary = \"x\"\n\ + groups = [\"default\"]\nfiles = [\n {{url = \"{MIRROR}/files/idna-3.4-py3-none-any.whl\", \ + hash = \"sha256:{}\"}},\n]\n\n[[package]]\nname = \"urllib3\"", + "a".repeat(64) + ), + ); + let cases = [ + // The issue's shape: `[[tool.pdm.source]] name = "pypi"` replaces PyPI. + ( + "replaced pypi", + tree(&[ + ("pdm.lock", lock.clone()), + ("pyproject.toml", replaced.clone()), + ]), + ), + // A supplementary index may have served the package. + ( + "extra index", + tree(&[("pdm.lock", lock.clone()), ("pyproject.toml", extra)]), + ), + // Project-level `pdm config pypi.url` (pdm.toml). + ( + "pdm.toml pypi.url", + tree(&[ + ("pdm.lock", lock.clone()), + ("pdm.toml", format!("[pypi]\nurl = \"{MIRROR}/simple\"\n")), + ]), + ), + // The legacy `.pdm.toml`, which PDM overlays on `pdm.toml`. + ( + ".pdm.toml pypi.url", + tree(&[ + ("pdm.lock", lock.clone()), + (".pdm.toml", format!("[pypi]\nurl = \"{MIRROR}/simple\"\n")), + ( + "pdm.toml", + "[pypi]\nurl = \"https://pypi.org/simple\"\n".to_string(), + ), + ]), + ), + // No config in the project (a user-global `pdm config`), but the + // lock's other packages show where PDM downloads from. + ("sibling package urls", tree(&[("pdm.lock", sibling)])), + ]; + for (label, input) in cases { + let (why, rewritten, after) = + pypi_refusal(&input, &[urllib3_dep()], &RestoreOptions::default()).await; + assert!( + why.contains("static_urls") && why.contains(MIRROR), + "{label}: {why}" + ); + assert!(why.contains("pdm.lock"), "{label}: {why}"); + assert_eq!( + after, rewritten, + "{label}: the hosted lock is left as it is" + ); + } + + // Controls. A PyPI-hosted static_urls lock beside a source that names + // PyPI itself still round-trips. + let pypi_lock = pdm_static_urls_lock(pypi_file_url); + let pypi_source = + pyproject("\n[[tool.pdm.source]]\nname = \"pypi\"\nurl = \"https://pypi.org/simple\"\n"); + let input = tree(&[("pdm.lock", pypi_lock), ("pyproject.toml", pypi_source)]); + assert_pypi_round_trip( + "pdm static_urls, PyPI source", + &input, + &[urllib3_dep()], + None, + ) + .await; + // A lock without static_urls records file names and hashes only, which + // a mirror of PyPI serves unchanged. + let input = tree(&[ + ("pdm.lock", fixture("pdm-native/2.29.2.lock")), + ("pyproject.toml", replaced), + ]); + assert_pypi_round_trip("pdm mirror, file names", &input, &[urllib3_dep()], None).await; +} + #[tokio::test] #[serial] async fn pdm_narrowed_lock_with_platform_wheels_is_refused() { diff --git a/docs/testing/pdm-compatibility.md b/docs/testing/pdm-compatibility.md index 4f8d372cd..31a87f722 100644 --- a/docs/testing/pdm-compatibility.md +++ b/docs/testing/pdm-compatibility.md @@ -39,7 +39,10 @@ Both modes retain the package version, extras, groups, markers and the lock unchanged (idempotent). Vendored rollback restores recorded fragments. Hosted rollback reconstructs upstream entries from registry metadata and may refuse after incompatible relocks; -it does not keep the original lockfile bytes. +it does not keep the original lockfile bytes. A `static_urls` lock whose project +installs from a private index or mirror (`[[tool.pdm.source]]`, `pdm.toml` +`pypi.url`, or mirror URLs on the lock's other packages) is refused rather than +given PyPI's file URLs (#413): restore `pdm.lock` from version control. After any unwind (`rollback`, `remove`, `vendor --revert`) PDM keeps the patched build installed, because the restored lock pins the same version: `pdm sync` and `pdm install` report nothing to do. The unwind warns