Fix PDM restore replacing a private index (#413)

crates/socket-patch-cli/CLI_CONTRACT.md

@@ -1028,7 +1028,7 @@ v5.0 replaces v4's per-purl reverts and whole-ledger reverse replay (`revert_rem
   * **vlt** — `vlt-lock.json`: slot [2] from the registry's `dist.integrity`, slot [3] per the lock's own convention (see the vlt hosted-mode contract); every hosted instance of the pin together.
   * **cargo** — `Cargo.lock` back on crates.io (source + the sparse index's checksum, `SOCKET_CRATES_INDEX`). When the lock already holds a crates.io block for the same `name@version` (a later dependent locked its own copy, #679), the Socket block merges into it instead: it is dropped (with a v1 lock's `[metadata]` line), and dependents' references take the spelling cargo writes, so no duplicate block is left that cargo cannot parse (#863), and no index lookup is needed; every `Cargo.toml` declaration loses its `registry = "socket-patch-<uuid>"` pin (the shorthand the rewriter produced collapses back); every `[registries.socket-patch-<uuid>]` block no manifest or lock still references leaves the project cargo config — including a superseded patch generation's block an earlier re-pin left behind (#864). A declaration it cannot unpin refuses.
   * **golang** — the hosted `replace` and the socket module's go.sum lines go; the upstream module's two go.sum lines come back, hashed from the module proxy (`SOCKET_GOPROXY`, else `GOPROXY` / `GONOPROXY` / `GOPRIVATE` as go reads them) and cross-checked against the checksum database (`SOCKET_GOSUMDB_URL`, else `sum.golang.org` unless `GOSUMDB=off` / `GONOSUMDB` / `GOPRIVATE` say go would not ask it). A `replace` the user had before the hosted run is not recorded anywhere, so the restore lands on the plain upstream module.
-  * **pypi** — `Pipfile.lock`, `requirements.txt` (+ in-root `-r` includes), Hatch PEP 508 direct references (`pyproject.toml` / `hatch.toml`), `poetry.lock`, `pdm.lock`, `uv.lock`, PEP 723 script locks and PEP 751 `pylock*.toml` (+ the paired `pyproject.toml` / script metadata): hashes re-derived from PyPI's JSON API (`SOCKET_PYPI_JSON_API`). A restored `requirements.txt` line gets `--hash` options only when the file is in pip's hash-checking mode. The mode is read off the file's other requirement lines (an `-e` / `--editable` line means unhashed). When every requirement is a hosted pin, it is read off the hosted line itself (`--hash` vs a `#sha256=` url fragment) (#410). Refused: a `pdm.lock` without `cross_platform`, or a uv / script / pylock lock, whose release has a wheel that is not pure Python 3 (which files the lock keeps is not re-derivable); a uv lock whose options filter files (`exclude-newer`, `no-binary`, `no-build`), or whose other registry packages name no registry, several, or one other than PyPI's simple index; a pylock whose other registry packages show neither an `index` nor (as `uv pip compile` writes them) only PyPI files with none, which restores the entry without an `index` too; uv 0.2 `[[distribution]]` locks. Restored artifact fields keep the spelling the lock's other entries show, including the `upload_time` that uv 0.6.15–0.6.17 write. A restored pylock entry's `upload-time`s are whole seconds, as uv writes them, unless the lock's other entries show fractions. Its artifacts come back in the TOML spelling the other entries use: uv's inline `wheels = [{ … }]`, or the standard tables `pip lock` writes (`[[packages.wheels]]` with a `[packages.wheels.hashes]` sub-table, `[packages.sdist]`). A `pip lock` file (`created-by = "pip"`) records only the artifact pip selected, so the entry is restored with only the release's wheel (its sdist when it has none), and a release with several wheels is refused. A transitive `override-dependencies` entry hosted mode added is removed (`upstream_uv_override_removed`). Hosted mode keeps no ledger, so it marks the entry it adds with a `# socket-patch hosted: …` comment line above it and removes only a marked entry; a user's own `<name>==<version>` override (which the hosted rewrite reuses rather than adding a second one) is kept, and the lock's `[manifest] overrides` record of it gets its specifier back.
+  * **pypi** — `Pipfile.lock`, `requirements.txt` (+ in-root `-r` includes), Hatch PEP 508 direct references (`pyproject.toml` / `hatch.toml`), `poetry.lock`, `pdm.lock`, `uv.lock`, PEP 723 script locks and PEP 751 `pylock*.toml` (+ the paired `pyproject.toml` / script metadata): hashes re-derived from PyPI's JSON API (`SOCKET_PYPI_JSON_API`). A restored `requirements.txt` line gets `--hash` options only when the file is in pip's hash-checking mode. The mode is read off the file's other requirement lines (an `-e` / `--editable` line means unhashed). When every requirement is a hosted pin, it is read off the hosted line itself (`--hash` vs a `#sha256=` url fragment) (#410). Refused: a `static_urls` `pdm.lock` whose project installs from an index other than PyPI (a `[[tool.pdm.source]]` in the sibling `pyproject.toml`, a project `pypi.url` (`.pdm.toml`, then `pdm.toml`), or, with neither, another package's file URL in the lock off PyPI's file host), since restoring PyPI's file URLs would make `pdm sync` bypass that index (#413); a `pdm.lock` without `cross_platform`, or a uv / script / pylock lock, whose release has a wheel that is not pure Python 3 (which files the lock keeps is not re-derivable); a uv lock whose options filter files (`exclude-newer`, `no-binary`, `no-build`), or whose other registry packages name no registry, several, or one other than PyPI's simple index; a pylock whose other registry packages show neither an `index` nor (as `uv pip compile` writes them) only PyPI files with none, which restores the entry without an `index` too; uv 0.2 `[[distribution]]` locks. Restored artifact fields keep the spelling the lock's other entries show, including the `upload_time` that uv 0.6.15–0.6.17 write. A restored pylock entry's `upload-time`s are whole seconds, as uv writes them, unless the lock's other entries show fractions. Its artifacts come back in the TOML spelling the other entries use: uv's inline `wheels = [{ … }]`, or the standard tables `pip lock` writes (`[[packages.wheels]]` with a `[packages.wheels.hashes]` sub-table, `[packages.sdist]`). A `pip lock` file (`created-by = "pip"`) records only the artifact pip selected, so the entry is restored with only the release's wheel (its sdist when it has none), and a release with several wheels is refused. A transitive `override-dependencies` entry hosted mode added is removed (`upstream_uv_override_removed`). Hosted mode keeps no ledger, so it marks the entry it adds with a `# socket-patch hosted: …` comment line above it and removes only a marked entry; a user's own `<name>==<version>` override (which the hosted rewrite reuses rather than adding a second one) is kept, and the lock's `[manifest] overrides` record of it gets its specifier back.
   * **gem** — `Gemfile.lock` / `gems.locked` + `Gemfile` / `gems.rb`: the spec moves back into the upstream `GEM` section (or the Socket remote leaves a merged section), the `source "<patch registry>" do … end` block is undone, the `CHECKSUMS` entry is re-pinned from the rubygems.org compact index (`SOCKET_RUBYGEMS_URL`) and the `DEPENDENCIES` pin loses its `!`, unless the declaration comes back inside the user's own `source "…" do` block (Bundler writes the `!` for any dependency declared in one, rubygems.org included), where the `!` stays. The declaration's original constraint is not recorded, so it comes back as the exact pin `gem "<name>", "<version>"`. A transitive gem (one the manifest never declared) gets an appended block with a blank line before it; the restore removes that block, its blank line and the `DEPENDENCIES` entry, so the pair comes back byte for byte. An appended block with no blank line before it (written by a release before this one) can't be told apart from an in-place rewrite, so it still comes back as the exact pin. Refused: an ambiguous upstream section, an upstream remote other than rubygems.org. The manifest pair can't make a committed bundler cache upstream: a `<name>-<version>.gem` left in Bundler's cache dir that isn't the upstream archive is named by `upstream_gem_stale_cache`, never deleted.
   * **composer** — `composer.lock`: `dist` and the deleted `source` block from packagist's composer v2 metadata (`SOCKET_PACKAGIST_URL`). Refused unless the entry is packagist-sourced and packagist still serves the lock's `dist.reference` for the version.
   * **maven** — `pom.xml` (the `-socket.<hex8>` version suffix, the added `<repository>` / `<dependencyManagement>` entry) and the `.mvn/maven.config` / `.mvn/checksums/checksums.sha256` lines hosted mode writes: **no network**, so it restores under `--offline` too. `.mvn` files holding anything else keep the resolver lines (`maven_trusted_checksums_left`).

crates/socket-patch-cli/tests/in_process_redirect_pdm.rs

@@ -464,6 +464,64 @@ async fn lock_only_pdm_project_redirects_attests_rescans_and_rolls_back() {
     assert_no_ledger(tmp.path());
 }
 
+/// #413: a `static_urls` lock on a project whose `[[tool.pdm.source]]`
+/// named `pypi` replaces PyPI with a private mirror. Rollback cannot know
+/// the mirror's file URLs, and restoring PyPI's would make `pdm sync`
+/// bypass the mirror, so both the dry run and the real rollback refuse
+/// (exit 1) and leave the hosted lock in place.
+#[tokio::test]
+#[serial]
+async fn static_urls_lock_on_a_private_index_is_not_rolled_back_to_pypi() {
+    const MIRROR: &str = "http://127.0.0.1:18780";
+    let server = MockServer::start().await;
+    mock_api(&server).await;
+    let tmp = tempfile::tempdir().unwrap();
+    let lock = LOCK
+        .replace(
+            "strategy = [\"inherit_metadata\"]",
+            "strategy = [\"inherit_metadata\", \"static_urls\"]",
+        )
+        .replace("{file = \"", &format!("{{url = \"{MIRROR}/files/"));
+    write_project(tmp.path(), &lock);
+    let pyproject = format!(
+        "{PYPROJECT}\n[[tool.pdm.source]]\nname = \"pypi\"\nurl = \"{MIRROR}/simple\"\nverify_ssl = false\n"
+    );
+    std::fs::write(tmp.path().join("pyproject.toml"), &pyproject).unwrap();
+    let lock_path = tmp.path().join("pdm.lock");
+
+    let code = run(hosted_args(tmp.path(), server.uri(), None)).await;
+    assert_eq!(code, 0, "hosted redirect must succeed");
+    let redirected = read(&lock_path);
+    assert!(redirected.contains(HOSTED_URL), "{redirected}");
+
+    mock_pypi(&server).await;
+    std::env::set_var("SOCKET_PYPI_JSON_API", format!("{}/pypi", server.uri()));
+    for dry_run in [true, false] {
+        let code = rollback::run(RollbackArgs {
+            targets: Vec::new(),
+            common: GlobalArgs {
+                patch_server_url: Some(PATCH_SERVER.to_string()),
+                dry_run,
+                ..global(tmp.path(), server.uri())
+            },
+            preserve_state: false,
+        })
+        .await;
+        assert_eq!(code, 1, "dry_run={dry_run}: the restore is refused");
+        let after = read(&lock_path);
+        assert!(
+            !after.contains("files.pythonhosted.org"),
+            "dry_run={dry_run}: PyPI URLs must never replace the mirror's: {after}"
+        );
+        assert_eq!(
+            after, redirected,
+            "dry_run={dry_run}: the lock is untouched"
+        );
+    }
+    std::env::remove_var("SOCKET_PYPI_JSON_API");
+    assert_eq!(read(&tmp.path().join("pyproject.toml")), pyproject);
+}
+
 /// A PDM project whose `pyproject.toml` names `hatchling` as its build
 /// backend is still a PDM project. The hatch rewriter claims every pypi uuid
 /// for such a project and then yields to the lock without confirming any, so

crates/socket-patch-core/src/patch/redirect/upstream/pypi_locks.rs

@@ -34,7 +34,8 @@ use toml_edit::{DocumentMut, Item};
 
 use super::client::PypiFile;
 use super::pypi::{
-    fetch_release_files, multiline_toml_array, pin_of, toml_quote, toml_value, universal_release,
+    fetch_release_files, is_pypi_simple, multiline_toml_array, pin_of, toml_quote, toml_value,
+    universal_release,
 };
 use super::{by_uuid, read_or_refuse, refuse_all_in, Ctx, FormatResult, HostedPin, View};
 use crate::crawlers::python_crawler::canonicalize_pypi_name;
@@ -309,6 +310,120 @@ pub(crate) async fn restore_poetry(
 
 // ── pdm.lock ─────────────────────────────────────────────────────────────────
 
+/// The `url`s of a `static_urls` lock's file entries outside `skip` (the
+/// hosted packages): inline `files` and lock_version 2 `[metadata.files]`.
+fn pdm_file_urls(doc: &DocumentMut, skip: &BTreeSet<usize>) -> Vec<String> {
+    fn urls_in(item: Option<&Item>, out: &mut Vec<String>) {
+        let Some(files) = item.and_then(Item::as_array) else {
+            return;
+        };
+        for file in files.iter().filter_map(|f| f.as_inline_table()) {
+            if let Some(url) = file.get("url").and_then(|u| u.as_str()) {
+                out.push(url.to_string());
+            }
+        }
+    }
+    let mut out = Vec::new();
+    if let Some(packages) = doc.get("package").and_then(Item::as_array_of_tables) {
+        for (index, package) in packages.iter().enumerate() {
+            if !skip.contains(&index) {
+                urls_in(package.get("files"), &mut out);
+            }
+        }
+    }
+    out
+}
+
+/// Whether `url` is one of PyPI's own file hosts.
+fn is_pypi_file_url(url: &str) -> bool {
+    reqwest::Url::parse(url).is_ok_and(|u| {
+        u.scheme() == "https"
+            && matches!(
+                u.host_str(),
+                Some("files.pythonhosted.org" | "pypi.org" | "pypi.python.org")
+            )
+    })
+}
+
+/// Why a `static_urls` `pdm.lock` cannot get PyPI's file URLs back (#413):
+/// such a lock records where PDM downloaded every file, and the restore
+/// only knows PyPI's. When the project installs from another index — a
+/// `[[tool.pdm.source]]` in its `pyproject.toml` (one named `pypi`
+/// replaces PyPI; any other may have served the package), a project
+/// `pypi.url` (`.pdm.toml`, then `pdm.toml`), or, failing both, the index
+/// the lock's other packages were downloaded from — writing PyPI's URLs
+/// would make `pdm sync` bypass that mirror (or fail where only the mirror
+/// is reachable).
+/// Like the uv and Pipenv restores, refuse rather than guess.
+async fn pdm_static_index_refusal(
+    view: &mut View<'_>,
+    rel: &str,
+    doc: &DocumentMut,
+    hits: &[LockHit],
+) -> Option<String> {
+    let dir = match rel.rfind('/') {
+        Some(i) => &rel[..=i],
+        None => "",
+    };
+    let mut foreign: Vec<(String, String)> = Vec::new();
+    let pyproject = format!("{dir}pyproject.toml");
+    if let Ok(Some(text)) = view.read(&pyproject).await {
+        if let Ok(project) = text.parse::<DocumentMut>() {
+            let sources = project
+                .get("tool")
+                .and_then(|t| t.get("pdm"))
+                .and_then(|p| p.get("source"))
+                .and_then(Item::as_array_of_tables);
+            for source in sources.into_iter().flatten() {
+                let url = source.get("url").and_then(Item::as_str).unwrap_or("");
+                if !is_pypi_simple(url) {
+                    foreign.push((
+                        url.to_string(),
+                        format!("`[[tool.pdm.source]]` in {pyproject}"),
+                    ));
+                }
+            }
+        }
+    }
+    // PDM overlays the legacy `.pdm.toml` on `pdm.toml`, so the first of
+    // the two that sets `pypi.url` is the project's.
+    for name in [".pdm.toml", "pdm.toml"] {
+        let config_rel = format!("{dir}{name}");
+        let Ok(Some(text)) = view.read(&config_rel).await else {
+            continue;
+        };
+        let Ok(config) = text.parse::<DocumentMut>() else {
+            continue;
+        };
+        let Some(url) = config
+            .get("pypi")
+            .and_then(|p| p.get("url"))
+            .and_then(Item::as_str)
+        else {
+            continue;
+        };
+        if !is_pypi_simple(url) {
+            foreign.push((url.to_string(), format!("`pypi.url` in {config_rel}")));
+        }
+        break;
+    }
+    if foreign.is_empty() {
+        let skip: BTreeSet<usize> = hits.iter().map(|h| h.index).collect();
+        if let Some(url) = pdm_file_urls(doc, &skip)
+            .into_iter()
+            .find(|u| !is_pypi_file_url(u))
+        {
+            foreign.push((url, format!("another package's file in {rel}")));
+        }
+    }
+    let (url, origin) = foreign.into_iter().next()?;
+    Some(format!(
+        "{rel} is a static_urls lock and the project installs from {url} ({origin}), not \
+         PyPI; the restore can only re-derive PyPI's file URLs, which would make `pdm sync` \
+         bypass that index, so the original URLs are not derivable"
+    ))
+}
+
 pub(crate) async fn restore_pdm(
     view: &mut View<'_>,
     pins: &[&HostedPin],
@@ -347,6 +462,14 @@ pub(crate) async fn restore_pdm(
             }
         };
         let static_urls = flags.iter().any(|f| f == "static_urls");
+        if static_urls {
+            if let Some(why) = pdm_static_index_refusal(view, rel, &doc, &hits).await {
+                for hit in &hits {
+                    result.refuse(&hit.uuid, why.clone());
+                }
+                continue;
+            }
+        }
         // lock_version 2 (PDM 0.x/1.x) always recorded every release file.
         let cross_platform = version == "2" || flags.iter().any(|f| f == "cross_platform");
         let released = fetch_release_files(&wanted(&hits, &result), ctx, &mut result).await;