diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index d5f8cecf0..5fde8db0d 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -1530,8 +1530,10 @@ command's integration tests check its output. | `vendor_lock_entry_pruned_kept` | revert `warnings[]` | vendor `--revert` / rollback (yarn berry, #737): vendoring dropped a lock entry only the patched package's implicit `node-gyp` dependency reached (or one descriptor of a shared entry), and the lock has re-added or changed that entry since; it is left as it is (not drift: the artifact is still removed). Run `yarn install` if the lock is incomplete. | | `pypi_{poetry,pdm,pipenv}_no_lockfile` | `failed` | vendor (pypi): a lock-less tool marker with no `requirements.txt` fallback — run ` lock`. | | `pypi_poetry_integrity_unverified` | `skipped` (warning) | vendor (pypi / poetry): the lock was written by Poetry < 1.4 (0.12 `[metadata.hashes]`, lock 1.0/1.1, or a 2.0 lock without a `@generated by Poetry X.Y.Z` header — 1.3 wrote those). That installer does not verify local wheel hashes (the committed wheel bytes are the protection) and does not replace an already-installed package at the same version; recreate the virtualenv or `pip uninstall` the package before `poetry install`, or upgrade Poetry. | +| `pypi_poetry_source_constraint` | `failed` / `skipped` (warning) | vendor (pypi / poetry): the pyproject pins the package to a named source (`source = "pypi"`), which Poetry records nothing for in the lock. Over a lock stamped by Poetry 1.4 – 2.2 (whose installer refuses that constraint on a file-sourced package, so every `poetry install` would fail) the package is refused before any write, `failed`; an already-wired lock is refused the same way instead of reading as in sync. Over any other lock the wiring stands and this is a warning naming the 1.4 – 2.2 range. The detail names the remedy: drop the `source` key or upgrade to Poetry >= 2.3, re-lock and re-run; when the lock is already wired it first says to run `socket-patch vendor --revert` (which reverts EVERY vendored package), since re-locking discards the wiring. | | `redirect_poetry_stale_install_risk` | `warnings[]` (warning) | scan `--mode hosted` (poetry): same writer test as above — a warm virtualenv keeps the upstream package after the redirect on Poetry < 1.4 (1.4+ re-installs from the new source); fresh installs pick up the patched wheel. Emitted once per rewritten lock, only on the run that rewrites it. | -| `redirect_poetry_entry_not_found` / `redirect_poetry_missing_sha256` / `redirect_poetry_lock_unsupported` | `warnings[]` (warning) | scan `--mode hosted` (poetry): the lock has no `[[package]]` at the granted version (uv-parity twin of `redirect_uv_entry_not_found`); the grant carries no SHA-256 (gated once per dep, not per lock); the lock is refused — Poetry 0.12 layout (URL sources ignored), an unsupported `lock-version`, a forked package listed at several versions, a user-authored `[package.source]` on another origin (an earlier Socket URL for the same wheel is superseded in place), a malformed `[metadata.files]`/`[metadata.hashes]`, or a wheel whose filename does not match the locked package. Exit code and `status` unchanged (hosted-refusal posture). A vendored → hosted takeover over a Poetry 0.x lock is retracted with this code (wet and `--dry-run`, see **Staged takeover**): the purl stays vendored and patched and is skipped with this code as `skipped` (hosted event). The detail names the remedy and its reach: run `socket-patch vendor --revert` (it reverts EVERY vendored package in the project, not just this one), upgrade to Poetry >= 1.0 and re-lock, then re-run `scan --mode hosted`. | +| `redirect_poetry_source_constraint` | `warnings[]` (warning) | scan `--mode hosted` (poetry): the pyproject pins the redirected package to a named source (`source = "pypi"`) and the lock was not written by Poetry 1.4 – 2.2 (those writers are refused with `redirect_poetry_lock_unsupported`, also when the lock is already redirected and before a vendored → hosted takeover reverts anything). The rewrite stands, and an already-redirected lock is still confirmed; the detail says a Poetry 1.4 – 2.2 installer would refuse the lock and to drop the `source` key if the project is installed with one. Exit code and `status` unchanged. | +| `redirect_poetry_entry_not_found` / `redirect_poetry_missing_sha256` / `redirect_poetry_lock_unsupported` | `warnings[]` (warning) | scan `--mode hosted` (poetry): the lock has no `[[package]]` at the granted version (uv-parity twin of `redirect_uv_entry_not_found`); the grant carries no SHA-256 (gated once per dep, not per lock); the lock is refused — Poetry 0.12 layout (URL sources ignored), an unsupported `lock-version`, a forked package listed at several versions, a user-authored `[package.source]` on another origin (an earlier Socket URL for the same wheel is superseded in place), a package the pyproject pins to a named source (`source = "pypi"`) over a lock stamped by Poetry 1.4 – 2.2, whose installer refuses that constraint on a URL-sourced package, a malformed `[metadata.files]`/`[metadata.hashes]`, or a wheel whose filename does not match the locked package. Exit code and `status` unchanged (hosted-refusal posture). A vendored → hosted takeover over a Poetry 0.x lock is retracted with this code (wet and `--dry-run`, see **Staged takeover**): the purl stays vendored and patched and is skipped with this code as `skipped` (hosted event). The detail names the remedy and its reach: run `socket-patch vendor --revert` (it reverts EVERY vendored package in the project, not just this one), upgrade to Poetry >= 1.0 and re-lock, then re-run `scan --mode hosted`. | | `redirect_pdm_refused` / `redirect_pdm_legacy_sync_required` | `warnings[]` (warning) | scan `--mode hosted` (pdm): the `pdm.lock` rewrite was refused — an unsupported `[metadata] lock_version` (the identity-losing `3.1` / `4.0`–`4.2` formats or an untested future format), an unsupported `strategy`, a package listed at several versions (fork) or absent, a user-authored `url`/`path`/VCS/`editable` source, hash-less or malformed `files`, or a wheel whose filename does not match the locked package (`redirect_pdm_refused`); or the lock was written in format `2` (PDM 0.12–1.4), whose upstream freshness bug lets `pdm install` regenerate the lock — use `pdm sync` (`redirect_pdm_legacy_sync_required`). A refused uuid is withheld from every other PyPI rewriter when `pdm.lock` is the install driver, and its patch is not confirmed. Exit code and `status` unchanged (hosted-refusal posture). | | `redirect_bun_lock_unsupported` | `warnings[]` (warning) | scan/get `--mode hosted` (bun): the text lock's `lockfileVersion` is not 0, 1 or 2 (a newer version: update socket-patch, re-locking would reproduce it; no integer: re-lock with Bun ≥ 1.2 — the shared gate's text, identical to vendored's `vendor_lockfile_version_unsupported`), or its `packages` section is not bun's single-line grammar. Nothing rewritten; exit 0 (hosted-refusal posture). | | `redirect_bun_workspace_unsupported` | `warnings[]` (warning) | scan/get `--mode hosted` (bun): a lockfileVersion-0 lock (Bun 1.1.39–1.1.45 `--save-text-lockfile`) holds `workspace:` packages; frozen installs of that grammar cannot keep the hosted tuple. Detail: "Bun version-0 workspace locks cannot preserve hosted tarballs on frozen installs; delete bun.lock and re-run `bun install` with Bun >= 1.2 (which writes lockfileVersion 1, accepted by hosted mode) — a plain in-place `bun install` bumps the version only when a workspace depends on another workspace (e.g. root -> member); otherwise it keeps version 0 or fails to resolve" (measured: Bun 1.2.0 keeps 0, 1.2.23–1.4.2 exit 1 "failed to resolve" on a root that does not depend on its members). Version-1/2 workspace locks are rewritten. Exit 0. | diff --git a/crates/socket-patch-cli/tests/e2e_vex_build/poetry.rs b/crates/socket-patch-cli/tests/e2e_vex_build/poetry.rs index c4406c679..6b45a22ec 100644 --- a/crates/socket-patch-cli/tests/e2e_vex_build/poetry.rs +++ b/crates/socket-patch-cli/tests/e2e_vex_build/poetry.rs @@ -336,8 +336,18 @@ fn empty_venv(project: &Path) { /// `six.py` bytes from a pristine install in a scratch sibling. `None` /// (after [`skip`]) when PyPI is unreachable. fn locked_project(poetry: &Poetry, project: &Path, probe: &Path) -> Option<(Vec, String)> { + locked_project_with(poetry, project, probe, PYPROJECT) +} + +/// [`locked_project`] of an arbitrary `pyproject`. +fn locked_project_with( + poetry: &Poetry, + project: &Path, + probe: &Path, + pyproject: &str, +) -> Option<(Vec, String)> { std::fs::create_dir_all(project).unwrap(); - std::fs::write(project.join("pyproject.toml"), PYPROJECT).unwrap(); + std::fs::write(project.join("pyproject.toml"), pyproject).unwrap(); let out = poetry.run(project, &["lock", "-n"]); if !out.status.success() { skip(&format!("poetry lock (needs PyPI): {}", text(&out))); @@ -1164,3 +1174,93 @@ fn poetry_vendored_revendors_to_a_superseding_patch() { "revert restores the pristine lock" ); } + +/// `six = {version = "1.16.0", source = "pypi"}` leaves no trace in the +/// lock, and Poetry 1.4 – 2.2 refuse to install a lock that then sources six +/// from a URL or file ("Repository \"pypi\" does not exist."). Over a lock +/// one of them wrote, both modes refuse and leave the lock installable; +/// other releases install the rewired lock, with a warning (#1282). +#[test] +#[ignore = "real Poetry + PyPI; run by the CI e2e matrix per Poetry release"] +fn poetry_source_pypi_constraint_never_breaks_install() { + let tmp = tempfile::tempdir().unwrap(); + let home = tmp.path().join("home"); + std::fs::create_dir_all(&home).unwrap(); + let Some(poetry) = Poetry::find(&home) else { + return; + }; + if poetry.major_minor() < (1, 1) { + println!("Poetry {} predates per-dependency sources", poetry.version); + return; + } + let pyproject = PYPROJECT.replace( + "six = \"1.16.0\"", + "six = {version = \"1.16.0\", source = \"pypi\"}", + ); + let project = tmp.path().join("proj"); + let Some((pristine, pristine_lock)) = + locked_project_with(&poetry, &project, &tmp.path().join("probe"), &pyproject) + else { + return; + }; + // The lock was written by this Poetry, so it is also the installer. + let refused = ((1, 4)..(2, 3)).contains(&poetry.major_minor()); + let mut patched = pristine.clone(); + patched.extend_from_slice(MARKER); + let wheel = build_wheel(&patched); + + for (mode, uuid, refusal) in [ + ("hosted", HOSTED_UUID, "redirect_poetry_lock_unsupported"), + ("vendored", VENDORED_UUID, "pypi_poetry_source_constraint"), + ] { + let work = tmp.path().join(mode); + fresh_checkout(&project, &work); + let service = PatchService::start(uuid, &pristine, &patched, &wheel); + let out = poetry.install(&work); + assert!( + out.status.success(), + "{mode}: pristine install: {}", + text(&out) + ); + let mut args = vec!["scan", "--mode", mode]; + if mode == "vendored" { + args.extend(["--vendor-source", "service"]); + } + let (code, env) = socket_patch(&work, &poetry, &service, &args); + let lock = std::fs::read_to_string(work.join("poetry.lock")).unwrap(); + let report = env.to_string(); + if refused { + assert_eq!( + lock, pristine_lock, + "{mode}: refused before any write: {env}" + ); + assert!( + report.contains(refusal) && report.contains("source ="), + "{mode}: names the constraint ({code:?}): {env}" + ); + } else { + assert_eq!(code, Some(0), "{mode}: {env}"); + assert_ne!(lock, pristine_lock, "{mode}: rewired: {env}"); + assert!( + report.contains("source_constraint"), + "{mode}: warns about Poetry 1.4 – 2.2: {env}" + ); + } + // Either way a fresh install works, and installs the patch whenever + // the lock was rewired. + let fresh = tmp.path().join(format!("{mode}-fresh")); + fresh_checkout(&work, &fresh); + let out = poetry.install(&fresh); + assert!( + out.status.success(), + "{mode}: fresh install: {}", + text(&out) + ); + let installed = std::fs::read(installed_six(&fresh).expect("six installed")).unwrap(); + assert_eq!( + installed, + *if refused { &pristine } else { &patched }, + "{mode}: installed six" + ); + } +} diff --git a/crates/socket-patch-core/src/patch/redirect/poetry.rs b/crates/socket-patch-core/src/patch/redirect/poetry.rs index 5ee901ff6..c875d9f20 100644 --- a/crates/socket-patch-core/src/patch/redirect/poetry.rs +++ b/crates/socket-patch-core/src/patch/redirect/poetry.rs @@ -8,8 +8,11 @@ use serde_json::Value; use toml_edit::DocumentMut; use super::{DepOverride, FileEdit, RewriteResult, RewriteWarning}; +use crate::crawlers::python_crawler::canonicalize_pypi_name; use crate::utils::poetry_lock::{ - generated_by_version, lock_version, rewrite_poetry_lock_all, LockBatch, LockStep, PoetryLockDep, + generated_by_version, lock_version, lock_wiring, pyproject_source_constraints, + rewrite_poetry_lock_all, source_constraint_advisory, source_constraint_breaks_writer, + source_constraint_refusal, LockBatch, LockStep, PoetryLockDep, }; /// Whether the lock was written by a Poetry release older than 1.4. Those @@ -56,6 +59,51 @@ pub(super) fn rewrite_poetry( } } for (path, original) in locks { + // A pyproject `source = "…"` constraint on the dep (#1282). Poetry + // 1.4 – 2.2 fail every install of a lock that sources it from a URL, + // so a lock one of them wrote is refused; other writers are warned. + // The constraint lives in the pyproject beside the lock. + let pyproject_path = format!( + "{}pyproject.toml", + path.strip_suffix("poetry.lock").unwrap_or("") + ); + let constraints = files + .get(&pyproject_path) + .map(|text| pyproject_source_constraints(text)) + .unwrap_or_default(); + let constraint = |dep: &DepOverride| constraints.get(&canonicalize_pypi_name(&dep.name)); + let mut usable = usable.clone(); + if !constraints.is_empty() && source_constraint_breaks_writer(original) { + usable.retain(|&(dep, _)| { + // A dep the lock doesn't list keeps its `entry_not_found` + // report; one it lists (even already redirected) is refused. + let Some(source) = constraint(dep) else { + return true; + }; + let Some(wired) = lock_wiring(original, &dep.name, &dep.version) else { + return true; + }; + result + .refused_python_lock_uuids + .insert(dep.patch_uuid.clone()); + result.warnings.push(RewriteWarning { + code: "redirect_poetry_lock_unsupported".into(), + detail: format!( + "{path}: {}", + source_constraint_refusal( + &pyproject_path, + path, + &dep.name, + &dep.version, + source, + "URL", + wired, + ) + ), + }); + false + }); + } let deps: Vec = usable .iter() .map(|&(dep, sha256)| PoetryLockDep { @@ -92,6 +140,17 @@ pub(super) fn rewrite_poetry( result .confirmed_python_lock_uuids .insert(dep.patch_uuid.clone()); + if let Some(source) = constraint(dep) { + result.warnings.push(RewriteWarning { + code: "redirect_poetry_source_constraint".into(), + detail: source_constraint_advisory( + &pyproject_path, + path, + &dep.name, + source, + ), + }); + } if !stale_warned { if let Some(format) = *writer_format.get_or_insert_with(|| pre_1_4_writer(&content)) @@ -129,6 +188,17 @@ pub(super) fn rewrite_poetry( result .confirmed_python_lock_uuids .insert(dep.patch_uuid.clone()); + if let Some(source) = constraint(dep) { + result.warnings.push(RewriteWarning { + code: "redirect_poetry_source_constraint".into(), + detail: source_constraint_advisory( + &pyproject_path, + path, + &dep.name, + source, + ), + }); + } } LockStep::NotFound => result.warnings.push(RewriteWarning { code: "redirect_poetry_entry_not_found".into(), @@ -273,6 +343,130 @@ mod equivalence_tests { g.finish(); } + /// A dependency the pyproject pins to a named source (`source = "pypi"`) + /// has no trace in the lock. Rewriting it to a URL source breaks every + /// `poetry install` on Poetry 1.4 – 2.2, so a lock stamped by one of + /// those is refused before any write; other writers are rewritten with a + /// warning (#1282). A sibling lock whose pyproject has no constraint is + /// rewritten as before. + #[test] + fn source_constrained_dependency_is_refused_for_poetry_1_4_to_2_2() { + const PYPROJECTS: &[&str] = &[ + "[tool.poetry.dependencies]\nurllib3 = {version = \"1.26.18\", source = \"pypi\"}\n", + "[tool.poetry.dependencies.URLLib3]\nversion = \"1.26.18\"\nsource = \"PyPI\"\n", + "[tool.poetry.group.dev.dependencies]\nurllib3 = {version = \"*\", source = \"pypi\"}\n", + "[tool.poetry.dev-dependencies]\nurllib3 = {version = \"*\", source = \"pypi\"}\n", + "[tool.poetry.dependencies]\nurllib3 = [\n {version = \"1.26.18\", python = \">=3.8\", source = \"pypi\"},\n {version = \"1.26.17\", python = \"<3.8\"},\n]\n", + "[[tool.poetry.dependencies.urllib3]]\nversion = \"1.26.18\"\npython = \">=3.8\"\nsource = \"pypi\"\n\n[[tool.poetry.dependencies.urllib3]]\nversion = \"1.26.17\"\npython = \"<3.8\"\n", + ]; + for version in VERSIONS.iter().filter(|version| !version.starts_with("0.")) { + let broken = matches!( + *version, + "1.4.2" | "1.5.1" | "1.6.1" | "1.7.1" | "1.8.5" | "2.0.1" | "2.1.4" | "2.2.1" + ); + for pyproject in PYPROJECTS { + let lock = fixture(version); + let files = BTreeMap::from([ + ("poetry.lock".to_string(), lock.clone()), + ("pyproject.toml".to_string(), pyproject.to_string()), + ("sub/poetry.lock".to_string(), lock.clone()), + ( + "sub/pyproject.toml".to_string(), + "[tool.poetry.dependencies]\nurllib3 = \"1.26.18\"\n".to_string(), + ), + ]); + let deps = [dep("urllib3", "1.26.18", Some(SHA), 0)]; + let got = run(rewrite_poetry, &files, &deps); + let what = format!("{version} {pyproject:?}: {:?}", got.warnings); + assert!(got.files.contains_key("sub/poetry.lock"), "{what}"); + let codes: Vec<&str> = got.warnings.iter().map(|w| w.code.as_str()).collect(); + if broken { + assert!(!got.files.contains_key("poetry.lock"), "{what}"); + assert!( + got.edits.iter().all(|edit| edit.path == "sub/poetry.lock"), + "{what}" + ); + assert!( + got.refused_python_lock_uuids.contains(&deps[0].patch_uuid), + "{what}" + ); + assert_eq!(codes, ["redirect_poetry_lock_unsupported"], "{what}"); + assert!(got.warnings[0].detail.contains("source ="), "{what}"); + // Nothing is wired yet, so re-locking is the whole remedy. + assert!(!got.warnings[0].detail.contains("rollback"), "{what}"); + } else { + assert!(got.files.contains_key("poetry.lock"), "{what}"); + assert!(got.refused_python_lock_uuids.is_empty(), "{what}"); + assert!( + codes.contains(&"redirect_poetry_source_constraint"), + "{what}" + ); + } + + // A lock a pre-#1282 run already redirected: still refused + // (never confirmed) on 1.4 – 2.2, still warned elsewhere. + let mut again = files.clone(); + again.insert("poetry.lock".into(), got.files["sub/poetry.lock"].clone()); + let got = run(rewrite_poetry, &again, &deps); + let codes: Vec<&str> = got.warnings.iter().map(|w| w.code.as_str()).collect(); + assert!(!got.files.contains_key("poetry.lock"), "{what}"); + if broken { + assert!( + got.refused_python_lock_uuids.contains(&deps[0].patch_uuid), + "{what}" + ); + assert_eq!(codes, ["redirect_poetry_lock_unsupported"], "{what}"); + // Re-locking would drop the live pin: roll back first. + let detail = &got.warnings[0].detail; + let rollback = detail + .find("socket-patch rollback pkg:pypi/urllib3@1.26.18") + .expect(detail); + assert!(rollback < detail.find("re-lock").unwrap(), "{detail}"); + } else { + assert!(got.refused_python_lock_uuids.is_empty(), "{what}"); + assert!( + codes.contains(&"redirect_poetry_source_constraint"), + "{what}" + ); + } + } + } + } + + /// A package pinned to a private index already carries that index as a + /// `[package.source]` in the lock. The rewriter refuses it as a + /// user-authored source, as before #1282: the source-pin refusal (whose + /// remedy is to drop the `source` key) must not take it over. + #[test] + fn private_index_pin_keeps_the_user_authored_source_refusal() { + for version in ["1.8.5", "2.2.1", "2.4.3"] { + let lock = fixture(version).replacen( + "\n[metadata]", + "\n[package.source]\ntype = \"legacy\"\nurl = \"https://pypi.corp.example/simple\"\nreference = \"corp\"\n\n[metadata]", + 1, + ); + let files = BTreeMap::from([ + ("poetry.lock".to_string(), lock), + ( + "pyproject.toml".to_string(), + "[tool.poetry.dependencies]\nurllib3 = {version = \"1.26.18\", source = \"corp\"}\n" + .to_string(), + ), + ]); + let deps = [dep("urllib3", "1.26.18", Some(SHA), 0)]; + let got = run(rewrite_poetry, &files, &deps); + assert!(got.files.is_empty(), "{version}: {:?}", got.warnings); + assert_eq!(got.warnings.len(), 1, "{version}: {:?}", got.warnings); + let detail = &got.warnings[0].detail; + assert_eq!(got.warnings[0].code, "redirect_poetry_lock_unsupported"); + assert!( + detail.contains("refusing to replace an existing Poetry source"), + "{version}: {detail}" + ); + assert!(!detail.contains("source ="), "{version}: {detail}"); + } + } + /// A dozen patched packages in one lock cost one whole-lock render and /// re-parse, not one per package (#760). #[test] diff --git a/crates/socket-patch-core/src/patch/redirect/pypi_takeover.rs b/crates/socket-patch-core/src/patch/redirect/pypi_takeover.rs index ca937cf07..c05f54aa9 100644 --- a/crates/socket-patch-core/src/patch/redirect/pypi_takeover.rs +++ b/crates/socket-patch-core/src/patch/redirect/pypi_takeover.rs @@ -14,8 +14,12 @@ use toml_edit::{DocumentMut, Item}; use super::requirements::preflight_requirements_takeover; use super::RewriteWarning; +use crate::crawlers::python_crawler::canonicalize_pypi_name; use crate::utils::fs::read_regular_to_string; -use crate::utils::poetry_lock::lock_version; +use crate::utils::poetry_lock::{ + lock_version, pyproject_source_constraints, source_constraint_breaks_writer, + source_constraint_refusal, Wired, +}; use crate::vendor::state::VendorEntry; /// Refuse a PyPI takeover the hosted rewriter cannot carry through once @@ -114,8 +118,10 @@ fn recorded_unit_version(unit: &str) -> Option { } /// Hosted mode refuses every Poetry 0.x lock (Poetry 0.12 ignores URL -/// sources), which vendored mode supports (#945). The revert never changes -/// the lock's format, so the lock on disk decides. +/// sources), which vendored mode supports (#945), and a package the +/// pyproject pins to a named source over a lock Poetry 1.4 – 2.2 wrote +/// (#1282). The revert never changes the lock's format, stamp or the +/// pyproject, so the files on disk decide. async fn preflight_poetry_takeover(root: &Path, entry: &VendorEntry) -> Result<(), RewriteWarning> { let lock_file = entry .wiring @@ -130,7 +136,7 @@ async fn preflight_poetry_takeover(root: &Path, entry: &VendorEntry) -> Result<( return Ok(()); }; if lock_version(&doc) != Ok("0") { - return Ok(()); + return preflight_poetry_source_constraint(root, lock_file, &text, entry).await; } Err(RewriteWarning { code: "redirect_poetry_lock_unsupported".into(), @@ -145,6 +151,52 @@ async fn preflight_poetry_takeover(root: &Path, entry: &VendorEntry) -> Result<( }) } +/// The hosted rewriter's source-pin refusal (#1282), predicted from the +/// pyproject beside the lock and the lock's writer stamp. +async fn preflight_poetry_source_constraint( + root: &Path, + lock_file: &str, + lock_text: &str, + entry: &VendorEntry, +) -> Result<(), RewriteWarning> { + if !source_constraint_breaks_writer(lock_text) { + return Ok(()); + } + let pyproject_file = format!( + "{}pyproject.toml", + lock_file.strip_suffix("poetry.lock").unwrap_or("") + ); + let Ok(pyproject) = read_regular_to_string(&root.join(&pyproject_file)).await else { + return Ok(()); + }; + let (name, version) = entry + .base_purl + .strip_prefix("pkg:pypi/") + .and_then(|rest| rest.split_once('@')) + .unwrap_or_default(); + let Some(source) = + pyproject_source_constraints(&pyproject).remove(&canonicalize_pypi_name(name)) + else { + return Ok(()); + }; + Err(RewriteWarning { + code: "redirect_poetry_lock_unsupported".into(), + detail: format!( + "{lock_file}: {} is kept vendored (not switched to hosted): {}", + entry.base_purl, + source_constraint_refusal( + &pyproject_file, + lock_file, + name, + version, + &source, + "URL", + Wired::Vendored + ), + ), + }) +} + #[cfg(test)] mod tests { use super::*; @@ -304,6 +356,43 @@ mod tests { .is_ok()); } + /// #1282: hosted mode refuses a source-pinned package over a lock + /// Poetry 1.4 – 2.2 wrote, so the takeover must keep it vendored rather + /// than revert it into neither mode. Other writers are admitted. + #[tokio::test] + async fn poetry_source_pinned_package_follows_the_writer() { + const PYPROJECT: &str = + "[tool.poetry.dependencies]\nsix = {version = \"1.16.0\", source = \"pypi\"}\n"; + for (stamp, refused) in [("1.8.5", true), ("2.2.1", true), ("2.3.4", false)] { + let tmp = tempfile::tempdir().unwrap(); + let lock = format!( + "# This file is automatically @generated by Poetry {stamp} and should not be \ + changed by hand.\n\n{POETRY_2}" + ); + std::fs::write(tmp.path().join("poetry.lock"), lock).unwrap(); + std::fs::write(tmp.path().join("pyproject.toml"), PYPROJECT).unwrap(); + let got = preflight_pypi_takeover(tmp.path(), &poetry_entry()).await; + if refused { + let err = got.unwrap_err(); + assert_eq!(err.code, "redirect_poetry_lock_unsupported", "{stamp}"); + assert!(err.detail.contains("kept vendored"), "{}", err.detail); + assert!(err.detail.contains("source = \"pypi\""), "{}", err.detail); + assert_remedy_reverts_first(&err.detail); + } else { + assert!(got.is_ok(), "{stamp}: {got:?}"); + } + // Without the pin every writer is admitted. + std::fs::write( + tmp.path().join("pyproject.toml"), + "[tool.poetry.dependencies]\nsix = \"1.16.0\"\n", + ) + .unwrap(); + assert!(preflight_pypi_takeover(tmp.path(), &poetry_entry()) + .await + .is_ok()); + } + } + #[tokio::test] async fn other_flavors_are_admitted() { let tmp = tempfile::tempdir().unwrap(); diff --git a/crates/socket-patch-core/src/utils/poetry_lock.rs b/crates/socket-patch-core/src/utils/poetry_lock.rs index 3cc377be3..c7280c770 100644 --- a/crates/socket-patch-core/src/utils/poetry_lock.rs +++ b/crates/socket-patch-core/src/utils/poetry_lock.rs @@ -12,6 +12,8 @@ //! panic: every table access here is guarded and degrades to an `Err`, which //! callers surface as a refusal warning. +use std::collections::BTreeMap; + use toml_edit::{value, Array, DocumentMut, InlineTable, Item, Table, TableLike, Value}; use crate::crawlers::python_crawler::canonicalize_pypi_name; @@ -186,6 +188,172 @@ pub fn generated_by_version(lock_text: &str) -> Option<(u64, u64)> { Some((major, minor)) } +/// Every `(key, spec)` entry of a pyproject's Poetry dependency tables: +/// `[tool.poetry.dependencies]`, the legacy `dev-dependencies` and every +/// `[tool.poetry.group.*.dependencies]`. +pub(crate) fn poetry_dependency_entries(doc: &DocumentMut) -> Vec<(&str, &Item)> { + let Some(poetry) = doc + .get("tool") + .and_then(Item::as_table_like) + .and_then(|tool| tool.get("poetry")) + .and_then(Item::as_table_like) + else { + return Vec::new(); + }; + let groups = poetry + .get("group") + .and_then(Item::as_table_like) + .into_iter() + .flat_map(|groups| groups.iter()) + .filter_map(|(_, group)| group.as_table_like()?.get("dependencies")); + ["dependencies", "dev-dependencies"] + .into_iter() + .filter_map(|table| poetry.get(table)) + .chain(groups) + .filter_map(Item::as_table_like) + .flat_map(|deps| deps.iter()) + .collect() +} + +/// The `source = ""` each Poetry dependency of a pyproject is pinned +/// to, keyed by PEP 503-canonical name. Covers the table and inline-table +/// forms and both spellings of multiple constraints (an inline array of +/// tables, or `[[tool.poetry.dependencies.]]`); the first constraint +/// naming a source wins. +/// +/// Poetry writes nothing into `poetry.lock` for a dependency pinned to PyPI +/// (`source = "pypi"`), so the lock alone can't show the constraint. An +/// unreadable pyproject reads as no constraints. +pub fn pyproject_source_constraints(pyproject: &str) -> BTreeMap { + let mut sources = BTreeMap::new(); + let Ok(doc) = pyproject.parse::() else { + return sources; + }; + let source_of = |table: &dyn TableLike| { + table + .get("source") + .and_then(Item::as_str) + .map(str::to_string) + }; + for (key, spec) in poetry_dependency_entries(&doc) { + let source = match spec { + Item::Value(Value::Array(constraints)) => constraints + .iter() + .filter_map(Value::as_inline_table) + .find_map(|table| source_of(table)), + Item::ArrayOfTables(constraints) => { + constraints.iter().find_map(|table| source_of(table)) + } + _ => spec.as_table_like().and_then(source_of), + }; + if let Some(source) = source { + sources.entry(canonicalize_pypi_name(key)).or_insert(source); + } + } + sources +} + +/// What `lock_text` already wires for `name` (PEP 503-canonical) at +/// `version`. `None` when the lock doesn't list it there (a rewrite would +/// not touch the lock), or when its `[package.source]` is one socket-patch +/// didn't write (a private index, a user's own path or URL): the rewriter +/// refuses that as a user-authored source on its own, and the source-pin +/// remedy (drop the `source` key) would be the wrong advice for it. +pub fn lock_wiring(lock_text: &str, name: &str, version: &str) -> Option { + let lock = lock_text.parse::().ok()?; + let canon = canonicalize_pypi_name(name); + let package = table_likes(lock.get("package")) + .into_iter() + .find(|package| { + package + .get("name") + .and_then(Item::as_str) + .is_some_and(|candidate| canonicalize_pypi_name(candidate) == canon) + && package.get("version").and_then(Item::as_str) == Some(version) + })?; + let Some(source) = package.get("source").and_then(Item::as_table_like) else { + return Some(Wired::No); + }; + let url = source.get("url").and_then(Item::as_str).unwrap_or(""); + match source.get("type").and_then(Item::as_str) { + Some("file") if url.starts_with(".socket/vendor/pypi/") => Some(Wired::Vendored), + Some("url") if url.contains("/patch/pypi/") => Some(Wired::Hosted), + _ => None, + } +} + +/// The refusal detail for a source-pinned package over a lock that +/// [`source_constraint_breaks_writer`] flags; `kind` is how the rewrite +/// would source it (`URL` hosted, `file` vendored). Dropping the key or +/// upgrading Poetry both mean re-locking, which discards whatever the lock +/// already wires: `wired` (the lock still carries a vendored or hosted pin +/// for the package) puts `socket-patch vendor --revert` or `rollback` first. +pub fn source_constraint_refusal( + pyproject_path: &str, + lock_path: &str, + name: &str, + version: &str, + source: &str, + kind: &str, + wired: Wired, +) -> String { + let first = match wired { + Wired::No => "", + Wired::Vendored => { + "run `socket-patch vendor --revert` (it reverts EVERY vendored package in the \ + project, not just this one), then " + } + Wired::Hosted => &format!( + "run `socket-patch rollback pkg:pypi/{name}@{version}` (only this package's \ + hosted pin; a bare `rollback` undoes every patch in the project), then " + ), + }; + format!( + "{pyproject_path} pins {name} to the {source:?} source (`source = {source:?}`), and \ + {lock_path} was written by Poetry 1.4 – 2.2, whose installer refuses that constraint \ + on a package the lock sources from a {kind}: every `poetry install` would fail. To \ + patch it, {first}drop the `source` key or upgrade to Poetry >= 2.3, re-lock, and \ + re-run" + ) +} + +/// What the lock already wires for a package refused by +/// [`source_constraint_refusal`], which decides the remedy's first step. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Wired { + /// Nothing yet: the lock still installs the upstream release. + No, + /// A vendored wheel, recorded in the vendor ledger. + Vendored, + /// A hosted URL pin. + Hosted, +} + +/// The advisory for a source-pinned package rewired over a lock that a +/// Poetry outside 1.4 – 2.2 wrote. +pub fn source_constraint_advisory( + pyproject_path: &str, + lock_path: &str, + name: &str, + source: &str, +) -> String { + format!( + "{pyproject_path} pins {name} to the {source:?} source (`source = {source:?}`): \ + Poetry 1.4 – 2.2 refuse to install {lock_path} with that constraint (1.1 – 1.3 and \ + 2.3+ install it); drop the `source` key if this project is installed with one of them" + ) +} + +/// Whether the lock's writer is a Poetry release (1.4 – 2.2) whose installer +/// refuses a pyproject `source = "…"` constraint on a package the lock +/// sources from a URL or file: every `poetry install` fails ("Repository +/// \"pypi\" does not exist."; 1.4 reports a solver failure). 1.1 – 1.3 and +/// 2.3+ install it. Keyed on the `@generated by Poetry X.Y` stamp, so it +/// judges the writer, which is usually but not always the installer. +pub fn source_constraint_breaks_writer(lock_text: &str) -> bool { + matches!(generated_by_version(lock_text), Some(v) if ((1, 4)..(2, 3)).contains(&v)) +} + /// Rewrite the `[[package]]` for `name`@`version` to install the wheel at /// `source_url` (`source_type` `"url"` for hosted, `"file"` for vendored), /// pinning `sha256`. Returns `Ok(None)` when the lock has no such entry (or @@ -651,6 +819,65 @@ fn poetry_lock_fragments_in( mod tests { use super::*; + /// Every place Poetry reads a per-dependency `source` (#1282). + #[test] + fn pyproject_source_constraints_reads_every_declaration_form() { + let pyproject = r#" +[tool.poetry.dependencies] +python = ">=3.8" +plain = "1.0" +Inline_Name = {version = "1.0", source = "pypi"} +arrayed = [{version = "1", python = "<3.8"}, {version = "2", source = "mirror"}] + +[[tool.poetry.dependencies.tabled]] +version = "1" +source = "PyPI" + +[tool.poetry.dependencies.table] +version = "1" +source = "pypi" + +[tool.poetry.dev-dependencies] +legacy = {version = "*", source = "pypi"} + +[tool.poetry.group.test.dependencies] +grouped = {version = "*", source = "pypi"} +"#; + let got = pyproject_source_constraints(pyproject); + let want: BTreeMap = [ + ("inline-name", "pypi"), + ("arrayed", "mirror"), + ("tabled", "PyPI"), + ("table", "pypi"), + ("legacy", "pypi"), + ("grouped", "pypi"), + ] + .into_iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect(); + assert_eq!(got, want); + assert!(pyproject_source_constraints("not = [toml").is_empty()); + assert!(pyproject_source_constraints("[project]\nname = \"x\"\n").is_empty()); + } + + #[test] + fn source_constraint_breaks_only_poetry_1_4_to_2_2_writers() { + let stamp = |v: &str| { + format!("# This file is automatically @generated by Poetry {v} and should not be changed by hand.\n") + }; + for (v, want) in [ + ("1.3.2", false), + ("1.4.0", true), + ("1.8.5", true), + ("2.2.1", true), + ("2.3.0", false), + ("2.4.3", false), + ] { + assert_eq!(source_constraint_breaks_writer(&stamp(v)), want, "{v}"); + } + assert!(!source_constraint_breaks_writer("[[package]]\n")); + } + const WHEEL: &str = "urllib3-1.26.18-py2.py3-none-any.whl"; const URL: &str = "https://patch.socket.dev/patch/pypi/urllib3/1.26.18/7e52b8b6-53f2-4dc8-860a-1ae7ebd8be0e/e828efa5-5c6d-43f3-9909-03f5ac232b98/urllib3-1.26.18-py2.py3-none-any.whl"; diff --git a/crates/socket-patch-core/src/vendor/pypi.rs b/crates/socket-patch-core/src/vendor/pypi.rs index 12b67afcb..bfa757343 100644 --- a/crates/socket-patch-core/src/vendor/pypi.rs +++ b/crates/socket-patch-core/src/vendor/pypi.rs @@ -1063,6 +1063,10 @@ async fn pypi_prelude<'p>( } Ok(PoetryTarget::Fresh) => { warnings.extend(project.warnings.iter().cloned()); + warnings.extend(super::pypi_poetry::source_constraint_warning( + &project, + &canon_name, + )); WiringPlan::Poetry(Box::new(project)) } Err(refusal) => { @@ -2422,7 +2426,10 @@ async fn fresh_pyproject_plan( let project = super::pypi_poetry::load_poetry_project(project_root).await?; match super::pypi_poetry::check_target_guards(&project, canon_name, version, uuid)? { PoetryTarget::Fresh => { - let warnings = project.warnings.clone(); + let mut warnings = project.warnings.clone(); + warnings.extend(super::pypi_poetry::source_constraint_warning( + &project, canon_name, + )); Ok((WiringPlan::Poetry(Box::new(project)), warnings)) } PoetryTarget::InSync => not_fresh("pypi_poetry_source_already_exists"), diff --git a/crates/socket-patch-core/src/vendor/pypi_poetry.rs b/crates/socket-patch-core/src/vendor/pypi_poetry.rs index b678bf631..f75d0531d 100644 --- a/crates/socket-patch-core/src/vendor/pypi_poetry.rs +++ b/crates/socket-patch-core/src/vendor/pypi_poetry.rs @@ -7,6 +7,7 @@ use toml_edit::{DocumentMut, Item}; use crate::crawlers::python_crawler::canonicalize_pypi_name; use crate::utils::fs::{atomic_write_bytes_preserving_mode, read_regular_to_string}; +use crate::utils::poetry_lock; use super::common::{ ensure_unchanged, item_get, lock_units_named, pep621_declared_names, record, refuse_symlinked, @@ -173,21 +174,10 @@ fn classify_dependency(p: &PoetryProject, canon_name: &str) -> &'static str { let Ok(doc) = PYPROJECT_MEMO.parse(text.as_bytes(), || text.parse::()) else { return "transitive"; }; - let mut declared: Vec = Vec::new(); - if let Some(poetry) = doc.get("tool").and_then(|t| item_get(t, "poetry")) { - for table in ["dependencies", "dev-dependencies"] { - if let Some(deps) = item_get(poetry, table).and_then(Item::as_table_like) { - declared.extend(deps.iter().map(|(k, _)| k.to_string())); - } - } - if let Some(groups) = item_get(poetry, "group").and_then(Item::as_table_like) { - for (_, group) in groups.iter() { - if let Some(deps) = item_get(group, "dependencies").and_then(Item::as_table_like) { - declared.extend(deps.iter().map(|(k, _)| k.to_string())); - } - } - } - } + let mut declared: Vec = poetry_lock::poetry_dependency_entries(&doc) + .into_iter() + .map(|(key, _)| key.to_string()) + .collect(); pep621_declared_names(&doc, &mut declared); if declared .iter() @@ -199,6 +189,54 @@ fn classify_dependency(p: &PoetryProject, canon_name: &str) -> &'static str { } } +/// The pyproject's `source = "…"` constraint on `canon_name`, if any. Poetry +/// records nothing in the lock for a package pinned to PyPI, so only the +/// pyproject shows it (#1282). +fn source_constraint(p: &PoetryProject, canon_name: &str) -> Option { + poetry_lock::pyproject_source_constraints(p.pyproject_text.as_deref()?).remove(canon_name) +} + +/// The advisory for a source-constrained package wired over a lock that a +/// Poetry outside 1.4 – 2.2 wrote: the writer installs it, but a 1.4 – 2.2 +/// installer would refuse it. [`check_target_guards`] refuses the 1.4 – 2.2 +/// writers outright. +pub(super) fn source_constraint_warning( + p: &PoetryProject, + canon_name: &str, +) -> Option { + let source = source_constraint(p, canon_name)?; + Some(VendorWarning::new( + "pypi_poetry_source_constraint", + poetry_lock::source_constraint_advisory("pyproject.toml", LOCK_FILE, canon_name, &source), + )) +} + +/// Refuse a source-constrained package over a lock written by Poetry +/// 1.4 – 2.2: its installer fails every install once the lock sources the +/// package from the vendored wheel (#1282). +fn source_constraint_guard( + p: &PoetryProject, + canon_name: &str, + version: &str, + wired: poetry_lock::Wired, +) -> Result<(), (&'static str, String)> { + match source_constraint(p, canon_name) { + Some(source) if poetry_lock::source_constraint_breaks_writer(&p.lock_text) => Err(( + "pypi_poetry_source_constraint", + poetry_lock::source_constraint_refusal( + "pyproject.toml", + LOCK_FILE, + canon_name, + version, + &source, + "file", + wired, + ), + )), + _ => Ok(()), + } +} + /// Target-specific guards (also re-run by [`wire_poetry`] right before /// writing). The orchestrator runs them pre-flight so a refusal happens /// before the wheel artifact is built. Lock names match by PEP 503 canonical @@ -240,6 +278,8 @@ pub(super) fn check_target_guards( return match parse_vendor_path(url) { // Ours, same patch generation: the in-sync hot path. Some(parts) if parts.eco == "pypi" && parts.uuid == record_uuid => { + // Wired before this guard existed: the install already fails. + source_constraint_guard(p, canon_name, version, poetry_lock::Wired::Vendored)?; Ok(PoetryTarget::InSync) } // Ours, but a STALE patch generation: wiring over it would lose @@ -278,6 +318,9 @@ pub(super) fn check_target_guards( ), )); } + // A pyproject `source = "…"` pin breaks every install of the wired lock + // on Poetry 1.4 – 2.2; refuse when one of those wrote the lock. + source_constraint_guard(p, canon_name, version, poetry_lock::Wired::No)?; Ok(PoetryTarget::Fresh) } @@ -847,6 +890,98 @@ content-hash = "4b42a89b7ff7b26511b06acdc458dbd85312e5083db8f212b017482bc68cdd01 } } + /// A pyproject `source = "pypi"` constraint on the vendored package has + /// no trace in the lock. Rewiring it to a file source breaks every + /// `poetry install` on Poetry 1.4 – 2.2, so a lock stamped by one of + /// those is refused before the wheel is built; other writers wire it with + /// a warning (#1282). + #[tokio::test] + async fn source_constrained_dependency_is_refused_for_poetry_1_4_to_2_2() { + const PYPROJECT: &str = "[tool.poetry.dependencies]\n\ + python = \">=3.8\"\n\ + urllib3 = {version = \"1.26.18\", source = \"pypi\"}\n"; + for (version, broken) in [ + ("0.12.17", false), + ("1.1.15", false), + ("1.3.2", false), + ("1.4.2", true), + ("1.6.1", true), + ("1.8.5", true), + ("2.0.1", true), + ("2.2.1", true), + ("2.3.4", false), + ("2.4.3", false), + ] { + let native = std::fs::read_to_string(format!( + "{}/tests/fixtures/poetry/{version}/poetry.lock", + env!("CARGO_MANIFEST_DIR") + )) + .unwrap(); + let tmp = write_project(&native, PYPROJECT).await; + let project = load_poetry_project(tmp.path()).await.unwrap(); + let guard = check_target_guards(&project, "urllib3", "1.26.18", UUID); + if broken { + let Err((code, detail)) = guard else { + panic!("{version}: wired a source-constrained dep"); + }; + assert_eq!(code, "pypi_poetry_source_constraint", "{version}"); + assert!(detail.contains("source = \"pypi\""), "{version}: {detail}"); + } else { + assert!( + matches!(guard, Ok(PoetryTarget::Fresh)), + "{version}: {guard:?}" + ); + let warning = source_constraint_warning(&project, "urllib3"); + assert!( + warning.is_some_and(|w| w.code == "pypi_poetry_source_constraint"), + "{version}" + ); + } + // A lock a pre-#1282 run already wired is refused the same way + // (its installs already fail), not reported in sync. + let wheel = "urllib3-1.26.18-py2.py3-none-any.whl"; + let wired = poetry_lock::rewrite_poetry_lock( + &native, + "urllib3", + "1.26.18", + "file", + &format!(".socket/vendor/pypi/{UUID}/{wheel}"), + wheel, + WHEEL_SHA, + ) + .unwrap() + .unwrap(); + let tmp = write_project(&wired, PYPROJECT).await; + let project = load_poetry_project(tmp.path()).await.unwrap(); + let guard = check_target_guards(&project, "urllib3", "1.26.18", UUID); + if broken { + let Err(("pypi_poetry_source_constraint", detail)) = guard else { + panic!("{version}: {guard:?}"); + }; + // Re-locking would drop the live wiring: revert it first. + let revert = detail.find("vendor --revert").expect(&detail); + assert!(revert < detail.find("re-lock").unwrap(), "{detail}"); + assert!(detail.contains("EVERY vendored package"), "{detail}"); + } else { + assert!( + matches!(guard, Ok(PoetryTarget::InSync)), + "{version}: {guard:?}" + ); + } + // Without the constraint every writer wires silently. + let plain = write_project(&native, PYPROJECT_DIRECT).await; + let project = load_poetry_project(plain.path()).await.unwrap(); + assert!( + matches!( + check_target_guards(&project, "urllib3", "1.26.18", UUID), + Ok(PoetryTarget::Fresh) + ), + "{version}" + ); + assert!(source_constraint_warning(&project, "urllib3").is_none()); + } + } + /// A newer ledger's record this build cannot replay (unknown kind / /// foreign file) is skipped with a warning; it must NOT hold the atomic /// write hostage — the fragments this build does understand are restored. diff --git a/docs/testing/poetry-compatibility.md b/docs/testing/poetry-compatibility.md index 5d7aa463e..dc554d072 100644 --- a/docs/testing/poetry-compatibility.md +++ b/docs/testing/poetry-compatibility.md @@ -36,6 +36,18 @@ rotation), an unsupported `lock-version`, a malformed `[metadata.files]` / `[metadata.hashes]` value, and a wheel whose filename does not match the locked package. +A dependency the pyproject pins to a named source (`six = {version = +"1.16.0", source = "pypi"}`, in the main, legacy dev or any group table) +leaves no trace in the lock. Poetry 1.4 – 2.2 refuse to install a lock that +then sources it from a URL or file (`Repository "pypi" does not exist.`; 1.4 +reports a solver failure); 1.1 – 1.3 and 2.3+ install it (measured, #1282). +So when the lock's `@generated by Poetry X.Y` stamp is 1.4 – 2.2, both modes +refuse that package before any write (`redirect_poetry_lock_unsupported` +hosted, `pypi_poetry_source_constraint` vendored); otherwise they rewire it +and warn (`redirect_poetry_source_constraint` / `pypi_poetry_source_constraint`) +that a 1.4 – 2.2 installer would refuse the lock. The stamp names the writer, +which is usually but not always the installer. + ## Installer boundaries (measured) | Poetry | Hosted | Vendored | Verifies the lock hash on install | Replaces an already-installed same-version package |