Skip to content

Commit 08c30b7

Browse files
committed
Keep Pipenv's index on hosted entries so rollback restores it exactly
The hosted Pipfile.lock rewrite dropped each entry's `index`, and the ledger-free upstream restore then guessed it from sibling registry entries (none -> the PyPI source name; all siblings index-less -> none). Pipenv's own choice cannot be re-derived from the lock or the Pipfile: for the same Pipfile it depends on the release and the locking environment (measured with real `pipenv lock`): shape 2018.11.26 2020-2022 2023.12.1-2026.8.0 direct pypi pypi pypi extras table pypi pypi (none) marker-excluded pypi (none) (none) transitive (none) (none) (none) So the backtest's rollbackRestoresLockBytes failed on 2022.12.19 extras (transitive pysocks sibling has no index -> index dropped) and on 2022.12.19 / 2026.8.0 marker-excluded (no siblings -> "pypi" added), and rollbackAfterRelockRetires failed on 2026.8.0 marker-excluded: the relock hybrid (our `file` kept, `version` and registry `hashes` restored, no `index`) was restored with an `index` Pipenv never wrote. The hosted rewrite now keeps `index` exactly as Pipenv wrote it (present or absent) and only drops `version`; the restore carries the entry's `index` back unchanged instead of choosing one, refusing when it (or the Pipfile's explicit index) does not name a PyPI source in `_meta.sources`. A `file` entry carrying `index` installs the referenced wheel itself (direct_url.json present) on Pipenv 2018.11.26, 2020.11.15, 2021.11.23, 2022.12.19, 2023.12.1, 2024.4.1, 2025.1.3 and 2026.8.0 (`install --deploy`, `sync`, `verify`), and a marker-excluded one stays uninstalled; Pipenv 7-11 ignore `index` on a `path` entry (convert_deps_to_pip skips it for file/path deps). Tests: real Pipenv 2018.11.26 / 2022.12.19 / 2026.8.0 locks for the extras and marker-excluded shapes (tests/fixtures/pipenv-shapes) round trip byte for byte in LF and CRLF, and the 2026.8.0 marker-excluded relock hybrid restores the pristine bytes. The backtest's allCategoriesRewritten now expects hosted entries to keep the pristine `index` (vendored still drops it). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018tncU7u7FK48tubFNA12UQ
1 parent e36ba5b commit 08c30b7

21 files changed

Lines changed: 537 additions & 76 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,21 @@ into the new version's section — see docs/releasing.md.
122122
files from version control. v4's `redirect_state_unreadable`,
123123
`redirect_pnpm_trust_scaffold_modified` and
124124
`redirect_npmrc_allow_remote_modified` are no longer emitted.
125+
- **The Pipenv hosted redirect keeps the entry's `index`.** A hosted
126+
`Pipfile.lock` entry is now `{"file" | "path", "hashes"}` plus every key
127+
but `version` exactly as Pipenv wrote it — `index` included, present or
128+
absent — so `rollback` / `remove` carry it back instead of guessing it
129+
from sibling entries. Pipenv records `index` by release, Pipfile spelling
130+
and locking environment (2018.11.26 writes it for a marker-excluded
131+
package, 2022.12.19 does not; 2022.12.19 writes it for an `extras` table,
132+
2023.12.1 and later do not; no release writes it for a transitive
133+
package), so no rule over the lock could re-derive it, and the guess left
134+
those rollbacks off by one key. Measured on Pipenv 2018.11.26, 2020.11.15,
135+
2021.11.23, 2022.12.19, 2023.12.1, 2024.4.1, 2025.1.3 and 2026.8.0: a
136+
`file` entry carrying `index` still installs the referenced wheel itself
137+
(`install --deploy`, `sync`, `verify`); Pipenv 7–11 ignore `index` on a
138+
`path` entry. The restore refuses when the entry's `index` (or the
139+
Pipfile's explicit one) does not name a PyPI source in `_meta.sources`.
125140
- **`list`, `vex`, `scan` and `repair` derive hosted state from the
126141
lockfiles.** `list` shows one entry per hosted pin: JSON
127142
`details.mode: "hosted"` plus `details.lockfiles: [<files wiring it>]`

‎crates/socket-patch-cli/CLI_CONTRACT.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -152,7 +152,7 @@ The rewriter reads a fixed set of candidate files from the project root: the npm
152152

153153
**Gem stale-install guard (additive warning — the canonical narrative; other mentions point here)**: the gem hosted rewrite is pure Gemfile/lock text, so a gem ALREADY materialized under the project's bundle paths keeps its upstream bytes — the next `bundle install` prints `Using <gem>` and never refetches, on **every** bundler major (live-verified 2026-08-19 on 1.17.3 / 2.7.2 / 4.0.18: bundler 4's CHECKSUMS verify at download time only, and nothing is downloaded; `bundle install --force`/`--redownload` re-install from the stale cached `.gem` instead of re-fetching — bundler 1 silently, bundler 4 with an exit-37 checksum refusal that still leaves the upstream bytes installed; the **verified** remedy is removing the installed dir + cache `.gem` + `specifications` entry, then `bundle install`). After the rewrite, a hosted run therefore probes the installed-gem discovery paths (the same ruby-crawler discovery `apply` uses, honoring `--global`/`--global-prefix` like scan's own discovery) for each confirmed gem redirect and judges the materialization against the patch record's `afterHash` file map. Judgment rules: records are found **by uuid** among this run's fetched records (v5.0: hosted mode persists no records, so a purl whose `/patches/view` fetch failed this run is not judged; the warning re-fires on every re-scan whose fetch succeeds, until the stale materialization is gone); a materialization with every file at `afterHash` is already patched and never warns (an agent→hosted migration stays quiet by construction), and when several confirmed variant purls resolve to one installed dir, ANY of them judging it patched keeps it quiet; staleness needs **positive evidence** — at least one record file whose bytes were actually read and hash to neither state's expectation — so missing or unreadable files never produce a warning. Warnings emit `redirect_gem_stale_install` (JSON `redirect.warnings[]` + a code-tagged stderr line) in three flavors: a PROJECT-LOCAL dir gets the verified delete-list remedy (installed dir, cache `.gem`, `specifications` entry — plus the project's committed `vendor/cache/<leaf>.gem` when present and not proven to be the patched artifact, since bundler installs from `vendor/cache` in preference to fetching); a SHARED gem-env home gets a caveat that the home is shared machine-wide and prefers migrating the project to a local bundle path over deleting shared files; and a committed `vendor/cache` archive whose sha256 differs from the patched artifact's warns standalone even with no installed dir at all (a fresh checkout with a committed stale cache re-materializes the upstream bytes forever). A stale-flagged purl is additionally **excluded from the same run's `--vex` `assume_applied` set** — the envelope must never attest a CVE its own warning says is live; the purl falls back to normal installed-tree verification (a patched install still attests, a stale one is omitted). The probe is read-only (nothing is deleted) and skipped on `--dry-run` — deliberately explicit, since nothing was rewritten. Exit code and `status` are unchanged (warning-only, the hosted-refusal posture); a same-run `--vex` may still fail on "nothing to attest" per the embedded-VEX contract.
154154

155-
**Pipenv hosted redirect (`Pipfile.lock`, pipfile-spec 6)**: every category other than `_meta` (`default`, `develop`, and Pipenv 2022+ named categories) that pins the package at the patched version is rewritten to the hosted reference — `{"file" | "path": "<artifact url>#sha256=<hex>", "hashes": ["sha256:<hex>"]}` with `markers`/`extras` preserved and `version`/`index` dropped; `_meta` (the Pipfile content hash) and the Pipfile itself are never touched, so `pipenv install --deploy`/`sync`/`verify` keep passing. The reference KEY depends on the installing Pipenv: releases 7–11 only install `path` references, 2018 and later `file` ones (0–6 write pipfile-spec < 6 and are refused). The release is probed once per command with `pipenv --version`, resolved on ABSOLUTE `PATH` entries only (a relative entry would run a `pipenv` planted in the scanned repository; `.bat`/`.cmd` shims are found through `PATHEXT` on Windows), only when a pypi patch actually targets an entry of the lock, and `SOCKET_PIPENV_MAJOR=<major>` pins the answer without spawning anything. An unknown installer selects `file` and warns `redirect_pipenv_installer_unknown` only when the lock was rewritten. **Refusal scope**: a pin/source CONFLICT (another version pinned, a foreign `file`/`path` source, a VCS/editable dependency) refuses the whole dependency atomically across categories as `redirect_pipenv_refused` AND vetoes the sibling Python rewriters (requirements.txt / uv.lock / pyproject) for that patch — the project's Pipenv install could not pick the patch up, so a half-redirected checkout is refused; anything else (no entry for the package, an old pipfile-spec, an unparseable lock, a digest-less patch) is `redirect_pipenv_skipped` and leaves the siblings alone (a stale Pipfile.lock in a uv/Poetry/requirements project must not block them). The veto applies to a LIVE lock only: a `Pipfile.lock` with no `Pipfile` beside it is abandoned, so its conflict refuses that file but never the siblings. Hash enforcement at install time is split by era — the `#sha256=` URL fragment is what Pipenv 2023+ verifies, the `hashes` list what 2018–2022 verify, Pipenv 11 either — so both are load-bearing. **Pipenv stale-install guard**: Pipenv never reinstalls a release that is already present (`pipenv install`, `install --deploy` and `sync` all exit 0 and keep the installed bytes — measured on 11.10.4, 2018.11.26 and 2026.8.0, hosted and vendored), so after the rewrite the run probes the Python crawler's site-packages (VIRTUAL_ENV, `./.venv`, `./venv`, Pipenv's out-of-tree `WORKON_HOME` venv; `--global`/`--global-prefix` honoured) for each confirmed Pipfile.lock redirect with the same rules as the gem guard (records by uuid from this run's fetch, PATCHED = `verify_patch_record` Ok, STALE needs positive evidence, read-only, skipped on `--dry-run`, stale purls excluded from the same-run `--vex` `assume_applied` set) and the Python stale-install guard (`redirect_pypi_stale_install`, see above) names the site-packages dir and the Pipenv-specific verified remedy: `pipenv run pip uninstall -y <pkg> && pipenv sync` (or `pipenv --rm && pipenv sync`) — NOT `pipenv uninstall`, which rewrites the Pipfile and re-locks the patch away. The vendored backend emits the twin `pypi_pipenv_stale_install` (`skipped` warning event). **Rollback** (v5.0, upstream restore): each hosted entry gets its registry shape back — `"version": "==<v>"`, the `index` its sibling registry entries (or the Pipfile) name, and every release file's sha256 from PyPI's JSON API (`SOCKET_PYPI_JSON_API`), sorted by filename as Pipenv records them; an entry that pins another version beside the hosted reference is refused with the `git checkout` remedy (see "Hosted unwind coverage"). A Pipfile names no project, so a same-run `--vex` on a Pipenv project needs `--vex-product` (or a git remote) to detect a product purl. **Discovery**: `Pipfile.lock` is part of the lockfile inventory (every category's `==` pins, with the lock's digest set as `Sha256AnyOf` integrity so a lock-only checkout can be vendored by fetching the pure wheel through PyPI's JSON API — only when `_meta.sources` name the public index; a private-index lock stays discovery-only and never reaches pypi.org), and Socket's own hosted / vendored references stay discoverable as the package they replace, so a re-scan of an already-redirected or already-vendored lock-only checkout re-confirms it (`--vex` attests, vendored reports `already_vendored`) instead of finding nothing.
155+
**Pipenv hosted redirect (`Pipfile.lock`, pipfile-spec 6)**: every category other than `_meta` (`default`, `develop`, and Pipenv 2022+ named categories) that pins the package at the patched version is rewritten to the hosted reference — `{"file" | "path": "<artifact url>#sha256=<hex>", "hashes": ["sha256:<hex>"]}` with `markers`/`extras`/`index` kept exactly as Pipenv wrote them (present or absent: whether Pipenv records `index` depends on its release, the Pipfile spelling and the locking environment, so only the entry itself knows) and `version` dropped; `_meta` (the Pipfile content hash) and the Pipfile itself are never touched, so `pipenv install --deploy`/`sync`/`verify` keep passing. The reference KEY depends on the installing Pipenv: releases 7–11 only install `path` references, 2018 and later `file` ones (0–6 write pipfile-spec < 6 and are refused). The release is probed once per command with `pipenv --version`, resolved on ABSOLUTE `PATH` entries only (a relative entry would run a `pipenv` planted in the scanned repository; `.bat`/`.cmd` shims are found through `PATHEXT` on Windows), only when a pypi patch actually targets an entry of the lock, and `SOCKET_PIPENV_MAJOR=<major>` pins the answer without spawning anything. An unknown installer selects `file` and warns `redirect_pipenv_installer_unknown` only when the lock was rewritten. **Refusal scope**: a pin/source CONFLICT (another version pinned, a foreign `file`/`path` source, a VCS/editable dependency) refuses the whole dependency atomically across categories as `redirect_pipenv_refused` AND vetoes the sibling Python rewriters (requirements.txt / uv.lock / pyproject) for that patch — the project's Pipenv install could not pick the patch up, so a half-redirected checkout is refused; anything else (no entry for the package, an old pipfile-spec, an unparseable lock, a digest-less patch) is `redirect_pipenv_skipped` and leaves the siblings alone (a stale Pipfile.lock in a uv/Poetry/requirements project must not block them). The veto applies to a LIVE lock only: a `Pipfile.lock` with no `Pipfile` beside it is abandoned, so its conflict refuses that file but never the siblings. Hash enforcement at install time is split by era — the `#sha256=` URL fragment is what Pipenv 2023+ verifies, the `hashes` list what 2018–2022 verify, Pipenv 11 either — so both are load-bearing. **Pipenv stale-install guard**: Pipenv never reinstalls a release that is already present (`pipenv install`, `install --deploy` and `sync` all exit 0 and keep the installed bytes — measured on 11.10.4, 2018.11.26 and 2026.8.0, hosted and vendored), so after the rewrite the run probes the Python crawler's site-packages (VIRTUAL_ENV, `./.venv`, `./venv`, Pipenv's out-of-tree `WORKON_HOME` venv; `--global`/`--global-prefix` honoured) for each confirmed Pipfile.lock redirect with the same rules as the gem guard (records by uuid from this run's fetch, PATCHED = `verify_patch_record` Ok, STALE needs positive evidence, read-only, skipped on `--dry-run`, stale purls excluded from the same-run `--vex` `assume_applied` set) and the Python stale-install guard (`redirect_pypi_stale_install`, see above) names the site-packages dir and the Pipenv-specific verified remedy: `pipenv run pip uninstall -y <pkg> && pipenv sync` (or `pipenv --rm && pipenv sync`) — NOT `pipenv uninstall`, which rewrites the Pipfile and re-locks the patch away. The vendored backend emits the twin `pypi_pipenv_stale_install` (`skipped` warning event). **Rollback** (v5.0, upstream restore): each hosted entry gets its registry shape back — `"version": "==<v>"`, the entry's own `index` carried back unchanged (refused unless it — and the Pipfile's explicit `index`, if any — names a PyPI source in `_meta.sources`), and every release file's sha256 from PyPI's JSON API (`SOCKET_PYPI_JSON_API`), sorted by filename as Pipenv records them; an entry that pins another version beside the hosted reference is refused with the `git checkout` remedy (see "Hosted unwind coverage"). A Pipfile names no project, so a same-run `--vex` on a Pipenv project needs `--vex-product` (or a git remote) to detect a product purl. **Discovery**: `Pipfile.lock` is part of the lockfile inventory (every category's `==` pins, with the lock's digest set as `Sha256AnyOf` integrity so a lock-only checkout can be vendored by fetching the pure wheel through PyPI's JSON API — only when `_meta.sources` name the public index; a private-index lock stays discovery-only and never reaches pypi.org), and Socket's own hosted / vendored references stay discoverable as the package they replace, so a re-scan of an already-redirected or already-vendored lock-only checkout re-confirms it (`--vex` attests, vendored reports `already_vendored`) instead of finding nothing.
156156

157157
**Mode ledgers (contract surfaces).** Vendored mode persists its state at a stable repo-relative path; external tools (and the depscan backend's GitHub-app PR flows) read and write it, so path + schema are part of the contract. Hosted mode (v5.0) persists nothing but its lockfile / config edits:
158158

‎crates/socket-patch-cli/tests/in_process_redirect_pipenv.rs‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -357,7 +357,12 @@ async fn lock_only_pipenv_project_redirects_attests_rescans_and_rolls_back() {
357357
serde_json::json!([format!("sha256:{}", sha256())]),
358358
"{redirected}"
359359
);
360-
assert!(entry.get("version").is_none() && entry.get("index").is_none(), "{entry}");
360+
assert!(entry.get("version").is_none(), "{entry}");
361+
assert_eq!(
362+
entry.get("index"),
363+
urllib3_entry(LOCK).get("index"),
364+
"Pipenv's own index is kept for rollback"
365+
);
361366
assert_eq!(
362367
entry["markers"],
363368
urllib3_entry(LOCK)["markers"],

‎crates/socket-patch-core/src/patch/redirect/pipenv.rs‎

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -376,7 +376,6 @@ fn plan(
376376
if object.get(source_key).and_then(Value::as_str) == Some(&url)
377377
&& object.get("hashes") == Some(&json!([format!("sha256:{sha}")]))
378378
&& !object.contains_key("version")
379-
&& !object.contains_key("index")
380379
{
381380
continue;
382381
}
@@ -388,9 +387,18 @@ fn plan(
388387
dep.name, dep.version
389388
)));
390389
}
390+
// `index` stays exactly as Pipenv wrote it (present or absent): it
391+
// is the one registry field the upstream restore cannot re-derive.
392+
// Whether Pipenv records it depends on the release, the Pipfile
393+
// spelling and the locking environment (2022.12.19 writes it for an
394+
// `extras` table, 2026.8.0 does not; neither writes it for a
395+
// marker-excluded package or a transitive one), so dropping it would
396+
// make `rollback` guess. On a `file`/`path` entry it only selects
397+
// the source group Pipenv installs the URL through — pip fetches
398+
// the URL itself either way (measured on 2018.11.26 through
399+
// 2026.8.0: install, `--deploy`, `sync`, `verify`).
391400
let mut new = object.clone();
392401
new.remove("version");
393-
new.remove("index");
394402
new.remove("file");
395403
new.remove("path");
396404
new.insert(source_key.into(), Value::String(url.clone()));
@@ -594,9 +602,10 @@ mod tests {
594602

595603
/// `pipenv lock --keep-outdated` (2022) rewrites our entry into a
596604
/// file+version+index hybrid that Pipenv still installs from: it is
597-
/// ours, so it is re-planned to the canonical shape instead of being
598-
/// refused as a foreign source (which also vetoed the sibling rewriters);
599-
/// a hybrid naming ANOTHER version is a real conflict.
605+
/// ours, so it is re-planned to the canonical shape (`version` dropped,
606+
/// Pipenv's `index` kept) instead of being refused as a foreign source
607+
/// (which also vetoed the sibling rewriters); a hybrid naming ANOTHER
608+
/// version is a real conflict.
600609
#[test]
601610
fn owned_hybrid_entries_are_replanned_not_refused() {
602611
let dep = dependency("urllib3", "1.26.18", "patch-one");
@@ -609,7 +618,7 @@ mod tests {
609618
assert!(!edits.is_empty(), "the hybrid is re-planned");
610619
let entry: Value = serde_json::from_str(&fixed).unwrap();
611620
assert!(entry["default"]["urllib3"].get("version").is_none());
612-
assert!(entry["default"]["urllib3"].get("index").is_none());
621+
assert_eq!(entry["default"]["urllib3"]["index"], json!("pypi"));
613622
assert!(entry["default"]["urllib3"]["file"].as_str().unwrap().contains("patch-one"));
614623

615624
value["default"]["urllib3"]["version"] = json!("==2.0.0");

0 commit comments

Comments
 (0)