Fix crawl of --system-site-packages venvs (#409)

[Mikola Lysenko (mikolalysenko)]

LLM Description written by Claude Code:claude-opus-5-5

Fixes #409

Summary

A venv made with --system-site-packages imports packages from its base interpreter (Debian dist-packages, the base site-packages, the user site). pip treats a release found there as already installed and leaves it in place. socket-patch only looked inside the venv, so:

• vex thought nothing was installed and attested not_affected from the lock pin, even though the code that runs is not patched.
• The hosted and vendored stale-install guards stayed silent.
• Agent mode said "not installed; run your package manager's install first", which can't help because the install has nothing to do.

This covers plain python -m venv --system-site-packages / virtualenv, Poetry's virtualenvs.options.system-site-packages, and Pipenv's --site-packages / PIPENV_SITE_PACKAGES. All of them write include-system-site-packages = true into pyvenv.cfg.

Root cause

Local Python discovery (find_local_venv_site_packages, non_hatch_local_venv_site_packages, and through them PythonCrawler::get_site_packages_paths) returned only the venv's own site-packages. It never read pyvenv.cfg, so the base interpreter's site dirs behind a system-site venv were never crawled.

Fix

• Discovery (add_system_site_packages): for each discovered venv whose pyvenv.cfg enables system site-packages, the base interpreter's site dirs are appended after the venv's own, in sys.path order:

  • the user site, unless PYTHONNOUSERSITE is set;
  • Debian's /usr/local/lib/pythonX.Y/dist-packages when the prefix is /usr;
  • <prefix>/lib{64,}/pythonX.Y/site-packages;
  • <prefix>/lib/python3/dist-packages.

  The prefix comes from home, following a symlinked interpreter to its stdlib landmark (Homebrew, Debian /usr/local/bin/python3 -> /usr/bin/python3.11). Only the venv's own X.Y (including a t free-threaded suffix) is probed. Everything is derived from pyvenv.cfg and the filesystem: the project's interpreter is never executed, because it is untrusted project content. pyvenv.cfg is parsed like CPython's site.venv(): keys are case-insensitive, the last one wins, and a missing key counts as true.

• Shadowing (PythonCrawler::set_search_order): Python imports the first copy of a name on sys.path, and pip treats that copy as the installed one. So a base copy is hidden when the venv (or an earlier base dir) installs the same name. That case is common after pip install --force-reinstall in the venv, which leaves the base copy in place. Without hiding it, the stale guard would keep warning, vex would refuse forever, and agent apply would try to write into /usr/lib/python3/dist-packages. A base dir that several venvs share hides only what every one of them hides. get_site_packages_paths declares its own order. The two call sites that use the free discovery functions, the hosted stale probe and the vendored Pipenv stale probe, declare it explicitly. Global mode clears it, so every copy stays visible there.

Tests (red → green)

Issue / behavior	Test	Without fix (main)	With fix
#409: hosted scan + --vex over an unpatched base copy behind a system-site venv	in_process_redirect_poetry::stale_base_copy_behind_system_site_venv_warns_and_cannot_attest	FAILED: exit 0, attests (panicked at the exit-code assert)	ok: redirect_pypi_stale_install, exit 1, no VEX file
#409: base dirs discovered	python_crawler::tests::system_site_venv_adds_base_site_dirs_after_its_own	FAILED	ok
#409: base copy crawled/found (agent mode, vex copies)	python_crawler::tests::base_copy_behind_a_system_site_venv_is_crawled	FAILED	ok
Venv copy shadows base copy (no false stale warning, VEX attests)	in_process_redirect_poetry::venv_copy_shadows_stale_base_copy_behind_system_site_venv, python_crawler::tests::venv_copy_hides_the_base_copy_of_the_same_package	n/a (guards the fix)	ok
Isolated venv unchanged; user site / PYTHONNOUSERSITE; base-dir order; shared-base intersection; pyvenv.cfg parsing; symlinked prefix; Debian dirs	isolated_venv_never_reaches_its_base_interpreter, system_site_venv_reaches_the_user_site_unless_disabled, earlier_base_site_dir_hides_a_later_one, shared_base_copy_stays_visible_to_a_venv_without_its_own, pyvenv_cfg_is_read_like_site_py, base_prefix_*, base_site_dirs_cover_debian_dist_packages_for_the_venv_version_only	n/a	ok

I also checked the derived dirs by hand against a real Debian python3 -m venv --system-site-packages (Python 3.11): sys.path = venv site-packages, /usr/local/lib/python3.11/dist-packages, /usr/lib/python3/dist-packages, which matches the discovery order.

Local checks: cargo fmt --all -- --check ✅, cargo clippy --workspace --all-features -- -D warnings ✅, cargo test --workspace --all-features ✅ except tests that fail only because this sandbox runs as root, so permission-denied setups (chmod 0555, unremovable files) don't take effect. These are the *write_failure* / *unremovable* / *cleanup_failure* tests in covgap_commands_vendor, in_process_redirect, repair and the core lib (copy_tree::relax_loop_must_not_traverse_symlinked_root, vlt_heal::an_unremovable_hidden_lock_keeps_every_store_entry, pypi_poetry/pypi_requirements wire-failure tests). There is also e2e_bun_lockb::binary_shared_bundled_record_hosted_pin_is_managed, which fails the same way on origin/main here. None of them involve Python discovery, and CI runs them green on this head. The Python e2e suite (e2e_pypi) needs the live API, so CI covers it.

CI: green on f1ba00d (36 pass / 50 path-skipped). Bugbot: no issues on f1ba00d.

Decisions and follow-ups (for reviewers)

• Agent mode now patches the base copy that a system-site venv actually imports. On a distro interpreter that write is root-owned and fails loudly (EACCES), where before it was silently skipped. On a writable interpreter (pyenv, Homebrew, user site), it patches a copy other projects on that interpreter also import. This follows the existing rule that a Python project with no venv falls back to the global interpreters. If maintainers prefer agent mode to refuse instead of writing outside the venv, that's a small follow-up.
• /code-review high findings not fixed here:
  • Shadow state lives on the PythonCrawler instance, not in the discovery return type. That would be a cleaner API, but it would change every Python consumer's signature. get_site_packages_paths resets the state, and each of the two free-function callers sets it.
  • set_search_order lists the venv and base dirs once more, only when a system-site venv exists.
  • The vendored Hatch stale probe (hatch_stale_install_warning) still judges only Hatch's own env dirs. A Hatch env with system-packages = true gets the base copy judged by the hosted guard (generic remedy), but not by the vendored Hatch-specific warning.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MPfWuTkaymyECuH3w5YJ49

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit f1ba00d. Configure here.}

[Mikola Lysenko (mikolalysenko)]

Ready for review at f1ba00de.

• CI: 86/86 green (36 success, 50 skipped) on the current head.
• Bugbot: reviewed f1ba00d, no findings.
• Mergeable against main, no CHANGELOG.md change.
• Worth a look: the shadowing rule in PythonCrawler::set_search_order (a base dir shared by several venvs hides only what every one of them hides), and that discovery reads only pyvenv.cfg/the filesystem and never runs the project's interpreter.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Final review brief

What it does. A venv created with --system-site-packages (plain venv/virtualenv, Poetry's option, pipenv --site-packages) also imports from its base interpreter. Local Python discovery now reads pyvenv.cfg and appends the base interpreter's site dirs (user site, Debian dist-packages, <prefix>/lib/pythonX.Y/site-packages) after the venv's own. The stale-install guards, vex and agent mode therefore see an unpatched base copy. A base copy is hidden when the venv already has the same package, which is how sys.path resolves it. Fixes #409.

Risk: medium. The discovery and shadowing logic is self-contained and well tested. Two behavior changes are worth a deliberate yes: (1) agent-mode apply can now write into a base interpreter's site dir, outside the project venv. It fails with EACCES on a distro Python and succeeds on pyenv/Homebrew/user site. (2) socket_patch_core::crawlers::PythonCrawler goes from a unit struct to a struct with state, so external code that writes PythonCrawler instead of PythonCrawler::new()/::default() stops compiling. The interpreter itself is never run; everything comes from pyvenv.cfg and the filesystem.

Look here

• parse_pyvenv_cfg:`` parsed the way site.venv() does; a missing key counts as `true`
• base_prefix_of_home:``` prefix search through a symlinked interpreter
• shadowed_names_sync and set_search_order:```` what hides a base copy, including the case of a base dir shared by several venvs
• Callers that now declare the search order: hosted stale probe,````` vendored Pipenv stale probe

Verified

• Read the full diff (7 files, +835/−40) and the description; the description matches the code. CHANGELOG.md is untouched.
• Ran locally on f1ba00d: cargo test -p socket-patch-core --lib python_crawler (100 passed) and cargo test -p socket-patch-cli --test in_process_redirect_poetry system_site (2 passed). The PR table shows the In a --system-site-packages venv, pip keeps the base interpreter's unpatched copy after a hosted rewrite, no stale-install warning fires, and vex attests it as patched #409 tests failing on main.
• CI: 86/86 settled on f1ba00d (36 passed, 50 path-skipped), ci-ok and clippy green. Bugbot found no issues on f1ba00d. Mergeable, no unresolved threads.

Changes I made: none.

Open questions

• Should agent mode refuse instead of patching a base-interpreter copy that other projects share? The PR offers this as a small follow-up.

Auto-merge (squash) is armed, so approving sends this straight to the merge queue.

---

Generated by Claude Code