Skip to content

Include files under symlinked directories in the sdist - #830

Open
faresrafat3 wants to merge 1 commit into
pypa:mainfrom
faresrafat3:flit-509-follow-symlinks
Open

faresrafat3 wants to merge 1 commit into
pypa:mainfrom
faresrafat3:flit-509-follow-symlinks

Conversation

@faresrafat3

Copy link
Copy Markdown

Fixes #509.

A directory symlink nested inside an included directory was silently dropped
from the sdist, so this pyproject.toml:

[tool.flit.sdist]
include = ["docs/"]

with this tree:

docs/
  linked -> ../real_data      # symlink
real_data/
  data.txt

produced an sdist without docs/linked/data.txt. Reproduced by building the
sdist and listing the tar members — the file is absent.

Root cause

SdistBuilder.apply_includes_excludes collects the contents of each included
directory with os.walk, which does not follow directory symlinks by
default
:

for dirpath, dirs, dfiles in os.walk(osp.join(cfgdir_s, rel_d)):

Note that a top-level include = ["linked_data/"] naming the symlink directly
does work (the pattern matches the symlink itself), which is why the bug only
shows up for a symlink nested inside a walked directory.

Change

Walk with followlinks=True. On its own that would recurse forever on a
symlink cycle, so the real path of every visited directory is recorded and
repeats are pruned:

seen_dirs: set[str] = set()
for dirpath, dirs, dfiles in os.walk(
    osp.join(cfgdir_s, rel_d), followlinks=True
):
    real = os.path.realpath(dirpath)
    if real in seen_dirs:
        dirs[:] = []
        continue
    seen_dirs.add(real)

Verification

Built the sdist before and after:

tree before after
docs/linked -> real_data file missing docs/linked/data.txt present
docs/sub/loop -> docs (cycle) n/a terminates, 4 members, no duplicates

The cycle case was measured explicitly because followlinks=True is unsafe
without the guard — with the naive one-line change it produced 45 members with
deeply repeated loop/sub/loop/sub/... paths before the seen-set was added.

Two regression tests added to tests/test_sdist.py, both driving
apply_includes_excludes directly with a fixed file list so they do not depend
on git/hg being present: one asserts the symlinked directory's file is
included, the other asserts a symlink cycle terminates with no duplicate
entries.

Full suite: 353 passed, 1 failed — the failure (tests/test_upload.py::test_upload)
is pre-existing and reproduces identically on the unmodified tree (it needs the
flit CLI installed, which this environment lacks).

A directory symlink nested inside an included directory was silently
dropped from the sdist, because os.walk does not follow directory
symlinks by default:

    docs/            <- in sdist include
      linked -> ../real_data   <- skipped entirely

Walking with followlinks=True picks those files up. Because that alone
would recurse forever on a symlink cycle, the real path of each visited
directory is tracked and repeats are pruned.

Fixes pypa#509.
Copilot AI balanced review requested due to automatic review settings October 1, 2026 15:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@faresrafat3

Copy link
Copy Markdown
Author

Hi — following up on this PR. It includes files under symlinked directories in the sdist. Let me know if you'd like any changes or if there's anything I can help with.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

flit doesn't follow symlinks when pulling in include = in pyproject.toml

2 participants