zippapp/ macOS: .DS_Store files should be excluded from the generated archives

[TurtleWilly]

Bug report

Bug description:

When generating a zipapp under macOS then random ".DS_Store" files get included into the generated archive.

e.g.

$ python3.12 -m zipapp --output cfv.pyz --compress --python /usr/local/bin/python3 zipapptempdir
$ unzip -l cfv.pyz 
Archive:  cfv.pyz 
  Length      Date    Time    Name
---------  ---------- -----   ----
     6148  04-13-2024 00:17   .DS_Store
      186  04-13-2024 00:20   __main__.py
        0  04-13-2024 00:17   cfv/
     6148  04-13-2024 00:17   cfv/.DS_Store
        …
---------                     -------
   137783                     20 files

Those files quickly get regenerated by Finder, so it's hard to entirely avoid them getting added to the archive by trying to pre-delete them before executing zipapp. .DS_Store files can add significantly to the size of the archive, possible have privacy implications (e.g. leaked spotlight comments), and serve zero purpose for the python app.

CPython versions tested on:

3.12

Operating systems tested on:

macOS

Linked PRs

• gh-117829: add --include and --exclude flag to zipapp #120537

[Buggem]

I do not agree. I think by default they should be excluded but there should be an option to include them.

[gaogaotiantian]

Quietly ignoring a specific file on MacOS does not seem like the desirable behavior to me. Currently the bahavior of zipapp is "pack everything", which is easy to understand. Changing it to "pack everything except for files with name .DS_Store" is kind of weird. There are other commonly generated files, __pycache__ or .coverage for example. Who should be the person to decide what's on the list and what's not?

To achieve your goal, the Python API zipapp.create_archive provides a way to do it. You can use the argument filter to filter the paths. It should be easy to create your own script to generate the zipapp file with customized filtering. I don't believe changing the default behavior is the correct direction.

I can, somehow, imagine a new option to ignore all hidden files, which is probably more generic to users.

And this is definitely not a bug, it's a feature request, so I'll change the lables.

[Buggem]

Quietly ignoring a specific file on MacOS does not seem like the desirable behavior to me. Currently the bahavior of zipapp is "pack everything", which is easy to understand. Changing it to "pack everything except for files with name .DS_Store" is kind of weird. There are other commonly generated files, __pycache__ or .coverage for example. Who should be the person to decide what's on the list and what's not?

To achieve your goal, the Python API zipapp.create_archive provides a way to do it. You can use the argument filter to filter the paths. It should be easy to create your own script to generate the zipapp file with customized filtering. I don't believe changing the default behavior is the correct direction.

I can, somehow, imagine a new option to ignore all hidden files, which is probably more generic to users.

And this is definitely not a bug, it's a feature request, so I'll change the lables.

agree 100%

[ronaldoussoren]

Quietly ignoring a specific file on MacOS does not seem like the desirable behavior to me. Currently the bahavior of zipapp is "pack everything", which is easy to understand. Changing it to "pack everything except for files with name .DS_Store" is kind of weird. There are other commonly generated files, __pycache__ or .coverage for example. Who should be the person to decide what's on the list and what's not?

Adding a filter option to the zipapp command line interface could be useful, but as you write the hard bit is which name should be included in the default filter.

Two options (which could both be added)

• python -m zipapp --filter GLOB ...: Multi-use argument for specifying which names should be excluded (user needs to determine useful filters themselves)
• python -m zipapp --filter-junk ..: Filter some commonly seen files that should generally not be included, such as .DS_Store, __pycache__ and backup files (*~, ._*.swp, ...).

The second option is a nicer user interface for users, but is harder to get right (both naming and which filters to include).

Currently the only API implements something similar to the second option is dircmp, other APIs try to avoid the temptation to guess ;-).

[TurtleWilly]

The problem with the .DS_Store files (unlike other temporary junk mentioned here) is that they can come back randomly and quickly during the time the zippapp is created. E.g. I do run a cleanup before already after I noticed the issue with .DS_Store files ballooning the sizes of the generated archives:

find zipapptempdir -type f -name '*.py[co]' -delete -o -type d -name __pycache__ -delete -o -type f -name '.DS_Store' -delete && \
  python3.12 -m zipapp --output s3cmd.pyz --compress --python /usr/local/bin/python3.12 zipapptempdir

And in one case a .DS_Store file still managed to sneak into the archive. It's very hard to make this work reliable from a single user account. Maybe killing off Finder before, and then relaunching it afterwards or some other awkward kludge?

I'm reporting about the the zipapp app/module itself (as used in python3 -m zipapp from a user perspective) and I'm perfectly fine with some --filter REGEX or --reject GLOB option, if that's more suitable than just default filtering out the .DS_Store files. 👍

I would consider those files getting added as a privacy/security/ information leak issue (hence a bug report), e.g. when I plan to distribute generated zipapps to other users: they really don't need my .DS_Store files.

[gaogaotiantian]

If we plan to add a command line option to filter, I think we should avoid the name filter, unless we can make it comparable to the filter argument for create_archive. The filter argument is a function that returns a boolean indicating whether the path should be accepted. The only way I can think of to make the option filter work is for it to pass in a regular expression that provides a pattern for all accepted paths (not declined, or it will have opposite meaning to the argument).

A better way might be --include and --exclude which is also commonly used to provide both allow-list and deny-list.

I'm a bit surprised that Finder will generate the file in the background (when it's not opened in the OS) though.