Use Windows `IFileOperation` to implement copying/moving/deleting in pathlib

[barneygale]

Feature or enhancement

Proposal:

The Windows-only IFileOperation API allows us to copy, move and delete files and directories using the same code as explorer.exe. We could use this to provide pathlib.Path.copy(), move() and _delete() - it should be much faster, and in the case of _delete() it would work around #84324

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

[zooba]

This is true, but it also requires enabling COM (CoInitializeEx), which we historically try to avoid doing on the main thread in case the application wants to enable it differently (e.g. single-threaded vs. free-threaded). The WMI query in _wmi runs on its own thread to avoid touching the main thread's state.

I suspect the overhead of threading would probably take away the benefits here. If we were to go down this route, we'd probably have to figure out how to safely transition to either all CPython or just python.exe (i.e. embedders can choose) running with COM initialised. I dislike this kind of process-wide state being unpredictable (i.e. based on whether anything has done a copy/move/delete).

That said, I believe that Store packaged Python is already doing this, I just haven't tested exactly what the conditions are (specifically whether it's somehow active but the mode can be changed). I'm not aware of anyone hitting issues due to this, but it's possible they just worked around it with a different install. (Under PEP 773, we go back to never having COM initialised by default.)

Apart from that massive caveat, we'd also be formally introducing C++ code (which I've already kind of smuggled in anyway, but it's entirely restricted to the Windows side with only Python-level references, so it's very "clean"). I'm not opposed, but it's a consideration. We're not using the macro nonsense to try and emulate OOP from C 😉

Besides all that, I'd love to see it done. It's just not quite trivial in its impact, unfortunately.

[barneygale]

Thanks Steve. I haven't done anything with COM before so my opinion here is uninformed to say the least... but if figuring out how to enable COM in the main thread and introducing C++ code into Python on Windows is what's necessary to reliably delete directory trees on Windows, then I vote for doing it. I'm sick of wrapping shutil.rmtree() in a retry loop! 🙃

[zooba]

I'm sick of wrapping shutil.rmtree() in a retry loop!

Likewise, and some of my wrapping gets pretty intense 😆

It would be easier to build the retry loop into rmtree first, though. Likely faster as well, as loading COM can take a noticeable amount of time. The main reason you'd need to go to IFileOperation is to get GUI. Otherwise, either a fairly short retry loop is either going to work, or it's "permanently" locked and IFileOperation isn't going to help either (it might pop up a dialog telling the user which process has it locked, but we probably wouldn't enable GUI anyway).

In all my testing of the retry loop I linked above, 99% of the time it works instantly, and the rest it gets stuck at the initial rename and normally goes through 15-20 attempts (none of which exist, it's just the original path being locked) before succeeding and the actual remove just works. (I do the rename so that even if the removes fail, if the following operation is going to recreate the directory it can succeed.)

[aisk]

Hi @zooba, would it ok to create a private module like _wmi that initializes COM in a separate thread, or even a separate process, to make it only used for specific cases like shutil.rmtree() and avoid affecting the main thread?

[zooba]

You could, but it's probably something that would make more sense as its own library. I expect it would be fairly crippled under rmtree in order to match behaviour (e.g. no UI, which likely means no retries), and of course the performance will be worse. It also expects subtly different path formats, so may not be a drop-in replacement either.

Having it be separately installable - possibly even with a convenient way to patch shutil - would be a better starting point. As long as it's written in fairly pure C++ it wouldn't be hard to merge in later if we decide we want to keep it.