fix(client): recover from a locked tool directory on reinstall - #2231
Merged
Merged
Conversation
When a reinstall/uninstall can't delete a tool's install directory because a process holds a file under it open without FILE_SHARE_DELETE (ERROR_SHARING_VIOLATION, os error 32), remove_directory_with_retry now identifies the lock holders via Restart Manager, evicts the safe ones, retries, and falls back to delete-on-reboot instead of looping. Tool-generic; not gated on any tool_id. Kill safety: never terminate this process + its ancestor chain, PID 0/4, RmCritical, or the critical system-process allowlist; auto-terminate only the orphan-shell class and processes whose image is under the tool dir; SCM-stop a service only when its binary is under the tool dir. Every holder is logged before any action is taken. Also routes the single-tool uninstall through remove_directory_with_retry and adds the Win32_Storage_FileSystem windows feature for MoveFileExW. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Contributor
🦩 Flamingo Code ReviewNo findings on the current head. Mode: advisory Need another pass? Commits pushed after this review are not reviewed automatically.
Prefer typing? Comment React 👍/👎 on inline comments to teach the reviewer. Started 2026-09-17 16:35 UTC · updated 2026-09-17 16:36 UTC · workflow run |
MoveFileEx delay-until-reboot schedules a delete by path and never cancels it, so if the tool directory is legitimately recreated by a successful reinstall before the reboot, the next boot would delete the fresh files. The eviction + retry path already recovers the common case (a killable orphan-shell holder), and a refused holder clears on the next reboot via redelivery, so the fallback's marginal value did not justify that footgun. Removes schedule_delete_on_reboot and the Win32_Storage_FileSystem feature; the terminal failure now returns Err as it already did for non-sharing-violation errors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
danylo-babenko-flamingo
approved these changes
Sep 23, 2026
danylo-babenko-flamingo
approved these changes
Sep 23, 2026
mikhailm-coder
enabled auto-merge (squash)
September 23, 2026 15:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
A managed tool's install directory (
app_support_dir/<tool_id>) can't always be deleted during reinstall/uninstall: if any process holds a file under it open withoutFILE_SHARE_DELETE,remove_dir_allfails withos error 32(ERROR_SHARING_VIOLATION) and the client loops without recovering. In production this stranded a tool as un-reinstallable for hours until a human found and killed the lock holders. The client already had a Restart Manager helper but only logged holders, and only on command-spawn failures — never on the actual removal failure.This makes the shared removal path lock-aware and self-recovering, safely. It is tool-generic (not gated on any
tool_id); the mesh agent was the case that hit it, but every managed tool uses these paths.What changed
platform/lock_recovery.rs(new) — a platform-independent, unit-testedclassify_holderencoding the hard kill-safety rules, plus Windows-only eviction and aMoveFileExdelete-on-reboot fallback.platform/file_lock.rs— newLockHolder(pid, app name,RM_APP_TYPE, TS session id, restartable, service short name) andget_directory_lock_holders(dir), which registers every file under the directory with a single Restart Manager session.platform/uninstall.rs—remove_directory_with_retrynow, onos error 32, identifies the holders, evicts the safe ones, retries, and falls back to delete-on-reboot for a refused holder.services/tool_uninstall_service.rs— the single-tool uninstall now goes throughremove_directory_with_retry(was a plainremove_dir_all).Cargo.toml— adds theWin32_Storage_FileSystemwindows feature (forMoveFileExW).The reinstall install-sites already call
remove_directory_with_retry, so they inherit the new behavior.Kill safety (enforced by
classify_holder, unit-tested)Never terminated: this process and its entire ancestor chain; PID 0 and PID 4;
ApplicationType == RmCritical; the critical image allowlist (services,svchost,lsass,csrss,wininit,smss,winlogon,System). Auto-terminated: the orphan-shell class (cmd,powershell,pwsh,conhost) and processes whose image path is under the tool dir. A service holder is SCM-stopped (neverTerminateProcess) and only when its binary is under the tool dir; a foreign service is skipped. Everything else is logged and skipped. Every holder is logged (pid, name, image, TS session, RM app type, restartable) before any action.Recovery contract
The common case (a killable orphan shell) recovers within the retry loop and the reinstall completes normally. If a refused holder still blocks removal, the directory's contents are scheduled for delete-on-reboot and a clear terminal warning is logged; the operation returns an error so the reinstall does not proceed over a doomed directory. Redelivery is bounded by the consumer's 10-delivery budget, so it does not loop infinitely, and the next boot clears the directory.
Testing
classify_holder(self/ancestor, PID 0/4, RmCritical, critical names, orphan shells, under-dir process, under-dir vs foreign service, nested/sibling path matching).cargo clippy --all-targets -- -D warningsclean on macOS and on thex86_64-pc-windows-gnucross target.cargo check --lockedclean for both the host and Windowsopenframe-clientbin;Cargo.lockunchanged.CU-86akkdefm
🤖 Generated with Claude Code