Skip to content

fix(client): recover from a locked tool directory on reinstall - #2231

Merged
mikhailm-coder merged 4 commits into
mainfrom
hotfix/tool-reinstall-lock-recovery
Sep 23, 2026
Merged

mikhailm-coder merged 4 commits into
mainfrom
hotfix/tool-reinstall-lock-recovery

Conversation

@mikhailm-coder

Copy link
Copy Markdown
Contributor

What & why

A managed tool's install directory (app_support_dir/<tool_id>) can't always be deleted during reinstall/uninstall: if any process holds a file under it open without FILE_SHARE_DELETE, remove_dir_all fails with os error 32 (ERROR_SHARING_VIOLATION) and the client loops without recovering. In production this stranded a tool as un-reinstallable for hours until a human found and killed the lock holders. The client already had a Restart Manager helper but only logged holders, and only on command-spawn failures — never on the actual removal failure.

This makes the shared removal path lock-aware and self-recovering, safely. It is tool-generic (not gated on any tool_id); the mesh agent was the case that hit it, but every managed tool uses these paths.

What changed

  • platform/lock_recovery.rs (new) — a platform-independent, unit-tested classify_holder encoding the hard kill-safety rules, plus Windows-only eviction and a MoveFileEx delete-on-reboot fallback.
  • platform/file_lock.rs — new LockHolder (pid, app name, RM_APP_TYPE, TS session id, restartable, service short name) and get_directory_lock_holders(dir), which registers every file under the directory with a single Restart Manager session.
  • platform/uninstall.rs — remove_directory_with_retry now, on os error 32, identifies the holders, evicts the safe ones, retries, and falls back to delete-on-reboot for a refused holder.
  • services/tool_uninstall_service.rs — the single-tool uninstall now goes through remove_directory_with_retry (was a plain remove_dir_all).
  • Cargo.toml — adds the Win32_Storage_FileSystem windows feature (for MoveFileExW).

The reinstall install-sites already call remove_directory_with_retry, so they inherit the new behavior.

Kill safety (enforced by classify_holder, unit-tested)

Never terminated: this process and its entire ancestor chain; PID 0 and PID 4; ApplicationType == RmCritical; the critical image allowlist (services, svchost, lsass, csrss, wininit, smss, winlogon, System). Auto-terminated: the orphan-shell class (cmd, powershell, pwsh, conhost) and processes whose image path is under the tool dir. A service holder is SCM-stopped (never TerminateProcess) and only when its binary is under the tool dir; a foreign service is skipped. Everything else is logged and skipped. Every holder is logged (pid, name, image, TS session, RM app type, restartable) before any action.

Recovery contract

The common case (a killable orphan shell) recovers within the retry loop and the reinstall completes normally. If a refused holder still blocks removal, the directory's contents are scheduled for delete-on-reboot and a clear terminal warning is logged; the operation returns an error so the reinstall does not proceed over a doomed directory. Redelivery is bounded by the consumer's 10-delivery budget, so it does not loop infinitely, and the next boot clears the directory.

Testing

  • 12 unit tests for classify_holder (self/ancestor, PID 0/4, RmCritical, critical names, orphan shells, under-dir process, under-dir vs foreign service, nested/sibling path matching).
  • cargo clippy --all-targets -- -D warnings clean on macOS and on the x86_64-pc-windows-gnu cross target.
  • cargo check --locked clean for both the host and Windows openframe-client bin; Cargo.lock unchanged.

CU-86akkdefm

🤖 Generated with Claude Code

When a reinstall/uninstall can't delete a tool's install directory because a
process holds a file under it open without FILE_SHARE_DELETE
(ERROR_SHARING_VIOLATION, os error 32), remove_directory_with_retry now
identifies the lock holders via Restart Manager, evicts the safe ones, retries,
and falls back to delete-on-reboot instead of looping. Tool-generic; not gated
on any tool_id.

Kill safety: never terminate this process + its ancestor chain, PID 0/4,
RmCritical, or the critical system-process allowlist; auto-terminate only the
orphan-shell class and processes whose image is under the tool dir; SCM-stop a
service only when its binary is under the tool dir. Every holder is logged
before any action is taken.

Also routes the single-tool uninstall through remove_directory_with_retry and
adds the Win32_Storage_FileSystem windows feature for MoveFileExW.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@michaelassraf

Copy link
Copy Markdown
Contributor

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

🦩 Flamingo Code Review

No findings on the current head.

Mode: advisory


Need another pass? Commits pushed after this review are not reviewed automatically.

  • Review the new commits — the commits added since this review
  • Review the whole diff again — ignoring what was already reviewed

Prefer typing? Comment @flamingo-review, or @flamingo-review full. To review every push on this pull request, add the flamingo-review-always label.

React 👍/👎 on inline comments to teach the reviewer.

Started 2026-09-17 16:35 UTC · updated 2026-09-17 16:36 UTC · workflow run

MoveFileEx delay-until-reboot schedules a delete by path and never cancels it, so
if the tool directory is legitimately recreated by a successful reinstall before
the reboot, the next boot would delete the fresh files. The eviction + retry path
already recovers the common case (a killable orphan-shell holder), and a refused
holder clears on the next reboot via redelivery, so the fallback's marginal value
did not justify that footgun.

Removes schedule_delete_on_reboot and the Win32_Storage_FileSystem feature; the
terminal failure now returns Err as it already did for non-sharing-violation errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mikhailm-coder
mikhailm-coder enabled auto-merge (squash) September 23, 2026 15:50
@mikhailm-coder
mikhailm-coder merged commit bd078df into main Sep 23, 2026
13 of 14 checks passed
@mikhailm-coder
mikhailm-coder deleted the hotfix/tool-reinstall-lock-recovery branch September 23, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants