fix(cache): fully remove downloaded models

[Minamiyama]

• Resolve nested cache symlinks recursively so ModelScope and Hugging Face payload files are removed instead of leaving real weights behind.

• Preserve files referenced by another cached model and unlink local model_uri entries without touching user-managed source directories.

• Restrict recursive directory deletion to Xinference-managed cache and tensorizer roots to prevent arbitrary path removal.

• Prune empty Hub directories up to, but never including, each managed download root; stop when sibling models or residual files remain.

• Delete linked files before cache trees and tolerate concurrent cleanup races while still reporting real deletion failures.

• Treat a false cache-delete API result as an error in WebUI and add localized messages for English, Chinese, Japanese, and Korean.

• Cover nested targets, shared blobs, local sources, unmanaged paths, and ModelScope parent pruning with focused worker tests.

Summary

• Remove actual downloaded model files referenced by nested Xinference cache
  symlinks.
• Preserve files still referenced by another cached model.
• Prune empty model directories without deleting Hub root directories.
• Prevent local model_uri sources and unmanaged directories from being
  removed.
• Report backend deletion failures correctly in WebUI.

Root Cause

Xinference recursively mirrors downloaded Hub snapshots into its cache using
symlinks. However, WorkerActor.list_deletable_models() only inspected the
first directory level.

For models containing nested directories, deleting the cache removed the
Xinference symlinks but left most ModelScope or Hugging Face payload files on
disk. Disk space was therefore not reclaimed.

The WebUI also ignored a backend response containing result: false and always
displayed a successful deletion notification.

This behavior was not Docker-specific. It affected Docker, local, and virtual
environment deployments; bind-mounted Docker storage only made the remaining
files easier to observe.

Changes

Backend

• Recursively collect nested symlink file targets without following directory
  symlinks.
• Delete external payload files before removing the Xinference cache tree.
• Preserve targets still referenced by another Xinference cache entry.
• Treat a root cache symlink as a user-provided model_uri and unlink only the
  cache entry.
• Reject cache paths outside XINFERENCE_CACHE_DIR.
• Restrict recursive directory deletion to Xinference cache and tensorizer
  roots.
• Tolerate files already removed by concurrent cleanup.
• Remove empty parent directories with os.rmdir after deleting downloaded
  files.
• Stop parent cleanup when a directory contains another model, revision,
  lock file, incomplete download, or other residual content.
• Preserve the managed download roots:
  • Hugging Face cache root
  • ModelScope models root
  • OpenMind cache root

For example:

/root/.xinference/modelscope/models/
└── ZhipuAI--GLM-Image/
    └── snapshots/
        └── master/

After all files for this model are deleted, empty directories are removed up
through ZhipuAI--GLM-Image, while the modelscope/models root remains.

WebUI

• Check the result field returned by the cache deletion API.
• Display an error instead of a false success notification when deletion fails.
• Add localized failure messages for English, Chinese, Japanese, and Korean.

Safety

• Shared payload files remain when another cached model references them.
• User-managed model_uri source directories are never recursively deleted.
• Empty parent cleanup uses os.rmdir, not shutil.rmtree.
• Parent cleanup never removes a configured Hub root.
• Non-empty directories stop cleanup naturally.

[gemini-code-assist]

Code Review

This pull request improves the model cache deletion process in Xinference. On the frontend, it adds error handling and localized error messages when cache deletion fails. On the backend, it introduces safer cache cleanup mechanisms, including resolving nested symlinks, preserving shared model files, and pruning empty parent directories within managed download caches, backed by comprehensive unit tests. The review feedback highlights two key issues: first, an unexpected OSError during empty directory pruning could abort the deletion process midway and leave the cache tracker in an inconsistent state; second, using XDG_CACHE_HOME directly as the OpenMind cache root is risky as it could lead to pruning directories in the shared system-wide cache.

[qinxuye]

Two additional cache-deletion findings:

[qinxuye]

One cache-deletion safety issue remains:

[qinxuye]

LGTM