Skip to content

feat(venv): install Jina flash-attn wheel from find-links - #5392

Open
m199369309 wants to merge 4 commits into
xorbitsai:mainfrom
m199369309:feat/jina-flash-attn-find-links
Open

feat(venv): install Jina flash-attn wheel from find-links#5392
m199369309 wants to merge 4 commits into
xorbitsai:mainfrom
m199369309:feat/jina-flash-attn-find-links

Conversation

@m199369309

Copy link
Copy Markdown
Collaborator

Summary

  • pin flash-attn==2.8.3.post1+cvte1 for jina-embeddings-v3
  • install the Jina flash-attn package separately from configured local Find Links using a binary-only, no-dependency reinstall
  • validate the installed distribution version, virtualenv ownership, and CUDA import in a fresh process
  • fall back to PyTorch native attention with a warning when no Find Links source is configured
  • keep flash-attn in the regular package installation path for every non-Jina model
  • keep the complete original package list in virtualenv setup fingerprints so flash-attn version/source changes invalidate setup deduplication

Dependencies

Depends on #5387.

Transitively depends on #5386.

This branch temporarily contains the dependency commits. It will be rebased onto official main after the prerequisite PRs merge.

Safety and compatibility

  • the dedicated hook is limited to jina-embeddings-v3
  • configured Find Links installation or post-install validation failures are fatal
  • absence of Find Links is non-fatal and preserves Jina's native-attention fallback
  • no global --no-index is added
  • non-Jina models retain their existing flash-attn installation behavior
  • no Wheel binaries or deployment-specific paths are included

Validation

  • pytest -q xinference/core/tests/test_virtual_env_manager.py xinference/core/tests/test_worker.py — 150 passed
  • focused Jina flash-attn and model-spec tests — 15 passed
  • pre-commit run --files ... — passed
  • python -m json.tool xinference/model/embedding/model_spec.json — passed
  • git diff --check — passed

@XprobeBot XprobeBot added this to the v3.x milestone Aug 19, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for worker-local wheel directories (find_links) during virtual environment package installation at model launch time. It adds validation to ensure local paths are absolute, readable, and reside within administrator-configured allowed roots. Additionally, it implements a dedicated post-install hook for installing flash-attn from local wheels for jina-embeddings-v3 models. Feedback on the changes suggests addressing a Python 3.9 compatibility issue with importlib.metadata.Distribution.locate_file and improving error reporting by logging or raising subprocess stdout and stderr when validation or installation fails.

Comment thread xinference/core/virtual_env_manager.py
Comment thread xinference/core/virtual_env_manager.py
Comment thread xinference/core/virtual_env_manager.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants