Skip to content

[Docs] Add LMCache service example - #4074

Open
DongDongJu wants to merge 1 commit into
dstackai:masterfrom
DongDongJu:master
Open

[Docs] Add LMCache service example#4074
DongDongJu wants to merge 1 commit into
dstackai:masterfrom
DongDongJu:master

Conversation

@DongDongJu

Copy link
Copy Markdown

Summary

  • Add an LMCache inference example that runs an LMCache multiprocess server alongside vLLM.
  • Configure an in-memory L1 cache and bounded fs_native L2 storage with LRU eviction.
  • Document process lifecycle, readiness checks, persistent instance storage, GPU requirements, and CUDA compatibility.
  • Add repeat-request instructions for verifying warm-cache reuse through LMCache logs.
  • Add the example to the documentation navigation and examples index.

This is intentionally a recipe-only integration. It does not add LMCache-specific provisioning, cache discovery, or multi-replica cache sharing.

Validation

  • uv run pytest
    • 3,229 passed
    • 1,713 skipped
    • 1 unrelated deprecation warning
  • Focused service and configuration tests: 140 passed
  • Strict MkDocs build using the CI Python 3.11 environment
  • uv run ruff check .
  • uv run ruff format . --check
  • Pyright: 0 errors
  • Pre-commit hooks and git diff --check
  • Parsed the embedded dstack configuration and JSON arguments
  • Validated the command with bash -n and ShellCheck
  • Checked the LMCache and vLLM configuration against their versioned schemas

A live deployment and warm-cache request were not run because they require GPU capacity and model access. The example includes steps for performing this validation after deployment.

AI Assistance

This PR was developed with assistance from OpenAI Codex. Codex helped audit the LMCache and vLLM configuration, draft the documentation, and run validation. The author reviewed the final changes and test results.

Signed-off-by: DongDongJu <commisori28@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant