[jeff3071]

Summary

Update the idle prefix-cache (MAX_CACHED_TOKENS) retention policy at runtime.

KVCACHED_MAX_CACHED_TOKENS remains the startup value. Any process with access to the engine's existing native ledger directory can subsequently publish an update through:

from kvcached.control import set_prefix_cache_retention_limit

result = set_prefix_cache_retention_limit(
    limit_tokens, instance_name="my-instance"
)

Before

sequenceDiagram
    participant Operator
    participant Engine
    participant PrefixCache

    Operator->>Engine: Start with KVCACHED_MAX_CACHED_TOKENS
    Engine->>PrefixCache: Configure fixed retention limit
    PrefixCache->>PrefixCache: Evict after request completion
    Note over Engine,PrefixCache: Changing the policy requires restarting the engine

Loading

After

sequenceDiagram
    participant Caller
    participant Control as kvcached.control
    participant Ledger as Existing Native Ledger
    participant Scheduler as Engine / Scheduler
    participant Policy as Retention Policy
    participant Cache as Prefix Cache
    participant Eviction as Existing Eviction Path

    Caller->>Control: set_prefix_cache_retention_limit(tokens, instance_name=...)
    Control->>Ledger: Publish limit and retention revision on each GPU
    Control-->>Caller: Publication status

    Scheduler->>Ledger: Read matching assignments across GPUs
    Note over Scheduler: SGLang TP ranks receive the same update
    Scheduler->>Policy: Update process-local limit
    Policy->>Cache: Enforce new limit
    Cache->>Eviction: Evict currently evictable excess
    Eviction-->>Cache: Active or locked KV remains protected

    Note over Caller,Eviction: Acceptance does not imply completed eviction or physical GPU memory release

    Scheduler->>Cache: Request finishes later
    Cache->>Policy: Read current limit
    Cache->>Eviction: Evict newly evictable excess

Loading

The implementation reuses the existing vLLM and SGLang eviction paths. Active KV is excluded through the engines' existing ref-count and lock eligibility rules.

Updates use the existing native ledger, with eviction performed on the engine or scheduler thread, including while idle. Engines consume only matching retention revisions and values across all GPUs. Callers can inspect per-GPU publication status and republish the desired limit to repair partial failures.

Current limitations

• SGLang runtime updates require pp_size=1. TP ranks apply updates together. With pipeline parallelism, publication may return accepted, but schedulers do not apply the update.
• SGLang HiCache is not supported yet. In-flight transfers remain protected, but a transfer-completion unlock can bypass the current cache_finished_req enforcement hook. A later request transition will enforce the policy, but immediate enforcement after every HiCache transition is not guaranteed.
• The API updates logical prefix retention only. Returning successfully does not guarantee that physical GPU memory has already been unmapped or returned.

Related to #585 (comment) .

[shipiyouniao]

Could you clarify the failure semantics in the docstring for set_prefix_cache_retention_limit()? The new policy is stored before eviction is attempted, so an eviction exception does not roll back the limit. It would also help to state explicitly that the update is process-local. This is just an API documentation clarification; I don't think it needs transactional rollback.