[jeff3071]

Stack on #603.

Diff against #603

Summary

HiCache write-through acks can make tokens evictable after cache_finished_req() has already checked retention. The idle cache can then exceed either the startup or runtime token cap.

Enforce retention after check_hicache_events() and loading_check() return, using native eviction. The load hook waits for unpinning and covers calls from is_load_back_event_done() outside the event poll.

Before

Example: a zero token cap, with 956 tokens protected by the final write-through copy.

sequenceDiagram
    participant S as Scheduler
    participant C as Prefix cache

    S->>C: cache_finished_req(req)
    C->>C: Check retention cap: 0 evictable tokens
    Note over C: Copy still protects 956 tokens
    S->>C: check_hicache_events()
    C->>C: writing_check(): synchronize ack, release copy lock
    C->>C: loading_check()
    C-->>S: Return
    Note over C: 956 tokens now evictable, exceeding cap 0

Loading

After

sequenceDiagram
    participant S as Scheduler
    participant H as Retention wrapper
    participant C as Prefix cache

    S->>H: check_hicache_events()
    H->>C: Original check_hicache_events()
    C->>C: writing_check(): synchronize ack, release copy lock
    C->>H: loading_check()
    H->>C: Original loading_check()
    Note over C: Complete any load unlocks and unpinning
    C-->>H: Return
    H->>C: Enforce retention cap
    C->>C: Native evict(EvictParams(num_tokens=956))
    H-->>C: Return
    C-->>H: Original event poll returns
    H->>C: Enforce retention cap (already satisfied)
    H-->>S: Return

Loading

Test

Real GPU before/after runs: RTX A2000 12GB, SGLang 0.5.20, Qwen2.5-0.5B-Instruct, TP=, HiCache page_first / write_through.

Evictable tokens after copy completion	Before	After
Startup cap 0	956	0
Runtime cap 256, after a new request	958	0
Unlimited cap -1	957	957

All 10 requests completed, generating 320 tokens. Every paired request produced identical complete output token IDs. The patched repeated request restored 924 tokens from the host; real write/load acks were observed without injecting cache nodes or completion events.