Skip to content

fix(sglang): reclaim the shrinking cache component - #573

Draft
shipiyouniao wants to merge 13 commits into
ovg-project:mainfrom
shipiyouniao:fix/sglang-component-shrink
Draft

shipiyouniao wants to merge 13 commits into
ovg-project:mainfrom
shipiyouniao:fix/sglang-component-shrink

Conversation

@shipiyouniao

@shipiyouniao shipiyouniao commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Shrinking a managed SWA pool can remain deferred while FULL prefix nodes are locked, even when their old SWA windows can be reclaimed. The shrink callback previously checked only full_evictable_size() and sent only num_tokens; it could return before requesting SWA eviction.

The fix retains each manager's FULL/SWA ownership and requests the matching native eviction component. It stops when the physical target is met, respects grouped frees and pending cache transfers, and retains active windows and permanent backing. This also handles HiSparse's logical allocator and SWA-owned C4 state.

The previous reclaimer used FULL availability for every discovered manager. Native SGLang supports independent SWA eviction while the FULL prefix remains protected.

Before:

sequenceDiagram
  participant S as Scheduler
  participant R as SWA shrink callback
  participant C as Native cache
  S->>R: Apply lower SWA limit
  R->>C: full_evictable_size()
  C-->>R: 0 (FULL protected)
  R-->>S: Return, SWA limit stays pending
Loading

After:

sequenceDiagram
  participant S as Scheduler
  participant R as SWA shrink callback
  participant C as Native cache
  S->>R: Apply lower SWA limit
  R->>C: swa_evictable_size()
  R->>C: evict(swa_num_tokens=needed)
  C-->>R: Release unprotected SWA references
  R-->>S: Recheck physical target
Loading

Stacked on #569 and its existing dependencies, with one new fix commit. Existing PR heads are unchanged. This does not add cache residency policy, async eviction or scheduling policy.

Validation at 3612369cdd700d606f1731f8a2f8f6636b049491: 532 targeted tests; the Python 3.11 CPU suite passed 1,644 tests with 14 optional native-probe/SGLang skips; GPU-free C++ tests, all-file pre-commit and MyPy 3.10–3.13 passed. On one T4 using Python 3.12, 18 native SGLang component tests and 4 exact-placement tests passed. Dedicated private Codex review completed with no actionable findings. The complete CPU version matrix is left to GitHub CI; it is not claimed as a completed private check.

The T4 fixture uses real SGLang 0.5.20 cache, allocator and physical backing with fixed small pools. It does not load model weights, execute attention/compressor kernels, capture graphs or measure serving performance. Full DSV4/MTP serving and distributed validation have not been run; this PR remains draft.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants