[jthomson04]

Overview:

Every SelectionService call that carries a prompt converts the whole token_ids list on the calling thread, under the GIL; in Ray Serve LLM's KVAwareRouter that is select_and_reserve on every request. pyo3's generic Vec<u32> extraction walks the list through the sequence iterator and takes and drops a reference to every element, which costs more than converting the ints. This PR reads a plain list of ints by index with borrowed items and falls back to the existing extraction for anything else.

99k-token prompt, GB200 Grace, Python 3.12	Before	After
token_ids conversion	7.9 ns/token	3.4 ns/token
select_and_reserve call, CPU on the calling thread	0.83 ms	0.37 ms
Ray-style embedded ingress, one replica, req/s (GIL-bound)	214.8	238.7 (+11%)

Setup:

• Stack: main at e7ccd47 with perf(kv-router): parse selection requests without serde flatten or path tracking #17238 and perf(kv-router): share ZMQ contexts across services sockets; dial replay per gap #17253 (both since merged) and feat(select-svc): Expose session affinity replication for external hosts #16969.
• Workload: an emulation of Ray's KVAwareRouter/KVTokenTracker ([serve][llm] Session pinning for KVAwareRouter ray-project/ray#67224) with 1,024 mocker workers publishing ZMQ KV events and 8,192 concurrent AgentX sessions (mean ISL 99k tokens).
• The hit ratio was unchanged (0.930 / 0.935).

Details:

• lib/bindings/python/rust/llm/kv.rs: what the fast path does.
  • extract_token_ids reads an exact list with PyList_GetItem (borrowed) plus PyLong_AsUnsignedLong on exact ints.
  • It falls back on any other element type (including bool and objects with __index__), and on a value that is negative or above u32::MAX.
  • Only an exact list takes the fast path. List subclasses (whose __iter__ the generic conversion honors), tuples and other sequences use the existing conversion.
  • In every fallback case the existing extract::<Vec<u32>>() accepts the input or raises the same ValueError as before.
• Safety. The GIL is held for the whole loop, and the loop runs no Python code, so the borrowed items stay valid.
• Scope. This covers every prompt-carrying binding call: select, select_and_reserve, create_reservation, overlap_scores and potential_loads.

Where should the reviewer start?

extract_token_ids in lib/bindings/python/rust/llm/kv.rs.

Related Issues

• Relates to perf(kv-router): parse selection requests without serde flatten or path tracking #17238

Validation

• cargo clippy --features select-service -- -D warnings in lib/bindings/python
• pytest lib/bindings/python/tests/test_kv_bindings.py -k selection_service: 16 passed. The select test also sends a tuple (generic path). The malformed-payload test covers negative, out-of-range and non-int ids, and a list subclass whose iterator yields an invalid id (it fails without the exact-list check).

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Python requests now handle lists of valid integer token IDs more reliably, while retaining support for other accepted token sequences.
  • Invalid token IDs—including negative, out-of-range, and non-integer values—now produce a clear ValueError identifying token_ids.
  • Worker selection continues to support tuple token sequences and lists containing booleans.

[jthomson04]

/ok to test 1ff5678

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: ai-dynamo/dynamo
• Branch: jthomson04/select-token-list-read
• File: .coderabbit.yaml
• Commit: 1ff5678
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: a2859ca7-18fa-457a-ad23-2b6232d299c8

Commits

Reviewing files that changed from the base of the PR and between 8222ff0 and 1ff5678.

Files selected for processing: 2

• lib/bindings/python/rust/llm/kv.rs
• lib/bindings/python/tests/test_kv_bindings.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Walkthrough

The Python binding now attempts direct conversion of valid integer lists for token_ids before using generic extraction. Tests cover tuple and boolean-containing token sequences, and verify errors for negative, out-of-range, and string token IDs.

Changes

Python token ID conversion

Layer / File(s)	Summary
Token conversion and validation lib/bindings/python/rust/llm/kv.rs, lib/bindings/python/tests/test_kv_bindings.py	depythonize_prompt_request attempts direct conversion for lists of exact integers that fit in u32. Other inputs use the existing generic extraction path. Tests cover tuple and boolean-containing inputs, and malformed token IDs that raise ValueError mentioning token_ids.

Priority: Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: Minimal · up to 1ff56

No merge-blocking issue is identified; the token-conversion change is ready for normal checks.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title clearly and concisely describes the main change: improving token_ids list conversion performance in Python bindings.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description check		The pull request description includes all required sections: Overview, Details, reviewer starting point, and Related Issues. It identifies Relates to #17238 and provides clear implementation details, …

Full details: Description check

Explanation

The pull request description includes all required sections: Overview, Details, reviewer starting point, and Related Issues. It identifies Relates to #17238 and provides clear implementation details, scope, safety considerations, benchmarks, and validation results.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[jthomson04]

/ok to test 9c4b219

[devin-ai-integration]

✅ Dynamo PR CI passed — run 40400231408 (attempt 1) on 9c4b219648

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 40400231408. Updated on every full-CI run of this PR.}