[pardeep-singh]

Description

The controller already tracks which multiplexed models each replica has loaded (DeploymentReplica._multiplexed_model_ids), but the Serve details don't expose it, so there's no way to see from the API or dashboard which replica holds which model.

This adds a multiplexed_model_ids field to ReplicaDetails:

• schema.py: multiplexed_model_ids: Tuple[str, ...] = Field(default=(), ...). It lists the models loaded or loading on the replica, as last reported to the controller, and is empty for deployments that don't use multiplexing. A tuple keeps the frozen ReplicaDetails hashable. The description notes that after a controller restart it stays empty until the replica's model cache next changes.
• deployment_state.py: DeploymentReplica.record_multiplexed_model_ids also calls update_actor_details(multiplexed_model_ids=tuple(sorted(...))). The IDs are sorted because the replica reports it from a set; this keeps the output stable and lets the unchanged-value fast path from [Serve] Eliminate per-replica-per-tick Pydantic rebuild in update_actor_details #60833 skip rebuilding the details. The routing copy is unchanged.
• multiplex.py: _ModelMultiplexWrapper now counts in-flight loads per model ID (_model_load_tasks is a Counter). load_model decrements the count in a finally block, and when no request for the ID is left and the model isn't loaded, it re-pushes. That covers a load that failed or was cancelled, including a cancellation while waiting for the cache lock, and a cancelled request no longer drops an ID that another request is still loading. unload_model_lru also sets the push flag right after it evicts a model, so an unload that fails or is cancelled still re-pushes. shutdown(), its other caller, stops the metrics pusher before it unloads, so a stopping replica doesn't push after the controller has removed it. Before, a failed load never re-pushed and a cancelled one left the ID in the loading set, so the controller (and routing) kept a stale ID.

The field shows up wherever ReplicaDetails does: get_serve_instance_details(), the REST /api/serve/applications/ response and the dashboard API. serve.status() only carries per-deployment replica counts, so it isn't affected.

Because the details are serialized with exclude_unset=True, a replica that never reports (or reports none) leaves the field unset. It's omitted from the JSON and parses back as (), so the existing exact-JSON tests don't change.

Related issues

Part of #65347 (the in-replica accessor the issue also asks for isn't included).

Additional information

This builds on @upadhyay74aman's #65370. I kept its schema field and the update_actor_details hook-up, adapted to current master after #60833. I left out the in-replica serve.get_multiplexed_model_ids() accessor and its api.py / __init__.py changes, to keep this to the details field.

Why this isn't a duplicate: #65370 was closed by the stale bot without a maintainer review, and no open PR references #65347.

API change: this adds a field to a stability="stable" public schema. It's additive with a default, so old clients parsing new JSON and new clients parsing old JSON both work.

Tests:

• serve/tests/unit/test_deployment_state.py: the new test_record_multiplexed_model_ids_updates_replica_details drives record_request_routing_info and checks list_replica_details(). It covers:

  • unset before any report;
  • sorted IDs after a report;
  • an evicted model dropping out on the next report;
  • a routing-stats-only report leaving the IDs unchanged;
  • hash() on the details.

  It failed before the change with AttributeError: 'ReplicaDetails' object has no attribute 'multiplexed_model_ids'.

• serve/tests/test_multiplex.py:

  • test_failed_load_drops_model_id and test_cancelled_load_drops_model_id[loading|waiting_for_lock] check that the ID leaves the reported set and a re-push is flagged.
  • test_cancelled_waiter_keeps_in_flight_model_id cancels a request waiting on the lock while another loads the same model. It checks the ID stays reported and no re-push is flagged.
  • test_failed_eviction_pushes_with_queued_request makes the evicted model's unload fail while another request for the incoming model is queued, and checks a re-push is flagged.
  • test_shutdown_stops_pushing_model_ids runs the pusher on a short interval, calls shutdown() with two models loaded and checks nothing is pushed afterwards.

  Each fails on master's multiplex.py or the change's first version.

• serve/tests/test_multiplex.py::test_request_routing_info also checks the REST-shaped get_serve_details() dict and the parsed ServeInstanceDetails through the LRU sequence model1 → {1, 2} → {2, 3}.

pytest python/ray/serve/tests/unit/test_deployment_state.py   # 337 passed, 12 skipped
pytest python/ray/serve/tests/test_multiplex.py -k 'TestMultiplexWrapper or TestBasicAPI or request_routing_info'  # 17 passed (local CPU cluster)
pre-commit run --files <changed files>                        # all hooks pass, incl. mypy and pyrefly (ray serve)

Notes:

• A stopped replica in recent_dead_replicas keeps its last reported list. I can clear it on stop if you'd prefer.

AI assistance: I used an AI coding assistant to write this change, its tests and this description.

[gemini-code-assist]

Code Review

This pull request exposes multiplexed model IDs in the replica's ReplicaDetails, updating the schema and tests to verify this behavior. Feedback suggests adding a safety check to handle the case where multiplexed_model_ids is None to avoid a potential TypeError when sorting.

[johntaylor-cell]

Thanks @pardeep-singh. Here are my comments:

1. Failed and cancelled loads leave stale IDs in the field: the except Exception path never re-pushes, and CancelledError skips it entirely. A try/finally in load_model that discards the ID and sets _push_multiplexed_replica_info fixes both.
2. After a controller restart the field stays empty until the replica's cache changes. Please say that in the field description.
3. A List on the frozen ReplicaDetails makes hash() raise. Tuple[str, ...], set via tuple(sorted(...)), keeps it hashable.
4. "Part of [Serve] Expose multiplexed_model_ids on ReplicaDetails object #65347" rather than "Closes", since the in-replica accessor isn't included.

[pardeep-singh]

@johntaylor-cell thanks for the review. All four are addressed in f340d07:

1. load_model now removes the ID from the loading set in a finally block and re-pushes when the load failed or was cancelled. That also covers a cancellation while it waits for the cache lock. New tests: test_failed_load_drops_model_id and test_cancelled_load_drops_model_id[loading|waiting_for_lock]. All three fail on the previous multiplex.py.
2. The field description now says it stays empty after a controller restart until the replica's model cache next changes.
3. The field is Tuple[str, ...], set via tuple(sorted(...)), and the unit test calls hash() on the details.
4. The description now says "Part of [Serve] Expose multiplexed_model_ids on ReplicaDetails object #65347".

test_deployment_state.py: 337 passed. test_multiplex.py wrapper and routing-info tests: 15 passed. Could you take another look when you have a moment?

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit f340d07. Configure here.}

[johntaylor-cell]

Thanks @pardeep-singh, all four look good. A few small things:

1. An eviction that fails or is cancelled after popitem never re-pushes if another request for the same model is queued. Setting the push flag right after popitem fixes it.
2. Tests don't pin when the finally re-arms, an unconditional re-arm passes them all. Please assert the flag after the cancelled waiter and add a case for 1.
3. test_collect_model_ids still sets _model_load_tasks to a set, it's a Counter now.
4. The description still describes the set-based fix.

[pardeep-singh]

@johntaylor-cell thanks, all four are addressed in b3e9698:

1. unload_model_lru sets _push_multiplexed_replica_info right after popitem, so a failed or cancelled unload still re-pushes. The flag load_model set after the unload is gone, since it's now redundant. shutdown(), the other caller, now sets the flag too.
2. test_cancelled_waiter_keeps_in_flight_model_id clears the flag before cancelling the waiter and asserts it stays clear after the cancellation and after the first load finishes, so an unconditional re-arm fails it. The new test_failed_eviction_pushes_with_queued_request queues two requests for "2" behind the cache lock. The first evicts "1", whose __del__ raises, and the test checks the flag is set. It fails on 9df5804.
3. test_collect_model_ids now sets _model_load_tasks to a Counter.
4. The description now describes the per-ID count and the eviction push.

test_multiplex.py -k 'TestMultiplexWrapper or TestBasicAPI or request_routing_info': 16 passed.

[johntaylor-cell]

Thanks @pardeep-singh. All four look good. One last thing is that shutdown() now sets the push flag while the pusher keeps running, so stopping replicas can push after removal and the controller logs "not found" warnings and errors. Please stop metrics_pusher first in shutdown(), with a test.

[pardeep-singh]

Thanks @johntaylor-cell. Done in 308a983: shutdown() now awaits metrics_pusher.graceful_shutdown() before it unloads, so a stopping replica no longer pushes after removal. The new test_shutdown_stops_pushing_model_ids runs the pusher on a short interval and checks nothing is pushed once shutdown() starts; it failed before the change (an empty ID list was pushed).