[hello-peter-tang]

Why are these changes needed?

With gang scheduling, GangSchedulingAutoscalingPolicy rounds the autoscaler's decision up to a whole gang, but AutoscalingState.apply_bounds() clips the decision to [min_replicas, max_replicas] without regard to gang size. When a bound is not a multiple of gang_size, the clip undoes the rounding:

• min = max = 3 with gang_size = 2: the target is held at 3, rejected by the scheduler on every tick (num_replicas_to_add 3 is not divisible by gang_size 2), and the deployment eventually fails with DEPLOY_FAILED after exhausting constructor retries.
• min = 3, max = 4: a scale-down can never stop a partial gang, so the target flaps between 3 and 4 every downscale_s tick.

The YAML schema already validates autoscaling bounds against gang_size, but the Python API skipped the check whenever autoscaling_config was set, so @serve.deployment(...) and Deployment.options(...) accepted these configurations.

What is the change?

Validate the bounds the user explicitly sets at both Python entry points, before handle_num_replicas_auto() merges in defaults:

• min_replicas, max_replicas, and initial_replicas must be multiples of gang_size.
• Bounds left at their defaults are not checked. Defaults (e.g. min_replicas=1, max_replicas=100) are safe because scaling decisions are rounded up to whole gangs, and checking merged defaults would break existing gang deployments and proto round-trips.
• Both dicts and AutoscalingConfig instances are handled; for instances only fields present in model_fields_set are checked.

Both reproduction configurations from #67494 now fail immediately at decoration time with:

ValueError: autoscaling_config.min_replicas (3) must be a multiple of gang_size (2).

Runtime alignment of capacity-adjusted bounds (e.g. max_replicas=8, gang_size=4, target_capacity=75 -> 6 replicas) is deliberately left for a follow-up: choosing between rounding up (may exceed the requested capacity by up to gang_size - 1 replicas) and rounding down (may undershoot, and can turn a non-zero capacity into zero replicas) needs a decision, which I've asked about in the issue.

Related issue number

Part of #67494

Tests

• Added tests for unaligned min_replicas, max_replicas, and initial_replicas (dict and AutoscalingConfig inputs), defaults, and the Deployment.options() path.
• python/ray/serve/tests/unit/test_config.py + test_schema.py: 219 passed.
• python/ray/serve/tests/unit/test_autoscaling_policy.py + test_application_state.py: 184 passed.
• black and ruff clean on the touched files.

[gemini-code-assist]

Code Review

This pull request introduces validation to ensure that explicitly set autoscaling bounds (min_replicas, max_replicas, and initial_replicas) are multiples of the gang_size when gang scheduling is configured. The feedback highlights two key issues: a potential TypeError in the validation function if autoscaling_config is None, and a validation bypass in the deployment options method when only one of the configs is updated. Both issues can be resolved using the provided code suggestions.

[gemini-code-assist]

The validate_gang_scheduling_bounds function signature allows autoscaling_config to be None (typed as Optional[Union[Dict, AutoscalingConfig]]). However, if None is passed, the function will bypass the isinstance check and execute the else block, where name in autoscaling_config will raise a TypeError: argument of type 'NoneType' is not iterable. Adding an early return guard for None prevents this potential runtime crash.

def validate_gang_scheduling_bounds(
    gang_size: int,
    autoscaling_config: Optional[Union[Dict, AutoscalingConfig]],
) -> None:
    """Validate that explicitly set autoscaling bounds are multiples of gang_size.

    Bounds left at their defaults are not checked: gang scheduling rounds the
    autoscaler's decisions up to whole gangs, so the defaults are safe.
    """
    if autoscaling_config is None:
        return

    if isinstance(autoscaling_config, AutoscalingConfig):

[GNITOAHC]

mypy reports:

python/ray/serve/_private/config.py:698: error: Value of type "dict[Any, Any] | AutoscalingConfig | None" is not indexable  [index]
python/ray/serve/_private/config.py:700: error: Unsupported right operand type for in (...)  [operator]

Both callers already filter out None, so either an early return like the suggestion here, or dropping Optional from the signature, should fix it.

[gemini-code-assist]

When calling options(), if a user updates only gang_scheduling_config or only autoscaling_config, the validation can be bypassed because it only checks autoscaling_config if it is explicitly passed in the current call (i.e., not DEFAULT.VALUE). To make this robust, we should resolve both gang_config_for_validation and autoscaling_config_for_validation by falling back to the existing deployment config, and then validate them if both are active.

        # Validate the user-provided bounds before handle_num_replicas_auto()
        # merges in defaults, so only explicitly set bounds are checked.
        gang_config_for_validation = (
            gang_scheduling_config
            if gang_scheduling_config is not DEFAULT.VALUE
            else self._deployment_config.gang_scheduling_config
        )
        autoscaling_config_for_validation = (
            autoscaling_config
            if autoscaling_config is not DEFAULT.VALUE
            else self._deployment_config.autoscaling_config
        )
        if (
            gang_config_for_validation is not None
            and autoscaling_config_for_validation is not None
        ):
            if isinstance(gang_config_for_validation, dict):
                gang_size = gang_config_for_validation.get("gang_size")
            else:
                gang_size = gang_config_for_validation.gang_size
            if gang_size is not None:
                validate_gang_scheduling_bounds(
                    gang_size, autoscaling_config_for_validation
                )

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit e54570b. Configure here.}

[cursor]

Locals pollute user-configured option names

Medium Severity

Assigning gang_config_for_validation and gang_size before user_configured_option_names is taken from locals() records those helper names as user-set options. Every Deployment.options() call then includes gang_config_for_validation, and @serve.deployment includes gang_size when both gang and autoscaling configs are provided.

Additional Locations (1)

• python/ray/serve/api.py#L750-L762

 

^{Reviewed by Cursor Bugbot for commit e54570b. Configure here.}

[GNITOAHC]

Gang scheduling docs only says num_replicas has to be a multiple of gang_size. Since this PR adds a new error for autoscaling bounds, should we add a line there too? I believe it's locating in doc/source/serve/advanced-guides/gang-scheduling.md.

[hello-peter-tang]

Thanks @GNITOAHC — pushed f669815 addressing both points:

• Added an early return in validate_gang_scheduling_bounds when autoscaling_config is None, so mypy can narrow the Optional union before indexing/iterating it.
• Documented the autoscaling-bound requirement next to the num_replicas rule in gang-scheduling.md, with a link to the Autoscaling section.

microcheck is rerunning on the new head.