[dragongu]

Description

Support retrying actor tasks in a fresh process when they raise exceptions that cannot be recovered from by retrying in the same process.

For example, a CUDA error can leave the GPU context unusable, or an engine failure can leave an actor with a dead subprocess. The actor process may remain alive, but subsequent attempts in that process continue to fail. Recovering requires restarting the actor rather than retrying the task in place.

This PR adds an actor-only restart_on_exceptions option. Users specify case-insensitive substring patterns to match exception type names, messages, and __cause__/__context__ chains. When an exception matches, Ray terminates the actor worker, reconstructs the actor if its restart budget permits, and retries the failed task if its task-retry budget permits.

The option is disabled by default and reuses the existing max_restarts and max_task_retries settings. Non-matching exceptions retain their existing behavior.

Additional information

Usage

import ray

@ray.remote(
    max_restarts=3,
    max_task_retries=2,
    restart_on_exceptions=["CUDA error", "EngineDeadError"],
)
class ModelWorker:
    def predict(self, batch):
        ...

restart_on_exceptions selects which exceptions require process-level recovery; max_restarts limits actor reconstruction, and max_task_retries limits retries of failed calls. Setting restart_on_exceptions alone does not enable either budget.

Behavior and implementation

• Matching exceptions trigger a process restart instead of an in-process exception retry, including when retry_exceptions=True is enabled.
• Reconstruction reruns the constructor; it does not restore application state. In-flight tasks may fail or be retried according to their retry policy. Methods with external side effects require the same idempotency or checkpointing guarantees as existing actor retries.
• Persistent matching exceptions consume the restart budget. Finite budgets and specific patterns are recommended to avoid restart loops and unintended matches.
• Exception handling covers constructors, ordinary methods, sync/async streaming generators, static generators, and legacy dynamic generators. Exceptions whose __str__ fails are formatted safely.
• Async restart requests are propagated to the task-execution thread before worker exit. This avoids raising SystemExit on the shared event loop, which can leave the task thread waiting indefinitely for its future.

This PR is Core-only. Ray Data integration is deferred to a follow-up.

Testing

bazel build //:_raylet

PYTHONPATH=/tmp/ray-restart-source-brxtncvd:/data/code/github/ray/python/ray/tests \
  .venv/bin/python -m pytest \
  python/ray/tests/test_actor_restart_on_exceptions.py -v

• Native rebuild passed.
• 22 passed in 55.67 seconds, using the original test file and repository conftest with current source and freshly built native artifacts in an isolated package.
• Coverage includes option validation, exception matching/chains/cycles, safe formatting, constructor/method recovery, restart-budget exhaustion, retry precedence, and partial-output replay of the same sync/async stream invocation.
• Applicable pre-commit hooks passed, including commit-time checks on all 17 changed files. git diff --check passed.

The isolated package avoids the older Ray wheel installed in the shared virtual environment, which cannot load the current conftest without the source PYTHONPATH. Protobuf staging applied the existing BUILD import rewrites using Python because unzip is unavailable. A full package build and the complete Core regression suites have not been validated.

[gemini-code-assist]

Code Review

This pull request implements the restart_on_exceptions feature for Ray actors, allowing actors to automatically restart on a fresh process when specific exception substring patterns are matched. The changes span across documentation, Python option validation, Cython bindings, C++ task specifications, and protobuf definitions, alongside comprehensive unit tests. The feedback suggests a minor but important improvement in python/ray/_raylet.pyx to explicitly check if __cause__ is not None instead of using the or operator, preventing potential issues with custom exceptions that evaluate to False in a boolean context.

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

^{Reviewed by Cursor Bugbot for commit 09134f6. Configure here.}