Skip to content

[Core] After a failed ray:// connect, a local ray.init() raises a bare AssertionError #67142

Open
#67407
Open
[Core] After a failed ray:// connect, a local ray.init() raises a bare AssertionError#67142
#67407
Assignees
justinyeh1995
Labels
P3Issue moderate in impact or severitybugSomething that is supposed to be working; but isn'tcoreIssues that should be addressed in Ray Coregood-first-issueGreat starter issue for someone just starting to contribute to Ray

Description

@MortalHappiness
MortalHappiness
opened on Oct 18, 2026

What happened + What you expected to happen

After ray.init("ray://...") fails, the process stays in Ray Client mode, and a later local ray.init() in that process raises a bare AssertionError that doesn't say what went wrong:

ray:// connect failed: ConnectionError ray client connection timeout
local ray.init() failed: AssertionError

The assertion is assert ray.experimental.internal_kv._internal_kv_initialized() in _record_stats in python/ray/_private/node.py. It fails because _ClientContext.connect in python/ray/util/client/__init__.py calls _explicitly_enable_client_mode() before connecting, and on failure its except calls only the per-context disconnect(), which never resets the flag. Only the module-level ray.util.disconnect() calls _explicitly_disable_client_mode().

Expected: after the first ray:// connect fails, a local ray.init() still fails, but with an error that says Ray Client is already initialized. instead of a bare AssertionError. A later ray.init("ray://<live address>", allow_multiple=True) already connects on master and must keep working.

As a workaround, call ray.util.disconnect() after the failed connect.

Versions / Dependencies

Reproduces on master 18b179a, Python 3.12.13, Linux.

Reproduction script

Run everything in one shell with auth off on both the head and the driver, and keep the head running:

export RAY_AUTH_MODE=disabled
ray start --head
python repro.py

If the driver doesn't set RAY_AUTH_MODE=disabled, the local ray.init() raises AuthenticationError instead, because the driver uses token auth against a cluster that has it disabled. If no head is running, the last call times out because nothing listens on port 10001.

import ray
from ray._common.network_utils import find_free_port

try:
    # Nothing listens on this port.
    ray.init(f"ray://127.0.0.1:{find_free_port()}")
except ConnectionError as e:
    print("ray:// connect failed:", type(e).__name__, e)

try:
    ray.init()  # Today: bare AssertionError in node.py _record_stats
except Exception as e:
    print("local ray.init() failed:", type(e).__name__, e)

# Connects on master today; must keep working after the fix.
ray.init("ray://127.0.0.1:10001", allow_multiple=True)
print("allow_multiple=True connect succeeded")

Issue Severity

Low: It annoys or frustrates me.

Activity

  1. MortalHappiness
    added
    bugSomething that is supposed to be working; but isn't
    triageNeeds triage (eg: priority, bug/not-bug, and owning component)
    on Oct 18, 2026
  2. github-project-automation
    added this to Ray Data OSSon Oct 18, 2026
  3. github-project-automation
    moved this to Todo in Ray Data OSSon Oct 18, 2026
  4. MortalHappiness
    added
    P3Issue moderate in impact or severity
    coreIssues that should be addressed in Ray Core
    good-first-issueGreat starter issue for someone just starting to contribute to Ray
    and removed
    triageNeeds triage (eg: priority, bug/not-bug, and owning component)
    on Oct 18, 2026

  5. justinyeh1995 commented on Oct 18, 2026

    @justinyeh1995
    justinyeh1995
    on Oct 18, 2026
    Contributor

    /take I would like to look into this issue.

  6. machichima
    assigned
    justinyeh1995
    on Oct 18, 2026
  7. MortalHappiness
    changed the title [-][Core] A failed ray:// connect leaves Ray Client mode on, breaking a later local ray.init()[/-] [+][Core] After a failed ray:// connect, ray.init() raises a bare AssertionError and an allow_multiple=True retry times out[/+] on Oct 20, 2026

  8. justinyeh1995 commented on Oct 24, 2026

    @justinyeh1995
    justinyeh1995
    on Oct 24, 2026
    Contributor

    Hi @MortalHappiness, I noticed the second expected behavior is resolved on master while reproducing the errors with the script below

    # stored as ~/adhoc/repro_67142_point_2.py
import ray
from ray._common.network_utils import find_free_port

try:
    ray.init(f"ray://localhost:{find_free_port()}")
except ConnectionError as e:
    print(f"[repro] step 1: failed as expected: {type(e).__name__}: {e}")

try:
    ray.init()
except Exception as e:
    print(f"[repro] step 2: local ray.init() failed: {type(e).__name__}: {e}")

try:
    ray.init("ray://127.0.0.1:10001", allow_multiple=True)
    print("[repro] step 3: CONNECTION SUCCEEDED")
except Exception as e:
    print(f"[repro] step 3: CONNECTION FAILED: {type(e).__name__}: {e}")
    > RAY_AUTH_MODE=disabled ray start --head
> RAY_AUTH_MODE=disabled python ~/adhoc/repro_67142_point_2.py

2026-10-24 17:17:36,358 INFO client_builder.py:262 -- Passing the following kwargs to ray.init() on the server: log_to_driver
/home/justinyeh1995_gmail_com/ray/python/ray/util/client/worker.py:342: UserWarning: Ray Client connection timed out. Ensure that the Ray Client port on the head node is reachable from your local machine. See https://docs.ray.io/en/latest/cluster/ray-client.html#step-2-check-ports for more information.
  warnings.warn(
2026-10-24 17:18:06,407 INFO worker.py:1842 -- Connecting to existing Ray cluster at address: 10.10.0.3:6379...
2026-10-24 17:18:06,419 INFO client_builder.py:262 -- Passing the following kwargs to ray.init() on the server: log_to_driver
SIGTERM handler is not set because current thread is not the main thread.
[repro] step 1: failed as expected: ConnectionError: ray client connection timeout
[repro] step 2: local ray.init() failed: AssertionError:
[repro] step 3: CONNECTION SUCCEEDED

    I haven't looked into what've changed between cd9a46e and master but this could suggest the fix can scope down to just adding the error message.

  9. github-project-automation
    moved this from Todo to In progress in Ray Data OSSon Oct 25, 2026

  10. justinyeh1995 commented on Oct 26, 2026

    @justinyeh1995
    justinyeh1995
    on Oct 26, 2026
    Contributor

    Hi @Kissshot-2003, I've already taken this issue last week and I intent to finish it. Thanks.

  11. justinyeh1995 commented on Oct 27, 2026

    @justinyeh1995
    justinyeh1995
    on Oct 27, 2026
    Contributor

    Hi @MortalHappiness, I'd like to clarify on the expected behavior when the first ray:// connection succeeds and a following ray.init() is called.
    Currently, it will not raise any issue despite it is using client mode under the hood. I assume this should still hold after the fix but I'd like to confirm with you first before jumping into a conclusion.

    I'm a bit unsure cause the expected error message in the description says Ray Client is already initialized, which makes even more sense on the successful execution path.

    Thank you

  12. MortalHappiness commented on Oct 27, 2026

    @MortalHappiness
    MortalHappiness
    on Oct 27, 2026
    MemberAuthor

    Did you run my reproduction script? I ran it again myself and it still fails on master branch. What I want is:

    • For ray.init() # Today: bare AssertionError in node.py _record_stats line, it should not raise AssertionError. Instead, it should be another error type containing Ray Client is already initialized in the error message.
    • For ray.init("ray://127.0.0.1:10001", allow_multiple=True) # Today: ConnectionError, ray client connection timeout line, it should not raise error. It should connect successfully.

  13. justinyeh1995 commented on Oct 28, 2026

    @justinyeh1995
    justinyeh1995
    on Oct 28, 2026
    Contributor

    Thanks for clarifying!

    Did you run my reproduction script? I ran it again myself and it still fails on master branch. What I want is:

    Yes, to confirm this again, I ran the script on master at a450e8d44b938a7523f8d647517e3485b116b233 after rebuilding with bazel run //:gen_ray_pkg.

    ray.init("ray://127.0.0.1:10001", allow_multiple=True) connected without an exception. I ran it on python 3.10 and linux.

    Image
    • For ray.init() # Today: bare AssertionError in node.py _record_stats line, it should not raise AssertionError. Instead, it should be another error type containing Ray Client is already initialized in the error message.

    Got it. I'll narrow #67407 so it only raises after a failed ray:// connection. Thank you.

  14. MortalHappiness
    changed the title [-][Core] After a failed ray:// connect, ray.init() raises a bare AssertionError and an allow_multiple=True retry times out[/-] [+][Core] After a failed ray:// connect, a local ray.init() raises a bare AssertionError[/+] on Oct 28, 2026

  15. MortalHappiness commented on Oct 28, 2026

    @MortalHappiness
    MortalHappiness
    on Oct 28, 2026
    MemberAuthor

    I updated the issue description, please check agian.

  16. justinyeh1995 commented on Oct 28, 2026

    @justinyeh1995
    justinyeh1995
    on Oct 28, 2026
    Contributor

    Thanks. I'll update the pr accordingly.

  17. Kissshot-2003
    added a commit that references this issue on Oct 29, 2026

    [core] Reset client mode after a failed Ray Client connect

    7499532
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P3Issue moderate in impact or severitybugSomething that is supposed to be working; but isn'tcoreIssues that should be addressed in Ray Coregood-first-issueGreat starter issue for someone just starting to contribute to Ray

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions