-
Notifications
You must be signed in to change notification settings - Fork 84
Fix potential connection storm #398
Copy link
Copy link
Closed
Labels
affects/nonePR/issue: this bug affects none version.PR/issue: this bug affects none version.process/fixedProcess of bugProcess of bugseverity/noneSeverity of bugSeverity of bugtype/bugType: something is unexpectedType: something is unexpected
Description
Activity
Metadata
Metadata
Assignees
Labels
affects/nonePR/issue: this bug affects none version.PR/issue: this bug affects none version.process/fixedProcess of bugProcess of bugseverity/noneSeverity of bugSeverity of bugtype/bugType: something is unexpectedType: something is unexpected
This affects 5.2.0.
NebulaPool creates multiple clients. When
strictly_server_healthyis set to false, each client creates a connection to every server. Consider a scenario where one Graphd instance goes down. When callingget_client, iftest_on_borrowis set to true andping_before_executeis set to false, the client will send a ping to one host in a round-robin manner. If the ping happens to be sent to the failed host, the ping will fail, and the client will be evicted, causing all of that client’s connections to all hosts to be closed. Since the ping target is effectively random, it is possible that all clients are evicted because they ping the failed server. When the number of clients in the pool drops below a threshold, the pool will recreate clients. However, the newly created clients may again be closed due to ping failures. This entire process can lead to a connection storm.