Conversation
…inedByRemote Motivation: akka/akka-core#32117 (backport of akka/akka-core#32050, issue akka/akka-core#31095) fixed InboundQuarantineCheck replying with a Quarantined control message for later inbound messages from a harmlessly quarantined (gracefully shut down) node, which made the remote node down itself via DownSelfQuarantinedByRemote. Pekko already stops propagating harmless quarantines by default (apache#1555, propagate-harmless-quarantine-events = off), but the multi-jvm coverage for this scenario was missing. Modification: - DowningWhenOtherHasQuarantinedThisActorSystemSpec: quarantine the second node directly instead of relying on split brain resolver timing, add a fourth node and a test that a node shutting down during a network partition does not trigger ThisActorSystemQuarantinedEvent on the surviving side. - Association / InboundQuarantineCheck: log at info when a Quarantined control message is sent, and mention "Harmless quarantine" in the harmless quarantine log message, matching Akka. - TestContext: pass harmless = false explicitly. Result: Multi-jvm coverage of the DownSelfQuarantinedByRemote scenarios, including harmless quarantine during a network partition. No behavioural change beyond log messages. Tests: - sbt "project cluster" "MultiJvm/testOnly org.apache.pekko.cluster.DowningWhenOtherHasQuarantinedThisActorSystem" (4 nodes, 4 tests each, all passed) - sbt "remote/testOnly org.apache.pekko.remote.artery.HarmlessQuarantineSpec org.apache.pekko.remote.artery.OutboundIdleShutdownSpec org.apache.pekko.remote.artery.RemoteMessageSerializationSpec" (14 passed) - native scalafmt run on the changed files References: Refs apache#1555 - port of akka/akka-core#32117 (the InboundQuarantineCheck fix was already present)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Port of akka/akka-core#32117 (backport of akka/akka-core#32050, issue akka/akka-core#31095). That fix stopped
InboundQuarantineCheckreplying with aQuarantinedcontrol message for later inbound messages from a harmlessly quarantined (gracefully shut down) node, which made the remote node down itself viaDownSelfQuarantinedByRemote- notably during a network partition.Pekko already stops propagating harmless quarantines by default: #1555 added
pekko.remote.artery.propagate-harmless-quarantine-events = off, andQuarantinedTimestamp.harmless/newQuarantined(harmless)already exist, so no MiMa filters are needed. What was missing was the multi-jvm coverage of the end-to-end cluster scenario.Modification
DowningWhenOtherHasQuarantinedThisActorSystemSpec: ported the Akka rewrite. Quarantinesseconddirectly viaRARP(system).provider.quarantine(...)instead of relying on split brain resolver timing, assertssecondreceivesThisActorSystemQuarantinedEventand terminates within 5 s (shorter thanstable-after, so it is not normal downing), adds afourthnode and a new case "not be triggered by another node shutting down during network partition" - the actual #31095 scenario. Pekko's Arterypendingguard andThrottlerTransportAdapter.Directionusage are kept.Association/InboundQuarantineCheck: log at info when aQuarantinedcontrol message is sent, and mention "Harmless quarantine." in the harmless quarantine log message, matching Akka. Log-only.TestContext: passharmless = falseexplicitly, as in Akka.Result
Multi-jvm coverage of the
DownSelfQuarantinedByRemotescenarios, including harmless quarantine during a network partition. No behavioural change beyond log messages.Tests
sbt "project cluster" "MultiJvm/testOnly org.apache.pekko.cluster.DowningWhenOtherHasQuarantinedThisActorSystem"- 4 nodes, 4 tests each, all passedsbt "remote/testOnly org.apache.pekko.remote.artery.HarmlessQuarantineSpec org.apache.pekko.remote.artery.OutboundIdleShutdownSpec org.apache.pekko.remote.artery.RemoteMessageSerializationSpec"- 14 passedReferences
Refs #1555 - port of akka/akka-core#32117 (the
InboundQuarantineCheckfix was already present). Akka source is Apache-2.0 licensed as of 2023-09-20.