Conversation
…ard remove by live register SessionRegistry.add() invoked the kicked register's callback inside the tenantSessions.compute() block. The real register answers kick() by unregistering itself synchronously on the same thread, re-entering compute for the same key; on the same-owner takeover branch the nested remove() deleted the mapping the add had just installed, silently losing the new session from the dictionary (subsequent kill could not find it). Two changes: - collect the kick and perform it after leaving the compute block; the notification is best-effort (the previous stream may already be closed) - remove() only drops the dict entry when the removed register is still the live one for that owner, so the delayed teardown of a taken-over register keeps the new session (and the counters) intact Fixes apache#304
…eardown testKickCallbackReenteringRemoveKeepsNewSession and testReAddSameOwnerWithReenteringKickKeepsNewSession cover a kick callback that re-enters remove() on the same thread; testDelayedTeardownOfTakenOverRegisterKeepsNewSession covers the old register's delayed teardown landing after the new register was added. The first three fail on the unfixed code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SessionRegistry.add()invokes the kicked register's callback inside thetenantSessions.compute()block. The real register answerskick()by unregistering itself synchronously on the same thread, re-enteringcompute()for the same key — on the same-owner takeover branch that nestedremove()deletes the mappingadd()has just installed, silently losing the new session from the dictionary (a subsequentkillcannot find it).Related failure mode: when a taken-over register is torn down late (its stream is closed by the client only after it receives the Quit), its
remove()can delete the mapping of the replacement session registered by the same clientId — session-dictionary counts drift.Trigger: same-clientId rapid reconnect, routine for diskless clients that reboot (and thus reconnect) frequently.
Fix
Commit 1 (
fix):remove()only drops the dictionary entry when the removed register is still the live one for that owner, so a delayed teardown cannot delete the new session (or its counters).Test evidence
Commit 2 (
test) adds three regression tests:testKickCallbackReenteringRemoveKeepsNewSession— kick callback re-entersremove()on the same thread; the new session must survive.testReAddSameOwnerWithReenteringKickKeepsNewSession— same owner re-registered with a different register; the nested remove must not delete the fresh mapping.testDelayedTeardownOfTakenOverRegisterKeepsNewSession— the old register's teardown lands after the takeover; the entry and counters must stay with the live register.All three fail on the unfixed code; the full session-dict suite (18/18) passes with the fix.
Production validation
Deployed in a rolling 6-node cluster (wholesale
lib/replacement, one node at a time, zero rollbacks). The rolling restarts forced all 13 live clients (including the bridge replicas) through same-clientId rapid reconnects; reconnects landed cleanly with no session-dictionary inconsistency observed.Fixes #304