Skip to content

Pub/Sub: don't orphan a subscribe that races the last unsubscribe of the same channel - #3363

Open
aradalvand wants to merge 1 commit into
seredis:mainfrom
aradalvand:fix/concurrent-last-unsubscribe-orphans-subscriber
Open

aradalvand wants to merge 1 commit into
seredis:mainfrom
aradalvand:fix/concurrent-last-unsubscribe-orphans-subscriber

Conversation

@aradalvand

@aradalvand aradalvand commented Nov 19, 2026 •

Copy link
Copy Markdown

Fixes #3362.

Subscribing a handler, and unsubscribing a channel's last handler, each take several steps that weren't atomic with respect to each other. When they interleave on the same channel, the new handler can be added to a Subscription that is concurrently being removed. Nothing routes messages to it anymore, and the server has been told to stop sending them. SubscribeAsync still completes successfully, so the handler silently never fires.

Changes

Registration lock. Subscription gets a registration lock and a "retired" flag.

  • Adding a handler (TryAdd) and retiring the subscription after its last handler is removed (RetireIfEmpty) are now mutually exclusive.
  • A subscribe that finds the subscription retired looks it up again, and gets a fresh one (ConnectionMultiplexer.AddToSubscription).

Ordering on the wire. While still holding that lock, the retiring side first issues the unsubscribe command to the server, and only then removes the subscription from the multiplexer.

  • Until the unsubscribe command has been issued, a racing subscribe still finds the old subscription, so it has to wait for the lock.
  • Its subscribe command is therefore issued after the unsubscribe command. Without this, the subscribe could overtake it, leaving the server unsubscribed while the client considers itself subscribed. I observed this second variant too, once the first was fixed.
  • This relies on commands to the same bridge being written in the order they're issued.

Multi-node unsubscribe. MultiNodeSubscription.UnsubscribeFromServerAsync now issues the command to every server before awaiting any of them, so that all of them are issued while the lock is held.

Unsubscribe-all. UnsubscribeAll/UnsubscribeAllAsync go through the same retire-then-remove path.

Things worth a reviewer's eye

Testing

  • New SubscriptionRegistrationRaceUnitTests. It uses the in-process test server: 300 attempts at unsubscribing a channel's last handler while another handler subscribes to the same channel, then publishing and checking that it arrives.
    • On main, it consistently misses some messages.
    • With this change it passes. In a temporary diagnostic variant, it lost 0 of 900 attempts, and 0 in the strictly sequential and single-thread pipelined variants.
  • Against a real Redis 8 server, with the repro from Pub/Sub: a subscribe racing the last unsubscribe of the same channel can be silently orphaned #3362: 8 of 300 orphaned with 3.4.20, 0 of 600 with this branch.
  • Existing tests: all *UnitTests classes pass (1221), and all *PubSub* classes pass against a local primary/replica (91 passed, 93 skipped, needing cluster or sentinel). Both on net10.0.

Checklist

  • I fully and freely contribute this code in accordance with the project license (and am legally able to do so)
  • I take responsibility for this contribution's quality and correctness, including any portions produced with AI assistance (see CONTRIBUTING.md).

…the same channel

Subscribing (looking up the channel's subscription, then adding the handler)
and unsubscribing the last handler (removing it, removing the now-empty
subscription from the multiplexer, then unsubscribing from the server) were
not atomic with respect to each other. Interleaved, the handler could be added
to a subscription that was being removed: nothing routes messages to it
anymore, and the server is told to stop sending them; yet the subscribe
completes successfully, so the handler silently never fires.

A per-subscription registration lock now makes adding a handler and retiring
the subscription mutually exclusive; a subscribe that finds the subscription
retired looks up a fresh one instead. The retiring side also issues the
unsubscribe command to the server (and only then removes the subscription from
the multiplexer) while holding the lock, so that the racing subscribe's command
is issued after it, rather than being able to overtake it and leave the server
unsubscribed while the client considers itself subscribed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XUNs4qcAQaGCRpkwmr8LnY
@aradalvand
aradalvand marked this pull request as ready for review November 19, 2026 17:47
@aradalvand

Copy link
Copy Markdown
Author

Some additional evidence, in case it helps the review: I hit both variants through a higher-level usage pattern, not just the minimal repro in #3362, and measured the effect of this branch.

Setup

  • What each iteration does:
    • A sharded channel (RedisChannel.Sharded) has exactly one subscribed handler (A).
    • At the same instant, a new handler (B) is subscribed (SubscribeAsync, awaited), and A is unsubscribed (UnsubscribeAsync). A is the channel's last handler, so this sends SUNSUBSCRIBE.
    • Then a message is published, and B must receive it within 2s.
  • Redis: 8.10.2, standalone, in Docker. Client on .NET (net10.0 build), Linux.
  • Builds compared, with the same test binary and only StackExchange.Redis.dll/RESPite.dll swapped:
    • the released 3.4.20 package;
    • main at f797d55 (this PR's base), built Release;
    • this PR's head, bb0d88f, built Release.
  • What's recorded on a miss:
    • whether the channel is still in the multiplexer's subscription map (read by reflection);
    • PUBSUB SHARDNUMSUB for the channel.

Results

Build Iterations B missed the message
3.4.20 (NuGet) 4,800 79
main @ f797d55 1,800 5
this PR @ bb0d88f 1,800 0

Every miss had one of two fingerprints. They match the two variants described in this PR:

  • Channel absent from the multiplexer's map, SHARDNUMSUB = 0. B's handler was added to the Subscription that was being retired: the orphan case.
  • Channel present in the map, SHARDNUMSUB = 0. The client considers itself subscribed, but the server processed the SUNSUBSCRIBE after B's SSUBSCRIBE: the overtaking case.

The second variant reproduces on the released 3.4.20 and comes in bursts. Across three 1,500-iteration runs, it accounted for 41 of 42 misses in one run, none in another, and the bulk of a third. In the 1,800 iterations against main, all misses were the first variant. With this PR, neither showed up.

Two more observations:

  • No event ever fires in either case. So application-level resync logic keyed on ConnectionRestored/ConfigurationChanged can't recover from it; the handler simply stays deaf.
  • In practice, this showed up as rare missed updates. Many short-lived "wait for the next update on key X" subscriptions on the same channel kept creating this interleaving.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pub/Sub: a subscribe racing the last unsubscribe of the same channel can be silently orphaned

1 participant