Repository navigation
Description
When unsubscribing a channel's last handler fails, for example with a BUSY reply while a long script runs, the client has already forgotten the subscription, but the server is still subscribed. The server keeps pushing that channel's messages to the connection, and the client silently discards them. Nothing reconciles the two until the subscription connection is replaced. A second UnsubscribeAsync can't fix it either, because there's no longer anything to unsubscribe client-side, so nothing is sent.
Repro
StackExchange.Redis 3.4.20, standalone Redis 8.10.2. Also reproduces with the head of #3363 (bb0d88f), which doesn't touch this.
var admin = await ConnectionMultiplexer.ConnectAsync("localhost:6379,allowAdmin=true");
var conn = await ConnectionMultiplexer.ConnectAsync("localhost:6379");
var subscriber = conn.GetSubscriber();
var channel = RedisChannel.Literal("leak-test");
var received = 0;
Action<RedisChannel, RedisValue> handler = (_, _) => Interlocked.Increment(ref received);
await subscriber.SubscribeAsync(channel, handler);
async Task<long> ServerSideSubscribers() => (long)((RedisResult[])(await admin.GetDatabase().ExecuteAsync("PUBSUB", "NUMSUB", "leak-test"))!)[1];
// make the server reply BUSY to everything for ~1.5s
await admin.GetServer("localhost:6379").ExecuteAsync("CONFIG", "SET", "busy-reply-threshold", "100");
var busy = admin.GetDatabase().ScriptEvaluateAsync("local s=redis.call('TIME') local a=s[1]*1000000+s[2] while true do local t=redis.call('TIME') if (t[1]*1000000+t[2])-a > 1500000 then break end end return 1");
await Task.Delay(300);
try { await subscriber.UnsubscribeAsync(channel, handler); } // last handler
catch (Exception e) { Console.WriteLine(e.Message); } // BUSY Redis is busy running a script...
await busy;
Console.WriteLine(subscriber.SubscribedEndpoint(channel)); // (null)
Console.WriteLine(await ServerSideSubscribers()); // 1
Console.WriteLine(await admin.GetSubscriber().PublishAsync(channel, "hello")); // 1 - delivered to this client
// ...but `received` stays 0, and a second UnsubscribeAsync sends nothing: PUBSUB NUMSUB stays at 1Output, for both 3.4.20 and bb0d88f:
subscribed: server-side subscribers = 1
UnsubscribeAsync (while BUSY): threw RedisServerException: BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.
after: client-side SubscribedEndpoint = null, server-side subscribers = 1
publish: server delivered to 1 client(s); handler invoked 0 time(s)
after a second UnsubscribeAsync: server-side subscribers = 1
Cause
Unsubscribe/UnsubscribeAsync first call UnregisterSubscription. When the last handler is removed, that removes the Subscription from the multiplexer (TryRemoveSubscription) (RedisSubscriber.cs#L330-L350). Only after that do they issue the server-side unsubscribe (#L317-L324). If that command fails, nothing restores the client-side record or retries the unsubscribe.
Which failures leave it subscribed:
- An error reply, where the server declined to run it:
BUSY, but presumably alsoLOADING,MASTERDOWNand so on. Those are the "known not applied" kinds inFaultContext. - A timeout: the outcome is ambiguous. In my tests, a timed-out unsubscribe that's still in the pipe is eventually applied once the connection recovers.
- Not a connection drop: the new connection simply doesn't re-subscribe.
Impact
- The leaked subscription costs bandwidth and server-side buffers until the connection is replaced. For a busy channel, that can be indefinite.
- It isn't visible from the client:
SubscribedEndpointandIsConnected(channel)say unsubscribed. - It's most likely during exactly the conditions where callers tear things down: a slow or overloaded server.
Possible directions
- Keep the client-side
Subscription(marked as retiring) until the server confirms the unsubscribe, and re-issue it on failure, or on the next opportunity, such as a reconnect or heartbeat. - Or, when the unsubscribe fails with a "not applied" kind, put the subscription back so that a later
UnsubscribeAsyncstill has something to act on.
This interacts with the ordering in #3363, where retiring a subscription and issuing its unsubscribe happen under one lock. So it may be easiest to address on top of that.
Activity
Pub/Sub: retry a last unsubscribe the server declined, rather than le…