Stackexchange.Redis timeouts & socketfailures

Question

I am using Azure Redis (using Stackexchange.Redis) as a cache storage and its generally working fine. But I am getting timeouts errors now and then and I can't nail down why it is happening.

My redis connection settings:

value="dev.redis.cache.windows.net,ssl=true,password=secret,abortConnect=false,syncTimeout=3000"

I am getting all these exception in the same second (multiple calls): [I get these on GET operations aswell. Almost all these exceptions are on StringSet & StringGet. I rarely get exceptions on HashSets or HashGets]

Timeout performing SET {key}, inst: 1, mgr: ExecuteSelect, queue: 6, qu=0, qs=6, qc=0, wr=0/0, in=0/0
SocketFailure on SET
SocketFailure on SET
No connection is available to service this operation: SET 

I am guessing that setting the object is taking longer than expected, this could be due to the object being large so I could potentially increase the synctimeout but would that be hiding some other problem?

I am only getting these exceptions on synchronous calls to stackexchange.redis, I have not seen an exception when the call is asynchronous.

Stacktrace:

StackExchange.Redis.RedisConnectionException: SocketFailure on SET
   at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) i
   at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) 
   at StackExchange.Redis.RedisDatabase.StringSet(RedisKey key, RedisValue value, Nullable`1 expiry, When when, CommandFlags flags) 
at calling method

Edit: I am using StackExchange.Redis 1.0.414 package and I am using MessagePack to serialize my objects

Solution

Timeouts are typically caused by one of a few things. Here are some examples

1. Client or server CPU hitting 100%
2. Poorly configured ThreadPool settings, combined with bursts of traffic
3. Clients sending expensive commands to the server.
4. Maxing out your network Bandwidth (on client or on server)

Tips for Client side issues: https://gist.github.com/JonCole/db0e90bedeb3fc4823c2

Tips for server side issues: https://gist.github.com/JonCole/9225f783a40564c9879d

I would recommend upgrading to a newer version of the StackExchange.Redis also. Version 1.1.603 has some more detailed diagnostic info in the timeout error message that may help you identify some of common client-side the things I listed above.

As for Socket failures, a couple of common causes for connection drops between the client and server that I have seen are:

1. Scaling the client - I have seen brief client side connectivity issues when scaling client apps in Azure.
2. When Redis is patched, there will be some connection blips. Azure Redis patching is explained here: https://gist.github.com/JonCole/317fe03805d5802e31cfa37e646e419d