How to solve a Redis timeout on client side with StackExchange.Redis?

Question

I'm using StackExchange.Redis client in a c# wcf application : I use only the synchronous command to get and set values. The problem is I have a timeout with this curious log :

Timeout performing EXISTS DataKey:50, 
inst: 1, queue: 1, qu: 0, qs: 1, qc: 0, wr: 0, wq: 0, in: 0, ar: 0, clientName: Machine1, 
serverEndpoint: redis-server:6381, keyHashSlot: 7984, 
IOCP: (Busy=1,Free=799,Min=8,Max=800), 
WORKER: (Busy=5,Free=795,Min=8,Max=800) 
(Please take a look at this article for some common client-side issues that can 
cause timeouts: http://stackexchange.github.io/StackExchange.Redis/Timeouts) 

If I understand that correctly it means that my get value is queued because there is five worker threads ? Using netstat, I see that my application is opening two physical connections to the server . I have made sure to have enought threads available in my threadpool. In my connection settings I have a syncTimeout=3000... If I use the redis-cli, I could get the value of the key in 0.64 secondes.

Can anyone help please? What can I do? Does I have to use async in my code all the way or find another redis client lib ?

Solution 1

There are a few solutions. First of all they have a writeup on the github repo about the solutions they propose, found here.

What I've found though is just use the async functions as much as you can, it is only the synchronous functions that timeout. It seems to me that the timeouts are a design decision on their part based on the idea that they don't want to block your code if something goes wrong. I don't buy that, and so the work around I use is to use the async functions and then wait for the task. So instead of db.StringGet("thestring") I just do db.StringGetAsync("thestring").Result. That is, if I cannot use await for whatever reason.

You might want to look into using async/await as much as possible anyway. You might also want to use FireAndForget if you that is appropriate. You can also use ContinueWith. They talk about those solutions here.

Solution 2

From "WORKER: (Busy=5,Free=795,Min=8,Max=800)", I see that there are 5 Busy worker threads and a minimum of only 8 worker threads. I suspect that if you look at the whole set of errors that you received, you might find that Busy is greater than the Min count in some cases. This would indicate threadpool throttling. The link also provides some solutions on increasing the Min Thread counts for Worker and IOCP threads.

Solution 3

fixing RedisTimeout exception is always a challenge, there any many factors that cause timeout Exception,

in your log you can see for IOCP thread there are 1 busy threads and your system is configured to allow 8 minimum threads. so it's not the case because 1 is not > 8

once I had the same situation and I fixed timeout exceptions by increasing syncTimeout in the connection string, you could try to increase it

I also realized that number of open connections affect Redis performance, you can look at open connections by running this command

info clients

I had around 90 open connections at the time that timeout exception was thrown, I restarted the server to reset open connection and timeout exceptions disappeared

Comments

[Dypso]

I have already read those solutions and I will give a try with the async version and then wait for the task... We are not using async code because my team is not well versed in async/await pattern and we fear to have many bugs due to asynchronicity... On another hand calling the asyncronous function and wait on it seems weird to me : doing so I would pay the time to switch on another thread, wait and then switch back to the main thread no ?

[Sebastian Zander]

I'm not sure what you mean, but it is no different than what happens with the synchronous functions. All that happens is that the current thread waits until the call is completed and then continues, but that is exactly what happens with the synchronous calls.

[PvPlatten]

You have no idea how much this answer helped me. Thank you!

[Topher Birth]

To backup Sebastian's statement: Asynchronous is not the same as parallelism.