StackExchange.Redis with Azure Redis is unusably slow or throws timeout errors

Question

I'm moving all of my existing Azure In-Role cache use to Redis and decided to use the Azure Redis preview along with the StackExchange.Redis library (https://github.com/StackExchange/StackExchange.Redis). I wrote all the code for it without much problem, but when running it is absolutely unusably slow and constantly throws timeout errors (my timeout period is set to 15 seconds).

Here is the relevant code for how I am setting up the Redis connection and using it for simple operations:

    private static ConnectionMultiplexer _cacheService;
    private static IDatabase _database;
    private static object _lock = new object();

    private void Initialize()
    {
        if (_cacheService == null)
        {
            lock (_lock)
            {
                if (_cacheService == null)
                {
                    var options = new ConfigurationOptions();
                    options.EndPoints.Add("{my url}", 6380);
                    options.Ssl = true;
                    options.Password = "my password";
                    // needed for FLUSHDB command
                    options.AllowAdmin = true;

                    // necessary?
                    options.KeepAlive = 30;
                    options.ConnectTimeout = 15000;
                    options.SyncTimeout = 15000;

                    int database = 0;

                    _cacheService = ConnectionMultiplexer.Connect(options);
                    _database = _cacheService.GetDatabase(database);
                }
            }
        }

    }

    public void Set(string key, object data, TimeSpan? expiry = null)
    {
        if (_database != null)
        {
            _database.Set(key, data, expiry: expiry);
        }
    }

    public object Get(string key)
    {
        if (_database != null)
        {
            return _database.Get(key);
        }
        return null;
    }

Performing very simple commands like Get and Set often time out or take 5-10 seconds to complete. Seems like it kind of negates the whole purpose of using it as a cache if it's WAY slower than actually fetching the real data from my database :)

Am I doing anything obviously incorrect?

Edit: here are some stats that I pulled from the server (using Redis Desktop Manager) in case that sheds some light on anything.

Server
redis_version:2.8.12
redis_mode:standalone
os:Windows  
arch_bits:64
multiplexing_api:winsock_IOCP
gcc_version:0.0.0
process_id:2876

tcp_port:6379
uptime_in_seconds:109909
uptime_in_days:1
hz:10
lru_clock:16072421
config_file:C:\Resources\directory\xxxx.Kernel.localStore\1\redis_2092_port6379.conf

Clients
connected_clients:5
client_longest_output_list:0
client_biggest_input_buf:0
client_total_writes_outstanding:0
client_total_sent_bytes_outstanding:0
blocked_clients:0

Memory
used_memory:4256488
used_memory_human:4.06M
used_memory_rss:67108864
used_memory_rss_human:64.00M
used_memory_peak:5469760
used_memory_peak_human:5.22M
used_memory_lua:33792
mem_fragmentation_ratio:15.77
mem_allocator:dlmalloc-2.8

Persistence
loading:0
rdb_changes_since_last_save:72465
rdb_bgsave_in_progress:0
rdb_last_save_time:1408471440
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok

Stats
total_connections_received:25266
total_commands_processed:123389
instantaneous_ops_per_sec:10
bytes_received_per_sec:275
bytes_sent_per_sec:65
bytes_received_per_sec_human:

Edit 2: Here are the extension methods I'm using for Get/Set -- they are very simple methods that just turn an object into JSON and call StringSet.

    public static object Get(this IDatabase cache, string key)
    {
        return DeserializeJson<object>(cache.StringGet(key));
    }

    public static void Set(this IDatabase cache, string key, object value, TimeSpan? expiry = null)
    {
        cache.StringSet(key, SerializeJson(value), expiry: expiry);
    }

Edit 3: here are a couple example error messages:

    A first chance exception of type 'System.TimeoutException' occurred in StackExchange.Redis.dll
    Timeout performing GET MyCachedList, inst: 11, queue: 1, qu=1, qs=0, qc=0, wr=0/1, in=0/0

    A first chance exception of type 'System.TimeoutException' occurred in StackExchange.Redis.dll
    Timeout performing GET MyCachedList, inst: 1, queue: 97, qu=0, qs=97, qc=0, wr=0/0, in=3568/0

Solution 1

Here is the recommended pattern, from the Azure Redis Cache documentation:

private static Lazy<ConnectionMultiplexer> lazyConnection = new Lazy<ConnectionMultiplexer>(() => {
    return ConnectionMultiplexer.Connect("mycache.redis.cache.windows.net,abortConnect=false,ssl=true,password=...");
});

public static ConnectionMultiplexer Connection {
    get {
        return lazyConnection.Value;
    }
}

A few important points:

• It uses Lazy<T> to handle thread-safe initialization
• It sets "abortConnect=false", which means if the initial connect attempt fails, the ConnectionMultiplexer will silently retry in the background rather than throw an exception.
• It does not check the IsConnected property, since ConnectionMultiplexer will automatically retry in the background if the connection is dropped.

Solution 2

I was having similar issues. Redis cache was unusually slow but was definitely caching. In some cases, it took 20-40 seconds to load a page.

I realized that the cache server was in a different location than the site's. I updated the cache server to live in the same location as the website and now everything works as expected.

That same page now loads in 4-6 seconds.

Good luck to anyone else who's having these issues.

Solution 3

In our case the issue is when using SSL connection. You're showing that your desktop manager is running on the non-SSL port, but your code is using SSL.

A quick benchmark on our Azure redis without SSL, retrieving around 80k values with an LRANGE command (also with .net and StackExchange.Redis) is basically instand. When SSL is used, the same query takes 27 seconds.

• WebApp: Standard S2

• Redis: Standard 1 GB

Edit: Checking the SLOWLOG, Redis itself seems to actually hit slowlog with it's 14ms time it takes or so to grab the rows, but this is far from the actual transfer with SSL enabled. We ended up with a premium Redis to have some sort of security between Redis and Web Apps.

Solution 4

The problem is how the connection object created and used. we faced exact problem initially and fixed with a single connection object getting used across all web requests. And we check is it null or connected in session start for graceful re creating object. that fixed the issue.

Note: Also check in which Zone of Azure your Redis Cache instance is running and Which Zone your Web Server exist. It is better to maintain both in Same Zone

In Global.ascx.cs file

public static ConnectionMultiplexer RedisConnection;
public static IDatabase RedisCacheDb;

protected void Session_Start(object sender, EventArgs e)
    {
        if (ConfigurationManager.ConnectionStrings["RedisCache"] != null)
        {
            if (RedisConnection == null || !RedisConnection.IsConnected)
            {
                RedisConnection = ConnectionMultiplexer.Connect(ConfigurationManager.ConnectionStrings["RedisCache"].ConnectionString);
            }
            RedisCacheDb = RedisConnection.GetDatabase();
        }
    }

Solution 5

It worked in my case. Don't forget to increase the SyncTimeout. The default is 1 second.

private static Lazy<ConnectionMultiplexer> ConnectionMultiplexerItem = new Lazy<ConnectionMultiplexer>(() =>
{
    var redisConfig = ConfigurationOptions.Parse("mycache.redis.cache.windows.net,abortConnect=false,ssl=true,password=...");

    redisConfig.SyncTimeout = 3000;

    return ConnectionMultiplexer.Connect(redisConfig);
});

Check if you your Azure Redis Cache and the Client in the same region in Azure. For example, you might be getting timeouts when your cache is in East US but the client is in West US and the request doesn't complete in synctimeout time or you might be getting timeouts when you are debugging from your local development machinex. It’s highly recommended to have the cache and in the client in the same Azure region. If you have a scenario to do a cross region calls, you would want to set the synctimeout to a higher value.

Read more: https://azure.microsoft.com/en-us/blog/investigating-timeout-exceptions-in-stackexchange-redis-for-azure-redis-cache/

View more solutions