How to cache large objects using Redis cache

Question

Our current caching implementation caches large amounts of data in report objects (50MB in some cases).

We’ve moved from memory cache to file cache and use ProtoBuf to serialize and de-serialize. This works well, however we are now experimenting with Redis cache. Below is an example of how much longer it takes for Redis than using the file system. (Note: using protobuf instead of JsonConvert improves set time to 15 seconds and get time to 4 seconds in the below example, when setting a byte array).

// Extremely SLOW – caching using Redis (JsonConvert to serialize/de-serialize)
IDatabase cache = Connection.GetDatabase();

// 23 seconds!
cache.StringSet("myKey", JsonConvert.SerializeObject(bigObject));

// 5 seconds!
BigObject redisResult = JsonConvert.DeserializeObject<BigObject>(cache.StringGet("myKey")); 




// FAST - caching using file system (protobuf to serialize/de-serialize)
IDataAccessCache fileCache = new DataAccessFileCache();

// .5 seconds
fileCache.SetCache("myKey",bigObject); 

// .5 seconds                                          
BigObject fileResult = fileCache.GetCache<BigObject>("myKey");                              

Thanks in advance for any help.

ps. I didn’t find an answer from similar questions asked. Caching large objects - LocalCache performance

or

Caching large objects, reducing impact of retrieval times

Solution

Redis actually is not designed for storing large objects (many MB) because it is a single-thread server. So, one request will be fast enough, but a few requests will be slow because they all will be processed by one thread. In the last versions some optimizations were done.

Speed of RAM and memory bandwidth seem less critical for global performance especially for small objects. For large objects (>10 KB), it may become noticeable though. Usually, it is not really cost-effective to buy expensive fast memory modules to optimize Redis. https://redis.io/topics/benchmarks

So, you can use Jumbo frames or buy a faster memory if it is possible. But actually it won't help significantly. Consider using Memcached instead. It is multi-threaded and can be scaled out horizontally to support large amount of data. Redis can be scaled only with master-slave replication.