Redis and Memcache or just Redis?

Question

I'm using memcached for some caching in my Rails 3 app through the simple Rails.cache interface and now I'd like to do some background job processing with redis and resque.

I think they're different enough to warrant using both. On heroku though, there are separate fees to use both memcached and redis. Does it make sense to use both or should I migrate to just using redis?

I like using memcached for caching because least recently used keys automatically get pushed out of the cache and I don't need the cache data to persist. Redis is mostly new to me, but I understand that it's persistent by default and that keys do not expire out of the cache automatically.

EDIT: Just wanted to be more clear with my question. I know it's feasible to use only Redis instead of both. I guess I just want to know if there are any specific disadvantages in doing so? Considering both implementation and infrastructure, are there any reasons why I shouldn't just use Redis? (I.e., is memcached faster for simple caching?) I haven't found anything definitive either way.

Solution 1

Assuming that migrating from memcached to redis for the caching you already do is easy enough, I'd go with redis only to keep things simple.

In redis persistence is optional, so you can use it much like memcached if that is what you want. You may even find that making your cache persistent is useful to avoid lots of cache misses after a restart. Expiry is available also - the algorithm is a bit different from memcached, but not enough to matter for most purposes - see http://redis.io/commands/expire for details.

Solution 2

I'm the author of redis-store, there is no need to use directly Redis commands, just use the :expires_in option like this:

ActionController::Base.cache_store = :redis_store, :expires_in => 5.minutes

The advantage of using Redis is fastness, and with my gem, is that you already have stores for Rack::Cache, Rails.cache or I18n.

Solution 3

I've seen a few large rails sites that use both Memcached and Redis. Memcached is used for ephemeral things that are nice to keep hot in memory but can be lost/regenerated if needed, and Redis for persistent storage. Both are used to take a load off the main DB for reading/write heavy operations.

More details:

Memcached: used for page/fragment/response caching and it's ok to hit the memory limit on Memcached because it will LRU (least recently used) to expire the old stuff, and frequently keep accessed keys hot in memory. It's important that anything in Memcached could be recreated from the DB if needed (it's not your only copy). But you can keep dumping things into it, and Memcached will figure which are used most frequently and keep those hot in memory. You don't have to worry about removing things from Memcached.

redis: you use this for data that you would not want to lose, and is small enough to fit in memory. This usually includes resque/sidekiq jobs, counters for rate limiting, split test results, or anything that you wouldn't want to lose/recreate. You don't want to exceed the memory limit here, so you have to be a little more careful about what you store and clean up later.

Redis starts to suffer performance problems once it exceeds its memory limit (correct me if I'm wrong). It's possible to solve this by configuring Redis to act like Memcached and LRU expire stuff, so it never reaches its memory limit. But you would not want to do this with everything you are keeping in Redis, like resque jobs. So instead of people often keep the default, Rails.cache set to use Memcached (using the dalli gem). And then they keep a separate $redis = ... global variable to do redis operations.

# in config/application.rb
config.cache_store = :dalli_store  # memcached

# in config/initializers/redis.rb
$redis = $redis = Redis.connect(url: ENV['REDIS_URL'])

There might be an easy way to do this all in Redis - perhaps by having two separate Redis instances, one with an LRU hard memory limit, similar to Memcache, and another for persistent storage? I haven't seen this used, but I'm guessing it would be doable.

Solution 4

I would consider checking out my answer on this subject:

Rails and caching, is it easy to switch between memcache and redis?

Essentially, through my experience, I would advocate for keeping them separate: memcached for caching and redis for data structures and more persistant storage

Solution 5

I asked the team at Redis Labs (who provide the Memcached Cloud and Redis Cloud add ons) about which product they would recommend for Rails caching. They said that in general they would recommend Redis Cloud, that Memcached Cloud is mainly offered for legacy purposes, and pointed out that their Memcached Cloud service is in fact build on top of Redis Cloud.

View more solutions

Comments

[markquezada]

Thanks for the answer and the relevant docs. I edited my question a bit to make it more clear what I'm looking for. I didn't think about the cache persistence after a restart... that is a nice feature. (Although I'm not sure how useful it is considering I'd be using redis to go with heroku.)

[Tom Clarkson]

That's not entirely true - redis uses multiple instances rather than multiple threads, so comparing a single core redis instance with a multi core memcached instance isn't a particularly relevant test. Besides, when benchmarks are available showing both systems to be the fastest the difference is unlikely to matter in the real world.

[Ludger Sprenker]

The multi process approach of redis scales better on multi core systems than the (default) approach of memcached with multi-threading: antirez.com/post/update-on-memcached-redis-benchmark.html

[markquezada]

Although I initially planned to consolidate them based on the original answer to my question, I have since come to basically the same conclusion. I'm using memcache for basic caching and redis for resque.

[markquezada]

Thanks for the response Luca. I'm assuming you can override :expires_in directly when you need to store something persistently? I guess the question is really about using both memcached type functionality and using redis as a persistent store alongside each other.

[Luca Guidi]

I don't think it makes sense to have both memcached and redis alongside, since they are in the same NoSQL segment: both are a k/v store. Redis PROS: 1. faster than memcached 2. more powerful commands 3. no cache warmup needed 4. useful for solving other problems (eg. queues with Resque) CONS: 1. you need an external gem 2. after a restart, the server doesn't accepts commands while reading data from append file Memcached PROS: baked in Rails CONS: 1. slower than Redis 2. cache warmup.

[ColinM]

This major shortcoming of Redis is really downplayed a lot. If you are looking for high-concurrency and have many cores, Memcached can run circles around Redis. Sharding is not a good solution since you have to implement consistent hashing in your application and you won't get perfect distribution between Redis instances. For example if shard A caches your application's config that is used in every request, shard A will get more requests than the other shards which are not hit for every request.

[Chris Vincent]

@LucaGuidi I'm presently using RedisStore for my Rails cache. I can't figure out how to set both the Redis URL and the default :expires_in. Can you help?

[Mindey I.]

If you want to keep some some items ephemeral (fragment caching) and some items persistent in redis, do you need to create two separate redis instances?

[Anirudhan J]

Like me, If you are facing issue setting :expires_in using redis_store refer stackoverflow.com/questions/20907247/…

[Marklar]

So no need for both memcached and redis as mentioned in Brian's answer? "people often keep the default Rails.cache set to use memcached (using the dalli gem). And then they keep a separate $redis = ... global variable to do redis operations."

[jwadsack]

While we use Redis for both jobs queues and cache we run them with different configurations for specifically the reasons that @BrianArmstrong states. Cache is ephemeral so we configure it with it being fast, but it's ok to lose things in memory and drop LRU. For the queue, we really don't want to lose jobs so we configure it with persistence so that it's recoverable.

[Marklar]

@jwadsack do you have a post where you show how you implemented this configuration?

[jwadsack]

@Marklar Good idea. I do now: ballardhack.wordpress.com/2015/09/30/…