How to count the number of keys matching a pattern?

Question

How can I find the count of all the keys that has a matching pattern.

For example, there are two keys abc:random-text-1 and abc:random-text-2 . The common pattern here isabc: . So, here the count is 2.

How can I do this in redis?

Solution 1

DISCLAIMER I hope this old answer haven't damaged any production systems, with millions of keys. If you still want to still count the matching keys of redis in production for some reason, better use scan with a match pattern.

If you simply search with KEYS, with your redis client, you will get a number list of all you matching keys, right?

e.g.

KEYS abc:*

will give you

1) abc:random-text-1
2) abc:random-text-2

or you can run the following:

./redis-cli KEYS "abc:*" | wc -l

and you will get 2 as an output.

Solution 2

From here:

eval "return #redis.pcall('keys', 'abc:*')" 0

It's not O(1), but at least the count is done on the server side.

Solution 3

From the command line, redis-cli --scan --pattern 'abc:*' | wc -l

Solution 4

By considering the performance, I would not recommend you use KEYS

Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases. This command is intended for debugging and special operations, such as changing your keyspace layout. Don't use KEYS in your regular application code. If you're looking for a way to find keys in a subset of your keyspace, consider using sets.

I would suggest you considering scan, if your redis version > 2.8.0. But it rely on which data type you are going to use.

Here is an simple example from redis doc:

redis 127.0.0.1:6379> sadd myset 1 2 3 foo foobar feelsgood
(integer) 6
redis 127.0.0.1:6379> sscan myset 0 match f*
1) "0"
2) 1) "foo"
   2) "feelsgood"
   3) "foobar"

Solution 5

If it's a one-time thing, you can use KEYS as described by x_maras, but you shouldn't use that in your code since KEYS will scan every key in the entire database each time it's called.

If you want to do it frequently, there is no "good" way exactly as you've written because it will always be fairly inefficient to scan every key (even using SCAN, since it would be doing the same thing as KEYS just in a safer manner).

However, if the patterns you need are known ahead of time, you can keep a set of every key that matches the pattern.

SET abc:random-text-1 "blah"
SADD patterns:abc abc:randomtext-1

SET abc:random-text-2 "more blah"
SADD patterns:abc abc:randomtext-2

SCARD patterns:abc
// (integer) 2

SORT patterns:abc BY nosort GET *
// 1) "blah"
// 2) "more blah"

View more solutions

Comments

[Amin Shojaei]

Counting keys in the Redis in production is a bad idea, forever.

[Natim]

This is exactly what I was looking for.

[hasen]

How would you use scan to count keys?

[Joshua]

Either your can read official doc: redis.io/commands/scan. Or google it: stackoverflow.com/questions/33166812/…

[hasen]

Well, Googling took me here :) It would be helpful if you could edit your answer to include a simple command using scan for counting the number of keys matching a pattern.

[Joshua]

okay, I take your suggestion. :-) It might be easier for the others too.

[Viren]

@Joshua Looking at scan api does not look like you can find the total number of count in one attempt.

[E. Bavoux]

That's exactly what I needed ! Thanks

[philraj]

If you want to count keys using SCAN, you'll need to store the keys on each iteration, and use your preferred method of filtering out duplicates (SCAN does not guarantee values will only appear once), probably in a data structure that allows fast lookup of existing values. For giant sets of keys, this will be impractical since you'll need to store all keys in memory until the SCAN is complete, so you'll need to consider either keeping a secondary key as a counter which you update on SET/DEL, or storing your data in a type that allows easy counting (e.g. in a Redis hash).

[Pankaj Chauhan]

Your answer is perfect but need to explain in details eg. Your redis keys 127.0.0.1:6379> keys abc* 1) "abc123" 2) "abc456" 3) "abc234" Now try to access this: From the command line, redis-cli --scan --pattern 'abc*' | wc -l Response below: localhost@username~$ redis-cli --scan --pattern 'abc*' | wc -l 3

[kivagant]

And what if one has millions of keys?

[x_maras]

That's an old answer. I should actually add a disclaimer here. So do not use this in a production setup with millions of keys. Better use the SCAN cursor with MATCH

[dizzyf]

Note that this still uses the blocking keys command, which can cause performance hiccups on large instances.

[Clintm]

This should be the accepted answer. Requires redis with version >= 2.8.0

[Arpit Agrawal]

You should never run this command on production, this is the blocking command and if you run this command all request will starve and may get timeout if this command runs longer.

[x_maras]

Thanks! I noticed as well and added a disclaimer on top, a year ago. When I answered 6+ years ago I didn't consider that someone might run this on production.

[Ulysse BN]

@x_maras why only put a text disclaimer, I guess you could add the command related (./redis-cli --scan --pattern "abc:*" | wc -l), couldn't you? Or point to an answer giving this.

[Ram]

2 seconds for 600K keys, fast enough

[Amin Shojaei]

That's O(n) where n is number of keys. Really bad for millions of keys!

[JSTR]

Will this be advisable to do when the keys expire? for example, im trying to get the number of active users by storing to redis. that key has expiration. will it still affect the production? Thanks

[Anatolii Stepaniuk]

Never(!!) call this in production, no matter how small your data set is!

[warvariuc]

@AnatoliiStepaniuk Never say "never".

[Anatolii Stepaniuk]

@warvariuc unless you want to risk putting your system down. There is a safe option with SCAN command. Do you have any reason not to use it?

[warvariuc]

@AnatoliiStepaniuk I don't know. Everyone has his reasons. But saying "never (!!)" is an absolute statement. The correct approach is to explain your reasoning and allow people decide for themselves when they need to choose. (I think this answer was written before better options were available).

[warvariuc]

Do I understand correctly that all matching records are fetched to the client?

[Pham Hung]

Hey, what if we need select database?