Querying in redis

Question

Recently I am learning redis and honestly very impressed and dying to use it. One of the things that keep bothering me is "how do I query redis". To be specific I am trying to resolve following

Say I have a millions of hashes stored as below

usage:1 = {created: 20100521, quantity:9, resource:1033, user:1842, ...}
usage:2 = {created: 20100812, quantity:3, resource:7233, user:1842, ...}
usage:3 = {created: 20100927, quantity:4, resource:1031, user:76, ...}

Please note that there are many keys in hashes I have shown only 4. Now that I want to find records in specific date range, by user, by resource or for a user in given period.

I suspect that there are redis specific patterns to retrieve such data. I am a python programmer. I did look at redisco(ohm port) which supports some quering but I am not sure if it does get all the data and then filters in python.

Solution 1

For Redis, it's best to understand what sort of query patterns you want over your data before you decide how you're going to store it.

For example, if you want to do a date range query over a set of data, you can store that data as a sorted set where the keys are the data items you want to query over, and the score is a unix timestamp.

In your example above, I might store your example hash as:

 user_to_resource:i = user:j                   # key -> value forward map
 resources => (resource:i, created_timestamp)  # sorted set
 count_resource:i = quantity                   # key -> value quantity map

That is, I'd have many forward and reverse maps depending on the query patterns I'd like to support.

Solution 2

the queries you mention are highly dependant on time. In this instance you would be wise to use a sorted set. You could use the datetime stamp as the score for each entry.

For example, you could do the following:

hmset usage:1 created 20100521 quantity 9 resource 1033 user 1842
hmset usage:2 created 20100812 quantity 3 resource 7233 user 1842
hmset usage:3 created 20100927 quantity 4 resource 1031 user 76

zadd usage 20200521 1
zadd usage 20100812 2
zadd usage 20100927 3

To retrieve everything:

sort usage get 
# get usage:*->created get usage:*->quantity get usage:*->resource get usage:*->user

or

lrange usage 0 -1

To get the indexes of a range:

zrangebyscore usage 20100800 20100900

For queries based on a hash key value, there is a useful addition to redis which allows the use of scripts written in lua. You could easily write a simple lua script within a python heredoc and use the redis.eval method to pass the script to redis. The script could be a loop which filters based on the value you are looking for.

Comments

[Shekhar]

Thought it would be useful to share this pythonik.blogspot.com/2010/11/redis-patterns-search.html Disclaimer: Pointer to my own blog