Limit list length in redis

Question

I'm using redis lists and pushing to new items to a list. The problem is I really only need the most recent 10 items in a list.

I'm using lpush to add items to a list and lrange to get the most recent 10.

Is there anyway to drop items after a certain number? I'll end up with lists that may have 1,000's of items and can cause performance issues with latency.

Thank you!

Solution 1

After every lpush, call ltrim to trim the list to 10 elements

See http://redis.io/commands/ltrim

Solution 2

You can use LTRIM intermittently after any LPUSH, no need to call LTRIM after every LPUSH as that would add to overall latency in your app ( though redis is really fast, but you can save lots of LPUSH operations )

Here is a pseudo code to achieve an LTRIM on approximately every 5th LPUSH:

LPUSH mylist 1
random_int = some random number between 1-5
if random_int == 1:  # trim my list with 1/5 chance
   LTRIM mylist 0 10

Though your list may grow to be a few elements more than 10 elements at times, but it will surely get truncated at regular intervals. This approach is good for most practical purposes and saves a lot of LTRIM operations, keeping your pushes fast.

Solution 3

The following code,

• pushes the item to the list,
• keep the size fixed to 10,
• and returns the most recent 10 elements

in a transaction.

MULTI
LPUSH list "item1"
LTRIM list 0 9
LRANGE list 0 9
EXEC

Solution 4

No one has ever mentioned the real solution about storing only most 10 recent items.

Let's create a sample list with 15 items (here just numbers):

RPUSH list 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Now indicate offset from the end of the list:

LTRIM list -10 -1

Show list

LRANGE list 0 -1

 1) "6"
 2) "7"
 3) "8"
 4) "9"
 5) "10"
 6) "11"
 7) "12"
 8) "13"
 9) "14"
10) "15"

Now you can add new items and run trim:

RPUSH list 16
LTRIM list -10 -1

 1) "7"
 2) "8"
 3) "9"
 4) "10"
 5) "11"
 6) "12"
 7) "13"
 8) "14"
 9) "15"
10) "16"

Solution 5

Just an alternative. According to official doc of LPUSH, it returns the length of the list after the push operations. You can set a threshold length like k (in your case k > 10) and call LTRIM when returned length is bigger than k. Sample pseudo code as follows:

len = LPUSH mylist xxx 
if len > k:   
  LTRIM mylist 0 9
LRANGE mylist 0 9

It's more controllable than random method. Greater k triggers less LTRIM but with more memory cost. You can adjust k according to how often you want to call LTRIM since calling extra command is more expensive.

View more solutions

Comments

[bornSwift]

Wouldn't retrieving the current list length and only inserting if the length is below 10 be a better performing solution for large entities?

[Terry Lewis]

@bornSwift, wouldn't your solution keep the 10 earliest entries instead of the 10 most recent?

[bornSwift]

@TerryLewis, yes good point. Didn't see the "most recent" in the question.. oops

[FeignMan]

You can almost smell when a piece of advice comes straight from someone's experiences in production ;)

[Roy Miloh]

@PaulT.Rawkeen why? it can be achieved with Reids builtin commands (wrapped in transaction), I don't see a reason to make it a lua script.

[fiatjaf]

How can you be sure that getting a random number and checking if it is something isn't more expensive than an LTRIM call?

[DhruvPathak]

@fiatjaf Mostly because of LTRIM being an over the network operation, however if both operations are part of a single Lua script, we might have to benchmark. However, I do strongly feel, that the random number operation is going to be simpler than LTRIM in both cases.

[George Leung]

@DhruvPathak If one is using a language/library that supports pipelining, then it is not necessary to write a script to save the round trip time.