Create a buffer in python

Question

I do not have much experience with Python and need to figure out a few things regarding buffering:

I want to generate a list in which I can allocate a certain amount of "buffer" space (do not know the specifics on this), to generate a list of integers. If the 'buffer' gets full I'm assuming it needs a flush command? Or how would you clear the buffer to continue putting things into that buffer?

Here is a sample of my code:

for i in range(0,500):
    randoms = random.randint(0,100000)
    looplist.append(randoms)

What I would want is in looplist, to be a sort of buffer I assume? in which if the maximum buffer space in looplist gets full, it would need to clear out (is there a pause during that time? or what happens) the list to continue re-generating integers to that list/ buffer.

Part 2 question: Would like explanation as simple of how a buffer could work for python? or does memory management of python just disable need to allocate own buffers? (can we still do it if we want too?)

I will edit my question if it seems to broad scope, trying to be as descriptive as I know how.

Solution 1

In Python, you don't have to care about memory management. You don't need to reserve "buffers", just assign the variables you want, and use them.

If you assign too many (for your RAM), you might run into MemoryErrors, indicating you don't have enough memory.

You could use a Queue as some kind of buffer, though, e.g.

import random
from Queue import Queue, Full

q = Queue(100)

for i in range(10):
    #fill queue
    while True:
        try:
            q.put_nowait(random.randint(1,10))
        except Full:
            break
    #take some values from queue
    print "Round", i,
    number_of_values_to_get = random.randint(0,20)
    print "getting %i values." % number_of_values_to_get
    for j in range(number_of_values_to_get):
        value = q.get()
        print "  got value", value

I create a Queue of size 100, i.e., a maximum of 100 entries can be stored in it. If you try to put on a full queue, it will raise the corresponding exception, so you better catch it. (This is only True if you use put_nowait or put with a timeout, just look at the docs for more details.)

In this example, for ten rounds, I fill a "buffer" (queue) of 100 random integers. Then, I pick between 0 and 20 values from this buffer and print them.

I hope this helps. The main use-case for queues is multithreaded program execution, they are thread-safe. So maybe you'll want to fill the queue from one thread and read it from another. If you don't want multithreading, collections.deque might be a faster alternative, but they are not thread-safe.

Solution 2

use a queue / list and make indices fall off on first-in first-out basis whenever the queue / list size is sufficiently large

Comments

[Bain]

Would a queue be able to lets say, allow the generation of integers to continue while placed into the queue? So Could i set up a queue on : Queue of 1, while integers are coming into queue, does the queue continue FIFO?

[Aditya Sihag]

i don't follow your question. try initializing a list L = []. Then run your loop slapping on values with L.append(), check if length of L, len(L) is sufficiently big. If it is then, snip of the first element, with L.pop(0)

[Bain]

Don't think that answers the buffer part. I want a buffer to allocate a certain amount of space for an integer generation to run on it. Then the system would "flush" the buffer space allow more integers to flow into it.

[Aditya Sihag]

@Bain Python is dynamically typed; to mimic your behaviour you can initialize a list with the desired size, and instead of calling pop on it, you can circulate values. This would basically mean you won't waste time in memory management of the list when elements were previously being popped off.