Iteration over list slices

Question

I want an algorithm to iterate over list slices. Slices size is set outside the function and can differ.

In my mind it is something like:

for list_of_x_items in fatherList:
    foo(list_of_x_items)

Is there a way to properly define list_of_x_items or some other way of doing this using python 2.5?

---

edit1: Clarification Both "partitioning" and "sliding window" terms sound applicable to my task, but I am no expert. So I will explain the problem a bit deeper and add to the question:

The fatherList is a multilevel numpy.array I am getting from a file. Function has to find averages of series (user provides the length of series) For averaging I am using the mean() function. Now for question expansion:

edit2: How to modify the function you have provided to store the extra items and use them when the next fatherList is fed to the function?

for example if the list is lenght 10 and size of a chunk is 3, then the 10th member of the list is stored and appended to the beginning of the next list.

---

Related:

• What is the most “pythonic” way to iterate over a list in chunks?

Solution 1

Answer to the last part of the question:

question update: How to modify the function you have provided to store the extra items and use them when the next fatherList is fed to the function?

If you need to store state then you can use an object for that.

class Chunker(object):
    """Split `iterable` on evenly sized chunks.

    Leftovers are remembered and yielded at the next call.
    """
    def __init__(self, chunksize):
        assert chunksize > 0
        self.chunksize = chunksize        
        self.chunk = []

    def __call__(self, iterable):
        """Yield items from `iterable` `self.chunksize` at the time."""
        assert len(self.chunk) < self.chunksize
        for item in iterable:
            self.chunk.append(item)
            if len(self.chunk) == self.chunksize:
                # yield collected full chunk
                yield self.chunk
                self.chunk = [] 

Example:

chunker = Chunker(3)
for s in "abcd", "efgh":
    for chunk in chunker(s):
        print ''.join(chunk)

if chunker.chunk: # is there anything left?
    print ''.join(chunker.chunk)

Output:

abc
def
gh

Solution 2

If you want to divide a list into slices you can use this trick:

list_of_slices = zip(*(iter(the_list),) * slice_size)

For example

>>> zip(*(iter(range(10)),) * 3)
[(0, 1, 2), (3, 4, 5), (6, 7, 8)]

If the number of items is not dividable by the slice size and you want to pad the list with None you can do this:

>>> map(None, *(iter(range(10)),) * 3)
[(0, 1, 2), (3, 4, 5), (6, 7, 8), (9, None, None)]

It is a dirty little trick

---

OK, I'll explain how it works. It'll be tricky to explain but I'll try my best.

First a little background:

In Python you can multiply a list by a number like this:

[1, 2, 3] * 3 -> [1, 2, 3, 1, 2, 3, 1, 2, 3]
([1, 2, 3],) * 3 -> ([1, 2, 3], [1, 2, 3], [1, 2, 3])

And an iterator object can be consumed once like this:

>>> l=iter([1, 2, 3])
>>> l.next()
1
>>> l.next()
2
>>> l.next()
3

The zip function returns a list of tuples, where the i-th tuple contains the i-th element from each of the argument sequences or iterables. For example:

zip([1, 2, 3], [20, 30, 40]) -> [(1, 20), (2, 30), (3, 40)]
zip(*[(1, 20), (2, 30), (3, 40)]) -> [[1, 2, 3], [20, 30, 40]]

The * in front of zip used to unpack arguments. You can find more details here. So

zip(*[(1, 20), (2, 30), (3, 40)])

is actually equivalent to

zip((1, 20), (2, 30), (3, 40))

but works with a variable number of arguments

Now back to the trick:

list_of_slices = zip(*(iter(the_list),) * slice_size)

iter(the_list) -> convert the list into an iterator

(iter(the_list),) * N -> will generate an N reference to the_list iterator.

zip(*(iter(the_list),) * N) -> will feed those list of iterators into zip. Which in turn will group them into N sized tuples. But since all N items are in fact references to the same iterator iter(the_list) the result will be repeated calls to next() on the original iterator

I hope that explains it. I advice you to go with an easier to understand solution. I was only tempted to mention this trick because I like it.

Solution 3

If you want to be able to consume any iterable you can use these functions:

from itertools import chain, islice

def ichunked(seq, chunksize):
    """Yields items from an iterator in iterable chunks."""
    it = iter(seq)
    while True:
        yield chain([it.next()], islice(it, chunksize-1))

def chunked(seq, chunksize):
    """Yields items from an iterator in list chunks."""
    for chunk in ichunked(seq, chunksize):
        yield list(chunk)

Solution 4

Use a generator:

big_list = [1,2,3,4,5,6,7,8,9]
slice_length = 3
def sliceIterator(lst, sliceLen):
    for i in range(len(lst) - sliceLen + 1):
        yield lst[i:i + sliceLen]

for slice in sliceIterator(big_list, slice_length):
    foo(slice)

sliceIterator implements a "sliding window" of width sliceLen over the squence lst, i.e. it produces overlapping slices: [1,2,3], [2,3,4], [3,4,5], ... Not sure if that is the OP's intention, though.

Solution 5

Do you mean something like:

def callonslices(size, fatherList, foo):
  for i in xrange(0, len(fatherList), size):
    foo(fatherList[i:i+size])

If this is roughly the functionality you want you might, if you desire, dress it up a bit in a generator:

def sliceup(size, fatherList):
  for i in xrange(0, len(fatherList), size):
    yield fatherList[i:i+size]

and then:

def callonslices(size, fatherList, foo):
  for sli in sliceup(size, fatherList):
    foo(sli)

View more solutions

Comments

[hughdbrown]

I think you'll get lists of irregular size with that. No telling exactly what the OP wanted, but this might be closer: for i in xrange(0, len(fatherList)-size, 1):

[hughdbrown]

Correction: for i in xrange(len(fatherList) - size + 1):

[Alex Martelli]

"irregular size"? they're all of len size except possibly for the last one -- and nonoverlapping. Consider what happens with your code when the len is 6 and the size is 7, say... 5 slices of decreasing length...

[Alex Martelli]

Better;-) but with len 6 and size 8 you still have the same problem.

[ThomasH]

One interesting thing is also that you're silently making use of the * unpacking operator's low precedence, which allows funny constructs like `` f(LN)``, and which I find nowhere documented.

[aryeh]

There needs to be another yield self.chunk after the for loop to account for chunks that aren't the full chunksize. Otherwise this won't work on arbitrary list sizes.

[jfs]

@aryeh: __call__ may be called more than once, therefore you can't just yield leftovers after the for loop. That's the point of the class. It answers "to store the extra items and use them when the next fatherList is fed to the function" part of the question as explicitly stated at the top of the answer.

[aryeh]

My mistake, I didn't read the question/example carefully enough. I thought it was just for grouping. I personally dislike the extra check at the end and would vote to add an option to call whether to save extra items or flush them at the end, so you wouldn't need to possibly duplicate the body of the for loop in a following if statement.

[e-satis]

Very beautiful. And you can actually make it smarter by adding a parameter to choose the type of each chunk: gist.github.com/1275417.

[lukmdo]

I was searing for something nice that would work out of the box as it looks as TOTALY general problem. My note: Nice&simple implementation but not to use with iterators... :( No len(list(fatherList)) is not it.

[jfs]

@ThomasH: f(*args) is an extended call syntax in Python. You can put anything that evaluates to an iterable on the right side of asterisk (*) e.g., f(*[1]+g())

[ThomasH]

@J.F.Sebastian Yes, thanks. What struck me was just the operator precedence for this. Your example resolves to f(*([1]+g())).

[lkostka]

Kinda pythonic, I love it

[Tim-Erwin]

In Python 3 it is next(it) instead of it.next(). For those wondering: islice doesn't raise a StopIteration, that's why we call next.

[davidA]

@Tim-Erwin I tried it in python 3.6 by changing to next(it) however I get the warning DeprecationWarning: generator 'ichunked' raised StopIteration. I don't understand where this is being raised, as it's not explicit.

[Tim-Erwin]

@meowsqueak It is raised in next(). That's expected in Python 3.5 and below, deprecated in 3.6, broken in 3.7. I added a version compatible with 3.7+ in the answer.

[Tim-Erwin]

@meowsqueak Others didn't find my edit useful and rejected it. You can find it as a separate answer down below.

[mit]

Does what some of the other solutions do in more lines, maybe a little slower, depending on the use case. This could be improved using islice.