Python Multidimensional Arrays - most efficient way to count number of non-zero entries

Question

Hi there on a Saturday Fun Night,

I am getting around in python and I am quite enjoying it.

Assume I have a python array:

x = [1, 0, 0, 1, 3]

What is the fastest way to count all non zero elements in the list (ans: 3) ? Also I would like to do it without for loops if possible - the most succint and terse manner possibe, say something conceptually like

[counter += 1 for y in x if y > 0]

Now - my real problem is that I have a multi dimensional array and what I really want to avoid is doing the following:

for p in range(BINS):
    for q in range(BINS):
        for r in range(BINS):
            if (mat3D[p][q][r] > 0): some_feature_set_count += 1

From the little python I have seen, my gut feeling is that there is a really clean syntax (and efficient) way how to do this.

Ideas, anyone?

Solution 1

For the single-dimensional case:

sum(1 for i in x if i)

For the multi-dimensional case, you can either nest:

sum(sum(1 for i in row if i) for row in rows)

or do it all within the one construct:

sum(1 for row in rows
      for i in row if i)

Solution 2

If you are using numpy as suggested by the fact that you're using multi-dimensional arrays in Python, the following is similar to @Marcelo's answer, but a tad cleaner:

>>> a = numpy.array([[1,2,3,0],[0,4,2,0]])
>>> sum(1 for i in a.flat if i)
5
>>>

Solution 3

If you go with numpy and your 3D array is a numpy array, this one-liner will do the trick:

numpy.where(your_array_name != 0, 1, 0).sum()

example:

In [23]: import numpy

In [24]: a = numpy.array([ [[0, 1, 2], [0, 0, 7], [9, 2, 0]], [[0, 0, 0], [1, 4, 6], [9, 0, 3]], [[1, 3, 2], [3, 4, 0], [1, 7, 9]] ])

In [25]: numpy.where(a != 0, 1, 0).sum()
Out[25]: 18

Comments

[slezica]

+1 for dominating the syntax. @original poster: as for efficiency, your code is as efficient as you can get. It has the minimum complexity, both spatial and temporal, requiered for the task (if you are using Python < 3.0, use xrange instead of range, though) Also, I personally feel your code, though longer and less pythonesque, is much clearer than any syntactic trick.

[Alex S]

That requires O(N) space, which is not ideal for large arrays, plus it's invalid Python syntax. I think you meant x != 0.

[aaronasterling]

the use of sum for counting elements is pretty idiomatic. You should get used to it so that you don't have to construct lists just to throw them away like you do here. Also, I think you mean x is not 0 which should really be x != 0 which, itself should really be x

[Jochen Ritzel]

Something that also works (because True == 1) is sum(bool(i) for i in x) .. but its more useful for counting with other predicates like i>0

[aaronasterling]

@Santiago You should time the two alternatives. This will probably smoke OP's code.

[Josh Smeaton]

Both fair points, thanks for the heads up. Changed the condition but left the len so others can see what not to do. List comprehensions are generators right? I may have forgotten that point.

[Alex S]

@THC4k: It will work, but it isn't as efficient (especially for a sparse matrix) because it has to yield a value for every element.

[Alex S]

@Santiago: Thanks for the up-vote, but what do you mean by "syntactic trick"?

[Alex S]

@THC4k: (WRT to your amended comment) AFAICT, neither your approach nor mine is any better than the other at handling arbitrary predicates. What are you getting at?

[aaronasterling]

You really shouldn't shadow sum. It's a builtin.

[slezica]

@Marcelo I meant that, while I love one-liners (and that was a good one!), I'm always careful that they don't obscure the code. OP said he was just starting with python, so maybe he's better off with simpler, nested loops, at least for now :). BTW where are you from?

[Alex S]

@Santiago: Born in Chile; raised in Australia; spent a brief and tumultuous couple of years in the U.S.; currently living in Melbourne.

[Alex S]

@Santiago: Back on topic, list comprehensions are idiomatic Python and I think users should be taught how to use them from day one, since they result in much cleaner, and often more efficient code. I always teach people this stuff as early as possible and I find that, though they have to go through a brief mental readjustment, they invariably become better programmers in the process, producing better code with less effort.

[Alex S]

If you have numpy, you can just write np.sum(a) (assuming the conventional import numpy as np).