remove None value from a list without removing the 0 value

Question

This was my source I started with.

My List

L = [0, 23, 234, 89, None, 0, 35, 9]

When I run this :

L = filter(None, L)

I get this results

[23, 234, 89, 35, 9]

But this is not what I need, what I really need is :

[0, 23, 234, 89, 0, 35, 9]

Because I'm calculating percentile of the data and the 0 make a lot of difference.

How to remove the None value from a list without removing 0 value?

Solution 1

>>> L = [0, 23, 234, 89, None, 0, 35, 9]
>>> [x for x in L if x is not None]
[0, 23, 234, 89, 0, 35, 9]

Just for fun, here's how you can adapt filter to do this without using a lambda, (I wouldn't recommend this code - it's just for scientific purposes)

>>> from operator import is_not
>>> from functools import partial
>>> L = [0, 23, 234, 89, None, 0, 35, 9]
>>> filter(partial(is_not, None), L)
[0, 23, 234, 89, 0, 35, 9]

Solution 2

A list comprehension is likely the cleanest way:

>>> L = [0, 23, 234, 89, None, 0, 35, 9
>>> [x for x in L if x is not None]
[0, 23, 234, 89, 0, 35, 9]

There is also a functional programming approach but it is more involved:

>>> from operator import is_not
>>> from functools import partial
>>> L = [0, 23, 234, 89, None, 0, 35, 9]
>>> list(filter(partial(is_not, None), L))
[0, 23, 234, 89, 0, 35, 9]

Solution 3

Using list comprehension this can be done as follows:

l = [i for i in my_list if i is not None]

The value of l is:

[0, 23, 234, 89, 0, 35, 9]

Solution 4

@jamylak answer is quite nice, however if you don't want to import a couple of modules just to do this simple task, write your own lambda in-place:

>>> L = [0, 23, 234, 89, None, 0, 35, 9]
>>> filter(lambda v: v is not None, L)
[0, 23, 234, 89, 0, 35, 9]

Solution 5

For Python 2.7 (See Raymond's answer, for Python 3 equivalent):

Wanting to know whether something "is not None" is so common in python (and other OO languages), that in my Common.py (which I import to each module with "from Common import *"), I include these lines:

def exists(it):
    return (it is not None)

Then to remove None elements from a list, simply do:

filter(exists, L)

I find this easier to read, than the corresponding list comprehension (which Raymond shows, as his Python 2 version).

View more solutions

Comments

[mgilson]

The less elegant filter version: filter(lambda x: x is not None, L) -- You could get rid of the lambda using partial and operator.is_not I think, but it's probably not worth it since the list-comp is so much cleaner.

[jamylak]

@mgilson Oh wow I didn't even know is_not existed! I thought it was only is_, I'm gonna add that in just for fun

[mgilson]

@jamylak -- Yeah. It actually bothers me that is_not exists and not_in doesn't exist. I actually think that not_in should be turned into a magic method __not_contains__ ... see a question I asked a while back and a comment I made to an answerer ... and still don't feel like it is resolved.

[jamylak]

@mgilson I think under that same assumption I just assumed it didn't exist. I guess you can just use filterfalse or something depending on the use case

[mgilson]

@jamylak -- Yeah. My main problem is that x > y does not imply not x <= y in python because you can do anything in __lt__ and __le__, so why should x not in y imply not x in y (especially since not in has it's own bytecode?)

[jamylak]

@mgilson right, that's a good point, Python isn't perfect though

[jamylak]

+1 Do you recommend the use of __ne__ like that as opposed to partial and ne?

[Raymond Hettinger]

@jamylak Yes, it is faster, a bit easier to write, and a bit more clear.

[Admin]

Consider using the operator module.

[Laurent LAPORTE]

Please, give some details information to the OP, and not just a code.

[med_abidi]

I did. What you think?

[Laurent LAPORTE]

Well, this doesn't answer the OP question. Consider this answer instead: stackoverflow.com/a/16096769/1513933

[med_abidi]

Yes you are right. There was a problem with the filter partial.

[DrMcCleod]

What is __ne__?

[Raymond Hettinger]

@DrMcCleod The expression x != y internally calls x.__ne__(y) where the ne stands for "not equal". So, None.__ne__ is a bound method that returns True when called with any value other than None. For example, bm = None.__ne__ called with bm(10) returns NotImplemented which as true value, and bm(None) returns False.

[jamylak]

I would prefer Raymonds solution for Python 3, and then the list comprehension for Python 2. But if I did have to go this route, I would rather partial(is_not, None) than this solution. I believe this will be slower (although thats not too important). But with a couple of imports of python modules, no need for a custom defined function in this case

[jamylak]

Not a fan of this at all, the whole advantage you claim with this solution is that the list might be so huge that building duplicate list in memory could be costly. Well then your solution will be even more costly because you're scanning the entire list for L.count(None) and then you're calling .remove(None) multiple times which makes this O(N^2) The situation you are trying to solve should not be dealt with in this way, the data should be restructured into a Database or file instead if it's that memory intensive.

[jamylak]

Would be interested if you had a practical example where this answer is the best solution, I tend to think that there would be a better approach in all cases. For example numpy would be able to handle this type of operation in a more optimised manner

[jamylak]

This discussion is getting too abstract now, I don't think you'd be able to give me one real life example in your years of experience where this answer is the correct approach over restructuring the data as i mentioned before.

[Marc]

if only M = L.filter(None)

[Qaswed]

This solution is already found in the top answer, or am I missing something?

[Kevin]

True, but not all real world situations allow flexibility of transforming the data. For example, pumping "legacy" geospatial data through a one-off analysis on a system without much memory. Another example is programming time vs runtime. Might not care if something takes all night to run so long as it was inexpensive to write. What about if writing a plugin or library? You may not be the one deciding what the data looks like. With this answer, I am bringing attention to considering memory and knowing the data, but I point out it probably does not matter most of the time.

[Kevin]

Best-case is O(n). Count n with nothing found. All None would be O(2n). Count == n ops; then each remove comparison is found on the first item, so total comparisons is n. A single None at the end of the list is also O(2n). Count is n; one additional pass through the list until None is found in the last position n. So, I think worst-case is when the back half the list is entirely None. I think that is O(n + ((n / 2) + 1)^2). Another real world scenario is embedded systems with tiny amounts of memory. Micropython exists specifically for those.

[Piotr Dobrogost]

That NotImplemented is true in boolean context is really bizarre. See Make NotImplemented unusable in boolean context Python issue.

[Michel de Ruiter]

Python 3.9 says: DeprecationWarning: NotImplemented should not be used in a boolean context

[ToolmakerSteve]

Presumably "# second" needs logic to handle ValueError, indicating there were no more to remove.

[Kevin]

@ToolmakerSteve, no need the "second" because the end-case logic is covered by only removing the number of L.count(None) items. It cannot make extra calls to remove.

[ToolmakerSteve]

The custom defined function isn't merely for this case. I wouldn't have defined a function if it were! I'm saying that I find it more readable, in many situations, to say "if exists(something)", instead of saying "if something is not None". Its closer to how I think, and avoids the "double-negative" of saying "not None".

[jamylak]

I see what you mean about avoiding the double negative, actually in the definition of exists return (it is not None) is a clear way to define it. (maybe I'd remove the brackets but that's just a small thing anyway)

[sfink]

It seems like the in-place approach would be better as w=0; for e in L: if e is not None: L[w] = e; w += 1; del L[w:]. (Sorry, I don't know how to format that in a comment. You can't really do one-line Python. The del is after the loop.) It's in-place and O(n). θ(n) in fact.

[Kevin]

@sfink, that is a viable in-place implementation. That takes the implicit list restructuring out of the List.remove() operation and does it explicitly. I think by switching to is None and using a front and a back pointer, instead of only one, you could get rid of the redundant assignments. The my answer is simply pointing out that sometimes memory size may need to be taken into account, and not only processor operation counts.

[mongotop]

using pandas is a great idea! Thank you @sedrak and welcome to the platform