In R, how to filter lists of lists?

Question

According to the manual, Filter works on vectors, and it happens to work also on lists, eg.:

z <- list(a=1, b=2, c=3)
Filter(function(i){
  z[[i]] > 1
}, z)
$b
[1] 2

$c
[1] 3

However, it doesn't work on lists of lists, eg.:

z <- list(z1=list(a=1,b=2,c=3), z2=list(a=1,b=1,c=1), z3=list())
Filter(function(i){
  if(length(z[[i]])>0){
    if(z[[i]]$b > 1)
      TRUE
    else
      FALSE
  }
  else
    FALSE
}, z)
Error in z[[i]] : invalid subscript type 'list'

What is the best way then to filter lists of lists without using nested loops? It could also be lists of lists of lists...

(I tried with nested lapply's instead, but couldn't manage to make it work.)

Edit: in the 2nd example, here is what I want to obtain:

list(z1=list(a=1,b=2,c=3))

that is, without z$z2 because z$z2$b < 1, and without z$z3 because it is empty.

Solution 1

I think you should use:

Filter(function(x){length(x)>0 && x[["b"]] > 1},z)

The predicate (the function you are using to filter z) applies to the elements of z, not their indexes.

Solution 2

I had never used Filter prior to your question, so this was a good exercise for first thing in the morning :)

There are at least a couple of things going on that are tripping you up (I think).

Let's start with your first simple anonymous function, but let's make it stand alone so it's easier to read:

f <- function(i){
        z[[i]] > 1
     }

It should jump out at you that this function takes one argument, i, yet in the function it calls z. That's not very good "functional" programming :)

So start by changing that function to:

f <- function(i){
        i > 1
     }

And you'll see Filter will actually run against a list of lists:

 z <- list(z1=list(a=1,b=2,c=3), z2=list(a=1,b=1,c=1))
 Filter( f, z)

but it returns:

> Filter( f, z)
$z2
$z2$a
[1] 1

$z2$b
[1] 1

$z2$c
[1] 1


$<NA>
NULL

which isn't exactly what you want. Honestly I can't grok why it returns that result, maybe someone can explain it to me.

@DWin was barking up the right tree when he said that there should be a recursive solution. I hacked up a first stab at a recursive function, but you'll need to improve on it:

fancyFilter <- function(f, x){
  if ( is.list( x[[1]] ) ) #only testing the first element... bad practice
    lapply( x, fancyFilter, f=f ) #recursion FTW!!
  else
    return( lapply(x, Filter, f=f ) )
}

fancyFilter looks at the first element of the x passed to it and if that element is a list, it recursively calls fancyFilter on each element of the list. But what if element #2 is not a list? That's the sort of thing you should test and tease out whether it matters for you. But the result of fancyFilter seems to look like what you are after:

> fancyFilter(f, z)
$z1
$z1$a
numeric(0)

$z1$b
[1] 2

$z1$c
[1] 3


$z2
$z2$a
numeric(0)

$z2$b
numeric(0)

$z2$c
numeric(0)

You may want to add some logic to clean up the output so the FALSE results don't get molested into numeric(0). And, obviously, I did an example using only your simple function, not the more complex function you used in the second example.

Solution 3

The modern tidy solution to this problem would be:

library(tidyverse)
z <- list(z1=list(a=1,b=2,c=3), z2=list(a=1,b=1,c=1), z3=list())

Then simply:

tibble(disc = z, Names = names(z)) %>% 
  hoist(disc, c = "c") %>%
  filter(c == 3) %>%
  unnest_wider(disc) %>% 
  split(.$Names) %>% map(select, -Names) %>% 
  map(as.list)

Note this is now super flexible, and easily allows other filtering, e.g. if a == 1

Comments

[tflutre]

thanks, your solution works on the example I provided, but it is not easily scalable for lists of lists of lists, etc. Whatsoever, if there is no better answers, I will accept yours.

[IRTFM]

I think a recursive answer is possible. I can even replicate the structure of such an answer, but I'm having difficulty filling in the nodes. I would wait.

[tflutre]

thanks a lot! I managed to use a polished version of your answer on 2-level lists, but it doesn't work on 3-level lists. As a result, I get an empty "named list", and I don't know what that is... but SO doesn't allow me to ask a new question. Should I first accept your answer?