Appending a list in a loop (R)

Question

I want to use a loop to read in multiple csv files and append a list in R.

path = "~/path/to/csv/"
file.names <- dir(path, pattern =".csv")
mylist=c()

for(i in 1:length(file.names)){

  datatmp <- read.csv(file.names[i],header=TRUE, sep=";", stringsAsFactors=FALSE)
  listtmp = datatmp[ ,6]
  finallist <- append(mylist, listtmp)
}
finallist

For each csv file, the desired column has a different length. In the end, I want to get the full appended list with all values in that certain column from all csv files.

I am fairly new to R, so I am not sure what I'm missing...

Solution

There are four errors in your approach.

First, file.names <- dir(path, pattern =".csv") will extract just file names, without path. So, when you try to import then, read.csv() doesn't find.

Building the path

You can build the right path including paste0():

path = "~/path/to/csv/"
file.names <- paste0(path, dir(path, pattern =".csv"))

Or file.path(), which add slashes automaticaly.

path = "~/path/to/csv"
file.names <- file.path(path, dir(path, pattern =".csv"))

And another way to create the path, for me more efficient, is that suggested in the answer commented by Tung.

file.names <- list.files(path = "~/path/to/csv", recursive = TRUE,
                            pattern = "\\.csv$", full.names = TRUE)

This is better because in addition to being all in one step, you can use within a directory containing multiple files of various formats. The code above will match all .csv files in the folder.

Importing, selecting and creating the list

The second error is in mylist <- c(). You want a list, but this creates a vector. So, the correct is:

mylist <- list()

And the last error is inside the loop. Instead of create other list when appending, use the same object created before the loop:

for(i in 1:length(file.names)){
  datatmp <- read.csv(file.names[i], sep=";", stringsAsFactors=FALSE)
  listtmp = datatmp[, 6]
  mylist <- append(mylist, list(listtmp))
}
mylist

Another approach, easier and cleaner, is looping with lapply(). Just this:

mylist <- lapply(file.names, function(x) {
  df <- read.csv(x, sep = ";", stringsAsFactors = FALSE)
  df[, 6]
})

Hope it helps!

Comments

[Oka]

and what error do you get?

[Tung]

See this answer

[FCouncil]

The final list only consists of the column values from the very last csv file