mc.cores > 1 is not support on windows

10,147

Solution 1

If all you want to do is make it so this code doesn't run in parallel, you just need to tell it to use 1 core, then it will use lapply under the hood.

ll <- parallel::mclapply(waydf$geometry$coordinates, st_linestring, mc.cores = 1)

Or just swap mclapply out for lapply.

ll <- lapply(waydf$geometry$coordinates, st_linestring)

Solution 2

You'll have to clarify what st_linestring is or does because you're trying to pass the contents of waydf$geometry$coordinates to it, but haven't specified any arguments, such as st_linestring(waydf$geometry$coordinates[i])

In Windows, you would use parLapply instead of mclapply.

# original
ll <- parallel::mclapply( waydf$geometry$coordinates , st_linestring,
                         mc.cores =parallel::detectCores() - 1  )

# replace the above with all of the below
library(parallel)
cl <- makeCluster(detectCores())
cl <- clusterEvalQ(cl, { library(sf) })  # you need to export packages as well
# cl <- clusterExport(cl, "st_linestring")  # each worker is a new environment, you will need to export variables/functions to
ll <- parallel::parLapply(cl, waydf$geometry$coordinates, function(i) st_linestring)    # if st_linestring takes arguments then st_linestring(x)
stopCluster(cl)

Edit since st_linestring is a function from the package sf, it is sufficient to export sf

2nd edit

rpl <- unlist( lapply( waydf$geometry$coordinates , nrow ) ) # row per line
 waydf <- waydf[ rpl > 1 , ]

library(parallel)
cl <- makeCluster(detectCores())
cl <- clusterEvalQ(cl, { library(sf) })  # you need to export packages as well
# cl <- clusterExport(cl, "st_linestring")  # each worker is a new environment, you will need to export variables/functions to
ll <- parallel::parLapply(cl, waydf$geometry$coordinates, function(i) st_linestring)    # if st_linestring takes arguments then st_linestring(x)
stopCluster(cl)

outdf <- sf::st_sf(
line_geometry = sf::st_sfc( ll , crs = epsg ) ,
osm_id = waydf$id
)

Solution 3

The problem is that you do cl <- ... in all those rows; you keep redefining that variable to be something else. You should only assign cl once and then reuse it.

library("parallel")
cl <- makeCluster(detectCores())
clusterEvalQ(cl, { library("sf") })
clusterExport(cl, "st_linestring")
res <- parallel::parLapply(cl, X = waydf$geometry$coordinates, 
                         fun = function(i) st_linestring)
stopCluster(cl)

The message Error in checkCluster(cl): not a valid cluster that you get with your code is because after you do cl <- clusterEvalQ(cl, { library("sf") }) it is no longer a cluster object.

Share:
10,147
Maryam Koulaei
Author by

Maryam Koulaei

Find myself!

Updated on June 22, 2022

Comments

  • Maryam Koulaei
    Maryam Koulaei over 1 year

    enter image description hereI am new in R programming and I have code like below and I know that the windows does not support multicore but I don't know how to change this part of the code. Can someone suggest me an equivalent code without using the mc.cores feature?

    rpl <- unlist( lapply( waydf$geometry$coordinates , nrow ) ) # row per line
     waydf <- waydf[ rpl > 1 , ]
    ll <- parallel::mclapply( waydf$geometry$coordinates , st_linestring,
                             mc.cores =parallel::detectCores() - 1  )
    outdf <- sf::st_sf(
    line_geometry = sf::st_sfc( ll , crs = epsg ) ,
    osm_id = waydf$id
    )
    
  • Maryam Koulaei
    Maryam Koulaei about 6 years
    I do still get the same error :(. how can I run this function without multicore feature?
  • CPak
    CPak about 6 years
    Sorry, but did you replace mclapply with parLapply? What is your error? Please show
  • Maryam Koulaei
    Maryam Koulaei about 6 years
    Error in makeCluster(detectCores()) : could not find function "makeCluster"
  • Maryam Koulaei
    Maryam Koulaei about 6 years
    now , i got this error: Error in checkCluster(cl) : not a valid cluster in this line: cl <- parallel::clusterExport(cl, "st_linestring")
  • CPak
    CPak about 6 years
    Since st_linestring is in sf package, you do not need to export it. I commented out the line
  • Maryam Koulaei
    Maryam Koulaei about 6 years
    this is the new error: Error in detectCores() : could not find function "detectCores"
  • Maryam Koulaei
    Maryam Koulaei about 6 years
    YEs, I loaded the parallel and the problem of detectingCores is solved. But I get again not a valid cluster error in the last line : Error in checkCluster(cl) : not a valid cluster
  • Maryam Koulaei
    Maryam Koulaei about 6 years
    could you please how can I run this code without parallelizing? in a very simple way.
  • CPak
    CPak about 6 years
    I'm assuming your code works. You can replace mclapply with the answer provided above (see after the 2nd edit). I recommend you start a new R session before running this code.
  • Maryam Koulaei
    Maryam Koulaei about 6 years
    I added a picture of my code with the error. I have no idea why do I still get this error.
  • CPak
    CPak about 6 years
    Did you start a new R session? My guess is that you loaded a cluster before using doMC. And could you type cl and show the output?
  • CPak
    CPak about 6 years
    You should see something like socket cluster with 8 nodes on host ‘localhost’ when typing cl
  • Maryam Koulaei
    Maryam Koulaei about 6 years
    yes, I restart a new R session each time I change the code.
  • Maryam Koulaei
    Maryam Koulaei about 6 years
    @ HenrikB, hey. thanks for the answer. I tried to run your code but i got this error: Error in get(name, envir = envir) : object 'st_linestring' not found