mc.cores > 1 is not support on windows
Solution 1
If all you want to do is make it so this code doesn't run in parallel, you just need to tell it to use 1 core, then it will use lapply
under the hood.
ll <- parallel::mclapply(waydf$geometry$coordinates, st_linestring, mc.cores = 1)
Or just swap mclapply
out for lapply
.
ll <- lapply(waydf$geometry$coordinates, st_linestring)
Solution 2
You'll have to clarify what st_linestring
is or does because you're trying to pass the contents of waydf$geometry$coordinates
to it, but haven't specified any arguments, such as st_linestring(waydf$geometry$coordinates[i])
In Windows, you would use parLapply
instead of mclapply
.
# original
ll <- parallel::mclapply( waydf$geometry$coordinates , st_linestring,
mc.cores =parallel::detectCores() - 1 )
# replace the above with all of the below
library(parallel)
cl <- makeCluster(detectCores())
cl <- clusterEvalQ(cl, { library(sf) }) # you need to export packages as well
# cl <- clusterExport(cl, "st_linestring") # each worker is a new environment, you will need to export variables/functions to
ll <- parallel::parLapply(cl, waydf$geometry$coordinates, function(i) st_linestring) # if st_linestring takes arguments then st_linestring(x)
stopCluster(cl)
Edit since st_linestring is a function from the package sf, it is sufficient to export sf
2nd edit
rpl <- unlist( lapply( waydf$geometry$coordinates , nrow ) ) # row per line
waydf <- waydf[ rpl > 1 , ]
library(parallel)
cl <- makeCluster(detectCores())
cl <- clusterEvalQ(cl, { library(sf) }) # you need to export packages as well
# cl <- clusterExport(cl, "st_linestring") # each worker is a new environment, you will need to export variables/functions to
ll <- parallel::parLapply(cl, waydf$geometry$coordinates, function(i) st_linestring) # if st_linestring takes arguments then st_linestring(x)
stopCluster(cl)
outdf <- sf::st_sf(
line_geometry = sf::st_sfc( ll , crs = epsg ) ,
osm_id = waydf$id
)
Solution 3
The problem is that you do cl <- ...
in all those rows; you keep redefining that variable to be something else. You should only assign cl
once and then reuse it.
library("parallel")
cl <- makeCluster(detectCores())
clusterEvalQ(cl, { library("sf") })
clusterExport(cl, "st_linestring")
res <- parallel::parLapply(cl, X = waydf$geometry$coordinates,
fun = function(i) st_linestring)
stopCluster(cl)
The message Error in checkCluster(cl): not a valid cluster
that you get with your code is because after you do cl <- clusterEvalQ(cl, { library("sf") })
it is no longer a cluster
object.
Comments
-
Maryam Koulaei over 1 year
I am new in R programming and I have code like below and I know that the windows does not support multicore but I don't know how to change this part of the code. Can someone suggest me an equivalent code without using the mc.cores feature?
rpl <- unlist( lapply( waydf$geometry$coordinates , nrow ) ) # row per line waydf <- waydf[ rpl > 1 , ] ll <- parallel::mclapply( waydf$geometry$coordinates , st_linestring, mc.cores =parallel::detectCores() - 1 ) outdf <- sf::st_sf( line_geometry = sf::st_sfc( ll , crs = epsg ) , osm_id = waydf$id )
-
Maryam Koulaei about 6 yearsI do still get the same error :(. how can I run this function without multicore feature?
-
CPak about 6 yearsSorry, but did you replace
mclapply
withparLapply
? What is your error? Please show -
Maryam Koulaei about 6 yearsError in makeCluster(detectCores()) : could not find function "makeCluster"
-
Maryam Koulaei about 6 yearsnow , i got this error: Error in checkCluster(cl) : not a valid cluster in this line: cl <- parallel::clusterExport(cl, "st_linestring")
-
CPak about 6 yearsSince
st_linestring
is insf
package, you do not need to export it. I commented out the line -
Maryam Koulaei about 6 yearsthis is the new error: Error in detectCores() : could not find function "detectCores"
-
Maryam Koulaei about 6 yearsYEs, I loaded the parallel and the problem of detectingCores is solved. But I get again not a valid cluster error in the last line : Error in checkCluster(cl) : not a valid cluster
-
Maryam Koulaei about 6 yearscould you please how can I run this code without parallelizing? in a very simple way.
-
CPak about 6 yearsI'm assuming your code works. You can replace
mclapply
with the answer provided above (see after the 2nd edit). I recommend you start a new R session before running this code. -
Maryam Koulaei about 6 yearsI added a picture of my code with the error. I have no idea why do I still get this error.
-
CPak about 6 yearsDid you start a new R session? My guess is that you loaded a cluster before using
doMC
. And could you typecl
and show the output? -
CPak about 6 yearsYou should see something like
socket cluster with 8 nodes on host ‘localhost’
when typingcl
-
Maryam Koulaei about 6 yearsyes, I restart a new R session each time I change the code.
-
Maryam Koulaei about 6 years@ HenrikB, hey. thanks for the answer. I tried to run your code but i got this error: Error in get(name, envir = envir) : object 'st_linestring' not found