Returning data from forked processes
Solution 1
I wrapped all the solutions I found along the way (some other problems like user exiting + piping-buffers) into ruby parallel gem. Now it is as easy as:
results = Parallel.map([1,2,3],:in_processes=>4) do |i|
execute_something(i)
end
or
results = Parallel.map([1,2,3],:in_threads=>4) do |i|
execute_something(i)
end
Solution 2
We actually just had to handle this problem in Rails isolation testing. I posted about it some on my blog.
Basically, what you want to do is open a pipe in the parent and child, and have the child write to the pipe. Here's a simple way to run the contents of a block in a child process and get back the result:
def do_in_child
read, write = IO.pipe
pid = fork do
read.close
result = yield
Marshal.dump(result, write)
exit!(0) # skips exit handlers.
end
write.close
result = read.read
Process.wait(pid)
raise "child failed" if result.empty?
Marshal.load(result)
end
Then you could run:
do_in_child do
require "some_polluting_library"
SomePollutingLibrary.some_operation
end
Note that if you do a require in the child, you will not have access to that library in the parent, so you cannot return an object of that type using this method. However, you could return any type that's available in both.
Also note that a lot of the details here (read.close
, Process.wait2(pid)
) are mostly housekeeping details, so if you use this a lot you should probably move this out into a utility library that you can reuse.
Finally, note that this will not work on Windows or JRuby, since they don't support forking.
Solution 3
Thanks for all the answers, I got my solution up and running, still need to see how to handle non-forking environments, but for now it works :)
read, write = IO.pipe
Process.fork do
write.puts "test"
end
Process.fork do
write.puts 'test 2'
end
Process.wait
Process.wait
write.close
puts read.read
read.close
you can see it in action @ parallel_specs Rails plugin
Solution 4
Yes, you can create a subprocess to execute a block inside.
I recommend the aw
gem:
Aw.fork! { 6 * 7 } # => 42
Of course, it prevents from side effects:
arr = ['foo']
Aw.fork! { arr << 'FUU' } # => ["foo", "FUU"]
arr # => ["foo"]
Related videos on Youtube
Comments
-
grosser almost 4 years
If I do
Process.fork do x end
how can I know what x returned (e.g. true/fase/string) ?
(Writing to a file/database is not an option...)
-
grosser almost 15 yearsHow can this help in getting what the 'x' returned ?
-
mipadi almost 15 yearsThe child process runs in a separate process (obviously), so to get the value of whatever "x" does, you'll probably have to communicate via sockets, pipes, or something similar. I doubt you can, e.g., set variables inside the block that are reflected outside of the block, since the child process has separate memory, etc.
-
Yehuda Katz almost 15 yearsHave you seen the isolation test link I provided in my answer? It handles both forking and non-forking environments, and may already do everything you need.
-
Automatico about 10 yearsCan you pass objects, or even classes, as parameters here? I could really use a way to fork a process and pass an entire environment to it.