Continuously read from STDOUT of external process in Ruby

Question

I want to run blender from the command line through a ruby script, which will then process the output given by blender line by line to update a progress bar in a GUI. It's not really important that blender is the external process whose stdout I need to read.

I can't seem to be able to catch the progress messages blender normally prints to the shell when the blender process is still running, and I've tried a few ways. I always seem to access the stdout of blender after blender has quit, not while it's still running.

Here's an example of a failed attempt. It does get and print the first 25 lines of the output of blender, but only after the blender process has exited:

blender = nil
t = Thread.new do
  blender = open "| blender -b mball.blend -o //renders/ -F JPEG -x 1 -f 1"
end
puts "Blender is doing its job now..."
25.times { puts blender.gets}

Edit:

To make it a little clearer, the command invoking blender gives back a stream of output in the shell, indicating progress (part 1-16 completed etc). It seems that any call to "gets" the output is blocked until blender quits. The issue is how to get access to this output while blender is still running, as blender prints it's output to shell.

Solution 1

I've had some success in solving this problem of mine. Here are the details, with some explanations, in case anyone having a similar problem finds this page. But if you don't care for details, here's the short answer:

Use PTY.spawn in the following manner (with your own command of course):

require 'pty'
cmd = "blender -b mball.blend -o //renders/ -F JPEG -x 1 -f 1" 
begin
  PTY.spawn( cmd ) do |stdout, stdin, pid|
    begin
      # Do stuff with the output here. Just printing to show it works
      stdout.each { |line| print line }
    rescue Errno::EIO
      puts "Errno:EIO error, but this probably just means " +
            "that the process has finished giving output"
    end
  end
rescue PTY::ChildExited
  puts "The child process exited!"
end

And here's the long answer, with way too many details:

The real issue seems to be that if a process doesn't explicitly flush its stdout, then anything written to stdout is buffered rather than actually sent, until the process is done, so as to minimize IO (this is apparently an implementation detail of many C libraries, made so that throughput is maximized through less frequent IO). If you can easily modify the process so that it flushes stdout regularly, then that would be your solution. In my case, it was blender, so a bit intimidating for a complete noob such as myself to modify the source.

But when you run these processes from the shell, they display stdout to the shell in real-time, and the stdout doesn't seem to be buffered. It's only buffered when called from another process I believe, but if a shell is being dealt with, the stdout is seen in real time, unbuffered.

This behavior can even be observed with a ruby process as the child process whose output must be collected in real time. Just create a script, random.rb, with the following line:

5.times { |i| sleep( 3*rand ); puts "#{i}" }

Then a ruby script to call it and return its output:

IO.popen( "ruby random.rb") do |random|
  random.each { |line| puts line }
end

You'll see that you don't get the result in real-time as you might expect, but all at once afterwards. STDOUT is being buffered, even though if you run random.rb yourself, it isn't buffered. This can be solved by adding a STDOUT.flush statement inside the block in random.rb. But if you can't change the source, you have to work around this. You can't flush it from outside the process.

If the subprocess can print to shell in real-time, then there must be a way to capture this with Ruby in real-time as well. And there is. You have to use the PTY module, included in ruby core I believe (1.8.6 anyways). Sad thing is that it's not documented. But I found some examples of use fortunately.

First, to explain what PTY is, it stands for pseudo terminal. Basically, it allows the ruby script to present itself to the subprocess as if it's a real user who has just typed the command into a shell. So any altered behavior that occurs only when a user has started the process through a shell (such as the STDOUT not being buffered, in this case) will occur. Concealing the fact that another process has started this process allows you to collect the STDOUT in real-time, as it isn't being buffered.

To make this work with the random.rb script as the child, try the following code:

require 'pty'
begin
  PTY.spawn( "ruby random.rb" ) do |stdout, stdin, pid|
    begin
      stdout.each { |line| print line }
    rescue Errno::EIO
    end
  end
rescue PTY::ChildExited
  puts "The child process exited!"
end

Solution 2

use IO.popen. This is a good example.

Your code would become something like:

blender = nil
t = Thread.new do
  IO.popen("blender -b mball.blend -o //renders/ -F JPEG -x 1 -f 1") do |blender|
    blender.each do |line|
      puts line
    end
  end
end

Solution 3

STDOUT.flush or STDOUT.sync = true

Solution 4

Blender probably doesn't print line-breaks until it is ending the program. Instead, it is printing the carriage return character (\r). The easiest solution is probably searching for the magic option which prints line-breaks with the progress indicator.

The problem is that IO#gets (and various other IO methods) use the line break as a delimiter. They will read the stream until they hit the "\n" character (which blender isn't sending).

Try setting the input separator $/ = "\r" or using blender.gets("\r") instead.

BTW, for problems such as these, you should always check puts someobj.inspect or p someobj (both of which do the same thing) to see any hidden characters within the string.

View more solutions

Comments

[ehsanul]

I've tried this. The problem is the same. I get access to the output afterwards. I believe IO.popen starts by running the first argument as a command, and waits for it to end. In my case, the output is given by blender while blender is still processing. And then the block is invoked after, which doesn't help me.

[ehsanul]

Here's what I tried. It returns the output after blender is done: IO.popen( "blender -b mball.blend //renders/ -F JPEG -x 1 -f 1", "w+") do |blender| blender.each { |line| puts line; output += line;} end

[ehsanul]

Ok, I just tried it with an external ruby process to test, and you're right. Seems to be a blender issue. Thanks for the answer anyways.

[ehsanul]

I just inspected the output given, and it seems that blender uses a line break (\n), so that wasn't the issue. Thanks for the tip anyways, I'll keep that in mind next time I'm debugging something like this.

[ehsanul]

Turns out there is a way to get the output through ruby after all, even though blender doesn't flush its stdout. Details shortly in a separate answer, in case you are interested.

[jake]

This is great, but I believe the stdin and stdout block parameters should be swapped. See: ruby-doc.org/stdlib-1.9.3/libdoc/pty/rdoc/…

[Boris B.]

How to close the pty? Kill the pid?

[Sergiy Seletskyy]

Awesome answer. You helped me to improve my rake deploy script for heroku. It displays 'git push' log in real time and aborts task if 'fatal:' found gist.github.com/sseletskyy/9248357

[Pakman]

I originally tried to use this method but 'pty' isn't available in Windows. As it turns out, STDOUT.sync = true is all that's needed (mveerman's answer below). Here's another thread with some example code.

[user4581301]

Not lame! Worked for me.

[caram]

More precisely: STDOUT.sync = true; system('<whatever-command>')