Capturing stdout from a system() command optimally

Question

I'm trying to start an external application through system() - for example, system("ls"). I would like to capture its output as it happens so I can send it to another function for further processing. What's the best way to do that in C/C++?

Solution 1

From the popen manual:

#include <stdio.h>

FILE *popen(const char *command, const char *type);

int pclose(FILE *stream);

Solution 2

EDIT: misread question as wanting to pass output to another program, not another function. popen() is almost certainly what you want.

System gives you full access to the shell. If you want to continue using it, you can redirect it's output to a temporary file, by system("ls > tempfile.txt"), but choosing a secure temporary file is a pain. Or, you can even redirect it through another program: system("ls | otherprogram");

Some may recommend the popen() command. This is what you want if you can process the output yourself:

FILE *output = popen("ls", "r");

which will give you a FILE pointer you can read from with the command's output on it.

You can also use the pipe() call to create a connection in combination with fork() to create new processes, dup2() to change the standard input and output of them, exec() to run the new programs, and wait() in the main program to wait for them. This is just setting up the pipeline much like the shell would. See the pipe() man page for details and an example.

Solution 3

The functions popen() and such don't redirect stderr and such; I wrote popen3() for that purpose.

Here's a bowdlerised version of my popen3():

int popen3(int fd[3],const char **const cmd) {
        int i, e;
        int p[3][2];
        pid_t pid;
        // set all the FDs to invalid
        for(i=0; i<3; i++)
                p[i][0] = p[i][1] = -1;
        // create the pipes
        for(int i=0; i<3; i++)
                if(pipe(p[i]))
                        goto error;
        // and fork
        pid = fork();
        if(-1 == pid)
                goto error;
        // in the parent?
        if(pid) {
                // parent
                fd[STDIN_FILENO] = p[STDIN_FILENO][1];
                close(p[STDIN_FILENO][0]);
                fd[STDOUT_FILENO] = p[STDOUT_FILENO][0];
                close(p[STDOUT_FILENO][1]);
                fd[STDERR_FILENO] = p[STDERR_FILENO][0];
                close(p[STDERR_FILENO][1]);
                // success
                return 0;
        } else {
                // child
                dup2(p[STDIN_FILENO][0],STDIN_FILENO);
                close(p[STDIN_FILENO][1]);
                dup2(p[STDOUT_FILENO][1],STDOUT_FILENO);
                close(p[STDOUT_FILENO][0]);
                dup2(p[STDERR_FILENO][1],STDERR_FILENO);
                close(p[STDERR_FILENO][0]);
                // here we try and run it
                execv(*cmd,const_cast<char*const*>(cmd));
                // if we are there, then we failed to launch our program
                perror("Could not launch");
                fprintf(stderr," \"%s\"\n",*cmd);
                _exit(EXIT_FAILURE);
        }

        // preserve original error
        e = errno;
        for(i=0; i<3; i++) {
                close(p[i][0]);
                close(p[i][1]);
        }
        errno = e;
        return -1;
}

Solution 4

The most efficient way is to use stdout file descriptor directly, bypassing FILE stream:

pid_t popen2(const char *command, int * infp, int * outfp)
{
    int p_stdin[2], p_stdout[2];
    pid_t pid;

    if (pipe(p_stdin) == -1)
        return -1;

    if (pipe(p_stdout) == -1) {
        close(p_stdin[0]);
        close(p_stdin[1]);
        return -1;
    }

    pid = fork();

    if (pid < 0) {
        close(p_stdin[0]);
        close(p_stdin[1]);
        close(p_stdout[0]);
        close(p_stdout[1]);
        return pid;
    } else if (pid == 0) {
        close(p_stdin[1]);
        dup2(p_stdin[0], 0);
        close(p_stdout[0]);
        dup2(p_stdout[1], 1);
        dup2(::open("/dev/null", O_WRONLY), 2);

        /// Close all other descriptors for the safety sake.
        for (int i = 3; i < 4096; ++i) {
            ::close(i);
        }

        setsid();
        execl("/bin/sh", "sh", "-c", command, NULL);
        _exit(1);
    }

    close(p_stdin[0]);
    close(p_stdout[1]);

    if (infp == NULL) {
        close(p_stdin[1]);
    } else {
        *infp = p_stdin[1];
    }

    if (outfp == NULL) {
        close(p_stdout[0]);
    } else {
        *outfp = p_stdout[0];
    }

    return pid;
}

To read output from child use popen2() like this:

int child_stdout = -1;
pid_t child_pid = popen2("ls", 0, &child_stdout);

if (!child_pid) {
    handle_error();
}

char buff[128];
ssize_t bytes_read = read(child_stdout, buff, sizeof(buff));

To both write and read:

int child_stdin = -1;
int child_stdout = -1;
pid_t child_pid = popen2("grep 123", &child_stdin, &child_stdout);

if (!child_pid) {
    handle_error();
}

const char text = "1\n2\n123\n3";
ssize_t bytes_written = write(child_stdin, text, sizeof(text) - 1);

char buff[128];
ssize_t bytes_read = read(child_stdout, buff, sizeof(buff));

Solution 5

Actually, I just checked, and:

1. popen is problematic, because the process is forked. So if you need to wait for the shell command to execute, then you're in danger of missing it. In my case, my program closed even before the pipe got to do it's work.

2. I ended up using system call with tar command on linux. The return value from system was the result of tar.

So: if you need the return value, then not no only is there no need to use popen, it probably won't do what you want.

View more solutions