Getting stdin from a named pipe

Question

What I am trying to do is run python in a terminal window and redirect it's stdin from a named pipe. Then I write to the named pipe in another terminal and have that command execute on python.

Terminal 1:

mkfifo p1
python < p1

Terminal 2:

echo -n "print \"Hello World\"" > p1

What happens is - python prints Hello World and exits. What I want to do is keep python running to take a next command. How do I do this in the shell?

Solution 1

You need to

• Run python interactively even though its stdin is not a terminal: use python -i
• keep the writing end of the pipe open, otherwise python will detect EOF and exit.

So:

python -i < p1

And elsewhere:

# open p1 on fd 3 of the shell process¹ for instance so stdout is
# left untouched in case you still want to write messages there.
exec 3> p1

# write something on fd 3. For instance with ksh/zsh's print
# builtin:
print -u3 '1j*1j'

# or for commands such as echo that can only print things
# on stdout (fd 1), duplicate that fd 3 to their fd 1:
echo '1j*1j' >&3 # short for 1>&3

# after echo (a builtin² in virtually all shells) returns, fd 1
# is restored to what it was before, but fd 3 remains open on
# the writing end of the pipe, so python still doesn't see EOF
# there.

echo normal message on stdout
echo 1+1 >&3 # more instructions for python
...
# and when done, close that file descriptor so python sees the EOF
# (assuming that fd 3 was the only file descriptor left open on
# the writing end of that pipe):
exec 3>&-

In scripts especially, as an alternative, you could redirect a whole command group instead of manually opening and closing the fd with exec³.

{
  echo ... >&3
  ...
  ...
} 3> p1

---

^{¹ that fd 3 will be inherited by child processes, and (except in ksh93 which sets the close-on-exec flag on it) by other commands executed in those processes if any.}

^{² that also works for non-builtin commands of course. For non-builtins, the shell doesn't need to save and restore fd 1 as the redirection is only performed in the child process that is forked to execute the command. For external commands, except in ksh93 which does that automatically, you may actually want to close that fd 3 so it's not leaked to them and background processes they may eventually spawn (cmd >&3 3>&-).}

^{³ note that in that case, ksh93 does not set the close-on-exec flag on that fd 3.}

Solution 2

You can use tail -f to keep the fifo open after echo writes to it.

tail -n1 -f p1 | python

Why this works

python is reading from p1. When it reaches the end of the file, it stops reading. This is normal behavior for file reads, even if the file is a named pipe. tail with the -f (follow) flag will keep reading from a file after its end is reached.

Solution 3

You need to send the entire program at once.

When you call run python < p1 the shell is waiting for input before invoking python. That is, python doesn't even begin executing at all until the entire data stream has been read by the shell and then is passed in its entirety to python.

Even running python -u p1 instead (that is, unbuffered and read from file p1) python will try to read the entire file before it executes any of it.

Try this experiment.

Terminal 1:

mkfifo p1
python < p1

Terminal 2:

cat > p1
print "Hello World"
print "Hello World"

You'll see that you can send multiple lines but python in Term 1 does nothing. Now press ctrl+D. The entire program executes at once.

So, to summarize, if you want python to read from a pipe you need to send the entire program. You can't use python interactively this way.

Solution 4

Maybe the tail approach is better (more flexible) but as an alternative:

{ echo -n "print \"Hello World\""; cat; } > p1

View more solutions

Comments

[Lord Loh.]

I tried echo "print \"Hello World\" " > p1 in the second terminal and nothing happened - but the terminal was not blocked either. The terminal with python remained blocked till I ^c it and exit it and terminate python with a keyboard interrupt message being displayed by python.

[Hauke Laging]

@LordLoh. May be a buffering problem. Probably python will execute the command if enough output has been created so that the first line finally gets written to the FIFO. But as there is a working solution it wouldn't make sense to put effort in solving this problem.

[Lord Loh.]

Thanks! It worked. I am not familiar with what you have done. Would you please add some details to your answer to explain what is going on. What is exec 3> p1 doing and what is &3 & exec 3> &1? Thank you.

[Wildcard]

A question, would exec 3>&- work the same as exec 3>&1 here?

[Stéphane Chazelas]

@Wildcard I suspect I intended to write 3>&- here. 3>&1 would work as well but make little sense. Thanks

[Mael]

I used this tail -f trick when unpacking block-split tar archive through a named pipe. It worked wonderfully.

[The Quark]

Do I understand correctly that echo '1j*1j' >&3 creates an additional writing end to p1 coming from file-descriptor 1 of the echo process and that the EOF condition is actually seen by file-descriptor 1 of echo and closes this additional writing end, whereas in the OP this is the only writing end to be closed? That is, EOF condition is first seen by the file-descriptor, which makes it to close, and when all writing ends are closed an EOF condition is seen by python which closes?

[The Quark]

And do I understand correctly that file-descriptor 3 created by exec 3> p1 actually belongs to the current bash process?

[Stéphane Chazelas]

@TheQuark see if edit makes it any clearer.

[Stéphane Chazelas]

@TheQuark, but, yes, you're correct on both accounts.

[The Quark]

Thank you. It sort-of makes it clearer (plus it adds the interesting print -u3 and inheritance parts). But my question was in fact motivated by the following underlying question: why an EOF condition was not sent to python by the echo '1j*1j' >&3 command, what makes it different from echo -n "print \"Hello World\"" > p1? Understanding the behaviour comes from realising that >& creates an additional writing end (and not a chain like 1 > 3 > p1 > python) and that the EOF condition affects a specific file-descriptor (and not directly the named pipe). Thank you for answering that part too.