Piping and Redirection

Question

What is exact difference between piping and redirection?

Where should we use piping and where should we use redirection?

How they internally work?

Solution 1

Redirection is (mostly) for files (you redirect streams to/from files).

Piping is for processes: you pipe (redirect) streams from one process to another.

Essentially what you really do is "connect" one standard stream (usually stdout) of one process to standard stream of another process (usually stdin) via pipe.

Pipes have also the synchronization "side effect" : they block one process (on reading) when the other has nothing to write (yet) or when reading process cannot read fast enough (when the pipe's buffer is full).

Solution 2

Piping directs the output of a program to another program.

For example:

ls * | grep "name"

Pipes the names of all files in the current directory to grep. Re-direction directs or appends output to a file.

ls * > file  # writes all file names in current directory to the "file"
ls * >> file # appends all files names in current directory to the "file"

Piping saves you the hassle of having to write to a file, then read from a file to execute a program on the output of another program.

ls * > file
grep "name" file

is equivalent to

ls * | grep "name"

As for how it they work internally, I am only learning that my self now. But I found this link which offers some discussion on it.

How Does Piping Work in Linux?

You should use piping if you want to pass outputs between programs; use redirection if you want to write to a file.

Solution 3

Basically redirection and piping are a few ways among many to achieve Inter Process communication in Unix.

1. Redirection: Data is written to and read from a typical UNIX file. Any number of processes can interoperate. this must be used when sharing large data sets.

ls > FileName

2. Piping : Piping is a process where the output of one process is made the input of another. They were evolved in the most primitive forms of the Unix operating system. They provide unidirectional flow of communication between processes within the same system. A pipe is created by invoking the pipe system call, which creates a pair of file descriptors. [ For file descriptors read http://www.bottomupcs.com/file_descriptors.html ]

ls | grep $myName

It works on simple data sharing, such as producer and consumer.

Property Comparison: Piping is always uni-directional while redirection could be used to redirecting input as well as output.

ls > grep myFileName [ Redirecting output of first command to later one ] sort < fileName.txt [ Redirecting fileName.txt file as an input to command sort ]

One can also write below to use bi-directional redirect in single statement.

sort < fileName.txt > sortNewFile.txt

While Piping, it is always output of first command supplied to the later one and that to simulanoeously.

ls | grep myName | awk '{ print $NF }' [ multiple piping in a single statement ]

Note 1: command > fileName . If there is a command named fileName, that would make using redirection a lot harder and more error prone. One must check first, whether there's a command named like destination file.

Other ways to achieve IPC in Unix system are:

3. Named pipe
4. Signal
5. Shared memory
6. Socket

Solution 4

Redirection: send the output (stdout and/or stderr) of a command to a file Example : ls > your_file write the result of directory listing to a file named your_file

Piping: send the output to another command. Example ls | wcsend the same output (directory listing) to the command wc which count characters.

Solution 5

I have noticed that pipelining applies to the output of process substitution, but not redirection:

bash-3.2$ echo $'one\ntwo\nthree' | tee >(grep o) | cat > pipe
bash-3.2$ echo $'one\ntwo\nthree' | tee >(grep o) > redirect
bash-3.2$ one
two

bash-3.2$ cat pipe
one
two
three
one
two
bash-3.2$ cat redirect
one
two
three

View more solutions

Comments

[edge]

Thanks! Explained nicely.

[Karan Kaw]

Useful Read askubuntu.com/questions/172982/…

[tripleee]

Of course, you probably want ls *name* or perhaps simply printf '%s\n' *name* or for file in *name*