gnome-terminal starts with "grep: write error: Broken pipe" message

Question

I am running Ubuntu 14.04.3, it's uptodate. I don't know why, for a few days I began to take grep: write error: Broken pipe message on launching gnome-terminal . It seems to be harmless but it bothers me. How can I debug it?

EDIT: I moved aliases and functions each to separate files such as .bash_aliases and .bash_functions and added a command to load them from .bashrc

 if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
 fi

 if [ -f ~/.bash_functions ]; then
. ~/.bash_functions
 fi

If I don't load .bash_functions problem disappears.

I am trying to find the faulty one by disabling each function one by one.

This one gives me the same error but when I disable it I keep getting the same error, so I may have more faulty functions.

 ls -lt  $PWD| grep ^d | head -1 | cut -b 51- 

 grep:  development
 write error: Broken pipe

I wonder why I begin to take that error.

EDIT2:

I found a similar problem here boken pipe

The root of the problem also seems similar.

I tried the given test command in the link which have the same error:

 bash -c '(while echo foo; do :; done); echo status=$? >&2' |  head
 foo
 foo
 foo
 foo
 foo
 foo
 foo
 foo
 foo
 foo
 bash: line 0: echo: write error: Broken pipe
 status=0

EDIT3:

Though that unbuffer workaround I posted below as an answer to my own question works, I am not satisfied with it, but my knowledge about debugging is limited. Acoording to this link https://lists.gnu.org/archive/html/bug-coreutils/2007-11/msg00080.html it stems from SIGPIPE trap by another task, and this link https://lists.gnu.org/archive/html/bug-coreutils/2007-11/msg00154.html pinpoints the exact cause of the problem, it's one of the pam authentication module which I am in trouble with it recently.

Solution 1

After hours of struggling with the problem I found a working workaround (I hope so)

Problem seems to be deeper and complicated. Many people met the same bug. Fixing it is beyond my coverage.

Closest workaround posted here how-can-i-fix-a-broken-pipe-error by Andrew Beals at bottom like :

ls -lt $PWD|dd obs=1M | grep -m 1 ^d | cut -b 51-

is not neat.

When I intuited that it's related to pipe buffer I gave a shot to unbuffer command like :

 unbuffer ls -lt $PWD| grep -m 1 ^d | cut -b 51-

It works well.

I hope somebody posts the real cause of the problem.

EDIT: A bash Guru would suggest this simple solution , redirecting stderr to /dev/null

 ls -lt $PWD 2>/dev/null | grep -m 1 ^d | cut -b 51-

Solution 2

There's a great explanation of this problem on this Super User answer: How can I fix a Broken Pipe error?.

Commands in pipes are run asynchronously: this means that in a pipe such as command1 | command2 there's no guarantee that command1 will end before command2.

When using [...] | grep | head -n 1, head ends as soon as it has read one line; if this happens before grep has finished writing to the pipe, grep receives a SIGPIPE signal and errors out.

As explained in the answer below that Super User answer, a workaround is to pipe the output of what's before head in the pipeline to tail -n +1 first, which will ignore the SIGPIPE signal:

command | tail -n +1 | head -n 1

But in this case there's not even any need for head, since grep has an option to print only the first match:

[...] | grep -m 1

Comments

[muru]

Look for grep in your .bashrc, .profile, etc. And is it only in gnome-terminal? Not in xterm, or in the TTYs?

[kenn]

@muru I already checked .bashrc and .profile I haven't added any new functions recently. I have a number of grep and | (pipe) in functions in them but they are there for months.

[kenn]

@muru when I run your command shopt -s nullglob; ls -t */ | head -1 I get development/: ls: write error: Broken pipe output

[kenn]

@muru I posted output of ls -dt */ in development directory. Added it to my question.

[muru]

I meant, I wanted the output of printf '%q' */ in whichever directory you were originally in, and that I meant the replacement function would be: shopt -s nullglob; ls -dt */ | head -1. Sorry for the miscommunication.

[kenn]

@muru shopt -s nullglob; ls -dt */ | head -1 just runs fine and gives development/

[kenn]

Thank you for responding. How can I apply it to my particular case for ls -lt $PWD| grep ^d | head -1 | cut -b 51- and generalize it?

[kos]

@kenn I had a misconception about pipes and the answer wasn't technically accurate before, it should be now. Also see the update, there's no real need for grep | head.

[kos]

@kenn Meaning: ls -lt $PWD| grep -m 1 ^d | cut -b 51-.

[kenn]

ls -lt $PWD| grep -m 1 ^d | cut -b 51- still gives me the same error development ls: write error: Broken pipe :(

[kos]

@kenn Looks like the same is happening with cut as well. What about ls -lt $PWD| grep ^d | tail -n +1 | head -n 1 | cut -b 51-?

[kenn]

hey, it works :) how?

[kenn]

Sorry, I spoke too early, it fails

[kenn]

Sometimes it just runs well but it often fails

[kos]

@kenn Ah but the last error was from ls. Just for testing, try this: ls -lt $PWD| tail -n +1 | grep -m 1 ^d | cut -b 51-.

[kenn]

Right, I noticed that just after I posted the output. Above code gives me development tail: write error: Broken pipe tail: write error

[kos]

@kenn I'm starting to think that in this case the problem is with Bash or grep. Version of both?

[kenn]

GNU bash, version 4.3.11 and grep (GNU grep) 2.16

[kos]

@kenn Sorry, also the output of md5sum /bin/bash and md5sum /bin/grep, to check if one of the two is corrupted... Though a corruption normally causes way worse problems.

[kenn]

you are the master and I need your help. 9c96f6ac51aaf5deb8043d8c2330da69 /bin/bash and b0ce14813cf497a45c894d73fa309ba8 /bin/grep

[kos]

Let us continue this discussion in chat.

[Andrew Beals]

Killing stderr's output hides other error conditions from you. When working with large output today, I found that I had to increase the buffer size in dd again and though about the problem a little more. In the strictest sense, the pipe is "broken", but the shell shoudn't be telling the user about it and for the simple case of excess data, but not too much excess data as it's all sitting in buffers, SIGPIPE does not happen: cat /etc/passwd | grep -m1 ssh For "Big Data" problems here such as your example, I would change it to ls -lt $PWD | (grep -m 1 ^d ; dd of=/dev/null) | cut -b 51-

[kenn]

@AndrewBeals Thank you for further info and suggestion. Beside the point do you think Gnome developers are programmers or self-satisfying obfuscators?