Why am I getting some extra, weird characters when making a file from grep output?

Question

I am doing a very basic command that never gave me trouble in the past, but is inexplicably returning undesired characters now.

I am in BASH on linux, and simply want to search through a directory and make a file containing filenames that match a pattern:

ls | grep "*.file_ID" > my_list.txt

...This works fine, and if I cat the data:

   cat my_list.txt
       seriesA.file_ID
       seriesB.file_ID
       seriesC.file_ID

However, when I try to feed this file into downstream processes, I keep getting a weird errors, as if the file isn't properly formatted as a list of file names. When I open the file in vim to reveal any unnecessary characters, I find the file actually looks like this:

vi my_list.txt

^[[00mseriesA.file_ID^[[00m
^[[00mseriesB.file_ID^[[00m
^[[00mseriesC.file_ID^[[00m

For some reason, every line is started and ended with the characters ^[[00m. If I delete these characters, all of the downstream processes work fine. However, I need to have my scripts automatically make such a file list, so I can't keep going in and manually deleting these chars.

Does anyone know what is producing the ^[[00m characters? I don't have any idea where they are coming from, and need a to be able to generate files without them.

Thanks!

Solution 1

Don't use ls:

printf "%s\n" *.file_ID > my_list.txt

Solution 2

Probably your GREP_OPTIONS environment variable contains --color=always, which causes the output to be stuffed with control characters, even when piped to a file.

Use --color=auto instead.

http://www.gnu.org/software/grep/manual/html_node/Environment-Variables.html

Even better, don't use grep:

ls *.file_ID > my_list.txt

Solution 3

• usually it is automatically => grep --color=auto

• if it pipes into a csv file that looks like this ^[[00...^[[00m

• you would have to type this in the terminal:
  grep --color=auto "your regex" > example.csv

• if you want it to be a permanent situation where you do not have to type "--color=auto" every time, type this in the terminal:
  export GREP_OPTIONS='--color=auto'

more info: https://linuxcommando.blogspot.com/2007/10/grep-with-color-output.html

Solution 4

This should take care of it (assuming GNU find and no directory traversing):

find . -maxdepth 1 -type f -name "*.file_ID" -printf "%f\n" > my_list.txt

Example:

~> ls *file_ID*
a.file_ID  b.file_ID  c.file_ID

~> find . -maxdepth 1 -type f -name "*.file_ID" -printf "%f\n" > my_list.txt

~> cat my_list.txt
a.file_ID
b.file_ID
c.file_ID

As far as the "^[[00m" characters, check your ls options:

~> alias -p | grep "ls="

You may get something like:

alias ls='/bin/ls $LS_OPTIONS'

If so, check env for this:

~> env | grep LS_OP
LS_OPTIONS=-N --color=tty -T 0

The character string you're referencing is used to turn off colors, so your shell likely has been set to show colors. Removing and/or changing the ls alias should resolve it.

View more solutions

Comments

[chepner]

Or in the event of a large number of matching files that could overflow the command line, find . -name '*.file_ID' > my_list.txt (although, some extra work is needed to remove the leading './' that find will produce for each file).

[twalberg]

Or, even better... echo *.file_ID > ..., or for f in *.file_ID; do ....

[Ansel Halliburton]

This does not answer the question

[Ansel Halliburton]

You're right, the extra characters are not produced by grep, but by ls, which is why there is no problem with grep -o (as per the comments above)

[glenn jackman]

@Pumbaa80, the literal answer to the question is use command ls, but the sensible answer is "don't do that"

[Stuart]

@glenn jackman, The question was "Why...". Can you explain the reason this happens so others can apply the general solution?

[glenn jackman]

Most likely the ls command is actually an alias or function that adds the --color option. So the odd chars are actually terminal color codes. ls output is intended for human eyes, not as input to other programs.'