Why am I getting some extra, weird characters when making a file from grep output?

10,466

Solution 1

Don't use ls:

printf "%s\n" *.file_ID > my_list.txt

Solution 2

Probably your GREP_OPTIONS environment variable contains --color=always, which causes the output to be stuffed with control characters, even when piped to a file.

Use --color=auto instead.

http://www.gnu.org/software/grep/manual/html_node/Environment-Variables.html

Even better, don't use grep:

ls *.file_ID > my_list.txt

Solution 3

  • usually it is automatically => grep --color=auto

  • if it pipes into a csv file that looks like this ^[[00...^[[00m

  • you would have to type this in the terminal:
    grep --color=auto "your regex" > example.csv

  • if you want it to be a permanent situation where you do not have to type "--color=auto" every time, type this in the terminal:
    export GREP_OPTIONS='--color=auto'

more info: https://linuxcommando.blogspot.com/2007/10/grep-with-color-output.html

Solution 4

This should take care of it (assuming GNU find and no directory traversing):

find . -maxdepth 1 -type f -name "*.file_ID" -printf "%f\n" > my_list.txt

Example:

~> ls *file_ID*
a.file_ID  b.file_ID  c.file_ID

~> find . -maxdepth 1 -type f -name "*.file_ID" -printf "%f\n" > my_list.txt

~> cat my_list.txt
a.file_ID
b.file_ID
c.file_ID

As far as the "^[[00m" characters, check your ls options:

~> alias -p | grep "ls="

You may get something like:

alias ls='/bin/ls $LS_OPTIONS'

If so, check env for this:

~> env | grep LS_OP
LS_OPTIONS=-N --color=tty -T 0

The character string you're referencing is used to turn off colors, so your shell likely has been set to show colors. Removing and/or changing the ls alias should resolve it.

Share:
10,466

Related videos on Youtube

jake9115
Author by

jake9115

Updated on September 18, 2022

Comments

  • jake9115
    jake9115 less than a minute

    I am doing a very basic command that never gave me trouble in the past, but is inexplicably returning undesired characters now.

    I am in BASH on linux, and simply want to search through a directory and make a file containing filenames that match a pattern:

    ls | grep "*.file_ID" > my_list.txt
    

    ...This works fine, and if I cat the data:

       cat my_list.txt
           seriesA.file_ID
           seriesB.file_ID
           seriesC.file_ID
    

    However, when I try to feed this file into downstream processes, I keep getting a weird errors, as if the file isn't properly formatted as a list of file names. When I open the file in vim to reveal any unnecessary characters, I find the file actually looks like this:

    vi my_list.txt
    
    ^[[00mseriesA.file_ID^[[00m
    ^[[00mseriesB.file_ID^[[00m
    ^[[00mseriesC.file_ID^[[00m
    

    For some reason, every line is started and ended with the characters ^[[00m. If I delete these characters, all of the downstream processes work fine. However, I need to have my scripts automatically make such a file list, so I can't keep going in and manually deleting these chars.

    Does anyone know what is producing the ^[[00m characters? I don't have any idea where they are coming from, and need a to be able to generate files without them.

    Thanks!

  • chepner
    chepner about 9 years
    Or in the event of a large number of matching files that could overflow the command line, find . -name '*.file_ID' > my_list.txt (although, some extra work is needed to remove the leading './' that find will produce for each file).
  • twalberg
    twalberg about 9 years
    Or, even better... echo *.file_ID > ..., or for f in *.file_ID; do ....
  • user123444555621
    user123444555621 about 9 years
    This does not answer the question
  • user123444555621
    user123444555621 about 9 years
    You're right, the extra characters are not produced by grep, but by ls, which is why there is no problem with grep -o (as per the comments above)
  • glenn jackman
    glenn jackman about 9 years
    @Pumbaa80, the literal answer to the question is use command ls, but the sensible answer is "don't do that"
  • Stuart
    Stuart over 8 years
    @glenn jackman, The question was "Why...". Can you explain the reason this happens so others can apply the general solution?
  • glenn jackman
    glenn jackman over 8 years
    Most likely the ls command is actually an alias or function that adds the --color option. So the odd chars are actually terminal color codes. ls output is intended for human eyes, not as input to other programs.'