Add prefix to every line in text in bash

Question

Suppose there is a text file a.txt e.g.

aaa
bbb
ccc
ddd

I need to add a prefix (e.g. myprefix_) to every line in the file:

myprefix_aaa
myprefix_bbb
myprefix_ccc
myprefix_ddd

I can do that with awk:

awk '{print "myprefix_" $0}' a.txt 

Now I wonder if there is another way to do that in shell.

Solution 1

Pure bash:

while read line
do
    echo "prefix_$line"
done < a.txt

Solution 2

With sed:

$ sed 's/^/myprefix_/' a.txt
myprefix_aaa
myprefix_bbb
myprefix_ccc
myprefix_ddd

This replaces every line beginning ^ with myprefix_. Note that ^ is not lost, so this allows to add content to the beginning of each line.

You can make your awk's version shorter with:

$ awk '$0="myprefix_"$0' a.txt
myprefix_aaa
myprefix_bbb
myprefix_ccc
myprefix_ddd

or passing the value:

$ prefix="myprefix_"
$ awk -v prefix="$prefix" '$0=prefix$0' a.txt
myprefix_aaa
myprefix_bbb
myprefix_ccc
myprefix_ddd

It can also be done with nl:

$ nl -s "prefix_" a.txt | cut -c7-
prefix_aaa
prefix_bbb
prefix_ccc
prefix_ddd

---

Finally: as John Zwinck explains, you can also do:

paste -d'' <(yes prefix_) a.txt | head -n $(wc -l a.txt)

on OS X:

paste -d '\0' <(yes prefix_) a.txt | head -n $(wc -l < a.txt)

Solution 3

For reference, regarding the speed of the awk, sed, and bash solution to this question:

Generate a 800K input file in bash:

line="12345678901234567890123456789012345678901234567890123456789012345678901234567890"
rm a.txt
for i in {1..10000} ; do
    echo $line >> a.txt
done

Then consider the bash script timeIt

if [ -e b.txt ] ; then
    rm b.txt
fi
echo "Bash:"
time bashtest
rm b.txt
echo
echo "Awk:"
time awktest
rm b.txt
echo
echo "Sed:"
time sedtest

where bashtest is

while read line
do
    echo "prefix_$line" >> b.txt
done < a.txt

awktest is:

awk '$0="myprefix_"$0' a.txt > b.txt

and sedtest is:

sed 's/^/myprefix_/' a.txt > b.txt

I got the following result on my machine:

Bash:
real    0m0.401s
user    0m0.340s
sys 0m0.048s
Awk:
real    0m0.009s
user    0m0.000s
sys 0m0.004s
Sed:
real    0m0.009s
user    0m0.000s
sys 0m0.004s

It seems like the bash solution is much slower..

Solution 4

You can also use the xargs utility:

cat file | xargs -d "\n" -L1 echo myprefix_ 

The -d option is used to allow input line with trailing blanks (related to -L spec).

View more solutions

Comments

[paxdiablo]

+1 for the sed solution, where using awk or Perl is like swatting a mosquito with a thermo-nuclear warhead :-)

[Michael]

Thanks! I like the sed solution and did not know about nl.

[John Zwinck]

Or with horror: paste -d'' <(yes prefix_) a | head -n $(wc -l a). It would be a lot better if paste had an option to stop at the first EOF rather than continue until all EOFs.

[fedorqui]

Didn't know about the yes command, @JohnZwinck, it sounds cool. However, it is not working to me, just shows the file a.

[John Zwinck]

Here's a version that works on OS X: paste -d '\0' <(yes prefix_) a.txt | head -n $(wc -l < a.txt) - on Linux you probably need to say '' instead of '\0' but the rest should work.

[fedorqui]

That's such a trick, @JohnZwinck, now it works to me. I would add this to your current answer not to be hidden here in comments :)

[John Zwinck]

No, I like my current answer as it is--you're welcome to add that monstrosity to your answer though--I wrote it specifically in the spirit of your several examples! :)

[chepner]

+1 Spawning external programs is the purpose of a shell language. If you were calling awk once per line to prefix it, that would be overkill. But calling it once to process the entire file is fine.

[Psyrus]

This test would be more indicative of the performance differences if the bashtest were to process all lines in memory and then write to output file, rather than appending to the output file upon every line read.

[Psyrus]

For reference, the code I was using:for line in $(<a.txt); do; text+="prefix_$line\n"; done; echo -e "$text" > b.txt; Tried tweaking in various ways, all with either worse or similar results

[kvantour]

While the awk solutions are good, they are redefining $0 which indicates that awk will waste time with recomputing the fields and the variable NF. It might be better just to do '{print "prefix_" $0}' or avoid concatenation with '{printf "prefix_%s\n", $0}'

[Bruce]

Your nl solution is fabulous, and I improved it slightly to make it stable and universal, --------------------- nl -w1 -s" $prefix" | cut -d' ' -f2-