How to Replace a line in the same file by SED in Unix Shell scripting?

Question

In Reference to this question After getting the line identifier matching in first and second file I need to replace the line in first file with the line of second file.For that I am using SED as below. But Sed only replaces that line in a new file. How can I achieve the update in same file without temporary file (Because those are very huge files).

#!/bin/ksh
while read line
do
var=`echo $line|cut -c 3-25`
while read i
do
var1=`echo $i|cut -c 3-25`
if [ $var == $var1 ];
then
sed -i s/$line/$i/ t1
else

echo "not matched"
fi
done < t2
done < t1

Even without -i option also I am not getting the result. Please help me.

Edit: Or else suggest me any other optimal way of doing without temporary files or with any scripting languages(less preferred).

Solution 1

You shouldn't use sed like that. As your script now stands this is what it says:

1. For each line in t1
2. Step through all the lines in t2
3. If one of the lines in t2 matches the current line in t1, then step through all the lines in t1 and replace the matches
4. Go to the next line in t1 and repeat

That means that the entire t2 file is read each time one line is read from t1. This is incredibly inefficient.

It's not necessary to use echo and cut for substrings. In Bash and ksh you can do:

var=${line:3:23}

Note: cut uses character positions for the beginning and end of a range, while this shell construct uses starting position and character count so you have to adjust the numbers accordingly.

If t2 is a list of replacements to be made in t1, thus t2 is a "script" of sorts, then this might do what you need:

keystart=3
keylen=23
while read line
do
    var="${line:$keystart:$keylen}"
    if (( ${#var} == keylen ))    # need "$" then don't need "$"
    then
        sed -in "/^.\{$keystart\}$var/ c$line" t1    # use double-quote so vars get expanded
    fi
done < t2

This will find all the lines in t1 that each line in t2 matches and do the replacement.

If, however, t1 and t2 have a line-for-line correspondence and you want to make the substitution only where corresponding lines match, then this, using a temporary file, will be the most efficient:

tempfile=$(mktemp)
keystart=3
keylen=23
while read line1
do
    var1="${line1:$keystart:$keylen}"
    read line2 <&3    # use file descriptor 3 for input

    var2="${line2:$keystart:$keylen}"
    if [[ $var1 == $var2 && ${#var2} == $keylen ]]
    then
        echo "${line2}" >> $tempfile    # substitute when matched
    else
        echo "${line1}" >> $tempfile    # keep when not matched
    fi
done < t1 3< t2    # t1 is input on stdin, t2 is input on fd 3
mv t1 t1.bak && mv $tempfile t1

Solution 2

Have a look at sponge which is part of moreutils.

e.g.

% sed "s/root/toor/" /etc/passwd | grep -v joey | sponge /etc/passwd

Solution 3

You can't replace one line with another in place. Because lines usually has different length and will overlap each other. If all lines in your file are the same length then it can work. I also suggest you use more suitable language for this task (like Perl for example) because it will be really complicated code in Shell. I think that you should look for a solution with temporal files, because it'll easier to implement and it can be debugged easily. Just imagine what will you do if your huge file will be currupted due to the bug in the script.

Solution 4

Sed is designed to work in a pipeline - hence the name "Stream EDitor". Instead, you could use an ex script to edit the file in place. Ex is the line-based text editor on which vi was originally based (not as old as ed, the old bear skins and stone knives text editor, but almost). A simple example that you can modify for your purpose might be as follows:

ex t1 << EOF
$lineNum
s/^.*$/$newline/
w
q
EOF

This script first goes to the line indicated by $lineNum, substitutes the whole line beginning (^) to end ($) with the contents of $newline, then writes and quits. These commands are surrounded by "<<EOF" and "EOF", which constitute the "here" document, essentially setting up the scripted commands as stdin.

View more solutions