sed command in dry run

Question

How it is possible to make a dry run with sed?

I have this command:

find ./ -type f | xargs sed -i 's/string1/string2/g'

But before I really substitute in all the files, i want to check what it WOULD substitute. Copying the whole directory structure to check is no option!

Solution 1

Remove the -i and pipe it to less to paginate though the results. Alternatively, you can redirect the whole thing to one large file by removing the -i and appending > dryrun.out

I should note that this script of yours will fail miserably with files that contain spaces in their name or other nefarious characters like newlines or whatnot. A better way to do it would be:

while IFS= read -r -d $'\0' file; do
  sed -i 's/string1/string2/g' "$file"
done < <(find ./ -type f -print0)

Solution 2

I would prefer to use the p-option:

find ./ -type f | xargs sed 's/string1/string2/gp'

Could be combined with the --quiet parameter for less verbose output:

find ./ -type f | xargs sed --quiet 's/string1/string2/gp'

From man sed:

p:

Print the current pattern space.

--quiet:

suppress automatic printing of pattern space

Solution 3

I know this is a very old thread and the OP doesn't really need this answer, but I came here looking for a dry run mode myself, so thought of adding the below piece of advice for anyone coming here in future. What I wanted to do was to avoid stomping the backup file unless there is something really changing. If you blindly run sed using the -i option with backup suffix, existing backup file gets overwritten, even when there is nothing substituted.

The way I ended up doing is to pipe sed output to diff and see if anything changed and then rerun sed with in-place update option, something like this:

if ! sed -e 's/string1/string2/g' $fpath | diff -q $fpath - > /dev/null 2>&1; then
    sed -i.bak -e 's/string1/string2/g' $fpath
fi

As per OP's question, if the requirement is to just see what would change, then instead of running the in-pace sed, you could do the diff again with some informative messages:

if ! sed -e 's/string1/string2/g' $fpath | diff -q $fpath - > /dev/null 2>&1; then
    echo "File $fpath will change with the below diff:"
    sed -e 's/string1/string2/g' $fpath | diff $fpath -
fi

You could also capture the output in a variable to avoid doing it twice:

diff=$(sed -e 's/string1/string2/g' $fpath | diff $fpath -)
if [[ $? -ne 0 ]]; then
    echo "File $fpath will change with the below diff:"
    echo "$diff"
fi

Comments

[SiegeX]

@dmeu please see my updated answer as to how to do this properly; not the dry run but the real-deal.

[Jonathan Leffler]

Anything wrong with find . -type f -print0 | xargs -0 sed ...? That has the merit of executing sed once for many files instead of once per file. There is some overhead in the shell-only version - not an outrageous overhead, but some.

[dmeu]

@SiegeX no problem. white space has nothing lost in my files @Jonathan Leffler: please post a different answer if it really works ;-)

[SiegeX]

@Jonathan nothing wrong with that except the overhead in the subshell created as noted. If you wanted to go that route but forgo the subshell I would use find . -type f -exec perl -pi -e 's/.../g' {} + which has the xargs-like benefit and acts just like GNU's sed -i but being way more portable.

[Jonathan Leffler]

Anything 'find' executes is executed once-per-file, just like in the shell loop, and in contrast to xargs which will accumulate arguments until it reaches a limit, and then executes the command (sed in this case) once with many (possibly hundreds) of file names. If you need process substitution, you can do that for xargs too: xargs -0 sed ... < <(find . -type f -print0).

[SiegeX]

@Jonathan "Anything 'find' executes is executed once-per-file". Well, that depends on how you terminate -exec. If you terminate it with \; then yes, you're right. If you terminate it with + like I did in my comment then it acts just like xargs would.

[SiegeX]

@Jonathan Alternatively, if you'd rather depend on GNU sed then use perl you can accomplish the very same thing with: find . -type f -exec bash -c 'sed -i "s/foo/bar/g" "$@"' _ {} + but it's quite messy since you have to pass in the file names as parameters to the shell.

[Jonathan Leffler]

OK - there's no end to the inventiveness of GNU commands...it is impossible to keep up with all the additions. I didn't know about the trailing + option. Since I have to work on machines without the GNU toolset, it is more important for me to know what is portable to other platforms than to know every last possible extension to the the GNU toolset. Having said which, -print0 and -0 and -i are all GNU extensions too (3 options, 3 commands).

[SiegeX]

@Jonathan You'll be happy to know that terminating -exec with a + isn't a GNUism but is in the POSIX standard for find, see this link and search for -exec utility_name

[Jonathan Leffler]

Consider me suitably pleased - thank you. I wonder how many of the systems I use have caught up with that detail from POSIX 2004...HP-UX, Solaris seem to be OK; older AIX (5.x) seems not. 4/5 isn't too bad, I suppose.

[sleepycal]

Interesting, I don't fully understand how/why the syntax works, but it works. Thank you!

[MatrixManAtYrService]

I like this. Just a warning: if you're in a git repo you risk matching strings in your .git folder, which is probably not what you want to do. To avoid this, start with something a bit smarter than find ./ -type f. For example: ag -l | xargs sed... (you may need to install ag)

[tobek]

On Linux, the first command works, but the second command with --quiet flag appears to break handling of filenames with spaces in them (tons of No such file or directory errors for parts of filenames

[tobek]

At least on Linux, the command in this answer is not a dry run - it actually changes the files.

[SiegeX]

@tobek On Linux, I can tell you if you remove the -i it will 100% not alter the input file.

[Murmel]

@tobek Which version of sed do you use? It works at least with sed (GNU sed) 4.7

[tobek]

@Murmel huh I'm on sed (GNU sed) 4.8 on Arch. Just tried it again, copy-pasted exactly, getting those errors

[tobek]

@SiegeX ah I see - I used the code in the code block which does have the -i! Would be safer to supply code for the actual answer (a dry run sed) in that code block, and then specify that you should add the -i to make it modify files.