Replace multiline string in files

Question

I have a number of files I want to update by replacing one multi-line string with another multi-line string. Something along the lines of:

* Some text, 
* something else
* another thing

And I want to replace it with:

* This is completely
* different text

The result would be that after the replacement the file containing the first block of text, will now contain the second string (the rest of the file is unchanged).

Part of the problem is that I have to find the list of files to be updated in the file system. I guess I can use grep for that (though again that's not as easy to do with multiline strings) then pipe it in sed maybe?

Is there an easy way to do this? Sed is an option but it's awkward because I have to add \n etc. Is there a way to say "take the input from this file, match it in those files, then replace it with the content of this other file" ? I can use python if need be, but I want something quick and simple, so if there is a utility available, I would rather use that than write my own script (which I know how to do).

Solution 1

Substitute "Some...\n...Thing" by the contents of file "new" in one or more input files

perl -i -p0e 's/Some.*?thing\n/`cat new`/se' input.txt ...

1. -i to change input.txt directly
2. -p0 slurp input file file and print it in the end
3. s/regexp/.../s in regexp . is .|\n
4. s/.../exp/e replace by eval(exp)
5. new -- a file containing the replacement text (This is completely...different text)
6. if useful you can expand the original text s/Some text\n...\n...thing\n/...

Solution 2

sed -e :n -e '$!N;/\n.*\n/!{$!bn
};  s/some text,\n* *something else\n* *another thing/this is completely\
different text/;P;D' <infile

I fear you're going to have a difficult time coming up with a solution that suits you until you hammer out a concrete description of the problem - but that's what QA is best suited for, as I see it. Maybe this will give you an idea - it will always keep 3 lines in pattern space at a time - with a 2 line lookahead - while sliding forward through the input file only a line at a time.

It should be able to match your string whether it spans multiple lines or not - up to three, that is. But there are no provisions for mirroring that provision in the replacement - it always spans two lines as written.

Solution 3

While ripgrep specifically doesn't support inline replacement, I've found that its current --replace functionality is already useful for this use case:

rg --replace "$(cat new.txt)" --passthru --no-line-number \
--multiline --multiline-dotall 'Some.*?thing\n' multi.txt > output.txt

• --replace 'string' enables replacement mode and sets the replacement string. It can also include captured regex groups by using $1 etc.
• $(cat new.txt) passes the contents of the file new.txt as the replacement string.
• --passthru is needed since ripgrep usually only shows the lines matching the regex pattern. With this option it also shows all lines from the file that don't match.
• --no-line-number / -N is because by default ripgrep includes the line numbers in the output (useful when only the matching lines are shown).
• --multiline / -U enables multiline processing which is disabled by default.
• --multiline-dotall is only needed if you want the dot ('.') regex pattern to match newlines (\n).
• > output.txt is needed since inline replace isn't supported. With the --passthrough and no-line-number options the standard output matches the desired new file with replacements and can be saved as usual.

However, this command isn't as useful for processing multiple files, as it needs to be run separately per file.

Solution 4

Not to strong (because don't chech second string but it easy to settle) and can be is not posix compilant but very simple:

sed '/^Some text/{:1;/another thing$/!{N;b 1}
     s/.*/this is completely\ndifferent text/g}' input.txt

First command add lines from Some text until have met another thing then second line change it to other text.

NOTE Limitation is that Some text should always be followed by another thing.

View more solutions

Comments

[ventsyv]

The problem is that the string might be more than 2 lines (up to a dozen or so) and may contain other stuff that might need to be escaped, such as tabs, * etc.

[Kvothe]

How can I do the same with a file called say "before" to look for the (multi-line) contents of that file? I tried but it does not work.

[JJoao]

@Kvothe, we need more details... Assuming that "before" has no special chars, you can try perl -i -p0e ' $b= `cat before`; s/$b/Some thing\n/se' input.txt ...

[Kvothe]

And assuming the "before" contains all special characters (new lines, slashes, brackets) except ' and `.

[Lost Crotchet]

I get an error "Unrecognized character \xE4; marked by <-- HERE after do {<-- HERE near column 5 at -e line 2." when executing this command. There are unicode characters in the strings I am inputting. Does perl support unicode?

[tekknolagi]

This worked for me only if I do not use /e. I had to use only /s instead.

[JJoao]

@tekknolagi, without /e the new file would not be included. We woud get just the string cat new.