How do I match any character across multiple lines in a regular expression?

Question

For example, this regex

(.*)<FooBar>

will match:

abcde<FooBar>

But how do I get it to match across multiple lines?

abcde
fghij<FooBar>

Solution 1

It depends on the language, but there should be a modifier that you can add to the regex pattern. In PHP it is:

/(.*)<FooBar>/s

The s at the end causes the dot to match all characters including newlines.

Solution 2

Try this:

((.|\n)*)<FooBar>

It basically says "any character or a newline" repeated zero or more times.

Solution 3

The question is, can the . pattern match any character? The answer varies from engine to engine. The main difference is whether the pattern is used by a POSIX or non-POSIX regex library.

A special note about lua-patterns: they are not considered regular expressions, but . matches any character there, the same as POSIX-based engines.

Another note on matlab and octave: the . matches any character by default (demo): str = "abcde\n fghij<Foobar>"; expression = '(.*)<Foobar>*'; [tokens,matches] = regexp(str,expression,'tokens','match'); (tokens contain a abcde\n fghij item).

Also, in all of boost's regex grammars the dot matches line breaks by default. Boost's ECMAScript grammar allows you to turn this off with regex_constants::no_mod_m (source).

As for oracle (it is POSIX based), use the n option (demo): select regexp_substr('abcde' || chr(10) ||' fghij<Foobar>', '(.*)<Foobar>', 1, 1, 'n', 1) as results from dual

POSIX-based engines:

A mere . already matches line breaks, so there isn't a need to use any modifiers, see bash (demo).

The tcl (demo), postgresql (demo), r (TRE, base R default engine with no perl=TRUE, for base R with perl=TRUE or for stringr/stringi patterns, use the (?s) inline modifier) (demo) also treat . the same way.

However, most POSIX-based tools process input line by line. Hence, . does not match the line breaks just because they are not in scope. Here are some examples how to override this:

• sed - There are multiple workarounds. The most precise, but not very safe, is sed 'H;1h;$!d;x; s/\(.*\)><Foobar>/\1/' (H;1h;$!d;x; slurps the file into memory). If whole lines must be included, sed '/start_pattern/,/end_pattern/d' file (removing from start will end with matched lines included) or sed '/start_pattern/,/end_pattern/{{//!d;};}' file (with matching lines excluded) can be considered.
• perl - perl -0pe 's/(.*)<FooBar>/$1/gs' <<< "$str" (-0 slurps the whole file into memory, -p prints the file after applying the script given by -e). Note that using -000pe will slurp the file and activate 'paragraph mode' where Perl uses consecutive newlines (\n\n) as the record separator.
• gnu-grep - grep -Poz '(?si)abc\K.*?(?=<Foobar>)' file. Here, z enables file slurping, (?s) enables the DOTALL mode for the . pattern, (?i) enables case insensitive mode, \K omits the text matched so far, *? is a lazy quantifier, (?=<Foobar>) matches the location before <Foobar>.
• pcregrep - pcregrep -Mi "(?si)abc\K.*?(?=<Foobar>)" file (M enables file slurping here). Note pcregrep is a good solution for macOS grep users.

See demos.

Non-POSIX-based engines:

• php - Use the s modifier PCRE_DOTALL modifier: preg_match('~(.*)<Foobar>~s', $s, $m) (demo)

• c# - Use RegexOptions.Singleline flag (demo):
  - var result = Regex.Match(s, @"(.*)<Foobar>", RegexOptions.Singleline).Groups[1].Value;
  - var result = Regex.Match(s, @"(?s)(.*)<Foobar>").Groups[1].Value;

• powershell - Use the (?s) inline option: $s = "abcde`nfghij<FooBar>"; $s -match "(?s)(.*)<Foobar>"; $matches[1]

• perl - Use the s modifier (or (?s) inline version at the start) (demo): /(.*)<FooBar>/s

• python - Use the re.DOTALL (or re.S) flags or (?s) inline modifier (demo): m = re.search(r"(.*)<FooBar>", s, flags=re.S) (and then if m:, print(m.group(1)))

• java - Use Pattern.DOTALL modifier (or inline (?s) flag) (demo): Pattern.compile("(.*)<FooBar>", Pattern.DOTALL)

• kotlin - Use RegexOption.DOT_MATCHES_ALL : "(.*)<FooBar>".toRegex(RegexOption.DOT_MATCHES_ALL)

• groovy - Use (?s) in-pattern modifier (demo): regex = /(?s)(.*)<FooBar>/

• scala - Use (?s) modifier (demo): "(?s)(.*)<Foobar>".r.findAllIn("abcde\n fghij<Foobar>").matchData foreach { m => println(m.group(1)) }

• javascript - Use [^] or workarounds [\d\D] / [\w\W] / [\s\S] (demo): s.match(/([\s\S]*)<FooBar>/)[1]

• c++ (std::regex) Use [\s\S] or the JavaScript workarounds (demo): regex rex(R"(([\s\S]*)<FooBar>)");

• vba vbscript - Use the same approach as in JavaScript, ([\s\S]*)<Foobar>. (NOTE: The MultiLine property of the RegExp object is sometimes erroneously thought to be the option to allow . match across line breaks, while, in fact, it only changes the ^ and $ behavior to match start/end of lines rather than strings, the same as in JavaScript regex) behavior.)

• ruby - Use the /m MULTILINE modifier (demo): s[/(.*)<Foobar>/m, 1]

• rtrebase-r - Base R PCRE regexps - use (?s): regmatches(x, regexec("(?s)(.*)<FooBar>",x, perl=TRUE))[[1]][2] (demo)

• ricustringrstringi - in stringr/stringi regex funtions that are powered with the ICU regex engine. Also use (?s): stringr::str_match(x, "(?s)(.*)<FooBar>")[,2] (demo)

• go - Use the inline modifier (?s) at the start (demo): re: = regexp.MustCompile(`(?s)(.*)<FooBar>`)

• swift - Use dotMatchesLineSeparators or (easier) pass the (?s) inline modifier to the pattern: let rx = "(?s)(.*)<Foobar>"

• objective-c - The same as Swift. (?s) works the easiest, but here is how the option can be used: NSRegularExpression* regex = [NSRegularExpression regularExpressionWithPattern:pattern options:NSRegularExpressionDotMatchesLineSeparators error:&regexError];

• re2, google-apps-script - Use the (?s) modifier (demo): "(?s)(.*)<Foobar>" (in Google Spreadsheets, =REGEXEXTRACT(A2,"(?s)(.*)<Foobar>"))

NOTES ON (?s):

In most non-POSIX engines, the (?s) inline modifier (or embedded flag option) can be used to enforce . to match line breaks.

If placed at the start of the pattern, (?s) changes the bahavior of all . in the pattern. If the (?s) is placed somewhere after the beginning, only those .s will be affected that are located to the right of it unless this is a pattern passed to Python's re. In Python re, regardless of the (?s) location, the whole pattern . is affected. The (?s) effect is stopped using (?-s). A modified group can be used to only affect a specified range of a regex pattern (e.g., Delim1(?s:.*?)\nDelim2.* will make the first .*? match across newlines and the second .* will only match the rest of the line).

POSIX note:

In non-POSIX regex engines, to match any character, [\s\S] / [\d\D] / [\w\W] constructs can be used.

In POSIX, [\s\S] is not matching any character (as in JavaScript or any non-POSIX engine), because regex escape sequences are not supported inside bracket expressions. [\s\S] is parsed as bracket expressions that match a single character, \ or s or S.

Solution 4

If you're using Eclipse search, you can enable the "DOTALL" option to make '.' match any character including line delimiters: just add "(?s)" at the beginning of your search string. Example:

(?s).*<FooBar>

Solution 5

In many regex dialects, /[\S\s]*<Foobar>/ will do just what you want. Source

View more solutions

Comments

[Ben Doom]

This is dependent on the language and/or tool you are using. Please let us know what you are using, eg Perl, PHP, CF, C#, sed, awk, etc.

[Alan Moore]

No, don't do that. If you need to match anything including line separators, use the DOTALL (a.k.a. /s or SingleLine) modifier. Not only does the (.|\n) hack make the regex less efficient, it's not even correct. At the very least, it should match \r (carriage return) as well as \n (linefeed). There are other line separator characters, too, albeit rarely used. But if you use the DOTALL flag, you don't have to worry about them.

[opyate]

\R is the platform-independent match for newlines in Eclipse.

[Grace]

and what if i wanted just a new line and not all characters ?

[Paige Ruten]

@Grace: use \n to match a newline

[Potherca]

Depending on your line endings you might need ((.|\n|\r)*)<FooBar>

[Danubian Sailor]

He said he is using Eclipse. This is correct solution in my opinion. I have same problem and this solved it.

[acme]

Right - the question is about eclipse and so are the tags. But the accepted solution is a PHP solution. Yours should be the accepted solution...

[J. Costa]

This solve the problem if you are using the Objective-C [text rangeOfString:regEx options:NSRegularExpressionSearch]. Thanks!

[jeckhart]

@opyate You should post this as an answer as this little gem is incredibly useful.

[Josef Sábl]

stackoverflow.com/questions/1331815/…

[Allen]

Seems like this is invalid (Chrome): text.match(/a/s) SyntaxError: Invalid flags supplied to RegExp constructor 's'

[Allen]

From that link: "JavaScript and VBScript do not have an option to make the dot match line break characters. In those languages, you can use a character class such as [\s\S] to match any character." Instead of the . use [\s\S] (match spaces and non-spaces) instead.

[Allen]

The s flag is (now?) invalid, at least in Chrome/V8. Instead use /([\s\S]*)<FooBar>/ character class (match space and non-space] instead of the period matcher. See other answers for more info.

[Paul Draper]

Shouldn't those be backslashes?

[RandomInsano]

They go at the end of the Regular Expression, not within in. Example: /blah/s

[Derek 朕會功夫]

@Allen - JavaScript doesn't support the s modifier. Instead, do [^]* for the same effect.

[Ryan Buckley]

In Ruby, use the m modifier

[barclay]

This works in intelliJ's find&replace regex, thanks.

[frIT]

\R matches line endings in a platform-independent manner. In eclipse, at least, and some other tools.

[Manolis Agkopian]

Very funny, I tried this on gedit and I got a segmentation fault. Murphy's law at its finest.

[Morgan Touverey Quilling]

Because it is unsupported in JavaScript RegEx engines. The s flags exists in PCRE, the most complete engine (available in Perl and PHP). PCRE has 10 flags (and a lot of other features) while JavaScript has only 3 flags (gmi).

[Wiktor Stribiżew]

This is the worst regex for matching multiple line input. Please never use it unless you are using ElasticSearch. Use [\s\S]* or (?s).*.

[Wiktor Stribiżew]

Not anywhere, only in regex flavors supporting inline modifiers, and certainly not in Ruby where (?s) => (?m)

[ssc-hrep3]

You could try this instead. It won't match the inner brackets and also consider the optional\r.: ((?:.|\r?\n)*)<foobar>

[Ozkan]

This works. But it needs to be the first occurrence of <FooBar>

[Jan]

You should link to this excellent overview from your profile page or something (+1).

[Mohamad Hamouday]

If there are multiple values of <FooBar>, it will ignore all the values in the middle and only match the last <FooBar>

[Admin]

You may want to add this to the boost item: In the regex_constants namespace, flag_type_'s : perl = ECMAScript = JavaScript = JScript = ::boost::regbase::normal = 0 which defaults to Perl. Programmers will set a base flag definition #define MOD regex_constants::perl | boost::regex::no_mod_s | boost::regex::no_mod_m for thier regex flags to reflect that. And the arbitor is always the inline modifiers. Where (?-sm)(?s).* resets.

[Vamshi Krishna]

I am using .NET too and (\s|\S) seems to do the trick for me!

[NealWalters]

What to use for Powershell?

[Wiktor Stribiżew]

@VamshiKrishna In .NET, use (?s) to make . match any chars. Do not use (\s|\S) that will slow down performance.

[3limin4t0r]

I guess you mean JavaScript, not Java? Since you can just add the s flag to the pattern in Java and JavaScript doesn't have the s flag.

[Pasupathi Rajamanickam]

Anything for bash?

[Pasupathi Rajamanickam]

Can you also add for bash please?

[Wiktor Stribiżew]

@PasupathiRajamanickam Bash uses a POSIX regex engine, the . matches any char there (including line breaks). See this online Bash demo.

[Snow]

Such needless alternation can result in catastrophic backtracking in some situations. This isn't a good general pattern.

[Adam Wenger]

Thanks for the ?s modifier for PowerShell; cleaned up my fragile regex to something a bit sturdier.

[Sebastián Espinosa]

you are a legend

[Gwyneth Llewelyn]

You rock — this is the most exhaustive mini-tutorial on (relatively) complex regexp's that I've ever seen. You deserve that your answer becomes the accepted one! Kudos and extra votes for including Go in the answer!

[Wiktor Stribiżew]

Never use (.*?|\n)*? unless you want to end up with a catastrophic backtracking.

[lkahtz]

I like this. This is more general.

[Nolwennig]

In xml files, I use : ((.|\n|\r|\t)*)<FooBar> pattern

[Peter Mortensen]

@Wiktor Stribiżew: Why is it the worst? Will the other match newlines without modifiers?

[Peter Mortensen]

There is a multi-line mode for .NET regular expressions.

[Peter Mortensen]

This is specific to a particular platform. What programming language and platform is it? C# / .NET?

[Peter Mortensen]

What is the underlying regular expression engine for Eclipse? Something in Java/JDK?

[Peter Mortensen]

The first link somehow redirects to www.facebook.com (which I have blocked in the hosts file). Is that link broken or not?

[Peter Mortensen]

What programming language? Java?

[Wiktor Stribiżew]

@PeterMortensen Too many people have already reported peformance issues and even stack overflow errors when using this pattern, and I have even recorded a YT video with explanation of why it is that bad.

[Peter Mortensen]

It doesn't look right. Why two times ".*"? This may work for the sample input in the question, but what if "<FooBar>" is on line 42?

[Sian Lerk Lau]

I guess the owner decided to redirect it to the facebook page. I will remove it.

[Just Me]

(\r\n)* - super answer. thanks