Finding quoted strings with escaped quotes in C# using a regular expression

Question

I'm trying to find all of the quoted text on a single line.

Example:

"Some Text"
"Some more Text"
"Even more text about \"this text\""

I need to get:

• "Some Text"
• "Some more Text"
• "Even more text about \"this text\""

\"[^\"\r]*\" gives me everything except for the last one, because of the escaped quotes.

I have read about \"[^\"\\]*(?:\\.[^\"\\]*)*\" working, but I get an error at run time:

parsing ""[^"\]*(?:\.[^"\]*)*"" - Unterminated [] set.

How do I fix this?

Solution 1

What you've got there is an example of Friedl's "unrolled loop" technique, but you seem to have some confusion about how to express it as a string literal. Here's how it should look to the regex compiler:

"[^"\\]*(?:\\.[^"\\]*)*"

The initial "[^"\\]* matches a quotation mark followed by zero or more of any characters other than quotation marks or backslashes. That part alone, along with the final ", will match a simple quoted string with no embedded escape sequences, like "this" or "".

If it does encounter a backslash, \\. consumes the backslash and whatever follows it, and [^"\\]* (again) consumes everything up to the next backslash or quotation mark. That part gets repeated as many times as necessary until an unescaped quotation mark turns up (or it reaches the end of the string and the match attempt fails).

Note that this will match "foo\"- in \"foo\"-"bar". That may seem to expose a flaw in the regex, but it doesn't; it's the input that's invalid. The goal was to match quoted strings, optionally containing backslash-escaped quotes, embedded in other text--why would there be escaped quotes outside of quoted strings? If you really need to support that, you have a much more complex problem, requiring a very different approach.

As I said, the above is how the regex should look to the regex compiler. But you're writing it in the form of a string literal, and those tend to treat certain characters specially--i.e., backslashes and quotation marks. Fortunately, C#'s verbatim strings save you the hassle of having to double-escape backslashes; you just have to escape each quotation mark with another quotation mark:

Regex r = new Regex(@"""[^""\\]*(?:\\.[^""\\]*)*""");

So the rule is double quotation marks for the C# compiler and double backslashes for the regex compiler--nice and easy. This particular regex may look a little awkward, with the three quotation marks at either end, but consider the alternative:

Regex r = new Regex("\"[^\"\\\\]*(?:\\\\.[^\"\\\\]*)*\"");

In Java, you always have to write them that way. :-(

Solution 2

Regex for capturing strings (with \ for character escaping), for the .NET engine:

(?>(?(STR)(?(ESC).(?<-ESC>)|\\(?<ESC>))|(?!))|(?(STR)"(?<-STR>)|"(?<STR>))|(?(STR).|(?!)))+   

Here, a "friendly" version:

(?>                            | especify nonbacktracking
   (?(STR)                     | if (STRING MODE) then
         (?(ESC)               |     if (ESCAPE MODE) then
               .(?<-ESC>)      |          match any char and exits escape mode (pop ESC)
               |               |     else
               \\(?<ESC>)      |          match '\' and enters escape mode (push ESC)
         )                     |     endif
         |                     | else
         (?!)                  |     do nothing (NOP)
   )                           | endif
   |                           | -- OR
   (?(STR)                     | if (STRING MODE) then
         "(?<-STR>)            |     match '"' and exits string mode (pop STR)
         |                     | else
         "(?<STR>)             |     match '"' and enters string mode (push STR)
   )                           | endif
   |                           | -- OR
   (?(STR)                     | if (STRING MODE) then
         .                     |     matches any character
         |                     | else
         (?!)                  |     do nothing (NOP)  
   )                           | endif
)+                             | REPEATS FOR EVERY CHARACTER

Based on http://tomkaminski.com/conditional-constructs-net-regular-expressions examples. It relies in quotes balancing. I use it with great success. Use it with Singleline flag.

To play around with regexes, I recommend Rad Software Regular Expression Designer, which has a nice "Language Elements" tab with quick access to some basic instructions. It's based at .NET's regex engine.

Solution 3

"(\\"|\\\\|[^"\\])*"

should work. Match either an escaped quote, an escaped backslash, or any other character except a quote or backslash character. Repeat.

In C#:

StringCollection resultList = new StringCollection();
Regex regexObj = new Regex(@"""(\\""|\\\\|[^""\\])*""");
Match matchResult = regexObj.Match(subjectString);
while (matchResult.Success) {
    resultList.Add(matchResult.Value);
    matchResult = matchResult.NextMatch();
} 

Edit: Added escaped backslash to the list to correctly handle "This is a test\\".

Explanation:

First match a quote character.

Then the alternatives are evaluated from left to right. The engine first tries to match an escaped quote. If that doesn't match, it tries an escaped backslash. That way, it can distinguish between "Hello \" string continues" and "String ends here \\".

If either don't match, then anything else is allowed except for a quote or backslash character. Then repeat.

Finally, match the closing quote.

Solution 4

I recommend getting RegexBuddy. It lets you play around with it until you make sure everything in your test set matches.

As for your problem, I would try four /'s instead of two:

\"[^\"\\\\]*(?:\\.[^\"\\\\]*)*\"

Solution 5

The regular expression

(?<!\\)".*?(?<!\\)"

will also handle text that starts with an escaped quote:

\"Some Text\" Some Text "Some Text", and "Some more Text" an""d "Even more text about \"this text\""

View more solutions

Comments

[Joshua Lowry]

This gives me: "Some Text"; "Some more Text"; ""

[Tim Pietzcker]

Sorry for editing this post so much. But now I think I've got it elegant enough. And correct, too. I hope.

[Kamarey]

This regex not work with this text: \"Some Text\" Some Text "Some Text", and "Some more Text" an""d "Even more text about \"this text\""

[Joshua Lowry]

This is excellent! I think part of the issue was that I was not using the @ which added more complexity with having to slash all over the place.

[Joshua Lowry]

Is there a way this could work for multiple lines quoted strings?

[Alan Moore]

One of RegexBuddy's selling points is that it can automatically convert the regex to source code in whatever language you specify. In this case, it converts the "raw" regex "[^"\\]*(?:\\.[^"\\]*)*" to @"""[^""\\]*(?:\\.[^""\\]*)*""".

[Tim Pietzcker]

Well, texts that are enclosed in escaped quotes weren't part of the question; neither was doubling as another way of escaping quotes.

[Tim Pietzcker]

This doesn't handle escaped backslashes at the end of strings: "Hello\\".

[Joshua Lowry]

I like this explanation best.

[ridgerunner]

Sorry Tim, but "(\\"|\\\\|[^"])*" is no good. Yes, it matches valid quoted strings very well, but it strays off into the land of catastrophic backtracking when presented with a non-match string like: "\\\\\\\\\\\\\\\\\\\\\\\ (The options within an alternation group should be mutually exclusive if you apply a * or + to it) This regex can match a backslash in more than one way.

[Tim Pietzcker]

@ridgerunner: You're right, thanks. I have fixed the regex (by including the backslash in the negated character class). Now your "pathological string" fails in 85 instead of 750.000 steps.

[ridgerunner]

Sorry again, but: "(\\"|\\\\|[^"\\])*" does not match: "\n" or "\t". The pattern needed here is: "([^"\\]|\\.)*" which matches correctly (or better yet: "([^"\\]++|\\.)*" if the possessive quantifier is available). But Friedl's unrolled version of this expression is much faster. See Alan's answer. Have you read MRE3 yet? If not, I know you would enjoy it very much (if you are into regex - which I think you are).

[131]

Still, i tend to say ([^"\]|\\.)* is the best answer here. It's the most natural and fully working string, when Friedl's unrolled is about the same, with optimization (but redundency)

[Piotr Zierhoffer]

I can see that this answer is a bit buggy, for some reason. Please refer to stackoverflow.com/questions/20196740/…

[zx81]

Cruising through some of the answers that made you famous... Upvoting this one for making such a clear explanation out of the worst backslash soup! :)

[Kalpesh Rajai]

It is possible to same thing with Single quote (')

[Alan Moore]

@KalpeshRajai: Sure, just replace the double-quotes with single-quotes in my first regex. You don't even need to escape them (unless you're using a single-quoted string literal, which C# doesn't support).

[Kalpesh Rajai]

@AlanMoore: Thank You

[wp78de]

While I appreciate the effort this regex is noticeably slower than the original by Alan.

[Matt Davis]

I would upvote this 100 times if I could! Your explanation finally helped me accomplish what I was trying to do. Your verbatim regex works as advertised for traditional escape sequences, i.e., a backslash followed by a character (including a quotation mark). However, it does not handle the escape sequence for quotation marks used in verbatim strings, i.e., a double quotation mark. But again, your explanation helped me see the answer using alternation, e.g., @"""[^""\\]*(?:(?:\\.|"""")[^""\\]*)*""". Thank you so much!