Regex : how to get words from a string (C#)

Question

My input consists of user-posted strings.

What I want to do is create a dictionary with words, and how often they’ve been used. This means I want to parse a string, remove all garbage, and get a list of words as output.

For example, say the input is "#@!@LOLOLOL YOU'VE BEEN \***PWN3D*** ! :') !!!1einszwei drei !"

The output I need is the list:

• "LOLOLOL"
• "YOU'VE"
• "BEEN"
• "PWN3D"
• "einszwei"
• "drei"

I’m no hero at regular expressions and have been Googling, but my Google-kungfu seams to be weak …

How would I go from input to the wanted output?

Solution 1

Simple Regex:

\w+

This matches a string of "word" characters. That is almost what you want.

This is slightly more accurate:

\w(?<!\d)[\w'-]*

It matches any number of word characters, ensuring that the first character was not a digit.

Here are my matches:

1 LOLOLOL
2 YOU'VE
3 BEEN
4 PWN3D
5 einszwei
6 drei

Now, that's more like it.

EDIT:
The reason for the negative look-behind, is that some regex flavors support Unicode characters. Using [a-zA-Z] would miss quite a few "word" characters that are desirable. Allowing \w and disallowing \d includes all Unicode characters that would conceivably start a word in any block of text.

EDIT 2:
I have found a more concise way to get the effect of the negative lookbehind: Double negative character class with a single negative exclusion.

[^\W\d][\w'-]*(?<=\w)

This is the same as the above with the exception that it also ensures that the word ends with a word character. And, finally, there is:

[^\W\d](\w|[-']{1,2}(?=\w))*

Ensuring that there are no more than two non-word-characters in a row. Aka, It matches "word-up" but not "word--up", which makes sense. If you want it to match "word--up", but not "word---up", you can change the 2 to a 3.

Solution 2

You should look into Natural Language Processing (NLP), not regular expressions, and if you are targeting more than one spoken language, you need to factor that in as well. Since you're using C#, check out the SharpNLP project.

Edit: This approach is only necessary if you care about the semantic content of the words you're trying to split up.

Solution 3

Using the following

var pattern = new Regex(
  @"( [^\W_\d]              # starting with a letter
                            # followed by a run of either...
      ( [^\W_\d] |          #   more letters or
        [-'\d](?=[^\W_\d])  #   ', -, or digit followed by a letter
      )*
      [^\W_\d]              # and finishing with a letter
    )",
  RegexOptions.IgnorePatternWhitespace);

var input = "#@!@LOLOLOL YOU'VE BEEN *PWN3D* ! :') !!!1einszwei drei foo--bar!";

foreach (Match m in pattern.Matches(input))
  Console.WriteLine("[{0}]", m.Groups[1].Value);

produces output of

[LOLOLOL]
[YOU'VE]
[BEEN]
[PWN3D]
[einszwei]
[drei]
[foo]
[bar]

Solution 4

You don't necessarily need a regex for this, if tokenizing is all you're doing. First you could sanitize the string by removing all non-letter characters except for spaces and then do a Split() on the space character. That will work for most everything, although contractions may be tough. That should get you started at least.

View more solutions