"Stop words" list for English?

Question

I'm generating some statistics for some English-language text and I would like to skip uninteresting words such as "a" and "the".

• Where can I find some lists of these uninteresting words?
• Is a list of these words the same as a list of the most frequently used words in English?

update: these are apparently called "stop words" and not "skip words".

Solution 1

The magic word to put into Google is "stop words". This turns up a reasonable-looking list.

MySQL also has a built-in list of stop words, but this is far too comprehensive to my tastes. For example, at our university library we had problems because "third" in "third world" was considered a stop word.

Solution 2

these are called stop words, check this sample

Solution 3

Depending on the subdomain of English you are working in, you may have/wish to compile your own stop word list. Some generic stop words could be meaningful in a domain. E.g. The word "are" could actually be an abbreviation/acronym in some domain. Conversely, you may want to ignore some domain specific words depending on your application which you may not want to ignore in the domain of general English. E.g. If you are analyzing a corpus of hospital reports, you may wish to ignore words like 'history' and 'symptoms' as they would be found in every report and may not be useful (from a plain vanilla inverted index perspective).

Otherwise, the lists returned by Google should be fine. The Porter Stemmer uses this and the Lucene seach engine implementation uses this.

Solution 4

Get statistics about word frequency in large txt corpora. Ignore all words with frequency > some number.

Solution 5

I think I used the stopword list for German from here when I built a search application with lucene.net a while ago. The site contains a list for English, too, and the lists on the site are apparaently the ones that the lucene project use as default, too.

View more solutions

Comments

[Mark Harrison]

lol, this is just the work I'm trying to avoid!

[bobobobo]

Your link is out, archive: web.archive.org/web/20080501010608/http://www.dcs.gla.ac.uk/‌​…

[alexis]

The nltk (Natural Language Toolkit, a python library) comes with a bunch of resources including a stopword corpus (Porter et al.), "2,400 stopwords for 11 languages". You can use the stopword list independent of the toolkit.

[Hamman Samuel]

How do I access this corpus of 2,400 stopwords in NLTK?

[Keyur Padalia]

nltk.org/nltk_data

[gidim]

The English stop-words in NLTK are tokenized. So instead of "shouldn't" it lists "shouldn"