python: How to use POS (part of speech) features in scikit learn classfiers (SVM) etc

Question

I want to use part of speech (POS) returned from nltk.pos_tag for sklearn classifier, How can I convert them to vector and use it? e.g.

sent = "This is POS example"

tok=nltk.tokenize.word_tokenize(sent) pos=nltk.pos_tag(tok) print (pos)

This returns following [('This', 'DT'), ('is', 'VBZ'), ('POS', 'NNP'), ('example', 'NN')]

Now I am unable to apply any of the vectorizer (DictVectorizer, or FeatureHasher, CountVectorizer from scikitlearn to use in classifier

Pls suggest

Solution 1

If I'm understanding you right, this is a bit tricky. Once you tag it, your sentence (or document, or whatever) is no longer composed of words, but of pairs (word + tag), and it's not clear how to make the most useful vector-of-scalars out of that.

Most text vectorizers do something like counting how many times each vocabulary item occurs, and then making a feature for each one:

 the: 4, player: 1, bats: 1, well: 2, today: 3,...

The next document might have:

 the: 0, quick:5, flying:3, bats:1, caught:1, bugs:2

Both can be stored as arrays of integers so long as you always put the same key in the same array element (you'll have a lot of zeros for most documents) -- or as a dict. So a vectorizer does that for many "documents", and then works on that.

So your question boils down to how to turn a list of pairs into a flat list of items that the vectorizors can count.

The most trivial way is to flatten your data to

('This', 'POS_DT', 'is', 'POS_VBZ', 'POS', 'POS_NNP', 'example', 'POS_NN')

The usual counting would then get a vector of 8 vocabulary items, each occurring once. I renamed the tags to make sure they can't get confused with words.

That would get you up and running, but it probably wouldn't accomplish much. That's because just knowing how many occurrences of each part of speech there are in a sample may not tell you what you need -- notice that any notion of which parts of speech go with which words is gone after the vectorizer does its counting.

Running a classifier on that may have some value if you're trying to distinguish something like style -- fiction may have more adjectives, lab reports may have fewer proper names (maybe), and so on.

Instead, you could change your data to

('This_DT', 'is_VBZ', 'POS_NNP', 'example_NN')

That keeps each tag "tied" to the word it belongs with, so now the vectors will be able to distinguish samples where "bat" is used as a verbs, from samples where it's only used as a noun. That would tell you slightly different things -- for example, "bat" as a verb is more likely in texts about baseball than in texts about zoos.

And there are many other arrangements you could do.

To get good results from using vector methods on natural language text, you will likely need to put a lot of thought (and testing) into just what features you want the vectorizer to generate and use. It depends heavily on what you're trying to accomplish in the end.

Hope that helps.

Solution 2

I know this is a bit late, but gonna add an answer here.

Depending on what features you want, you'll need to encode the POST in a way that makes sense. I've had the best results with SVM classification using ngrams when I glue the original sentence to the POST sentence so that it looks like the following:

word1 word2 word3 ... wordn POST1 POST2 POST3... POSTn

Once this is done, I feed it into a standard ngram or whatever else and feed that into the SVM.

This method keeps the information of the individual words, but also keeps the vital information of POST patterns when you give your system a words it hasn't seen before but that the tagger has encountered before.

Solution 3

What about merging the word and its tag like 'word/tag' then you may feed your new corpus to a vectorizer that count the word (TF-IDF or word of bags) then make a feature for each one:

    wpt = nltk.WordPunctTokenizer()
    text = wpt.tokenize('Someone should have this ring to a volcano')
    text_tagged = nltk.pos_tag(text)
    new_text = []
    for word in text_tagged:
      new_text.append(word[0] + "/" + word[1])

    doc = ' '.join(new_text)

output for this is

   Someone/NN should/MD have/VB this/DT piece/NN of/IN shit/NN to/TO a/DT volcano/NN

Comments

[Suresh Mali]

Thanks that helps. I am trying following just POS tags, POS tags_word (as suggested by you) and concatenate all pos tags only(so that position of pos tag information is retained). Will see which one helps better

[Ivan Bilan]

Great suggestion. But what if I have other features (not vectorizers) that are looking for a specific word occurance?

[Ivan Bilan]

This is worth trying! Thanks