How do I tokenize a string sentence in NLTK?

148,662

This is actually on the main page of nltk.org:

>>> import nltk
>>> sentence = """At eight o'clock on Thursday morning
... Arthur didn't feel very good."""
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['At', 'eight', "o'clock", 'on', 'Thursday', 'morning',
'Arthur', 'did', "n't", 'feel', 'very', 'good', '.']
Share:
148,662
diegoaguilar
Author by

diegoaguilar

I'm a mexican backend web developer interested in performance and infrastructure development. I've been working with Node/express mainly for last 3 years doing API's design and implementation. Been pretty involved with MEAN apps too, and pretty involved with MongoDB in particular. I really can say I love express before frameworks. About languages, Right now I'm pretty involved with Javascript, with new ES6 and ES7 stuff. I've got experience with Java and Python too, can even work a bit in Android what I've liked. Would love to get to know Go, Elixir and these trending functional programming languages soon ;) Image processing and natural language processing are challenging topics for me, and I'd love to lead my career to high performance and availability aplications and stacks development. I rather do backend development but lately I've been trying React, which I enjoy. Apart from software development, I love Real Madrid and watching movies and series, of course music, and I also love food, like a lot.

Updated on July 08, 2022

Comments

  • diegoaguilar
    diegoaguilar almost 2 years

    I am using nltk, so I want to create my own custom texts just like the default ones on nltk.books. However, I've just got up to the method like

    my_text = ['This', 'is', 'my', 'text']
    

    I'd like to discover any way to input my "text" as:

    my_text = "This is my text, this is a nice way to input text."
    

    Which method, python's or from nltk allows me to do this. And more important, how can I dismiss punctuation symbols?