Python – tokenizer errors with nltk

nltkpython

I'm very new to Python, and am trying to learn in conjunction with using nltk.

I've been following some examples and testing things out, but it seems I am very limited in what I can do due to errors being returned by python.

I know nltk is installed and importing fine, because this code works

from nltk.sem import chat80
print chat80.items

However, 'from nltk.tokenizer import *' returns

'File "stdin", line1. 

I get similar errors when using any sort of "TOKEN=" or I'm guessing tokenization of anything.

I've installed python many times in the last few days, hoping a different version or better install might help.

I'm getting this error on windows7 using activePython2.6, though I've gotten similar err
ors with python 3.1 activePython3.1 and Python 2.6.
as well as on Mac OSx 10.5 with Python 2.5.

The mac is giving a bit more data with "Import Error: No module named tokenizer.

I'm just trying some of the introductory demos to nltk online, not even trying to write my own code yet, and I'm getting more errors than successes.

Best Answer

Looks like the nltp package doesn't have a tokenizer package.

A quick look on the NLTK website suggests that from nltp.tokenize import * is what you're after.

Related Topic