nltk分词

1.安装nltk

2.运行如下

>>>import nltk
>>> nltk.download('punkt')

3.代码：

import nltk
sentence= """At eight o'clock on Thursday morning
... Arthur didn't feel very good."""
tokens = nltk.word_tokenize(sentence)
print(tokens)

4.结果