1.WordNetLemmatizer提取词干

确定词根

import nltk
nltk.download('wordnet')
lemmatizer = WordNetLemmatizer()#确定词源
print(lemmatizer.lemmatize('gathering', 'v'))
print(lemmatizer.lemmatize('gathering', 'n'))

输出:

gather
gathering

2.word_tokenize分词

https://kite.com/python/docs/nltk.word_tokenize

分词：

import nltk
nltk.download('punkt')

sentence = "At eight o'clock on Thursday morning, Arthur didn't feel very good."
print(word_tokenize(sentence))

#
['At', 'eight', "o'clock", 'on', 'Thursday', 'morning', ',', 'Arthur', 'did', "n't", 'feel', 'very', 'good', '.']

2020-9-18更新————————

1.基本功能

https://www.jianshu.com/p/9d232e4a3c28

分句
分词
词性标注
命名实体识别

分句和分词：

#分句
sents = nltk.sent_tokenize("And now for something completely different. I love you.")#对于这种比较简单的句子，是可以处理的很好的。
word = []
for s in sents:
    print(s)
#在句子内部分词
for sent in sents:
    word.append(nltk.word_tokenize(sent))
print(word)

词性标注:

nltk.download('averaged_perceptron_tagger')
s="And now for something completely different."
text = nltk.word_tokenize(s)
print(text)
#词性标注
tagged = nltk.pos_tag(text)#这里需要是分词的结果，否则就会将单个char作为单位

tagged
[('And', 'CC'),
 ('now', 'RB'),
 ('for', 'IN'),
 ('something', 'NN'),
 ('completely', 'RB'),
 ('different', 'JJ'),
 ('.', '.')]

分块：

entities = nltk.chunk.ne_chunk(tagged)#如果使用这个函数的话，输入的变量必须是经过词性标注之后的，进行分块
print (entities)

输出：

Tree('S', [('And', 'CC'), ('now', 'RB'), ('for', 'IN'), ('something', 'NN'), ('completely', 'RB'), ('different', 'JJ'), ('.', '.')])

2.统计文本中词的概率分布FreqDist

from nltk.book import FreqDist#这个类是在book中的

text1 = nltk.word_tokenize("And now for something completely different. I love you. This is my friend. You are my friend.")

# FreqDist()获取在文本中每个出现的标识符的频率分布
fdist = FreqDist(text1)
print(fdist)
# 词数量
print(fdist.N())
# 不重复词的数量
print(fdist.B())

>>><FreqDist with 16 samples and 21 outcomes>
21
16

# 获取频率
print(fdist.freq('friend') * 100)
# 获取频数
print(fdist['friend'])
#出现次数最多的词
fdist.max()

>>>9.523809523809524
2
'.'

后面还有对文章进行词干化的代码，仔细看了一下，但是觉得目前用不到，所以就不贴了。

3.Text类和TextCollection类

前者是对单个文本的分析，后者是前者的集合，可以计算某一单词的逆文档频率IDF等。