gensim是python下一个极易上手的主题模型程序库(topic model)，网址在：http://radimrehurek.com/gensim/index.html
安装过程较为繁琐，参考http://radimrehurek.com/gensim/install.html 中的步骤。
我本机用的python2.7，需安装setuptools或者pip，然后通过这2个工具安装numpy和scipy，因为gensim里面依赖科学/数值计算。其中scipy还需要安装BLAS和LAPACK这2个包，所以说依赖的库非常之多。在安装LAPACK的时候还要注意修改make.inc中的编译参数，增加-fPIC，否则scipy安装会报错。这其中的弯路暂且不表，重点介绍安装完之后的怎么玩gensim。
gensim支持的模型由LSI，LDA，TFIDF等模型，由于主题模型要用到语料，所以在做预研的时候可以结合NLTK的丰富语料库来getting started。
以LSI为例，整个流程包括数据读取/预处理、训练和预测，数据读取和处理使用gensim的corpus包和dictionary包进行序列化，转成TFIDF格式后，交给model下的LsiModel训练，代码如下：
- 主函数
'''
Created on 2013-6-13
@author: william.hw
'''
import gensim, aequerycluster, logging, nltk, sys
def notpurepunc(word):
for ch in word:
if (ch <= '9' and ch >= '0') or (ch <= 'z' and ch >= 'a'):
return True
return False
if __name__ == '__main__':
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
logger = logging.getLogger('aequerycluster')
dictpath = 'aequerycluster.dict'
modelpath = 'aequerycluster.model'
logger.info("start...")
hivefile = sys.argv[1]
uniqlevel = sys.argv[2]
texts = []
count = 10
with open(hivefile) as finput:
for line in finput:
fields = line[1:-2].split("","")
if len(fields) != 2:
continue
count -= 1
if count < 0:
break
texts.append(fields[1].split())
logger.info("finish make texts")
# unique token
if uniqlevel == "full":
all_tokens = sum(texts, [])
tokens_once = set(word for word in set(all_tokens) if all_tokens.count(word) == 1)
texts = [[word for word in text if word not in tokens_once and len(word) > 1 and word.lower() not in nltk.corpus.stopwords.words('english') and notpurepunc(word)] for text in texts]
else:
texts = [[word for word in text if len(word) > 1 and word.lower() not in nltk.corpus.stopwords.words('english') and notpurepunc(word)] for text in texts]
logger.info("finish filter texts")
serdictionary = aequerycluster.ClusterUtil.serializeDictionary(texts, dictpath)
aqc = aequerycluster.AeQueryCluster()
mycorpus = [serdictionary.doc2bow(text) for text in texts]
mytfidf = gensim.models.TfidfModel(mycorpus)
corpus_tfidf = mytfidf[mycorpus]
lsimodel = aqc.train_tfidf(corpus_tfidf, dictpath, 10)
logger.info("finish train gensim")
lsimodel.print_topics(10,5)
lsimodel.save(modelpath)
index = gensim.similarities.MatrixSimilarity(lsimodel[corpus_tfidf])
newdoc = "Human computer interaction"
vec_bow = serdictionary.doc2bow(newdoc.lower().split())
vec_lsi = lsimodel[vec_bow]
sims = index[vec_lsi]
print list(enumerate(sims))
- 封装类
'''
Created on 2013-6-13
@author: william.hw
'''
import gensim
class ClusterUtil(object):
@staticmethod
def deserializedCorpus(serpath, serformat = "MM"):
if serformat == "SVM":
return gensim.corpora.SvmLightCorpus(serpath)
else:
return gensim.corpora.MmCorpus(serpath)
@staticmethod
def serializeCorpus(corpus, serpath, serformat = "MM"):
if serformat == "SVM":
gensim.corpora.SvmLightCorpus.serialize(serpath, corpus)
else:
gensim.corpora.MmCorpus.serialize(serpath, corpus)
@staticmethod
def deserializedDictionary(serpath):
return gensim.corpora.Dictionary.load(serpath)
@staticmethod
def serializeDictionary(texts, dictpath, save=False):
dictionary = gensim.corpora.Dictionary(texts)
if save:
dictionary.save(dictpath)
return dictionary
class AeQueryCluster(object):
def __init__(self):
self._model = None
def train(self, corpus, serdictionary, num_topics=2):
tfidf = gensim.models.TfidfModel(corpus)
corpus_tfidf = tfidf[corpus]
self._model = self.train_tfidf(corpus_tfidf, self._dictionary, num_topics)
return self._model
def train_tfidf(self, corpus_tfidf, serdictionary, num_topics=2):
self._dictionary = ClusterUtil.deserializedDictionary(serdictionary)
self._model = gensim.models.LsiModel(corpus_tfidf, num_topics, self._dictionary)
return self._model
def train_onedoc(self, corpus_tfidf):
self._model.add_documents(corpus_tfidf)
return self._model
首先要记得初始化logging，因为之后LSI训练的结果需要通过logging才能够打印出来调试。hivefile保存的是一行行的待聚类的doc，切词之后过滤掉标点符号和停用词(使用NLTK英语停用词库)。然后将其序列化到dictionary中，并将文本打散成词袋(doc2bow)，生成原始语料。
生成预料后需要将语料转为TDIDF，因为主题模型以每个token的TD和IDF作为训练的特征(我个人理解)，然后用词频和逆文档频率作为LSI模型的训练输入，通过设置主题数=10后，模型建立完毕，我们用print_topics(10,5)打印出每个主题，及其最显著的5个token代表。
2013-06-19 17:00:43,409 : INFO : topic #0(1.087): 0.210*"phone" + 0.187*"Windows" + 0.183*"Dual" + 0.183*"3G" + 0.171*"core" 2013-06-19 17:00:43,410 : INFO : topic #1(1.056): 0.441*"silver" + 0.289*"accessories" + 0.289*"tibetan" + 0.289*"jewelry" + 0.171*"pure" 2013-06-19 17:00:43,410 : INFO : topic #2(1.054): 0.419*"hair" + 0.222*"wave" + 0.222*"product,queen" + 0.222*"brazilian" + 0.222*"body" 2013-06-19 17:00:43,411 : INFO : topic #3(1.024): -0.239*"phone" + -0.164*"arrival" + -0.164*"many" + -0.164*"new" + -0.164*"wifi" 2013-06-19 17:00:43,412 : INFO : topic #4(1.000): 0.557*"Led" + 0.371*"Indoor" + 0.371*"Display" + 0.186*"Module" + 0.186*"Unit" 2013-06-19 17:00:43,412 : INFO : topic #5(1.000): -0.459*"capacity" + -0.459*"4g" + -0.459*"card" + -0.229*"memory" + -0.229*"4gb" 2013-06-19 17:00:43,413 : INFO : topic #6(0.960): 0.204*"Windows" + -0.155*"Leather" + -0.155*"X2" + -0.155*"x2" + -0.155*"Case" 2013-06-19 17:00:43,413 : INFO : topic #7(0.944): -0.217*"wave" + -0.217*"product,queen" + -0.217*"brazilian" + -0.217*"body" + -0.217*"3pcs/lot,queen" 2013-06-19 17:00:43,414 : INFO : topic #8(0.937): 0.263*"accessories" + 0.263*"tibetan" + 0.263*"jewelry" + -0.164*"s999" + -0.164*"pure" 2013-06-19 17:00:43,415 : INFO : topic #9(0.923): -0.219*"Windows" + 0.185*"smartphones" + 0.185*"WiFi" + 0.185*"S5" + 0.185*"Phone"
每个token之前的权重代表token对主题的影响和贡献，可以看到有些weight是负的，代表token对主题是反方向的。
如果需要预测，则先索引模型数据similarities.MatrixSimilarity，对于待预测的一个newdoc，同样也是预处理成词袋格式doc2bow，再跟索引数据进行相似度计算(用的是余弦夹角，值区间是-1到1)：
sims = index[vec_lsi] print list(enumerate(sims)) [(0, -2.2351742e-08), (1, 3.7252903e-09), (2, 0.99415344), (3, 7.4505806e-09), (4, -9.3132257e-09), (5, -1.4901161e-08), (6, 0.0), (7, -1.8626451e-09), (8, 3.7252903e-09), (9, 0.0)]
可以看到当前的这个doc相对第2个主题的相似度最大，为0.99415344。
美中不足的是，虽然gensim本身支持分布式计算，但是对于想直接利用hadoop的mapreduce进行云计算还没有找到好的解决方案，希望有志之士能一起探讨，找到解决方案。