【Spark机器学习速成宝典】模型篇06随机森林【Random Forests】（Python版）

# -*-coding=utf-8 -*-  
from pyspark import SparkConf, SparkContext
sc = SparkContext('local')

from pyspark.mllib.tree import RandomForest, RandomForestModel
from pyspark.mllib.util import MLUtils

# Load and parse the data file into an RDD of LabeledPoint.
data = MLUtils.loadLibSVMFile(sc, 'data/mllib/sample_libsvm_data.txt')
'''
每一行使用以下格式表示一个标记的稀疏特征向量
label index1:value1 index2:value2 ...

tempFile.write(b"+1 1:1.0 3:2.0 5:3.0\n-1\n-1 2:4.0 4:5.0 6:6.0")
>>> tempFile.flush()
>>> examples = MLUtils.loadLibSVMFile(sc, tempFile.name).collect()
>>> tempFile.close()
>>> examples[0]
LabeledPoint(1.0, (6,[0,2,4],[1.0,2.0,3.0]))
>>> examples[1]
LabeledPoint(-1.0, (6,[],[]))
>>> examples[2]
LabeledPoint(-1.0, (6,[1,3,5],[4.0,5.0,6.0]))
'''
# Split the data into training and test sets (30% held out for testing) 分割数据集，留30%作为测试集
(trainingData, testData) = data.randomSplit([0.7, 0.3])

# Train a RandomForest model. 训练决策树模型
#  Empty categoricalFeaturesInfo indicates all features are continuous. 空的categoricalFeaturesInfo意味着所有的特征都是连续的
#  Note: Use larger numTrees in practice. 注意：在实践中可以使用更多棵树
#  Setting featureSubsetStrategy="auto" lets the algorithm choose. featureSubsetStrategy="auto"的意思是让算法自己选择
model = RandomForest.trainClassifier(trainingData, numClasses=2, categoricalFeaturesInfo={},
                                     numTrees=3, featureSubsetStrategy="auto",
                                     impurity='gini', maxDepth=4, maxBins=32)

# Evaluate model on test instances and compute test error 评估模型
predictions = model.predict(testData.map(lambda x: x.features))
labelsAndPredictions = testData.map(lambda lp: lp.label).zip(predictions)
testErr = labelsAndPredictions.filter(
    lambda lp: lp[0] != lp[1]).count() / float(testData.count())
print('Test Error = ' + str(testErr)) #Test Error = 0.0
print('Learned classification forest model:')
print(model.toDebugString())
'''
TreeEnsembleModel classifier with 3 trees

  Tree 0:
    If (feature 517 <= 116.0)
     If (feature 489 <= 11.0)
      If (feature 437 <= 218.0)
       Predict: 0.0
      Else (feature 437 > 218.0)
       Predict: 1.0
     Else (feature 489 > 11.0)
      Predict: 1.0
    Else (feature 517 > 116.0)
     Predict: 1.0
  Tree 1:
    If (feature 456 <= 0.0)
     If (feature 471 <= 0.0)
      Predict: 1.0
     Else (feature 471 > 0.0)
      Predict: 0.0
    Else (feature 456 > 0.0)
     Predict: 0.0
  Tree 2:
    If (feature 377 <= 3.0)
     If (feature 212 <= 253.0)
      Predict: 0.0
     Else (feature 212 > 253.0)
      Predict: 1.0
    Else (feature 377 > 3.0)
     If (feature 299 <= 204.0)
      Predict: 1.0
     Else (feature 299 > 204.0)
      Predict: 0.0
'''
# Save and load model
model.save(sc, "myRandomForestClassificationModel")
sameModel = RandomForestModel.load(sc, "myRandomForestClassificationModel")
print sameModel.predict(data.collect()[0].features) #0.0
返回目录
【Spark机器学习速成宝典】模型篇06随机森林【Random Forests】（Python版）

目录

随机森林原理

随机森林代码(Spark Python)
