pyspark数据准备

鸢尾花数据集

1 5.1,3.5,1.4,0.2,Iris-setosa
2 4.9,3.0,1.4,0.2,Iris-setosa
3 4.7,3.2,1.3,0.2,Iris-setosa
4 4.6,3.1,1.5,0.2,Iris-setosa
5 5.0,3.6,1.4,0.2,Iris-setosa
6 5.4,3.9,1.7,0.4,Iris-setosa
7 4.6,3.4,1.4,0.3,Iris-setosa
8 5.0,3.4,1.5,0.2,Iris-setosa

转换成libsvm格式代码

 1 import sys
 2 
 3 file = sys.argv[1]
 4 
 5 def main():
 6     with open(file,'r') as df:
 7         for line in df:
 8             ss = line.strip().split(",")
 9             if ss[4]=="Iris-setosa":
10                 ss[4]=0
11             if ss[4]=="Iris-versicolor":
12                 ss[4]=1
13             if ss[4]=="Iris-virginica":
14                 ss[4]=2
15             print("%d 1:%.1f 2:%.1f 3:%.1f 4:%.1f"%(ss[4],float(ss[0]),float(ss[1]),float(ss[2]),float(ss[3])))
16 if __name__ == '__main__':
17     try:
18         main()
19     except Exception as e:
20         raise e

libsvm格式的鸢尾花数据集

 1 0 1:5.1 2:3.5 3:1.4 4:0.2
 2 0 1:4.9 2:3.0 3:1.4 4:0.2
 3 0 1:4.7 2:3.2 3:1.3 4:0.2
 4 0 1:4.6 2:3.1 3:1.5 4:0.2
 5 0 1:5.0 2:3.6 3:1.4 4:0.2
 6 0 1:5.4 2:3.9 3:1.7 4:0.4
 7 0 1:4.6 2:3.4 3:1.4 4:0.3
 8 0 1:5.0 2:3.4 3:1.5 4:0.2
 9 0 1:4.4 2:2.9 3:1.4 4:0.2
10 0 1:4.9 2:3.1 3:1.5 4:0.1
11 0 1:5.4 2:3.7 3:1.5 4:0.2

pyspark读取libsvm格式数据并转换

>>> from pyspark.mllib.util import MLUtils

>>> examples = MLUtils.loadLibSVMFile(sc, "data/mllib/sample_libsvm_data.txt")

　>>> examples.take(2)
　[Stage 26:> (0 + 1) / 1]

　[LabeledPoint(0.0, (4,[0,1,2,3],[5.1,3.5,1.4,0.2])), LabeledPoint(0.0, (4,[0,1,2
　,3],[4.9,3.0,1.4,0.2]))]