机器学习实战之树回归
机器学习实战 ch09 问题解决办法
最近在学习《机器学习实战（Machine Learning in Action）》，因为个人比较喜欢Python 3，而这本书里面的代码都是通过Python 2实现的，所以自己在调试的时候会改写成Python 3。
在前几章里问题都不是很大，但是在ch09 树回归这一章中出现了很多小问题，现在此做一个笔记，如果有错误望指正。
问题一
在使用书上ch09中的代码时，运行遇到的第一个问题就是类型错误，无法映射。错误提示为：TypeError: unsupported operand type(s) for /: 'map' and 'int'。
def loadDataSet(fileName): dataMat = [] fr = open(fileName) for line in fr.readlines(): curLine = line.strip().split(' ') fltLine = map(float, curLine) dataMat.append(fltLine) return dataMat
在loadDataSet这个方法中，使用了一个map方法来对从文本文件中读取的数据进行映射处理，也就是把读取到的string转换为float。这一个简单的类型转换按照书上实现的方法在Python 2中不会报错。
但是在Python 3中，map方法返回的是一个map对象，因此对于这个错误，解决办法很简单。
def loadDataSet(fileName): dataMat = [] fr = open(fileName, 'r') for line in fr.readlines(): curLine = line.strip().split(' ') fltLine = list(map(float, curLine)) # 方法1 # fltLine = [float(item) for item in curLine] # 方法2 dataMat.append(fltLine) fr.close() return dataMat
这里给出两种解决办法：
方法1，将map方法返回的map对象再转换为list对象就行了。
方法2，使用列表推导式做一个处理。
这里给出的两种解决办法的运行结果完全一样，具体使用看个人喜好吧，关于性能问题暂不作考虑。
问题二
解决了问题一之后，重新运行，又出现新的错误，说matrix类型不能被哈希。错误提示为：TypeError: unhashable type: 'matrix'。
在chooseBestSplit方法里有这么一条语句：
for splitVal in set((dataSet[:, featIndex])):
这里报错就是无法把matrix对象转换为set类型，这里找了一个解决办法是将这一句改为：
for splitVal in set((dataSet[:, featIndex].T.A.tolist())[0]):
解释一下就是对于矩阵中满足featIndex的一列，先将其转置（.T），然后转换为numpy array类型（.A），再转换为list类型（.tolist()）。
在一系列转换之后，就可以达到最初想要的效果了。
问题三
解决了问题二之后，再运行，出现新的错误。错误提示为：IndexError: index 0 is out of bounds for axis 0 with size 0。
在书中binSplitDataSet方法的三个参数为数据集、待切分的特征、特征值。该方法用于在给定特征和特征值的情况下，通过数组过滤的方式将数据集进行切分并返回两个子集。书中给出的实现为：
def binSplitDataSet(dataSet, feature, value): mat0 = dataSet[nonzero(dataSet[:,feature] > value)[0],:][0] mat1 = dataSet[nonzero(dataSet[:,feature] <= value)[0],:][0] return mat0,mat1
其实可以在测试的时候就发现，并不像书上给出的运行结果一样，mat0的值是一个一行的向量，mat1的值是一个三行的矩阵。
解决办法很简单，讲两条语句最后的[0]删去就行了。
def binSplitDataSet(dataSet, feature, value): mat0 = dataSet[nonzero(dataSet[:, feature] > value)[0], :] mat1 = dataSet[nonzero(dataSet[:, feature] <= value)[0], :] return mat0, mat1
解释一下，该方法使用的是数组过滤的办法，对于特征feature，先过滤出特征feature的值大于（或小于等于）value的。这里使用nonzero()方法返回的是一个元素均为boolean类型的list，而这个list[0]的值就是对应过滤出的元素下标，换句话说就是过滤出的值在原数组中的位置。最后一步是一个Python切片操作，通过dataSet[index, :]把对应的向量提取出来。
个人理解在这里需要得到的是被切分的两个子集，所以最后的[0]下标运算是多余的。
小结
在ch09中，9.3.2之前的问题就是这些。解决了这三个问题之后，程序可以正常执行，得出和书上一样的测试结果。
运行截图如下：
test9.py
#-*- coding：utf-8 import sys sys.path.append("regTrees.py") import regTrees from numpy import * import matplotlib.pyplot as plt # testMat = mat(eye(4)) # print(testMat) # mat0, mat1 = regTrees.binSplitDataSet(testMat, 1, 0.5) # print("test9") # print(mat0) # print("mat0 over") # print(mat1) # myDat = regTrees.loadDataSet("ex00.txt") # myMat = mat(myDat) # regTree = regTrees.createTree(myMat) # print(regTree) # # myDat1 = regTrees.loadDataSet("ex0.txt") # myMat1 = mat(myDat1) # regTree1 = regTrees.createTree(myMat1) # print(regTree1) # myDat2 = regTrees.loadDataSet("ex2.txt") # myMat2 = mat(myDat2) # # regTree2 = regTrees.createTree(myMat2) # # regTree2 = regTrees.createTree(myMat2, ops = (1000, 4)) # # print(regTree2) # # myTree = regTrees.createTree(myMat2, ops = (0, 1)) # myDatTest = regTrees.loadDataSet("ex2test.txt") # myMat2Test = mat(myDatTest) # myNewTree = regTrees.prune(myTree, myMat2Test) # print(myNewTree) trainMat = mat(regTrees.loadDataSet('bikeSpeedVsIq_train.txt')) testMat = mat(regTrees.loadDataSet('bikeSpeedVsIq_test.txt')) # 回归树 myTree = regTrees.createTree(trainMat, ops = (1, 20)) yHat = regTrees.createForeCast(myTree, testMat[:, 0]) resultCorr0 = corrcoef(yHat, testMat[:, 1], rowvar = 0) #corrcoef得到相关系数矩阵（向量的相似程度） print(resultCorr0) # 模型树 myTree = regTrees.createTree(trainMat, regTrees.modelLeaf, regTrees.modelErr, (1,20)) yHat = regTrees.createForeCast(myTree, testMat[:,0], regTrees.modelTreeEval) resultCorr1 = corrcoef(yHat, testMat[:,1], rowvar=0) print(resultCorr1) # 标准回归 ws, X, Y = regTrees.linearSolve(trainMat) print(ws) for i in range(shape(testMat)[0]) : yHat[i] = testMat[i, 0] * ws[1, 0] + ws[0, 0] resultCorr2 = corrcoef(yHat, testMat[:,1], rowvar=0) print(resultCorr2) print("over!")
regTrees.py
''' Created on Feb 4, 2011 Tree-Based Regression Methods @author: Peter Harrington ''' from numpy import * def loadDataSet(fileName): dataMat = [] fr = open(fileName, 'r') for line in fr.readlines(): curLine = line.strip().split(' ') fltLine = list(map(float, curLine)) # 方法1 # fltLine = [float(item) for item in curLine] # 方法2 dataMat.append(fltLine) fr.close() return dataMat def binSplitDataSet(dataSet, feature, value): mat0 = dataSet[nonzero(dataSet[:,feature] > value)[0],:] mat1 = dataSet[nonzero(dataSet[:,feature] <= value)[0],:] return mat0,mat1 def regLeaf(dataSet):#returns the value used for each leaf return mean(dataSet[:,-1]) def regErr(dataSet): return var(dataSet[:,-1]) * shape(dataSet)[0] def linearSolve(dataSet): #helper function used in two places m,n = shape(dataSet) X = mat(ones((m,n))); Y = mat(ones((m,1)))#create a copy of data with 1 in 0th postion X[:,1:n] = dataSet[:,0:n-1]; Y = dataSet[:,-1]#and strip out Y xTx = X.T*X if linalg.det(xTx) == 0.0: raise NameError('This matrix is singular, cannot do inverse, try increasing the second value of ops') ws = xTx.I * (X.T * Y) return ws,X,Y def modelLeaf(dataSet):#create linear model and return coeficients ws,X,Y = linearSolve(dataSet) return ws def modelErr(dataSet): ws,X,Y = linearSolve(dataSet) yHat = X * ws return sum(power(Y - yHat,2)) def chooseBestSplit(dataSet, leafType=regLeaf, errType=regErr, ops=(1,4)): tolS = ops[0]; tolN = ops[1] #if all the target variables are the same value: quit and return value if len(set(dataSet[:,-1].T.tolist()[0])) == 1: #exit cond 1 return None, leafType(dataSet) m,n = shape(dataSet) #the choice of the best feature is driven by Reduction in RSS error from mean S = errType(dataSet) bestS = inf; bestIndex = 0; bestValue = 0 for featIndex in range(n-1): for splitVal in set((dataSet[:, featIndex].T.A.tolist())[0]): mat0, mat1 = binSplitDataSet(dataSet, featIndex, splitVal) if (shape(mat0)[0] < tolN) or (shape(mat1)[0] < tolN): continue newS = errType(mat0) + errType(mat1) if newS < bestS: bestIndex = featIndex bestValue = splitVal bestS = newS #if the decrease (S-bestS) is less than a threshold don't do the split if (S - bestS) < tolS: return None, leafType(dataSet) #exit cond 2 mat0, mat1 = binSplitDataSet(dataSet, bestIndex, bestValue) if (shape(mat0)[0] < tolN) or (shape(mat1)[0] < tolN): #exit cond 3 return None, leafType(dataSet) return bestIndex,bestValue#returns the best feature to split on #and the value used for that split def createTree(dataSet, leafType=regLeaf, errType=regErr, ops=(1,4)):#assume dataSet is NumPy Mat so we can array filtering feat, val = chooseBestSplit(dataSet, leafType, errType, ops)#choose the best split if feat == None: return val #if the splitting hit a stop condition return val retTree = {} retTree['spInd'] = feat retTree['spVal'] = val lSet, rSet = binSplitDataSet(dataSet, feat, val) retTree['left'] = createTree(lSet, leafType, errType, ops) retTree['right'] = createTree(rSet, leafType, errType, ops) return retTree def isTree(obj): return (type(obj).__name__=='dict') def getMean(tree): if isTree(tree['right']): tree['right'] = getMean(tree['right']) if isTree(tree['left']): tree['left'] = getMean(tree['left']) return (tree['left']+tree['right'])/2.0 def prune(tree, testData): if shape(testData)[0] == 0: return getMean(tree) #if we have no test data collapse the tree if (isTree(tree['right']) or isTree(tree['left'])):#if the branches are not trees try to prune them lSet, rSet = binSplitDataSet(testData, tree['spInd'], tree['spVal']) if isTree(tree['left']): tree['left'] = prune(tree['left'], lSet) if isTree(tree['right']): tree['right'] = prune(tree['right'], rSet) #if they are now both leafs, see if we can merge them if not isTree(tree['left']) and not isTree(tree['right']): lSet, rSet = binSplitDataSet(testData, tree['spInd'], tree['spVal']) errorNoMerge = sum(power(lSet[:,-1] - tree['left'],2)) + sum(power(rSet[:,-1] - tree['right'],2)) treeMean = (tree['left']+tree['right'])/2.0 errorMerge = sum(power(testData[:,-1] - treeMean,2)) if errorMerge < errorNoMerge: print("merging") return treeMean else: return tree else: return tree def regTreeEval(model, inDat): return float(model) def modelTreeEval(model, inDat): n = shape(inDat)[1] X = mat(ones((1,n+1))) X[:,1:n+1]=inDat return float(X*model) def treeForeCast(tree, inData, modelEval=regTreeEval): if not isTree(tree): return modelEval(tree, inData) if inData[tree['spInd']] > tree['spVal']: if isTree(tree['left']): return treeForeCast(tree['left'], inData, modelEval) else: return modelEval(tree['left'], inData) else: if isTree(tree['right']): return treeForeCast(tree['right'], inData, modelEval) else: return modelEval(tree['right'], inData) def createForeCast(tree, testData, modelEval=regTreeEval): m=len(testData) yHat = mat(zeros((m,1))) for i in range(m): yHat[i,0] = treeForeCast(tree, mat(testData[i]), modelEval) return yHat
treeExplore.py
from numpy import * from tkinter import * import regTrees import matplotlib matplotlib.use('TkAgg') from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg from matplotlib.figure import Figure def reDraw(tolS,tolN): reDraw.f.clf() # clear the figure reDraw.a = reDraw.f.add_subplot(111) if chkBtnVar.get(): if tolN < 2: tolN = 2 myTree=regTrees.createTree(reDraw.rawDat, regTrees.modelLeaf, regTrees.modelErr, (tolS,tolN)) yHat = regTrees.createForeCast(myTree, reDraw.testDat, regTrees.modelTreeEval) else: myTree=regTrees.createTree(reDraw.rawDat, ops=(tolS,tolN)) yHat = regTrees.createForeCast(myTree, reDraw.testDat) reDraw.a.scatter(reDraw.rawDat[:,0].tolist(), reDraw.rawDat[:,1].tolist(), s=5) #use scatter for data set reDraw.a.plot(reDraw.testDat, yHat, linewidth=2.0) #use plot for yHat reDraw.canvas.show() def getInputs(): try: tolN = int(tolNentry.get()) except: tolN = 10 print("enter Integer for tolN") tolNentry.delete(0, END) tolNentry.insert(0,'10') try: tolS = float(tolSentry.get()) except: tolS = 1.0 print("enter Float for tolS") tolSentry.delete(0, END) tolSentry.insert(0,'1.0') return tolN,tolS def drawNewTree(): tolN,tolS = getInputs()#get values from Entry boxes reDraw(tolS,tolN) root=Tk() reDraw.f = Figure(figsize=(5,4), dpi=100) #create canvas reDraw.canvas = FigureCanvasTkAgg(reDraw.f, master=root) reDraw.canvas.show() reDraw.canvas.get_tk_widget().grid(row=0, columnspan=3) Label(root, text="tolN").grid(row=1, column=0) tolNentry = Entry(root) tolNentry.grid(row=1, column=1) tolNentry.insert(0,'10') Label(root, text="tolS").grid(row=2, column=0) tolSentry = Entry(root) tolSentry.grid(row=2, column=1) tolSentry.insert(0,'1.0') Button(root, text="ReDraw", command=drawNewTree).grid(row=1, column=2, rowspan=3) chkBtnVar = IntVar() chkBtn = Checkbutton(root, text="Model Tree", variable = chkBtnVar) chkBtn.grid(row=3, column=0, columnspan=2) reDraw.rawDat = mat(regTrees.loadDataSet('sine.txt')) reDraw.testDat = arange(min(reDraw.rawDat[:,0]), max(reDraw.rawDat[:,0]), 0.01) reDraw(1.0, 10) root.mainloop()