pandas:DataFrame数据对齐与缺失数据
DataFrame对象在运算时，同样会对数据对齐，结果的行索引和列索引分别为两个操作数的行索引与列索引的并集
DataFrame处理缺失数据的方法
1 dropna(axis=0,how='any') #清除缺失数据，axis=0表示按行进行清除,axis=1表示按列清楚,how=any表示如果有任意一个NaN就清除,how=all表示该行(列)中的所有值为NaN就清除 2 3 fillna()设置缺失值 4 isnull()是否为空 5 notnull()不为空 6 7 In [62]: df2 8 Out[62]: 9 open close high 10 0 22.074 20.657 22.503 11 1 20.750 20.489 20.944 12 2 20.300 19.593 20.384 13 3 19.426 19.977 20.308 14 4 19.995 20.520 20.706 15 5 20.353 20.273 20.454 16 6 20.264 20.101 20.353 17 7 19.999 19.739 19.999 18 8 19.783 19.818 19.982 19 9 19.558 19.841 19.911 20 21 In [63]: df3 22 Out[63]: 23 date open close low 24 0 2007-03-01 22.074 20.657 20.220 25 1 2007-03-02 20.750 20.489 20.256 26 2 2007-03-05 20.300 19.593 19.218 27 3 2007-03-06 19.426 19.977 19.315 28 4 2007-03-07 19.995 20.520 19.827 29 5 2007-03-08 20.353 20.273 20.167 30 6 2007-03-09 20.264 20.101 19.735 31 7 2007-03-12 19.999 19.739 19.646 32 8 2007-03-13 19.783 19.818 19.699 33 9 2007-03-14 19.558 19.841 19.333 34 35 In [64]: df4 = df2+df3 36 37 In [65]: df4 38 Out[65]: 39 close date high low open 40 0 41.314 NaN NaN NaN 44.148 41 1 40.978 NaN NaN NaN 41.500 42 2 39.186 NaN NaN NaN 40.600 43 3 39.954 NaN NaN NaN 38.852 44 4 41.040 NaN NaN NaN 39.990 45 5 40.546 NaN NaN NaN 40.706 46 6 40.202 NaN NaN NaN 40.528 47 7 39.478 NaN NaN NaN 39.998 48 8 39.636 NaN NaN NaN 39.566 49 9 39.682 NaN NaN NaN 39.116 50 51 In [66]: df4.dropna(axis=1,) 52 Out[66]: 53 close open 54 0 41.314 44.148 55 1 40.978 41.500 56 2 39.186 40.600 57 3 39.954 38.852 58 4 41.040 39.990 59 5 40.546 40.706 60 6 40.202 40.528 61 7 39.478 39.998 62 8 39.636 39.566 63 9 39.682 39.116 64 65 66 In [67]: df4.fillna(0) 67 Out[67]: 68 close date high low open 69 0 41.314 0 0.0 0.0 44.148 70 1 40.978 0 0.0 0.0 41.500 71 2 39.186 0 0.0 0.0 40.600 72 3 39.954 0 0.0 0.0 38.852 73 4 41.040 0 0.0 0.0 39.990 74 5 40.546 0 0.0 0.0 40.706 75 6 40.202 0 0.0 0.0 40.528 76 7 39.478 0 0.0 0.0 39.998 77 8 39.636 0 0.0 0.0 39.566 78 9 39.682 0 0.0 0.0 39.116 79 80 In [68]: df4.isnull() 81 Out[68]: 82 close date high low open 83 0 False True True True False 84 1 False True True True False 85 2 False True True True False 86 3 False True True True False 87 4 False True True True False 88 5 False True True True False 89 6 False True True True False 90 7 False True True True False 91 8 False True True True False 92 9 False True True True False 93 94 In [69]: df4.notnull() 95 Out[69]: 96 close date high low open 97 0 True False False False True 98 1 True False False False True 99 2 True False False False True 100 3 True False False False True 101 4 True False False False True 102 5 True False False False True 103 6 True False False False True 104 7 True False False False True 105 8 True False False False True 106 9 True False False False True
pandas常用方法(适用于Series和DataFrame)
1 In [89]: df5 2 Out[89]: 3 id date open close high low volume code 4 0 0 2007-03-01 22.074 20.657 22.503 20.220 1977633.51 601318 5 1 1 2007-03-02 20.750 20.489 20.944 20.256 425048.32 601318 6 2 2 2007-03-05 20.300 19.593 20.384 19.218 419196.74 601318 7 3 3 2007-03-06 19.426 19.977 20.308 19.315 297727.88 601318 8 4 4 2007-03-07 19.995 20.520 20.706 19.827 287463.78 601318 9 5 5 2007-03-08 20.353 20.273 20.454 20.167 130983.83 601318 10 6 6 2007-03-09 20.264 20.101 20.353 19.735 160887.79 601318 11 7 7 2007-03-12 19.999 19.739 19.999 19.646 145353.06 601318 12 8 8 2007-03-13 19.783 19.818 19.982 19.699 102319.68 601318 13 9 9 2007-03-14 19.558 19.841 19.911 19.333 173306.56 601318 14 15 mean(axis=0,skipna=False) # 求平均值 16 17 In [90]: df5.mean() 18 Out[90]: 19 id 4.5000 20 open 20.2502 21 close 20.1008 22 high 20.5544 23 low 19.7416 24 volume 411992.1150 25 code 601318.0000 26 dtype: float64 27 28 In [91]: df5['open'].mean() 29 Out[91]: 20.2502 30 31 sum(axis=1) 32 33 In [93]: df5.sum() # 求和 34 Out[93]: 35 id 45 36 date 2007-03-012007-03-022007-03-052007-03-062007-0... 37 open 202.502 38 close 201.008 39 high 205.544 40 low 197.416 41 volume 4.11992e+06 42 code 6013180 43 dtype: object 44 45 sort_index(axis,ascending,...) #按行或列索引排序 46 sort_values(by,axis,ascending) # 按值排序 47 48 In [99]: df5.sort_index(axis=0) 49 Out[99]: 50 id date open close high low volume code 51 0 0 2007-03-01 22.074 20.657 22.503 20.220 1977633.51 601318 52 1 1 2007-03-02 20.750 20.489 20.944 20.256 425048.32 601318 53 2 2 2007-03-05 20.300 19.593 20.384 19.218 419196.74 601318 54 3 3 2007-03-06 19.426 19.977 20.308 19.315 297727.88 601318 55 4 4 2007-03-07 19.995 20.520 20.706 19.827 287463.78 601318 56 5 5 2007-03-08 20.353 20.273 20.454 20.167 130983.83 601318 57 6 6 2007-03-09 20.264 20.101 20.353 19.735 160887.79 601318 58 7 7 2007-03-12 19.999 19.739 19.999 19.646 145353.06 601318 59 8 8 2007-03-13 19.783 19.818 19.982 19.699 102319.68 601318 60 9 9 2007-03-14 19.558 19.841 19.911 19.333 173306.56 601318 61 62 63 In [102]: df5.sort_values(['close','open']) 64 Out[102]: 65 id date open close high low volume code 66 2 2 2007-03-05 20.300 19.593 20.384 19.218 419196.74 601318 67 7 7 2007-03-12 19.999 19.739 19.999 19.646 145353.06 601318 68 8 8 2007-03-13 19.783 19.818 19.982 19.699 102319.68 601318 69 9 9 2007-03-14 19.558 19.841 19.911 19.333 173306.56 601318 70 3 3 2007-03-06 19.426 19.977 20.308 19.315 297727.88 601318 71 6 6 2007-03-09 20.264 20.101 20.353 19.735 160887.79 601318 72 5 5 2007-03-08 20.353 20.273 20.454 20.167 130983.83 601318 73 1 1 2007-03-02 20.750 20.489 20.944 20.256 425048.32 601318 74 4 4 2007-03-07 19.995 20.520 20.706 19.827 287463.78 601318 75 0 0 2007-03-01 22.074 20.657 22.503 20.220 1977633.51 601318
1 # apply(func, axis=0) #将自定义函数应用在各行或者各列上，func可返回标量或者Series 2 #applymap(func) #将函数应用在DataFrame各个元素上 3 #map(func) 将函数应用在Series各个元素上 4 In [108]: df2 5 Out[108]: 6 open close high low volume 7 0 22.074 20.657 22.503 20.220 1977633.51 8 1 20.750 20.489 20.944 20.256 425048.32 9 2 20.300 19.593 20.384 19.218 419196.74 10 3 19.426 19.977 20.308 19.315 297727.88 11 4 19.995 20.520 20.706 19.827 287463.78 12 5 20.353 20.273 20.454 20.167 130983.83 13 6 20.264 20.101 20.353 19.735 160887.79 14 7 19.999 19.739 19.999 19.646 145353.06 15 8 19.783 19.818 19.982 19.699 102319.68 16 9 19.558 19.841 19.911 19.333 173306.56 17 18 In [110]: df2.apply(lambda x:x.sum()) 19 Out[110]: 20 open 202.502 21 close 201.008 22 high 205.544 23 low 197.416 24 volume 4119921.150 25 dtype: float64 26 27 In [109]: df2.applymap(lambda x:x+1) 28 Out[109]: 29 open close high low volume 30 0 23.074 21.657 23.503 21.220 1977634.51 31 1 21.750 21.489 21.944 21.256 425049.32 32 2 21.300 20.593 21.384 20.218 419197.74 33 3 20.426 20.977 21.308 20.315 297728.88 34 4 20.995 21.520 21.706 20.827 287464.78 35 5 21.353 21.273 21.454 21.167 130984.83 36 6 21.264 21.101 21.353 20.735 160888.79 37 7 20.999 20.739 20.999 20.646 145354.06 38 8 20.783 20.818 20.982 20.699 102320.68 39 9 20.558 20.841 20.911 20.333 173307.56
pandas: 层次化索引
层次化索引是pandas的一项重要功能，它使我们能够在一个轴上拥有多个索引级别
1 In [114]: import numpy as np 2 In [115]: data = pd.Series(np.random.rand(9),index=[['a','a','a','b','b','b','c','c','c'],[ 3 ...: 1,2,3,1,2,3,1,2,3]]) 4 5 In [116]: data 6 Out[116]: 7 a 1 0.445620 8 2 0.584242 9 3 0.454314 10 b 1 0.439814 11 2 0.714734 12 3 0.415314 13 c 1 0.491325 14 2 0.411385 15 3 0.617076 16 dtype: float64 17 18 In [118]: data['a'] 19 Out[118]: 20 1 0.445620 21 2 0.584242 22 3 0.454314 23 dtype: float64 24 25 In [119]: data['c'] 26 Out[119]: 27 1 0.491325 28 2 0.411385 29 3 0.617076 30 dtype: float64