3 汇总和计算描述统计
pandas有一组用于常用的数学和统计方法。他们一般都是基于没有缺失数据而构建的。
下面是一些简约方法的选项：
In [81]: df = DataFrame([[1.4,np.nan],[7.1,-4.5],[np.nan,np.nan],[0.73,-1.3]],index=list('abcd'),columns=['one','two'])
In [82]: df
Out[82]:
one two
a 1.40 NaN
b 7.10 -4.5
c NaN NaN
d 0.73 -1.3
In [83]: df.sum()
Out[83]:
one 9.23
two -5.80
dtype: float64
In [84]: df.mean()
Out[84]:
one 3.076667
two -2.900000
dtype: float64
In [85]: df.mean(axis=1) # 指定方向
Out[85]:
a 1.400
b 1.300
c NaN
d -0.285
dtype: float64
In [86]: df.mean(axis=1, skipna=False) # 排除nan
Out[86]:
a NaN
b 1.300
c NaN
d -0.285
dtype: float64
下面是描述和汇总统计相关的方法：
In [88]: df.describe()
/Users/yangfeilong/anaconda/lib/python2.7/site-packages/numpy/lib/function_base.py:3834: RuntimeWarning: Invalid value encountered in percentile
RuntimeWarning)
Out[88]:
one two
count 3.000000 2.000000
mean 3.076667 -2.900000
std 3.500376 2.262742
min 0.730000 -4.500000
25% NaN NaN
50% NaN NaN
75% NaN NaN
max 7.100000 -1.300000
In [89]: df.max()
Out[89]:
one 7.1
two -1.3
dtype: float64
In [90]: df.min(axis=1)
Out[90]:
a 1.4
b -4.5
c NaN
d -1.3
dtype: float64
In [91]: df.quantile()
Out[91]:
one NaN
two NaN
dtype: float64
In [92]: s1 = Series(np.arange(100))
In [93]: s1.quantile()
Out[93]: 49.5
In [94]:
In [95]: s1.quantile(0.2)
Out[95]: 19.800000000000001
In [96]: s1.quantile(0.24)
Out[96]: 23.759999999999998
In [97]: s1.quantile(0.25)
Out[97]: 24.75
In [98]: s1.quantile(0.5)
Out[98]: 49.5
In [99]: s1.quantile()
Out[99]: 49.5
In [100]: s1.median()
Out[100]: 49.5
In [101]: s1.mad()
Out[101]: 25.0
In [102]: df = DataFrame(np.arange(100).reshape(10,10),columns=list('abcdefghij'))
In [103]: df
Out[103]:
a b c d e f g h i j
0 0 1 2 3 4 5 6 7 8 9
1 10 11 12 13 14 15 16 17 18 19
2 20 21 22 23 24 25 26 27 28 29
3 30 31 32 33 34 35 36 37 38 39
4 40 41 42 43 44 45 46 47 48 49
5 50 51 52 53 54 55 56 57 58 59
6 60 61 62 63 64 65 66 67 68 69
7 70 71 72 73 74 75 76 77 78 79
8 80 81 82 83 84 85 86 87 88 89
9 90 91 92 93 94 95 96 97 98 99
In [104]: df.mad()
Out[104]:
a 25.0
b 25.0
c 25.0
d 25.0
e 25.0
f 25.0
g 25.0
h 25.0
i 25.0
j 25.0
dtype: float64
In [105]: df.mad(axis=1)
Out[105]:
0 2.5
1 2.5
2 2.5
3 2.5
4 2.5
5 2.5
6 2.5
7 2.5
8 2.5
9 2.5
dtype: float64
In [106]: df.var(axis=1)
Out[106]:
0 9.166667
1 9.166667
2 9.166667
3 9.166667
4 9.166667
5 9.166667
6 9.166667
7 9.166667
8 9.166667
9 9.166667
dtype: float64
In [107]: df.var(axis=0)
Out[107]:
a 916.666667
b 916.666667
c 916.666667
d 916.666667
e 916.666667
f 916.666667
g 916.666667
h 916.666667
i 916.666667
j 916.666667
dtype: float64
In [108]: df.cummax()
Out[108]:
a b c d e f g h i j
0 0 1 2 3 4 5 6 7 8 9
1 10 11 12 13 14 15 16 17 18 19
2 20 21 22 23 24 25 26 27 28 29
3 30 31 32 33 34 35 36 37 38 39
4 40 41 42 43 44 45 46 47 48 49
5 50 51 52 53 54 55 56 57 58 59
6 60 61 62 63 64 65 66 67 68 69
7 70 71 72 73 74 75 76 77 78 79
8 80 81 82 83 84 85 86 87 88 89
9 90 91 92 93 94 95 96 97 98 99
In [109]: df
Out[109]:
a b c d e f g h i j
0 0 1 2 3 4 5 6 7 8 9
1 10 11 12 13 14 15 16 17 18 19
2 20 21 22 23 24 25 26 27 28 29
3 30 31 32 33 34 35 36 37 38 39
4 40 41 42 43 44 45 46 47 48 49
5 50 51 52 53 54 55 56 57 58 59
6 60 61 62 63 64 65 66 67 68 69
7 70 71 72 73 74 75 76 77 78 79
8 80 81 82 83 84 85 86 87 88 89
9 90 91 92 93 94 95 96 97 98 99
In [112]: df.diff()
Out[112]:
a b c d e f g h i j
0 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
1 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0
2 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0
3 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0
4 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0
5 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0
6 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0
7 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0
8 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0
9 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0 10.0
3.1 相关系数和协方差
相关概念：
- 相关系数：反映两变量间线性相关关系的统计指标称为相关系数
- 协方差：在概率论和统计学中，协方差用于衡量两个变量的总体误差。而方差是协方差的一种特殊情况，即当两个变量是相同的情况。
In [92]: s1 = Series(np.arange(100))
In [117]: s1.corr(s2)
Out[117]: 0.99999999999999989
In [118]: s2 = Series(np.arange(2,202,2))
In [119]: s1.corr(s2)
Out[119]: 0.99999999999999989
In [120]: s1.cov(s2)
Out[120]: 1683.3333333333335
In [102]: df = DataFrame(np.arange(100).reshape(10,10),columns=list('abcdefghij'))
In [122]: df.corr()
Out[122]:
a b c d e f g h i j
a 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
b 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
c 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
d 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
e 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
f 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
g 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
h 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
i 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
j 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
3.2 唯一值、值计数以及成员资格
unique:去重
In [124]: s1 = Series(list('ccbbabcddacbd'))
In [125]: s1.unique()
Out[125]: array(['c', 'b', 'a', 'd'], dtype=object)
value_counts:值计数
In [129]: s1.value_counts() # 默认降序排列
Out[129]:
b 4
c 4
d 3
a 2
dtype: int64
In [130]: pd.value_counts(s1,sort=False) # pd也可以直接调用
Out[130]:
a 2
c 4
b 4
d 3
dtype: int64
isin:判断矢量化的集合成员
In [134]: obj = Series(list('abcbcdcddcba'))
In [135]: mask = obj.isin(['b','c'])
In [136]: mask
Out[136]:
0 False
1 True
2 True
3 True
4 True
5 False
6 True
7 False
8 False
9 True
10 True
11 False
dtype: bool
In [137]: obj[mask]
Out[137]:
1 b
2 c
3 b
4 c
6 c
9 c
10 b
dtype: object
如下表：
形成一个相关列的柱状图
In [138]: data = DataFrame({'Qu1':[1,3,4,3,4],'Qu2':[2,3,1,2,3],'Qu3':[1,5,2,4,4]})
In [139]: data
Out[139]:
Qu1 Qu2 Qu3
0 1 2 1
1 3 3 5
2 4 1 2
3 3 2 4
4 4 3 4
In [143]: data.apply(pd.value_counts).fillna(0)
Out[143]:
Qu1 Qu2 Qu3
1 1.0 1.0 1.0
2 0.0 2.0 1.0
3 2.0 2.0 0.0
4 2.0 0.0 2.0
5 0.0 0.0 1.0
4 处理缺失数据
pandas使用非浮点数（NaN）来表示缺失数据，它只是表示缺少数据的一种标识。
In [144]: string_data = Series(['hello',np.nan,'world'])
In [145]: string_data
Out[145]:
0 hello
1 NaN
2 world
dtype: object
In [146]: string_data.isnull()
Out[146]:
0 False
1 True
2 False
dtype: bool
注意：python中的None值也会被当成Nan处理。
4.1 滤除缺失数据
纯手工处理永远是最好的，但是很麻烦，使用dropna来处理简单一些。
In [146]: string_data.isnull()
Out[146]:
0 False
1 True
2 False
dtype: bool
In [147]: data = Series([1,np.nan,3,np.nan])
In [148]: data
Out[148]:
0 1.0
1 NaN
2 3.0
3 NaN
dtype: float64
In [149]: data.dropna()
Out[149]:
0 1.0
2 3.0
dtype: float64
当然也可以使用bool索引来处理。
In [150]: data[data.notnull()]
Out[150]:
0 1.0
2 3.0
dtype: float64
DataFrame而言比较麻烦。
In [152]: df = DataFrame([[1,2,3],[np.nan,np.nan,np.nan],[3,4,np.nan],[2,3,4]])
In [153]: df
Out[153]:
0 1 2
0 1.0 2.0 3.0
1 NaN NaN NaN
2 3.0 4.0 NaN
3 2.0 3.0 4.0
In [154]: df.dropna()
Out[154]:
0 1 2
0 1.0 2.0 3.0
3 2.0 3.0 4.0
In [155]: df.dropna(how='all') # 只丢弃全部都是nan的行。
Out[155]:
0 1 2
0 1.0 2.0 3.0
2 3.0 4.0 NaN
3 2.0 3.0 4.0
In [164]: df[4] = np.nan
In [165]: df
Out[165]:
0 1 2 4
0 1.0 2.0 3.0 NaN
1 NaN NaN NaN NaN
2 3.0 4.0 NaN NaN
3 2.0 3.0 4.0 NaN
In [166]: df.dropna(axis=1,how='all')
Out[166]:
0 1 2
0 1.0 2.0 3.0
1 NaN NaN NaN
2 3.0 4.0 NaN
3 2.0 3.0 4.0
4.2 填充缺失数据
生成数据：
In [167]: df = DataFrame(np.random.randn(4,4),columns=list('abcd'))
In [168]: df
Out[168]:
a b c d
0 -0.010218 -0.256541 -0.507837 0.470124
1 0.293587 0.517149 -1.813092 -0.791727
2 0.434398 1.352332 0.012355 -1.687852
3 0.573836 -0.701182 -0.548737 0.022037
In [169]: df.ix[:2,2]
Out[169]:
0 -0.507837
1 -1.813092
2 0.012355
Name: c, dtype: float64
In [170]: df.ix[:2,2]= np.nan
In [171]: df.ix[:1,3]= np.nan
In [172]: df
Out[172]:
a b c d
0 -0.010218 -0.256541 NaN NaN
1 0.293587 0.517149 NaN NaN
2 0.434398 1.352332 NaN -1.687852
3 0.573836 -0.701182 -0.548737 0.022037
In [173]: df.fillna(0) #全部填充0
Out[173]:
a b c d
0 -0.010218 -0.256541 0.000000 0.000000
1 0.293587 0.517149 0.000000 0.000000
2 0.434398 1.352332 0.000000 -1.687852
3 0.573836 -0.701182 -0.548737 0.022037
In [176]: df.fillna({'c':0,'d':0.5}) #不同列填充不同的值
Out[176]:
a b c d
0 -0.010218 -0.256541 0.000000 0.500000
1 0.293587 0.517149 0.000000 0.500000
2 0.434398 1.352332 0.000000 -1.687852
3 0.573836 -0.701182 -0.548737 0.022037
#默认总是会返回新的对象，也可以在源对象上修改；
In [177]: _ = df.fillna({'c':0,'d':0.5},inplace=True)
In [178]: df
Out[178]:
a b c d
0 -0.010218 -0.256541 0.000000 0.500000
1 0.293587 0.517149 0.000000 0.500000
2 0.434398 1.352332 0.000000 -1.687852
3 0.573836 -0.701182 -0.548737 0.022037
同样，也可以使用其他选项
In [181]: df
Out[181]:
a b c d
0 -0.010218 -0.256541 NaN NaN
1 0.293587 0.517149 NaN NaN
2 0.434398 1.352332 NaN -1.687852
3 0.573836 -0.701182 -0.548737 0.022037
In [184]: df.fillna(method='bfill',limit=2)
Out[184]:
a b c d
0 -0.010218 -0.256541 NaN -1.687852
1 0.293587 0.517149 -0.548737 -1.687852
2 0.434398 1.352332 -0.548737 -1.687852
3 0.573836 -0.701182 -0.548737 0.022037
待续。。。