大数据量导出文件优化

33s,24s/4.88m;2表示测试2次,耗时分别为33秒、24秒,生成文件大小4.88M

数据量
导出文件格式
6w 10w 20w
csv 33s,24s/4.88m;2 59s,8.14m 250s/16.3m;3
dbf 37s,38s/40.1m;2 64s/66.8m 239s,240s,240s/133m;3
xlsx 40s/2.86m;2 63s/4.75m 246s,248s,248s/9.52m;3

10w数据测试更多细节

3.2(getData)18 3.2(getData)18 3.2(getData)34    
3.3(ObjToMap)11 3.3(ObjToMap&Mapping)16 3.3(ObjToMap&Mapping)26  
3.4(ComplexPropData)0 3.4(ComplexPropData)0 3.4(ComplexPropData)0  
3.5(Mapping)4 3.5(DataIntercept)0 3.5(DataIntercept)0  
3.6(DataIntercept)0 2.2 35   2.2 60      
2.2 34   2.3(addtitletosheet) 0 2.3(addtitletosheet) 0  
2.3(addtitletosheet) 0 2.4(addgridtosheet) 4 2.4(addgridtosheet) 9  
2.4(addgridtosheet) 3 2.5(serverpro) 40 2.5(serverpro) 69    
2.5(serverpro) 38 2.6(write) 1 2.6(write) 2    
2.6(write) 1   1.2(server) 42 1.2(server) 72    
1.2(server) 39 1.3 0   1.3 0      
1.3 0              

结论:10w数据做3次for循环和1次for循环耗时差别不大,所以慢的根本原因不在于此

更多细节:

begin:1464679204339 begin:1464680191678
totalCount:100000 totalCount:100000  
getDatas1:1464679207778 getDatas1:1464680194934
usetime:3s usetime:3s  
sortDatas :1464679207778 sortDatas :1464680194934
usetime:0s usetime:0s  
0-299999   0-299999    
getPagingData  :1464679207778 getPagingData  :1464680194934
usetime:0s usetime:0s  
3.1.2 3   3.1.2 3    
3.2(getData)31 3.2(getData)3  
3.3(ObjToMap&Mapping)14 3.3(ObjToMap&Mapping)38
3.4(ComplexPropData)0 3.4(ComplexPropData)0
3.5(DataIntercept)0 3.5(DataIntercept)0
2.2 46   2.2 41    
2.3(addtitletosheet) 0 2.3(addtitletosheet) 0
2.4(addgridtosheet) 7 2.4(addgridtosheet) 5
2.5(serverpro) 53 2.5(serverpro) 47  
2.6(write) 2 2.6(write) 2  
1.2(server) 56 1.2(server) 49  
1.3 0   1.3 0    

3.2和3.3变化很大,

代码不同:第一次测试,3.2步骤包含代理对象的集合强制类型转换代码collection = ((EntityList) collection).getTarget();

而第二次测试,3.3步骤移除了上述代码慢的代码为:targetMap=BeanToMapUtil.convertBean(obj);即将代理对象转换为map,第一次测试同样包含了该代码。

综上:大数据量时处理代理对象是很耗时的。

修改获取数据方式,直接获取原始对象数据:

begin:1464681604310   begin:1464681631666  
totalCount:100000   totalCount:100000  
getDatas1:1464681607939   getDatas1:1464681635058
usetime:3s   usetime:3s  
sortDatas :1464681607939   sortDatas :1464681635058
usetime:0s   usetime:0s  
0-299999     0-299999    
getPagingData  :1464681607939   getPagingData  :1464681635058
usetime:0s   usetime:0s  
3.2(getData)3   3.2(getData)3  
3.3(ObjToMap&Mapping)6   3.3(ObjToMap&Mapping)7
3.4(ComplexPropData)0   3.4(ComplexPropData)0  
3.5(DataIntercept)0   3.5(DataIntercept)0  
2.2 10     2.2 11    
2.3(addtitletosheet) 0   2.3(addtitletosheet) 0
2.4(addgridtosheet) 5   2.4(addgridtosheet) 5  
2.5(serverpro) 16   2.5(serverpro) 17  
2.6(write) 2   2.6(write) 2  
1.2(server) 18   1.2(server) 19  
1.3 0     1.3 0    


 
耗时减少近2/3。

结论:大数据量情况下,代理严重影响性能。

岁月留痕
原文地址:https://www.cnblogs.com/navigate/p/5531123.html