基于Python的大数据的分页模型代码

最近在写一个cmdb系统的分页,尽管Django本身有分页的模块儿,但是还是想实现一下自己心中想的分页的一种逻辑

因为,在我们工作当中,当我们的数据量超级大的时候,其实我们每次分页查询都不必将所有的数据查询出来,而是可以按阶段的查询,举个例子

每次查询5页数据,当需要第六页的时候,再次进行加载,为了更加明了,做了举例

首先我们规定好每次查询5页,每页4条数据  

1 db_data = list(range(1,100))
2 
3 final_page = len(db_data)
4 
5 one_page_num = 5 #每页4条
6 one_time_num = 5 #每次查5页

1、查询索引的逻辑 设定findIndex查询的索引

 1     # page = 1  int(1/5) + 1  findIndex = 1
 2     # page = 2  int(2/5) + 1  findIndex = 1
 3     # page = 3  int(3/5) + 1  findIndex = 1
 4     # page = 4  int(4/5) + 1  findIndex = 1
 5     # page = 5  int(5/5)  findIndex = 1
 6     # page = 6  int(6/5) + 1  findIndex = 2
 7     # page = 7  int(7/5) + 1  findIndex = 2
 8     # page = 8  int(8/5) + 1  findIndex = 2
 9     # page = 9  int(9/5) + 1  findIndex = 2
10     # page = 10  int(10/5)  findIndex = 2
11     # page = 11  int(11/5) + 1  findIndex = 3
12     # page = 12  int(12/5) + 1  findIndex = 3
13     # page = 13  int(13/5) + 1  findIndex = 3
14     # page = 14  int(14/5) + 1  findIndex = 3
15     # page = 15  int(15/5)  findIndex = 3

所以我们可以写如下代码

 1 if page/one_time_num > int(page/one_time_num):
 2     findIndex = int(page / one_time_num) + 1
 3 else:
 4     findIndex = int(page / one_time_num)
 5 #进行第一次查询,查询每次5页,每页4条的数据
 6 select_num = one_time_num * one_page_num #这里决一次定查询20条
 7 select_start = (findIndex - 1) * select_num  # 开始查询的索引
 8 select_down =  findIndex * select_num  # 结束查询的索引
 9 select_data = db_data[select_start: select_down]  #这里查询出了一次20条数据

这个时候是查询出20条数据了,但是,我们想要的具体的数据和页码对不上,所以做了以下的数据

 1 #下面对20条数据进行截取
 2     #findIndex 1
 3         # page = 1  1-(0*5)  gnPage = 1
 4         # page = 2  2-(0*5)  gnPage = 2
 5         # page = 3  3-(0*5)  gnPage = 3
 6         # page = 4  4-(0*5)  gnPage = 4
 7         # page = 5  5-(0*5)  gnPage = 5
 8     #findIndex 2
 9         # page = 6  6-(1*5)  gnPage = 1
10         # page = 7  7-(1*5)  gnPage = 2
11         # page = 8  8-(1*5)  gnPage = 3
12         # page = 9  9-(1*5)  gnPage = 4
13         # page = 10  10-(1*5)  gnPage = 5
14     #findIndex 3
15         # page = 11  11-(2*5)  gnPage = 1
16         # page = 12  12-(2*5)  gnPage = 2
17         # page = 13  13-(2*5)  gnPage = 3
18         # page = 14  14-(2*5)  gnPage = 4
19         # page = 15  15-(2*5)  gnPage = 5

所以我们可以得到以下代码来获取具体的一页数据

#设定 截取的索引位now_index
now_index = page - (findIndex-1)*one_time_num
#设定单页的截取起始
page_start = (now_index - 1) * one_page_num
page_end = now_index * one_page_num
#开始截取
page_data = select_data[page_start:page_end]

最后进行代码结合

 1 #coding:utf-8
 2 
 3 db_data = list(range(1,100))
 4 
 5 one_page_num = 5 #每页4条
 6 one_time_num = 5 #每次查5页
 7 
 8 
 9 while True:
10     page = int(input("page >>>"))  #输入页码
11 
12     #查询索引的逻辑 设定findIndex查询的索引
13     # page = 1  int(1/5) + 1  findIndex = 1
14     # page = 2  int(2/5) + 1  findIndex = 1
15     # page = 3  int(3/5) + 1  findIndex = 1
16     # page = 4  int(4/5) + 1  findIndex = 1
17     # page = 5  int(5/5)  findIndex = 1
18     # page = 6  int(6/5) + 1  findIndex = 2
19     # page = 7  int(7/5) + 1  findIndex = 2
20     # page = 8  int(8/5) + 1  findIndex = 2
21     # page = 9  int(9/5) + 1  findIndex = 2
22     # page = 10  int(10/5)  findIndex = 2
23     # page = 11  int(11/5) + 1  findIndex = 3
24     # page = 12  int(12/5) + 1  findIndex = 3
25     # page = 13  int(13/5) + 1  findIndex = 3
26     # page = 14  int(14/5) + 1  findIndex = 3
27     # page = 15  int(15/5)  findIndex = 3
28 
29     if page/one_time_num > int(page/one_time_num):
30         findIndex = int(page / one_time_num) + 1
31     else:
32         findIndex = int(page / one_time_num)
33     #进行第一次查询,查询每次5页,每页4条的数据
34     select_num = one_time_num * one_page_num #这里决一次定查询20条
35     select_start = (findIndex - 1) * select_num  # 开始查询的索引
36     select_down =  findIndex * select_num  # 结束查询的索引
37 
38     select_data = db_data[select_start: select_down]  #这里查询出了一次20条数据
39 
40     #下面对20条数据进行截取
41     #findIndex 1
42         # page = 1  1-(0*5)  gnPage = 1
43         # page = 2  2-(0*5)  gnPage = 2
44         # page = 3  3-(0*5)  gnPage = 3
45         # page = 4  4-(0*5)  gnPage = 4
46         # page = 5  5-(0*5)  gnPage = 5
47     #findIndex 2
48         # page = 6  6-(1*5)  gnPage = 1
49         # page = 7  7-(1*5)  gnPage = 2
50         # page = 8  8-(1*5)  gnPage = 3
51         # page = 9  9-(1*5)  gnPage = 4
52         # page = 10  10-(1*5)  gnPage = 5
53     #findIndex 3
54         # page = 11  11-(2*5)  gnPage = 1
55         # page = 12  12-(2*5)  gnPage = 2
56         # page = 13  13-(2*5)  gnPage = 3
57         # page = 14  14-(2*5)  gnPage = 4
58         # page = 15  15-(2*5)  gnPage = 5
59     #设定 截取的索引位now_index
60     now_index = page - (findIndex-1)*one_time_num
61     #设定单页的截取起始
62     page_start = (now_index - 1) * one_page_num
63     page_end = now_index * one_page_num
64     #开始截取
65     page_data = select_data[page_start:page_end]
66     #输出效果
67     print("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
68     print("当前是第%s次查询"%findIndex)
69     print("当前查询到的所有数据是 
 %s" % select_data)
70     print("当前是 %s 页" % page)
71     print("当前的数据是 
 %s " % page_data)
72     print("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")

大家试一下吧。

原文地址:https://www.cnblogs.com/bianjinhui/p/9998529.html