并行处理能大大提高数据的处理速度,它依赖于硬件资源、网络资源等环境.
并行处理的硬件资源环境分为5大类:
1.传统的单台计算机、单处理器、单内核的机器.(无法进行并行处理,但是并行程序还是可以运行的,此时就和普通程序一样了)
2.单台计算机、单处理器、多内核的机器.(目前本人的测试环境,没钱买多处理器的
)
3.单台计算机、多处理器、多内核的机器.
4.集群系统
5.分布式系统
PS:如何查看几个处理器、几核,常用的软件就是CPU-Z.
本人的测试机器CPU:
单CPU,双核心处理器
下面开始,编写一个普通的管道函数(这里演示的就是管道函数的并行处理):
CREATE OR REPLACE FUNCTION pipe_test
(
c_empno SYS_REFCURSOR
,p VARCHAR2 DEFAULT ','
) RETURN EMP_element
PIPELINED IS
v_element VARCHAR2(1000);
BEGIN
FETCH c_empno
INTO v_element;
LOOP
EXIT WHEN c_empno%NOTFOUND;
PIPE ROW(substr(v_element, 0, instr(v_element, p) - 1));
LOOP
v_element := substr(v_element
,instr(v_element, p) + 1
,length(v_element) - instr(v_element, p) + 1);
EXIT WHEN instr(v_element, p) = 0;
PIPE ROW(substr(v_element, 0, instr(v_element, p) - 1));
END LOOP;
PIPE ROW(v_element);
FETCH c_empno
INTO v_element;
END LOOP;
RETURN;
END pipe_test;
注意需要先建立类型EMP_element,如下:
CREATE OR REPLACE TYPE EMP_element as table of varchar2(100);
然后我们建立一个并行处理的管道函数:
CREATE OR REPLACE FUNCTION parallel_test
(
p_empno SYS_REFCURSOR
,p VARCHAR2 DEFAULT ','
) RETURN EMP_element
PIPELINED
PARALLEL_ENABLE(PARTITION p_empno BY ANY) IS
v_element VARCHAR2(1000);
BEGIN
FETCH p_empno
INTO v_element;
LOOP
EXIT WHEN p_empno%NOTFOUND;
PIPE ROW(substr(v_element, 0, instr(v_element, p) - 1));
LOOP
v_element := substr(v_element
,instr(v_element, p) + 1
,length(v_element) - instr(v_element, p) + 1);
EXIT WHEN instr(v_element, p) = 0;
PIPE ROW(substr(v_element, 0, instr(v_element, p) - 1));
END LOOP;
PIPE ROW(v_element);
FETCH p_empno
INTO v_element;
END LOOP;
RETURN;
END parallel_test;
留意下启用并行的关键字PARALLEL_ENABLE.
接下来我们构造大表进行测试,根据v$pq_sesstat视图的结果,我们来判断数据库系统是否真的进行了并行处理.
12:57:52 SYS@orcl> create table big_data as select 'Cc' a,'Dd' b from dual connect by level<1000000;
表已创建。
已用时间: 00: 00: 01.92
12:59:00 SYS@orcl> select count(*) from big_data;
COUNT(*)
----------
999999
已选择 1 行。
已用时间: 00: 00: 00.31
13:47:08 SYS@orcl> set autot trace exp stat
--普通管道函数耗时40.48秒
13:48:42 SYS@orcl> select SCOTT.pipe_test(CURSOR(SELECT a||','||b from big_data)) FROM DUAL;
已选择 1 行。
已用时间: 00: 00: 40.48
执行计划
----------------------------------------------------------
Plan hash value: 2452824241
------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 2 (0)| 00:00:01 |
| 1 | VIEW | | 8168 | 414K| 29 (0)| 00:00:01 |
| 2 | COLLECTION ITERATOR PICKLER FETCH| PIPE_TEST | 8168 | | 29 (0)| 00:00:01 |
| 3 | FAST DUAL | | 1 | | 2 (0)| 00:00:01 |
------------------------------------------------------------------------------------------------
Note
-----
- dynamic sampling used for this statement (level=2)
统计信息
----------------------------------------------------------
1000515 recursive calls
0 db block gets
1000325 consistent gets
1548 physical reads
0 redo size
6027769 bytes sent via SQL*Net to client
1214 bytes received via SQL*Net from client
10 SQL*Net roundtrips to/from client
7 sorts (memory)
0 sorts (disk)
1 rows processed
13:53:24 SYS@orcl> alter system flush shared_pool;
系统已更改。
已用时间: 00: 00: 01.43
--使用启用并行的管道函数,耗时40.32,与普通的管道函数耗时差不多
--奇怪的是执行计划了多了一个BIG_DATA的全表扫描操作了
13:53:26 SYS@orcl> select SCOTT.parallel_test(CURSOR(SELECT a||','||b from big_data)) FROM DUAL;
已选择 1 行。
已用时间: 00: 00: 40.32
执行计划
----------------------------------------------------------
Plan hash value: 1682567826
----------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
----------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 2 (0)| 00:00:01 |
| 1 | VIEW | | 8168 | 414K| 29 (0)| 00:00:01 |
| 2 | COLLECTION ITERATOR PICKLER FETCH| PARALLEL_TEST | 8168 | | 29 (0)| 00:00:01 |
| 3 | TABLE ACCESS FULL | BIG_DATA | 968K| 7565K| 422 (3)| 00:00:06 |
| 4 | FAST DUAL | | 1 | | 2 (0)| 00:00:01 |
----------------------------------------------------------------------------------------------------
Note
-----
- dynamic sampling used for this statement (level=2)
统计信息
----------------------------------------------------------
1000867 recursive calls
0 db block gets
1000353 consistent gets
1560 physical reads
0 redo size
6026847 bytes sent via SQL*Net to client
1006 bytes received via SQL*Net from client
8 SQL*Net roundtrips to/from client
9 sorts (memory)
0 sorts (disk)
1 rows processed
--通过v$pq_sesstat视图我们发现此时系统并没有进行并行处理,(其实通过执行计划也可看出没有使用并行)
13:54:21 SYS@orcl> select * from v$pq_sesstat;
STATISTIC LAST_QUERY SESSION_TOTAL
------------------------------ ---------- -------------
Queries Parallelized 0 0
DML Parallelized 0 0
DDL Parallelized 0 0
DFO Trees 0 0
Server Threads 0 0
Allocation Height 0 0
Allocation Width 0 0
Local Msgs Sent 0 0
Distr Msgs Sent 0 0
Local Msgs Recv'd 0 0
Distr Msgs Recv'd 0 0
已选择11行。
已用时间: 00: 00: 00.00
13:54:40 SYS@orcl> alter system flush buffer_cache;
系统已更改。
已用时间: 00: 00: 00.00
13:55:27 SYS@orcl> alter system flush shared_pool;
系统已更改。
已用时间: 00: 00: 01.40
--接下来我们加HINT,强制并行处理,此时耗时25.03秒,降低了一半
13:55:30 SYS@orcl> select /*+ parallel */scott.PARALLEL_TEST(CURSOR(SELECT a||','||b from big_data))
FROM DUAL;
已选择 1 行。
已用时间: 00: 00: 25.03
执行计划
----------------------------------------------------------
Plan hash value: 3127237831
-----------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | TQ |IN-OUT| PQ Distrib |
-----------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 2 (0)| 00:00:01 | | | |
| 1 | PX COORDINATOR | | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10000 | 8168 | 414K| 16 (0)| 00:00:01 | Q1,00 | P->S | QC (RAND) |
| 3 | VIEW | | 8168 | 414K| 16 (0)| 00:00:01 | Q1,00 | PCWP | |
| 4 | COLLECTION ITERATOR PICKLER FETCH| PARALLEL_TEST | 8168 | | 16 (0)| 00:00:01 | Q1,00 | PCWP | |
| 5 | PX BLOCK ITERATOR | | 968K| 7565K| 234 (3)| 00:00:03 | Q1,00 | PCWC | |
| 6 | TABLE ACCESS FULL | BIG_DATA | 968K| 7565K| 234 (3)| 00:00:03 | Q1,00 | PCWP | |
| 7 | FAST DUAL | | 1 | | 2 (0)| 00:00:01 | | | |
-----------------------------------------------------------------------------------------------------------------------------------
Note
-----
- dynamic sampling used for this statement (level=2)
- automatic DOP: Computed Degree of Parallelism is 2
统计信息
----------------------------------------------------------
1000941 recursive calls
4 db block gets
1000516 consistent gets
1842 physical reads
0 redo size
6026847 bytes sent via SQL*Net to client
1006 bytes received via SQL*Net from client
8 SQL*Net roundtrips to/from client
9 sorts (memory)
0 sorts (disk)
1 rows processed
--上面的执行计划及v$pq_sesstat都显示系统启用了并行处理,并行度为2
13:57:44 SYS@orcl> select * from v$pq_sesstat;
STATISTIC LAST_QUERY SESSION_TOTAL
------------------------------ ---------- -------------
Queries Parallelized 0 1
DML Parallelized 0 0
DDL Parallelized 0 0
DFO Trees 0 1
Server Threads 0 0
Allocation Height 0 0
Allocation Width 0 0
Local Msgs Sent 0 553
Distr Msgs Sent 0 0
Local Msgs Recv'd 0 553
Distr Msgs Recv'd 0 0
已选择11行。
已用时间: 00: 00: 00.01
为了使示例更加有说服力,我们再来试一下普通管道函数+HINT强制并行看看:
16:02:36 SYS@orcl> select /*+ parallel */SCOTT.pipe_test(CURSOR(SELECT a||','||b from big_data)) FROM DUAL;
已选择 1 行。
已用时间: 00: 00: 43.34
执行计划
----------------------------------------------------------
Plan hash value: 2452824241
------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 2 (0)| 00:00:01 |
| 1 | VIEW | | 8168 | 414K| 16 (0)| 00:00:01 |
| 2 | COLLECTION ITERATOR PICKLER FETCH| PIPE_TEST | 8168 | | 16 (0)| 00:00:01 |
| 3 | FAST DUAL | | 1 | | 2 (0)| 00:00:01 |
------------------------------------------------------------------------------------------------
Note
-----
- dynamic sampling used for this statement (level=2)
- automatic DOP: Computed Degree of Parallelism is 1
统计信息
----------------------------------------------------------
1003093 recursive calls
18 db block gets
1000973 consistent gets
1628 physical reads
0 redo size
6026843 bytes sent via SQL*Net to client
1006 bytes received via SQL*Net from client
8 SQL*Net roundtrips to/from client
100 sorts (memory)
0 sorts (disk)
1 rows processed
由于普通函数不支持并行,所以即使+HINT强制并行,系统还是没有使用并行处理.耗时依然为40多秒.
如果不使用函数,使用普通的SQL,又是可以使用HINT强制并行的,如下:
15:59:07 SYS@orcl> select /*+ parallel */ a||','||b from big_data;
已选择999999行。
已用时间: 00: 00: 04.09
执行计划
----------------------------------------------------------
Plan hash value: 2638980575
--------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | TQ |IN-OUT| PQ Distrib |
--------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 968K| 7565K| 234 (3)| 00:00:03 | | | |
| 1 | PX COORDINATOR | | | | | | | | |
| 2 | PX SEND QC (RANDOM)| :TQ10000 | 968K| 7565K| 234 (3)| 00:00:03 | Q1,00 | P->S | QC (RAND) |
| 3 | PX BLOCK ITERATOR | | 968K| 7565K| 234 (3)| 00:00:03 | Q1,00 | PCWC | |
| 4 | TABLE ACCESS FULL| BIG_DATA | 968K| 7565K| 234 (3)| 00:00:03 | Q1,00 | PCWP | |
--------------------------------------------------------------------------------------------------------------
Note
-----
- dynamic sampling used for this statement (level=2)
- automatic DOP: Computed Degree of Parallelism is 2
统计信息
----------------------------------------------------------
6 recursive calls
0 db block gets
1577 consistent gets
1520 physical reads
0 redo size
13467035 bytes sent via SQL*Net to client
733742 bytes received via SQL*Net from client
66668 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
999999 rows processed
通过上面的示例可以看到,并行处理可以大大加速数据的处理,几乎成倍的提升性能.
但并不是并行度为几,就会提示几倍的性能,它受硬件环境,优化器内部算法,PL/SQL引擎解析等等的约束.
总之,并行处理给我们带来的性能提升是很大的,所以在某些场景使用并行编程也是很必要的.