几台WEB服务器经常宕机，平时运行得好好的，负载也都在2以下，偶尔会突然负高上去就宕机，init 6也无法重启成功
只能echo 'b'>/proc/sysrq-trigger 硬重启才行,dmesg 没发现硬件方面有问题。
从top相关信息来看，每次宕机现象都是系统内核态占用非常高，相关nginx,php-cgi进程虽然不僵死但无法kill
每次宕机都会有kswaped0进程出现，但是从free上来看，内存还是有空闲，swap用得也并不多
top - 22:26:02 up 3 days, 8:06, 1 user, load average: 15.05, 15.12, 15.09
Tasks: 518 total, 16 running, 502 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 87.6%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16426616k total, 15985652k used, 440964k free, 102232k buffers
Swap: 16386292k total, 22552k used, 16363740k free, 8320464k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
553 root 20 -5 0 0 0 R 100.9 0.0 115:59.83 kswapd0
554 root 20 -5 0 0 0 R 100.9 0.0 115:55.12 kswapd1
3417 daemon 25 0 90164 48m 1052 R 100.6 0.3 142:19.56 nginx
3418 daemon 25 0 90840 48m 1048 R 100.6 0.3 143:54.26 nginx
3422 daemon 25 0 94196 52m 1052 R 100.6 0.3 144:34.68 nginx
3423 daemon 25 0 85680 43m 1048 R 100.4 0.3 141:31.40 nginx
3421 daemon 25 0 87720 45m 1056 R 100.1 0.3 143:13.23 nginx
3419 daemon 25 0 90148 48m 1044 R 99.6 0.3 144:06.64 nginx
16942 daemon 25 0 440m 55m 33m R 99.6 0.3 106:08.76 php-cgi
14476 daemon 25 0 432m 55m 33m R 97.3 0.3 102:58.61 php-cgi
8706 daemon 25 0 432m 60m 37m R 93.7 0.4 103:46.72 php-cgi
3416 daemon 25 0 88832 47m 1048 R 93.2 0.3 144:04.28 nginx
7672 daemon 25 0 439m 67m 37m R 82.4 0.4 104:21.30 php-cgi
8102 daemon 25 0 441m 62m 36m R 59.8 0.4 107:12.30 php-cgi
3420 daemon 25 0 89760 47m 1056 R 50.1 0.3 143:38.59 nginx
15963 root 16 0 13008 1428 820 R 22.6 0.0 0:04.34 top
3428 root 15 0 411m 4300 1172 S 2.1 0.0 2:33.80 php-cgi
15965 root 18 0 99.6m 1584 908 S 0.8 0.0 0:00.03 crond
3743 root 15 0 74812 1164 584 S 0.3 0.0 0:00.83 crond
15964 root 18 0 99.6m 1468 800 D 0.3 0.0 0:00.01 crond
top - 22:26:02 up 3 days, 8:06, 1 user, load average: 15.05, 15.12, 15.09
Tasks: 518 total, 16 running, 502 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 87.6%sy, 0.0%ni, 12.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16426616k total, 15985652k used, 440964k free, 102232k buffers
Swap: 16386292k total, 22552k used, 16363740k free, 8320464k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
553 root 20 -5 0 0 0 R 100.9 0.0 115:59.83 kswapd0
554 root 20 -5 0 0 0 R 100.9 0.0 115:55.12 kswapd1
3417 daemon 25 0 90164 48m 1052 R 100.6 0.3 142:19.56 nginx
3418 daemon 25 0 90840 48m 1048 R 100.6 0.3 143:54.26 nginx
3422 daemon 25 0 94196 52m 1052 R 100.6 0.3 144:34.68 nginx
3423 daemon 25 0 85680 43m 1048 R 100.4 0.3 141:31.40 nginx
3421 daemon 25 0 87720 45m 1056 R 100.1 0.3 143:13.23 nginx
3419 daemon 25 0 90148 48m 1044 R 99.6 0.3 144:06.64 nginx
16942 daemon 25 0 440m 55m 33m R 99.6 0.3 106:08.76 php-cgi
14476 daemon 25 0 432m 55m 33m R 97.3 0.3 102:58.61 php-cgi
8706 daemon 25 0 432m 60m 37m R 93.7 0.4 103:46.72 php-cgi
3416 daemon 25 0 88832 47m 1048 R 93.2 0.3 144:04.28 nginx
7672 daemon 25 0 439m 67m 37m R 82.4 0.4 104:21.30 php-cgi
8102 daemon 25 0 441m 62m 36m R 59.8 0.4 107:12.30 php-cgi
3420 daemon 25 0 89760 47m 1056 R 50.1 0.3 143:38.59 nginx
15963 root 16 0 13008 1428 820 R 22.6 0.0 0:04.34 top
3428 root 15 0 411m 4300 1172 S 2.1 0.0 2:33.80 php-cgi
15965 root 18 0 99.6m 1584 908 S 0.8 0.0 0:00.03 crond
3743 root 15 0 74812 1164 584 S 0.3 0.0 0:00.83 crond
15964 root 18 0 99.6m 1468 800 D 0.3 0.0 0:00.01 crond
free信息:
total used free shared buffers cached
Mem: 16426616 16245028 181588 0 159248 12046536
-/+ buffers/cache: 4039244 12387372
Swap: 16386292 28980 16357312
sar -r
03:20:01 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
09:30:02 PM 268628 16157988 98.36 149268 11969424 16357312 28980 0.18 2644
09:40:02 PM 256492 16170124 98.44 150628 11980476 16357312 28980 0.18 2644
09:50:02 PM 244752 16181864 98.51 152092 11991476 16357312 28980 0.18 2644
10:00:02 PM 233296 16193320 98.58 153516 12002516 16357312 28980 0.18 2644
10:10:02 PM 219404 16207212 98.66 155032 12013568 16357312 28980 0.18 2644
10:20:02 PM 204912 16221704 98.75 156452 12024544 16357312 28980 0.18 2644
10:30:02 PM 193128 16233488 98.82 158008 12037200 16357312 28980 0.18 2644
Average: 588844 15837772 96.42 216781 11964159 16358029 28263 0.17 1420
sar -u
03:20:01 PM CPU %user %nice %system %iowait %steal %idle
03:30:01 PM all 8.37 0.00 0.86 0.23 0.00 90.54
03:40:01 PM all 8.11 0.00 0.79 0.17 0.00 90.93
03:50:01 PM all 7.22 0.00 0.81 0.23 0.00 91.74
04:00:01 PM all 7.60 0.00 0.84 0.16 0.00 91.41
04:10:01 PM all 7.38 0.00 0.82 0.26 0.00 91.53
04:20:01 PM all 7.12 0.00 0.81 0.27 0.00 91.80
04:30:01 PM all 7.10 0.00 0.84 0.42 0.00 91.64
04:40:01 PM all 7.38 0.00 0.83 0.23 0.00 91.55
04:50:01 PM all 7.20 0.00 0.82 0.17 0.00 91.80
05:00:01 PM all 7.54 0.00 0.83 0.19 0.00 91.44
05:10:01 PM all 6.72 0.00 0.76 0.24 0.00 92.28
05:20:01 PM all 6.98 0.00 0.79 0.13 0.00 92.09
05:30:02 PM all 7.18 0.00 0.80 0.16 0.00 91.87
05:40:01 PM all 7.07 0.00 0.77 0.15 0.00 92.01
05:50:01 PM all 7.75 0.00 0.88 0.17 0.00 91.20
06:00:01 PM all 7.46 0.00 0.85 0.25 0.00 91.43
06:10:01 PM all 7.58 0.00 0.88 0.19 0.00 91.34
06:20:01 PM all 7.81 0.00 0.88 0.28 0.00 91.03
06:30:01 PM all 7.90 0.00 0.87 0.22 0.00 91.00
06:40:01 PM all 7.20 0.00 0.86 0.22 0.00 91.72
06:50:01 PM all 7.34 0.00 0.90 0.25 0.00 91.51
07:00:01 PM all 7.94 0.00 0.96 0.23 0.00 90.87
07:10:01 PM all 8.14 0.00 0.97 0.29 0.00 90.59
07:20:01 PM all 8.23 0.00 0.97 0.41 0.00 90.39
07:30:01 PM all 8.12 0.00 0.97 0.30 0.00 90.61
07:40:01 PM all 7.77 0.00 0.90 0.30 0.00 91.03
07:50:01 PM all 7.97 0.00 0.92 0.17 0.00 90.94
08:00:01 PM all 8.44 0.00 1.03 0.43 0.00 90.10
08:10:01 PM all 7.90 0.00 0.95 0.51 0.00 90.64
08:20:01 PM all 8.31 0.00 1.03 0.18 0.00 90.48
08:30:01 PM all 8.19 0.00 0.98 0.29 0.00 90.54
08:40:01 PM all 8.88 0.00 1.06 0.27 0.00 89.79
08:50:01 PM all 11.45 0.00 1.39 0.35 0.00 86.81
09:00:01 PM all 11.67 0.00 1.44 0.48 0.00 86.41
09:10:02 PM all 0.92 0.00 91.23 0.49 0.00 7.36
09:20:02 PM all 0.01 0.00 99.88 0.00 0.00 0.11
09:30:02 PM all 0.01 0.00 99.42 0.00 0.00 0.57
09:40:02 PM all 0.01 0.00 98.05 0.00 0.00 1.94
09:50:02 PM all 0.01 0.00 99.31 0.00 0.00 0.68
10:00:02 PM all 0.01 0.00 99.83 0.00 0.00 0.15
10:10:02 PM all 0.01 0.00 99.05 0.00 0.00 0.94
10:20:02 PM all 0.01 0.00 99.69 0.00 0.00 0.30
10:30:02 PM all 0.01 0.00 99.75 0.00 0.00 0.24
Average: all 4.86 0.00 7.11 0.16 0.00 87.87
系统、nginx、php-fpm相关信息
系统:
cat /etc/issue
CentOS release 5.5 (Final)
Kernel
on an m
nginx:
/usr/local/nginx/sbin/nginx -V
nginx version: nginx/0.8.53
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-50)
TLS SNI support disabled
configure arguments: --user= --group= --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-http_sub_module --with-md5=/usr/lib --with-sha1=/usr/lib --with-http_gzip_static_module
php -v
PHP 5.2.17p1 with Suhosin-Patch 0.9.7 (cli) (built: Jan 7 2012 15:14:25)
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies
with eAccelerator v0.9.6.1, Copyright (c) 2004-2010 eAccelerator, by eAccelerator