昨天晚上第十款做数据库迁移，数据库版本：Version: '5.1.61'，数据量27G左右
message报错信息如下：
Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480438] TsysProxy invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480444] Pid: 32626, comm: TsysProxy Not tainted 2.6.32.57-tlinux_xenU-1.1.rc10-default #1 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480447] Call Trace: Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480456] [<ffffffff810904ab>] oom_kill_process.clone.0+0xab/0x270 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480459] [<ffffffff81090788>] __out_of_memory+0x118/0x180 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480462] [<ffffffff81090872>] out_of_memory+0x82/0x170 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480467] [<ffffffff81098b40>] ? isolate_pages_global+0x0/0x230 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480470] [<ffffffff81094f09>] __alloc_pages_nodemask+0x629/0x640 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480476] [<ffffffff810bf376>] alloc_pages_current+0x76/0xd0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480481] [<ffffffff8108db8b>] __page_cache_alloc+0xb/0x10 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480484] [<ffffffff810969e1>] __do_page_cache_readahead+0xc1/0x210 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480487] [<ffffffff81096b4c>] ra_submit+0x1c/0x20 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480490] [<ffffffff8108e382>] filemap_fault+0x3f2/0x440 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480493] [<ffffffff810a480f>] __do_fault+0x4f/0x4e0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480500] [<ffffffff817a7266>] ? schedule_hrtimeout_range+0xb6/0x130 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480503] [<ffffffff810a6de8>] handle_mm_fault+0x1b8/0x9e0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480507] [<ffffffff81048e50>] ? default_wake_function+0x0/0x10 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480512] [<ffffffff810e3954>] ? mntput_no_expire+0x24/0xf0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480517] [<ffffffff81034460>] do_page_fault+0x110/0x2e0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480520] [<ffffffff817a87d5>] page_fault+0x25/0x30 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480521] Mem-Info: Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480523] Node 0 DMA per-cpu: Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480525] CPU 0: hi: 0, btch: 1 usd: 0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480527] CPU 1: hi: 0, btch: 1 usd: 0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480529] CPU 2: hi: 0, btch: 1 usd: 0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480530] CPU 3: hi: 0, btch: 1 usd: 0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480532] Node 0 DMA32 per-cpu: Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480534] CPU 0: hi: 186, btch: 31 usd: 0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480535] CPU 1: hi: 186, btch: 31 usd: 0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480537] CPU 2: hi: 186, btch: 31 usd: 0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480539] CPU 3: hi: 186, btch: 31 usd: 0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480540] Node 0 Normal per-cpu: Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480542] CPU 0: hi: 186, btch: 31 usd: 0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480544] CPU 1: hi: 186, btch: 31 usd: 0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480545] CPU 2: hi: 186, btch: 31 usd: 0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480547] CPU 3: hi: 186, btch: 31 usd: 0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480551] active_anon:6920538 inactive_anon:552717 isolated_anon:0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480552] active_file:142 inactive_file:214 isolated_file:32 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480553] unevictable:0 dirty:0 writeback:0 unstable:0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480554] free:35240 slab_reclaimable:3770 slab_unreclaimable:3676 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480555] mapped:2656 shmem:2366 pagetables:16355 bounce:0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480557] Node 0 DMA free:15836kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15260kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480565] lowmem_reserve[]: 0 3768 29570 29570 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480569] Node 0 DMA32 free:105924kB min:2804kB low:3504kB high:4204kB active_anon:2740028kB inactive_anon:547884kB active_file:56kB inactive_file:92kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3858656kB mlocked:0kB dirty:0kB writeback:0kB mapped:1356kB shmem:1068kB slab_reclaimable:1848kB slab_unreclaimable:2584kB kernel_stack:1080kB pagetables:7140kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480578] lowmem_reserve[]: 0 0 25802 25802 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480580] Node 0 Normal free:19200kB min:19200kB low:24000kB high:28800kB active_anon:24942124kB inactive_anon:1662984kB active_file:512kB inactive_file:764kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:26421600kB mlocked:0kB dirty:0kB writeback:0kB mapped:9268kB shmem:8396kB slab_reclaimable:13232kB slab_unreclaimable:12120kB kernel_stack:1424kB pagetables:58280kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:416 all_unreclaimable? no Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480589] lowmem_reserve[]: 0 0 0 0 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480592] Node 0 DMA: 3*4kB 2*8kB 2*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15836kB Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480599] Node 0 DMA32: 581*4kB 485*8kB 423*16kB 361*32kB 291*64kB 202*128kB 78*256kB 22*512kB 2*1024kB 0*2048kB 1*4096kB = 106380kB Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480606] Node 0 Normal: 3841*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 19460kB Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480613] 42760 total pagecache pages Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480615] 39775 pages in swap cache Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480617] Swap cache stats: add 577057, delete 537282, find 6631146/6638910 Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480618] Free swap = 0kB Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.480619] Total swap = 2097144kB Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.567838] 7680000 pages RAM Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.567841] 126998 pages reserved Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.567842] 14402 pages shared Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.567843] 7510002 pages non-shared Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.567848] Out of memory: kill process 28092 (mysqld_safe) score 3710915 or a child Mar 27 03:44:09 VM_163_210_tlinux kernel: [18465684.567851] Killed process 28250 (mysqld) vsz:34419948kB, anon-rss:29649368kB, file-rss:460kB Mar 27 04:40:31 VM_163_210_tlinux kernel: [18469067.168641] TCP: Possible SYN flooding on port 3306. Sending cookies. Check SNMP counters. Mar 27 06:35:01 VM_163_210_tlinux nslcd[1762]: [5c31df] ldap_result() failed: Can't contact LDAP server Mar 27 06:35:01 VM_163_210_tlinux nslcd[1762]: [5c31df] ldap_abandon() failed to abandon search: Other (e.g., implementation specific) error
mysql error log 如下：
Version: '5.1.61' socket: '/tmp/mysql.sock' port: 3306 Source distribution 150327 03:44:11 mysqld_safe Number of processes running now: 0 150327 03:44:11 mysqld_safe mysqld restarted
参考stack-flow上面的分析：http://stackoverflow.com/questions/22522307/mysqld-safe-mysqld-restarted-automatically
死循环导致的，究其原因可能是因为innodb_buffer_size配置有问题，系统硬件内存是30G，buffer_size配置的是28G，建议rein下次导数据的时候先把buffer_size调低，导完之后再调高。