故障现象
MySQL slave status详情
mysql> show slave statusG
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.10.10.101
Master_User: root
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000010
Read_Master_Log_Pos: 143861468
Relay_Log_File: slave-relay-bin.000525
Relay_Log_Pos: 128835579
Relay_Master_Log_File: mysql-bin.000010
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table: mysql.%,information_schema.%,test.%
Last_Errno: 0
Last_Error: Could not parse relay log event entry. The possck this by running 'mysqlbinlog' on the binary log), the slave's relay log iay log), a network problem, or a bug in the master's or slave's MySQL code. you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this
Skip_Counter: 0
Exec_Master_Log_Pos: 128835442
Relay_Log_Space: 143862393
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
1 row in set (0.00 sec)
mysql>
MySQL 错误日志中显示了问题的根源,磁盘空间满.
thatsit-mysql:/var/lib/mysql # grep -i disk mysqld.log-20160114 /usr/sbin/mysqld: Disk is full writing './mysql-bin.000319' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space) /usr/sbin/mysqld: Disk is full writing './mysql-bin.000320' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space) thatsit-mysql:/var/lib/mysql #
MySQL 错误日志中同样还记录了binlog执行截至的位置点.
thatsit-mysql:/var/lib/mysql # grep ERROR mysqld.log-20160114|grep stopped|tail -1
160114 13:16:16 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql-bin.000010' position 128835442
thatsit-mysql:/var/lib/mysql #
修复
① 清理MySQL datadir所在的磁盘
② 恢复MySQL SLAVE (STOP SLAVE → CHANGE MASTER → START SLAVE)
※: 位置点为上面从SLAVE STATUS的"Exec_Master_Log_Pos"或者错误日志中获取的.
mysql> stop slave;
Query OK, 0 rows affected (0.00 sec)
mysql>
mysql> CHANGE MASTER TO
-> MASTER_HOST='10.10.10.101',
-> MASTER_PORT=3306,
-> MASTER_USER='REP_USER',
-> MASTER_PASSWORD='REP_PASSWORD',
-> MASTER_LOG_FILE='mysql-bin.000010',
-> MASTER_LOG_POS=128835442;
Query OK, 0 rows affected (0.41 sec)
mysql> start slave;
Query OK, 0 rows affected (0.00 sec)
mysql>
操作后状态确认
mysql> show slave statusG
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.10.10.101
Master_User: root
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000010
Read_Master_Log_Pos: 144444705
Relay_Log_File: slave-relay-bin.000002
Relay_Log_Pos: 38259
Relay_Master_Log_File: mysql-bin.000010
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table: mysql.%,information_schema.%,test.%
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 128873466
Relay_Log_Space: 15609498
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 99299
1 row in set (0.00 sec)
mysql>
此故障至此修复完毕.