自动failover
slave1:
stop slave io_thread
slave2
stop slave io_thread
server1:
create database sbtest;
/usr/local/bin/sysbench --test=/root/sysbench-0.5/sysbench/tests/db/oltp.lua --oltp-table-size=10000 --oltp-tables-count=1 --mysql-user=root --mysql-password=123 -- mysql-
host=10.24.220.232 --mysql-port=3306 --num-threads=4 --max-requests=0 --max-time=30 --report-interval=3 prepare
/usr/local/bin/sysbench --test=/root/sysbench-0.5/sysbench/tests/db/oltp.lua --oltp-table-size=10000 --oltp-tables-count=1 --mysql-user=root --mysql-password=123 -- mysql-
host=10.24.220.232 --mysql-port=3306 --num-threads=4 --max-requests=0 --max-time=30 --report-interval=3 run
slave2:
start slave io_thread
slave1
start slave io_thread
server1
service mysql.server stop
Sun May 15 20:42:46 2016 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun May 15 20:42:46 2016 - [info] Reading application default configuration from /etc/masterha/app1.conf.. Sun May 15 20:42:46 2016 - [info] Reading server configuration from /etc/masterha/app1.conf.. Sun May 15 20:49:44 2016 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun May 15 20:49:44 2016 - [info] Reading application default configuration from /etc/masterha/app1.conf.. Sun May 15 20:49:44 2016 - [info] Reading server configuration from /etc/masterha/app1.conf.. :3306): 0 Sun May 15 20:42:46 2016 - [debug] Connected to: slave1(10.24.220.70:3306), user=root Sun May 15 20:42:46 2016 - [debug] Number of slave worker threads on host slave1(10.24.220.70:3306): 0 Sun May 15 20:42:46 2016 - [debug] Connected to: slave2(10.169.214.33:3306), user=root Sun May 15 20:42:46 2016 - [debug] Number of slave worker threads on host slave2(10.169.214.33:3306): 0 Sun May 15 20:42:46 2016 - [debug] Comparing MySQL versions.. Sun May 15 20:42:46 2016 - [debug] Comparing MySQL versions done. Sun May 15 20:42:46 2016 - [debug] Connecting to servers done. Sun May 15 20:42:46 2016 - [info] GTID failover mode = 1 Sun May 15 20:42:46 2016 - [info] Dead Servers: Sun May 15 20:42:46 2016 - [info] Alive Servers: Sun May 15 20:42:46 2016 - [info] server1(10.24.220.232:3306) Sun May 15 20:42:46 2016 - [info] slave1(10.24.220.70:3306) Sun May 15 20:42:46 2016 - [info] slave2(10.169.214.33:3306) Sun May 15 20:42:46 2016 - [info] Alive Slaves: Sun May 15 20:42:46 2016 - [info] slave1(10.24.220.70:3306) Version=5.7.11-log (oldest major version between slaves) log-bin:enabled Sun May 15 20:42:46 2016 - [info] GTID ON Sun May 15 20:42:46 2016 - [debug] Relay log info repository: FILE Sun May 15 20:42:46 2016 - [info] Replicating from 10.24.220.232(10.24.220.232:3306) Sun May 15 20:42:46 2016 - [info] Primary candidate for the new Master (candidate_master is set) Sun May 15 20:42:46 2016 - [info] slave2(10.169.214.33:3306) Version=5.7.11-log (oldest major version between slaves) log-bin:enabled Sun May 15 20:42:46 2016 - [info] GTID ON Sun May 15 20:42:46 2016 - [debug] Relay log info repository: FILE Sun May 15 20:42:46 2016 - [info] Replicating from 10.24.220.232(10.24.220.232:3306) Sun May 15 20:42:46 2016 - [info] Not candidate for the new Master (no_master is set) Sun May 15 20:42:46 2016 - [info] Current Alive Master: server1(10.24.220.232:3306) Sun May 15 20:42:46 2016 - [info] Checking slave configurations.. Sun May 15 20:42:46 2016 - [info] Checking replication filtering settings.. Sun May 15 20:42:46 2016 - [info] binlog_do_db= , binlog_ignore_db= Sun May 15 20:42:46 2016 - [info] Replication filtering check ok. Sun May 15 20:42:46 2016 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking. Sun May 15 20:42:46 2016 - [info] Checking SSH publickey authentication settings on the current master.. Sun May 15 20:42:46 2016 - [debug] SSH connection test to server1, option -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o BatchMode=yes -o ConnectTimeout=5, timeout 5 Sun May 15 20:42:46 2016 - [info] HealthCheck: SSH to server1 is reachable. Sun May 15 20:42:46 2016 - [info] server1(10.24.220.232:3306) (current master) +--slave1(10.24.220.70:3306) +--slave2(10.169.214.33:3306) Sun May 15 20:42:46 2016 - [info] Checking master_ip_failover_script status: Sun May 15 20:42:46 2016 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=server1 --orig_master_ip=10.24.220.232 --orig_master_port=3306 IN SCRIPT TEST====/sbin/ifconfig eth0:88 down==/sbin/ifconfig eth0:88 10.24.220.100/24=== Checking the Status of the script.. OK Sun May 15 20:42:46 2016 - [info] OK. Sun May 15 20:42:46 2016 - [warning] shutdown_script is not defined. Sun May 15 20:42:46 2016 - [debug] Disconnected from server1(10.24.220.232:3306) Sun May 15 20:42:46 2016 - [debug] Disconnected from slave1(10.24.220.70:3306) Sun May 15 20:42:46 2016 - [debug] Disconnected from slave2(10.169.214.33:3306) Sun May 15 20:42:46 2016 - [debug] SSH check command: exit 0 Sun May 15 20:42:46 2016 - [info] Set master ping interval 3 seconds. Sun May 15 20:42:46 2016 - [info] Set secondary check script: /usr/bin/masterha_secondary_check -s 10.169.214.33 -s 10.24.220.70 --user=root --master_host=10.24.220.232 --master_port=3306 Sun May 15 20:42:46 2016 - [info] Starting ping health check on server1(10.24.220.232:3306).. Sun May 15 20:42:46 2016 - [debug] Connected on master. Sun May 15 20:42:46 2016 - [debug] Set short wait_timeout on master: 6 seconds Sun May 15 20:42:46 2016 - [debug] Trying to get advisory lock.. Sun May 15 20:42:46 2016 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond.. Sun May 15 20:49:34 2016 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away) Sun May 15 20:49:34 2016 - [info] Executing SSH check script: exit 0 Sun May 15 20:49:34 2016 - [debug] SSH connection test to server1, option -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o BatchMode=yes -o ConnectTimeout=5, timeout 5 Sun May 15 20:49:34 2016 - [info] Executing secondary network check script: /usr/bin/masterha_secondary_check -s 10.169.214.33 -s 10.24.220.70 --user=root --master_host=10.24.220.232 --master_port=3306 --user=root --master_host=server1 --master_ip=10.24.220.232 --master_port=3306 --master_user=root --master_password=971448976 --ping_type=SELECT Sun May 15 20:49:35 2016 - [info] HealthCheck: SSH to server1 is reachable. Monitoring server 10.169.214.33 is reachable, Master is not reachable from 10.169.214.33. OK. Monitoring server 10.24.220.70 is reachable, Master is not reachable from 10.24.220.70. OK. Sun May 15 20:49:37 2016 - [info] Master is not reachable from all other monitoring servers. Failover should start. Sun May 15 20:49:37 2016 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111) Sun May 15 20:49:37 2016 - [warning] Connection failed 2 time(s).. Sun May 15 20:49:40 2016 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111) Sun May 15 20:49:40 2016 - [warning] Connection failed 3 time(s).. Sun May 15 20:49:43 2016 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111) Sun May 15 20:49:43 2016 - [warning] Connection failed 4 time(s).. Sun May 15 20:49:43 2016 - [warning] Master is not reachable from health checker! Sun May 15 20:49:43 2016 - [warning] Master server1(10.24.220.232:3306) is not reachable! Sun May 15 20:49:43 2016 - [warning] SSH is reachable. Sun May 15 20:49:43 2016 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.conf again, and trying to connect to all servers to check server status.. Sun May 15 20:49:43 2016 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun May 15 20:49:43 2016 - [info] Reading application default configuration from /etc/masterha/app1.conf.. Sun May 15 20:49:43 2016 - [info] Reading server configuration from /etc/masterha/app1.conf.. Sun May 15 20:49:43 2016 - [debug] Skipping connecting to dead master server1(10.24.220.232:3306). Sun May 15 20:49:43 2016 - [debug] Connecting to servers.. Sun May 15 20:49:44 2016 - [debug] Connected to: slave1(10.24.220.70:3306), user=root Sun May 15 20:49:44 2016 - [debug] Number of slave worker threads on host slave1(10.24.220.70:3306): 0 Sun May 15 20:49:44 2016 - [debug] Connected to: slave2(10.169.214.33:3306), user=root Sun May 15 20:49:44 2016 - [debug] Number of slave worker threads on host slave2(10.169.214.33:3306): 0 Sun May 15 20:49:44 2016 - [debug] Comparing MySQL versions.. Sun May 15 20:49:44 2016 - [debug] Comparing MySQL versions done. Sun May 15 20:49:44 2016 - [debug] Connecting to servers done. Sun May 15 20:49:44 2016 - [info] GTID failover mode = 1 Sun May 15 20:49:44 2016 - [info] Dead Servers: Sun May 15 20:49:44 2016 - [info] server1(10.24.220.232:3306) Sun May 15 20:49:44 2016 - [info] Alive Servers: Sun May 15 20:49:44 2016 - [info] slave1(10.24.220.70:3306) Sun May 15 20:49:44 2016 - [info] slave2(10.169.214.33:3306) Sun May 15 20:49:44 2016 - [info] Alive Slaves: Sun May 15 20:49:44 2016 - [info] slave1(10.24.220.70:3306) Version=5.7.11-log (oldest major version between slaves) log-bin:enabled Sun May 15 20:49:44 2016 - [info] GTID ON Sun May 15 20:49:44 2016 - [debug] Relay log info repository: FILE Sun May 15 20:49:44 2016 - [info] Replicating from 10.24.220.232(10.24.220.232:3306) Sun May 15 20:49:44 2016 - [info] Primary candidate for the new Master (candidate_master is set) Sun May 15 20:49:44 2016 - [info] slave2(10.169.214.33:3306) Version=5.7.11-log (oldest major version between slaves) log-bin:enabled Sun May 15 20:49:44 2016 - [info] GTID ON Sun May 15 20:49:44 2016 - [debug] Relay log info repository: FILE Sun May 15 20:49:44 2016 - [info] Replicating from 10.24.220.232(10.24.220.232:3306) Sun May 15 20:49:44 2016 - [info] Not candidate for the new Master (no_master is set) Sun May 15 20:49:44 2016 - [info] Checking slave configurations.. Sun May 15 20:49:44 2016 - [info] Checking replication filtering settings.. Sun May 15 20:49:44 2016 - [info] Replication filtering check ok. Sun May 15 20:49:44 2016 - [info] Master is down! Sun May 15 20:49:44 2016 - [info] Terminating monitoring script. Sun May 15 20:49:44 2016 - [info] Got exit code 20 (Master dead). Sun May 15 20:49:44 2016 - [info] MHA::MasterFailover version 0.56. Sun May 15 20:49:44 2016 - [info] Starting master failover. Sun May 15 20:49:44 2016 - [info] Sun May 15 20:49:44 2016 - [info] * Phase 1: Configuration Check Phase.. Sun May 15 20:49:44 2016 - [info] Sun May 15 20:49:44 2016 - [debug] Skipping connecting to dead master server1. Sun May 15 20:49:44 2016 - [debug] Connecting to servers.. Sun May 15 20:49:44 2016 - [debug] Connected to: slave1(10.24.220.70:3306), user=root Sun May 15 20:49:44 2016 - [debug] Number of slave worker threads on host slave1(10.24.220.70:3306): 0 Sun May 15 20:49:44 2016 - [debug] Connected to: slave2(10.169.214.33:3306), user=root Sun May 15 20:49:44 2016 - [debug] Number of slave worker threads on host slave2(10.169.214.33:3306): 0 Sun May 15 20:49:44 2016 - [debug] Comparing MySQL versions.. Sun May 15 20:49:44 2016 - [debug] Comparing MySQL versions done. Sun May 15 20:49:44 2016 - [debug] Connecting to servers done. Sun May 15 20:49:44 2016 - [info] GTID failover mode = 1 Sun May 15 20:49:44 2016 - [info] Dead Servers: Sun May 15 20:49:44 2016 - [info] server1(10.24.220.232:3306) Sun May 15 20:49:44 2016 - [info] Checking master reachability via MySQL(double check)... Sun May 15 20:49:44 2016 - [info] ok. Sun May 15 20:49:44 2016 - [info] Alive Servers: Sun May 15 20:49:44 2016 - [info] slave1(10.24.220.70:3306) Sun May 15 20:49:44 2016 - [info] slave2(10.169.214.33:3306) Sun May 15 20:49:44 2016 - [info] Alive Slaves: Sun May 15 20:49:44 2016 - [info] slave1(10.24.220.70:3306) Version=5.7.11-log (oldest major version between slaves) log-bin:enabled Sun May 15 20:49:44 2016 - [info] GTID ON Sun May 15 20:49:44 2016 - [debug] Relay log info repository: FILE Sun May 15 20:49:44 2016 - [info] Replicating from 10.24.220.232(10.24.220.232:3306) Sun May 15 20:49:44 2016 - [info] Primary candidate for the new Master (candidate_master is set) Sun May 15 20:49:44 2016 - [info] slave2(10.169.214.33:3306) Version=5.7.11-log (oldest major version between slaves) log-bin:enabled Sun May 15 20:49:44 2016 - [info] GTID ON Sun May 15 20:49:44 2016 - [debug] Relay log info repository: FILE Sun May 15 20:49:44 2016 - [info] Replicating from 10.24.220.232(10.24.220.232:3306) Sun May 15 20:49:44 2016 - [info] Not candidate for the new Master (no_master is set) Sun May 15 20:49:44 2016 - [info] Starting GTID based failover. Sun May 15 20:49:44 2016 - [info] Sun May 15 20:49:44 2016 - [info] ** Phase 1: Configuration Check Phase completed. Sun May 15 20:49:44 2016 - [info] Sun May 15 20:49:44 2016 - [info] * Phase 2: Dead Master Shutdown Phase.. Sun May 15 20:49:44 2016 - [info] Sun May 15 20:49:44 2016 - [info] Forcing shutdown so that applications never connect to the current master.. Sun May 15 20:49:44 2016 - [info] Executing master IP deactivation script: Sun May 15 20:49:44 2016 - [info] /usr/local/bin/master_ip_failover --orig_master_host=server1 --orig_master_ip=10.24.220.232 --orig_master_port=3306 --command=stopssh --ssh_user=root Sun May 15 20:49:44 2016 - [debug] Stopping IO thread on slave2(10.169.214.33:3306).. Sun May 15 20:49:44 2016 - [debug] Stopping IO thread on slave1(10.24.220.70:3306).. IN SCRIPT TEST====/sbin/ifconfig eth0:88 down==/sbin/ifconfig eth0:88 10.24.220.100/24=== Disabling the VIP on old master: server1 Sun May 15 20:49:44 2016 - [debug] Stop IO thread on slave2(10.169.214.33:3306) done. Sun May 15 20:49:44 2016 - [debug] Stop IO thread on slave1(10.24.220.70:3306) done. SIOCSIFFLAGS: Cannot assign requested address Sun May 15 20:49:44 2016 - [info] done. Sun May 15 20:49:44 2016 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master. Sun May 15 20:49:44 2016 - [info] * Phase 2: Dead Master Shutdown Phase completed. Sun May 15 20:49:44 2016 - [info] Sun May 15 20:49:44 2016 - [info] * Phase 3: Master Recovery Phase.. Sun May 15 20:49:44 2016 - [info] Sun May 15 20:49:44 2016 - [info] * Phase 3.1: Getting Latest Slaves Phase.. Sun May 15 20:49:44 2016 - [info] Sun May 15 20:49:44 2016 - [debug] Fetching current slave status.. Sun May 15 20:49:44 2016 - [debug] Fetching current slave status done. Sun May 15 20:49:44 2016 - [info] The latest binary log file/position on all slaves is log.000001:8278273 Sun May 15 20:49:44 2016 - [info] Retrieved Gtid Set: 191f7a9f-ffa2-11e5-a825-00163e00242a:1-3821 Sun May 15 20:49:44 2016 - [info] Latest slaves (Slaves that received relay log files to the latest): Sun May 15 20:49:44 2016 - [info] slave1(10.24.220.70:3306) Version=5.7.11-log (oldest major version between slaves) log-bin:enabled Sun May 15 20:49:44 2016 - [info] GTID ON Sun May 15 20:49:44 2016 - [debug] Relay log info repository: FILE Sun May 15 20:49:44 2016 - [info] Replicating from 10.24.220.232(10.24.220.232:3306) Sun May 15 20:49:44 2016 - [info] Primary candidate for the new Master (candidate_master is set) Sun May 15 20:49:44 2016 - [info] slave2(10.169.214.33:3306) Version=5.7.11-log (oldest major version between slaves) log-bin:enabled Sun May 15 20:49:44 2016 - [info] GTID ON Sun May 15 20:49:44 2016 - [debug] Relay log info repository: FILE Sun May 15 20:49:44 2016 - [info] Replicating from 10.24.220.232(10.24.220.232:3306) Sun May 15 20:49:44 2016 - [info] Not candidate for the new Master (no_master is set) Sun May 15 20:49:44 2016 - [info] The oldest binary log file/position on all slaves is log.000001:8278273 Sun May 15 20:49:44 2016 - [info] Retrieved Gtid Set: 191f7a9f-ffa2-11e5-a825-00163e00242a:1-3821 Sun May 15 20:49:44 2016 - [info] Oldest slaves: Sun May 15 20:49:44 2016 - [info] slave1(10.24.220.70:3306) Version=5.7.11-log (oldest major version between slaves) log-bin:enabled Sun May 15 20:49:44 2016 - [info] GTID ON Sun May 15 20:49:44 2016 - [debug] Relay log info repository: FILE Sun May 15 20:49:44 2016 - [info] Replicating from 10.24.220.232(10.24.220.232:3306) Sun May 15 20:49:44 2016 - [info] Primary candidate for the new Master (candidate_master is set) Sun May 15 20:49:44 2016 - [info] slave2(10.169.214.33:3306) Version=5.7.11-log (oldest major version between slaves) log-bin:enabled Sun May 15 20:49:44 2016 - [info] GTID ON Sun May 15 20:49:44 2016 - [debug] Relay log info repository: FILE Sun May 15 20:49:44 2016 - [info] Replicating from 10.24.220.232(10.24.220.232:3306) Sun May 15 20:49:44 2016 - [info] Not candidate for the new Master (no_master is set) Sun May 15 20:49:44 2016 - [info] Sun May 15 20:49:44 2016 - [info] * Phase 3.3: Determining New Master Phase.. Sun May 15 20:49:44 2016 - [info] Sun May 15 20:49:44 2016 - [info] Searching new master from slaves.. Sun May 15 20:49:44 2016 - [info] Candidate masters from the configuration file: Sun May 15 20:49:44 2016 - [info] slave1(10.24.220.70:3306) Version=5.7.11-log (oldest major version between slaves) log-bin:enabled Sun May 15 20:49:44 2016 - [info] GTID ON Sun May 15 20:49:44 2016 - [debug] Relay log info repository: FILE Sun May 15 20:49:44 2016 - [info] Replicating from 10.24.220.232(10.24.220.232:3306) Sun May 15 20:49:44 2016 - [info] Primary candidate for the new Master (candidate_master is set) Sun May 15 20:49:44 2016 - [info] Non-candidate masters: Sun May 15 20:49:44 2016 - [info] slave2(10.169.214.33:3306) Version=5.7.11-log (oldest major version between slaves) log-bin:enabled Sun May 15 20:49:44 2016 - [info] GTID ON Sun May 15 20:49:44 2016 - [debug] Relay log info repository: FILE Sun May 15 20:49:44 2016 - [info] Replicating from 10.24.220.232(10.24.220.232:3306) Sun May 15 20:49:44 2016 - [info] Not candidate for the new Master (no_master is set) Sun May 15 20:49:44 2016 - [info] Searching from candidate_master slaves which have received the latest relay log events.. Sun May 15 20:49:44 2016 - [info] New master is slave1(10.24.220.70:3306) Sun May 15 20:49:44 2016 - [info] Starting master failover.. Sun May 15 20:49:44 2016 - [info] From: server1(10.24.220.232:3306) (current master) +--slave1(10.24.220.70:3306) +--slave2(10.169.214.33:3306) To: slave1(10.24.220.70:3306) (new master) +--slave2(10.169.214.33:3306) Sun May 15 20:49:44 2016 - [info] Sun May 15 20:49:44 2016 - [info] * Phase 3.3: New Master Recovery Phase.. Sun May 15 20:49:44 2016 - [info] Sun May 15 20:49:44 2016 - [info] Waiting all logs to be applied.. Sun May 15 20:49:44 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:49:45 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:49:46 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:49:47 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:49:48 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:49:49 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:49:50 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:49:51 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:49:52 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:49:53 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:49:54 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:49:55 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:49:56 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:49:57 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:49:58 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:49:59 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:50:00 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:50:01 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:50:02 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:50:03 2016 - [debug] Sql Thread Done: 0, Worker Thread done: 0, Ended workers: 0 Sun May 15 20:50:04 2016 - [info] done. Sun May 15 20:50:04 2016 - [debug] Stopping slave IO/SQL thread on slave1(10.24.220.70:3306).. Sun May 15 20:50:04 2016 - [debug] done. Sun May 15 20:50:04 2016 - [info] Getting new master's binlog name and position.. Sun May 15 20:50:04 2016 - [info] log.000002:8235609 Sun May 15 20:50:04 2016 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='slave1 or 10.24.220.70', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx'; Sun May 15 20:50:04 2016 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: log.000002, 8235609, 191f7a9f-ffa2-11e5-a825-00163e00242a:1-3821 Sun May 15 20:50:04 2016 - [info] Executing master IP activate script: Sun May 15 20:50:04 2016 - [info] /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=server1 --orig_master_ip=10.24.220.232 --orig_master_port=3306 --new_master_host=slave1 --new_master_ip=10.24.220.70 --new_master_port=3306 --new_master_user='root' --new_master_password='971448976' Unknown option: new_master_user Unknown option: new_master_password IN SCRIPT TEST====/sbin/ifconfig eth0:88 down==/sbin/ifconfig eth0:88 10.24.220.100/24=== Enabling the VIP - 10.24.220.100/24 on the new master - slave1 Sun May 15 20:50:05 2016 - [info] OK. Sun May 15 20:50:05 2016 - [info] Setting read_only=0 on slave1(10.24.220.70:3306).. Sun May 15 20:50:05 2016 - [info] ok. Sun May 15 20:50:05 2016 - [info] ** Finished master recovery successfully. Sun May 15 20:50:05 2016 - [info] * Phase 3: Master Recovery Phase completed. Sun May 15 20:50:05 2016 - [info] Sun May 15 20:50:05 2016 - [info] * Phase 4: Slaves Recovery Phase.. Sun May 15 20:50:05 2016 - [info] Sun May 15 20:50:05 2016 - [info] Sun May 15 20:50:05 2016 - [info] * Phase 4.1: Starting Slaves in parallel.. Sun May 15 20:50:05 2016 - [info] Sun May 15 20:50:05 2016 - [info] -- Slave recovery on host slave2(10.169.214.33:3306) started, pid: 6629. Check tmp log /var/log/masterha/app1/slave2_3306_20160515204944.log if it takes time.. Sun May 15 20:50:06 2016 - [info] Sun May 15 20:50:06 2016 - [info] Log messages from slave2 ... Sun May 15 20:50:06 2016 - [info] Sun May 15 20:50:05 2016 - [info] Resetting slave slave2(10.169.214.33:3306) and starting replication from the new master slave1(10.24.220.70:3306).. Sun May 15 20:50:05 2016 - [debug] Stopping slave IO/SQL thread on slave2(10.169.214.33:3306).. Sun May 15 20:50:05 2016 - [debug] done. Sun May 15 20:50:05 2016 - [info] Executed CHANGE MASTER. Sun May 15 20:50:05 2016 - [debug] Starting slave IO/SQL thread on slave2(10.169.214.33:3306).. Sun May 15 20:50:06 2016 - [debug] done. Sun May 15 20:50:06 2016 - [info] Slave started. Sun May 15 20:50:06 2016 - [info] gtid_wait(191f7a9f-ffa2-11e5-a825-00163e00242a:1-3821) completed on slave2(10.169.214.33:3306). Executed 24 events. Sun May 15 20:50:06 2016 - [info] End of log messages from slave2. Sun May 15 20:50:06 2016 - [info] -- Slave on host slave2(10.169.214.33:3306) started. Sun May 15 20:50:06 2016 - [info] All new slave servers recovered successfully. Sun May 15 20:50:06 2016 - [info] Sun May 15 20:50:06 2016 - [info] * Phase 5: New master cleanup phase.. Sun May 15 20:50:06 2016 - [info] Sun May 15 20:50:06 2016 - [info] Resetting slave info on the new master.. Sun May 15 20:50:06 2016 - [debug] Clearing slave info.. Sun May 15 20:50:06 2016 - [debug] Stopping slave IO/SQL thread on slave1(10.24.220.70:3306).. Sun May 15 20:50:06 2016 - [debug] done. Sun May 15 20:50:06 2016 - [debug] SHOW SLAVE STATUS shows new master does not replicate from anywhere. OK. Sun May 15 20:50:06 2016 - [info] slave1: Resetting slave info succeeded. Sun May 15 20:50:06 2016 - [info] Master failover to slave1(10.24.220.70:3306) completed successfully. Sun May 15 20:50:06 2016 - [info] Deleted server1 entry from /etc/masterha/app1.conf . Sun May 15 20:50:06 2016 - [debug] Disconnected from slave1(10.24.220.70:3306) Sun May 15 20:50:06 2016 - [debug] Disconnected from slave2(10.169.214.33:3306) Sun May 15 20:50:06 2016 - [info] ----- Failover Report ----- app1: MySQL Master failover server1(10.24.220.232:3306) to slave1(10.24.220.70:3306) succeeded Master server1(10.24.220.232:3306) is down! Check MHA Manager logs at monitor:/var/log/masterha/app1/manager.log for details. Started automated(non-interactive) failover. Invalidated master IP address on server1(10.24.220.232:3306) Selected slave1(10.24.220.70:3306) as a new master. slave1(10.24.220.70:3306): OK: Applying all logs succeeded. slave1(10.24.220.70:3306): OK: Activated master IP address. slave2(10.169.214.33:3306): OK: Slave started, replicating from slave1(10.24.220.70:3306) slave1(10.24.220.70:3306): Resetting slave info succeeded. Master failover to slave1(10.24.220.70:3306) completed successfully.