本案例转载自李大玉老师分享
Ⅰ、问题背景
探活脚本连续8次探测,判断主库异常,触发切换(判断备机是否有延迟,kill原主,VIP飘到备机,设置新主可写)
切换后,业务还是异常,SQL查询没返回,DB连接数耗完,为了恢复业务,重启新主后业务恢复
两个问题
- 主库为什么会切换
- 切换后新主库为什么还是不可用
Ⅱ、问题排查
1、根据系统运行时间、系统日志以及服务器带外日志可排除服务器和数据库OOM
2、从每分钟的线程快照中发现故障时大量线程处于sending data和statistics状态,但前一分钟快照中未看到任何阻塞与锁等待
3、被阻塞的sql都是基于主键或者业务索引访问,理论上没问题,提取sql在从库执行一遍,很快,且当时没有产生慢日志,固排除sql执行效率慢导致阻塞
4、statistics是为sql生成执行计划的,会触发表的统计操作,而统计操作需要对表中page进行采样,会触发io,分析当时磁盘iops、吞吐量、cpu负载等,和前一天基本吻合,排除系统负载导致性能下降
思路中断~~~
复盘,开发反映业务切到新db,业务各个接口耗时变大,询问是否新库服务器性能不如老库
会后上机器发现实例上存在20个处于user sleep状态的sql,大概模型都是where id = '+(select 'rbzd' where 6910=6910 and sleep(300)+)',比较可疑,因为开发不会在程序中调用sleep函数
重点分析此sql
此sql不占用系统资源,但是写法可疑,类似sql注入
经查,sleep操作和innodb_thread_concurrency参数互斥,这样每秒只能处理24个sql
换言之当有24个线程进入引擎并处于sleep状态的话,其他线程是不能进入innodb引擎层,这里的24是和线上MySQL参数innodb_thread_concurrency被设置为24有关
Ⅲ、模拟故障
3.1 准备数据
CREATE TABLE `test` (
`id` int(11) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
insert into test select 1;
3.2 开24个线程访问此表
提前设置innodb_thread_concurrency设置为24模拟线上情况
[root@VM_42_63_centos ~]# for i in `seq 1 24`; do nohup mysql -S /tmp/mysql.sock3306 -e "select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+''" & done
3.3 观察一波
执行上一步脚本后,再开一个session访问test表会发现线程hang住,没返回,状态为sending data,如下:
(root@localhost) [test]> show processlist;
+----+------+-----------------+------+---------+------+--------------+----------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+------+-----------------+------+---------+------+--------------+----------------------------------------------------------------------------------------+
| 2 | root | localhost:38204 | NULL | Sleep | 0 | | NULL |
| 3 | root | localhost | test | Query | 0 | starting | show processlist |
| 4 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 5 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 6 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 7 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 8 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 9 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 10 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 11 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 12 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 13 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 14 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 15 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 16 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 17 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 18 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 19 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 20 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 21 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 22 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 23 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 24 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 25 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 26 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 27 | root | localhost | NULL | Query | 23 | User sleep | select id from test.test where id=''+(select 'rbzd' where 6910=6910 and sleep(300))+'' |
| 28 | root | localhost | NULL | Query | 11 | Sending data | select id from test.test |
+----+------+-----------------+------+---------+------+--------------+----------------------------------------------------------------------------------------+
27 rows in set (0.00 sec)
Ⅳ、故障回顾
1、sql注入,user sleep状态的sql累计到24条之后,其他sql就不能进入innodb进行操作,包括高可用探测程序,由于线程快照中过滤了sleep,导致没抓到注入的sql,加大了后续排查难度
2、主库的存活探测程序检查失败(探测方式为update一个innodb表),连续8次失败后,ha认为实例异常,则kill主,触发切换流程
3、切换到主库后,注入还在继续,所以同样的故障在新主上重演
4、重启主库后,仅有20个注入进入innodb且一直为user sleep,由于没达到24个触发阈值,所以业务表现正常,只是性能不及老主库,原因就是已经有20个线程在innodb层一直没退出,kill掉这些线程,业务恢复正常
Ⅴ、问题解决
- 由于线程快照过滤了sleep关键字,导致排查过程较长,后续要对user sleep状态线程保留在线程快照中
- 开发优化程序,防止sql注入