TCP的连接状态对于我们web服务器来说是至关重要的，尤其是并发量ESTAB；或者是syn_recv值，假如这个值比较大的话我们可以认为是不是受到了攻击，或是是time_wait值比较高的话，我们要考虑看我们内核是否需要调优，太高的time_wait值的话会占用太多端口，要是端口少的话后果不堪设想
一、TCP状态介绍
1、man netstat查看TCP的各种状态信息描述：
LISTEN -侦听来自远方TCP端口的连接请求；
SYN-SENT -在发送连接请求后等待匹配的连接请求；
SYN-RECEIVED -在收到和发送一个连接请求后等待对连接请求的确认；
ESTABLISHED -代表一个打开的连接，数据可以传送给用户；
FIN-WAIT-1 -等待远程TCP的连接中断请求，或先前的连接中断请求的确认；
FIN-WAIT-2 -从远程TCP等待连接中断请求；
CLOSE-WAIT -等待从本地用户发来的连接中断请求；
CLOSING -等待远程TCP对连接中断的确认；
LAST-ACK -等待原来发向远程TCP的连接中断请求的确认；
TIME-WAIT -等待足够的时间以确保远程TCP接收到连接中断请求的确认；
CLOSED -没有任何连接状态；
2、监控原理
[root@Node1 ~]# /bin/netstat -an|awk '/^tcp/{++S[$NF]}END{for(a in S) print a,S[a]}' //通过netstat获取相关值 LISTEN 10 ESTABLISHED 1 TIME_WAIT 178 [root@Node1 ~]#
二、监控脚本编写
1、编写脚本，放于 /etc/zabbix/zabbix_agentd.d/目录下
1 [root@Node1 zabbix_agentd.d]# cat tcp_status.sh 2 #!/bin/bash 3 #This script is used to get tcp and udp connetion status 4 #tcp status 5 metric=$1 6 tmp_file=/tmp/tcp_status.txt 7 /bin/netstat -an|awk '/^tcp/{++S[$NF]}END{for(a in S) print a,S[a]}' > $tmp_file 8 case $metric in 9 closed) 10 output=$(awk '/CLOSED/{print $2}' $tmp_file) 11 if [ "$output" == "" ];then 12 echo 0 13 else 14 echo $output 15 fi 16 ;; 17 listen) 18 output=$(awk '/LISTEN/{print $2}' $tmp_file) 19 if [ "$output" == "" ];then 20 echo 0 21 else 22 echo $output 23 fi 24 ;; 25 synrecv) 26 output=$(awk '/SYN_RECV/{print $2}' $tmp_file) 27 if [ "$output" == "" ];then 28 echo 0 29 else 30 echo $output 31 fi 32 ;; 33 synsent) 34 output=$(awk '/SYN_SENT/{print $2}' $tmp_file) 35 if [ "$output" == "" ];then 36 echo 0 37 else 38 echo $output 39 fi 40 ;; 41 established) 42 output=$(awk '/ESTABLISHED/{print $2}' $tmp_file) 43 if [ "$output" == "" ];then 44 echo 0 45 else 46 echo $output 47 fi 48 ;; 49 timewait) 50 output=$(awk '/TIME_WAIT/{print $2}' $tmp_file) 51 if [ "$output" == "" ];then 52 echo 0 53 else 54 echo $output 55 fi 56 ;; 57 closing) 58 output=$(awk '/CLOSING/{print $2}' $tmp_file) 59 if [ "$output" == "" ];then 60 echo 0 61 else 62 echo $output 63 fi 64 ;; 65 closewait) 66 output=$(awk '/CLOSE_WAIT/{print $2}' $tmp_file) 67 if [ "$output" == "" ];then 68 echo 0 69 else 70 echo $output 71 fi 72 ;; 73 lastack) 74 output=$(awk '/LAST_ACK/{print $2}' $tmp_file) 75 if [ "$output" == "" ];then 76 echo 0 77 else 78 echo $output 79 fi 80 ;; 81 finwait1) 82 output=$(awk '/FIN_WAIT1/{print $2}' $tmp_file) 83 if [ "$output" == "" ];then 84 echo 0 85 else 86 echo $output 87 fi 88 ;; 89 finwait2) 90 output=$(awk '/FIN_WAIT2/{print $2}' $tmp_file) 91 if [ "$output" == "" ];then 92 echo 0 93 else 94 echo $output 95 fi 96 ;; 97 *) 98 echo -e "\e[033mUsage: sh $0 [closed|closing|closewait|synrecv|synsent|finwait1|finwait2|listen|established|lastack|timewait]\e[0m" 99 esac
赋予脚本执行权限：
[root@Node1 ~]# chmod a+x /etc/zabbix/zabbix_agentd.d/tcp_status.sh //赋予执行权限 [root@Node1 ~]# ll /etc/zabbix/zabbix_agentd.d/tcp_status.sh -rwxr-xr-x 1 root root 2502 Jan 18 09:48 /etc/zabbix/zabbix_agentd.d/tcp_status.sh [root@Node1 ~]#
2、添加zabbix配置文件，放于 /etc/zabbix/zabbix_agentd.d/目录下（agent的配置文件 /etc/zabbix/zabbix_agentd.conf 中定义了其他key的包含目录）创建配置文件tcp_status.conf
[root@Node1 ~]# cat /etc/zabbix/zabbix_agentd.d/tcp_status.conf UserParameter=tcp.status[*],/etc/zabbix/zabbix_agentd.d/tcp_status.sh "$1" //脚本路径 [root@Node1 ~]#
3、确保配置Agent配置文件开启自定义参数UnsafeUserParameters=1
[root@Node1 ~]# grep -n "^[a-Z]" /etc/zabbix/zabbix_agentd.conf 13:PidFile=/var/run/zabbix/zabbix_agentd.pid 32:LogFile=/var/log/zabbix/zabbix_agentd.log 43:LogFileSize=0 57:DebugLevel=3 97:Server=172.17.21.208 138:ServerActive=172.17.21.208 149:Hostname=Node1.contoso.com 267:Include=/etc/zabbix/zabbix_agentd.d/*.conf 286:UnsafeUserParameters=1 //1代表允许，0代表关闭
4、重启zabbix-agent服务
[root@Node1 ~]# systemctl restart zabbix-agent.service
备注：因为脚本是把tcp的一些信息存放在/tmp/下，为了zabbix可以读取到我们设置zabbix可以读的权限，确保属主与属组都为zabbix即可
[root@Node1 ~]# chown zabbix.zabbix /tmp/tcp_status.txt //改变属主与属主 [root@Node1 ~]# ll /tmp/tcp_status.txt -rw-rw-r-- 1 zabbix zabbix 38 Jan 18 11:32 /tmp/tcp_status.txt
5、在zabbix servere服务器上测试,是否能正常获取数据
[root@Node3 ~]# zabbix_get -s 172.17.21.206 -p 10050 -k "tcp.status[listen]" 0 [root@Node3 ~]# zabbix_get -s 172.17.21.206 -p 10050 -k "tcp.status[timewait]" 183 [root@Node3 ~]# zabbix_get -s 172.17.21.206 -p 10050 -k "tcp.status[established]" 2 [root@Node3 ~]#
三、WEB界面配置
1、导入事先准备好的TCP Connection Status模板，Configuration-->Templates-->Import，选择相应的模板，点击Import即可
2、关联模板，Configuration-->Hosts-->Node1--Templates，选择模板,点击Updtae即可
3、查看TCP状态：
Monitoring-->Graphs-->Host(Node1)-->Grahp(TCP Status)
附模板：Template_TCP_Status.xml
<?xml version="1.0" encoding="UTF-8"?> <zabbix_export> <version>3.4</version> <date>2018-01-18T11:10:22Z</date> <groups> <group> <name>Templates</name> </group> </groups> <templates> <template> <template>Template TCP Connection Status</template> <name>Template TCP Connection Status</name> <description/> <groups> <group> <name>Templates</name> </group> </groups> <applications> <application> <name>TCP Status</name> </application> </applications> <items> <item> <name>CLOSED</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[closed]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>CLOSE_WAIT</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[closewait]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>CLOSING</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[closing]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>ESTABLISHED</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[established]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>FIN_WAIT1</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[finwait1]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>FIN_WAIT2</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[finwait2]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>LAST_ACK</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[lastack]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>LISTEN</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[listen]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>SYN_RECV</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[synrecv]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>SYN_SENT</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[synsent]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> <item> <name>TIME_WAIT</name> <type>0</type> <snmp_community/> <snmp_oid/> <key>tcp.status[timewait]</key> <delay>60s</delay> <history>90d</history> <trends>365d</trends> <status>0</status> <value_type>3</value_type> <allowed_hosts/> <units/> <snmpv3_contextname/> <snmpv3_securityname/> <snmpv3_securitylevel>0</snmpv3_securitylevel> <snmpv3_authprotocol>0</snmpv3_authprotocol> <snmpv3_authpassphrase/> <snmpv3_privprotocol>0</snmpv3_privprotocol> <snmpv3_privpassphrase/> <params/> <ipmi_sensor/> <authtype>0</authtype> <username/> <password/> <publickey/> <privatekey/> <port/> <description/> <inventory_link>0</inventory_link> <applications> <application> <name>TCP Status</name> </application> </applications> <valuemap/> <logtimefmt/> <preprocessing/> <jmx_endpoint/> <master_item/> </item> </items> <discovery_rules/> <httptests/> <macros/> <templates/> <screens/> </template> </templates> <triggers> <trigger> <expression>{Template TCP Connection Status:tcp.status[listen].last()}>500</expression> <recovery_mode>0</recovery_mode> <recovery_expression/> <name>There are too many TCP LISTEN status</name> <correlation_mode>0</correlation_mode> <correlation_tag/> <url/> <status>0</status> <priority>4</priority> <description/> <type>0</type> <manual_close>0</manual_close> <dependencies/> <tags/> </trigger> <trigger> <expression>{Template TCP Connection Status:tcp.status[timewait].last()}>10000</expression> <recovery_mode>0</recovery_mode> <recovery_expression/> <name>There are too many TCP TIME_WAIT status</name> <correlation_mode>0</correlation_mode> <correlation_tag/> <url/> <status>0</status> <priority>4</priority> <description/> <type>0</type> <manual_close>0</manual_close> <dependencies/> <tags/> </trigger> </triggers> <graphs> <graph> <name>TCP Status</name> <width>900</width> <height>200</height> <yaxismin>0.0000</yaxismin> <yaxismax>100.0000</yaxismax> <show_work_period>1</show_work_period> <show_triggers>1</show_triggers> <type>0</type> <show_legend>1</show_legend> <show_3d>0</show_3d> <percent_left>0.0000</percent_left> <percent_right>0.0000</percent_right> <ymin_type_1>0</ymin_type_1> <ymax_type_1>0</ymax_type_1> <ymin_item_1>0</ymin_item_1> <ymax_item_1>0</ymax_item_1> <graph_items> <graph_item> <sortorder>0</sortorder> <drawtype>0</drawtype> <color>C80000</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[closed]</key> </item> </graph_item> <graph_item> <sortorder>1</sortorder> <drawtype>0</drawtype> <color>00C800</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[closewait]</key> </item> </graph_item> <graph_item> <sortorder>2</sortorder> <drawtype>0</drawtype> <color>0000C8</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[closing]</key> </item> </graph_item> <graph_item> <sortorder>3</sortorder> <drawtype>0</drawtype> <color>C800C8</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[established]</key> </item> </graph_item> <graph_item> <sortorder>4</sortorder> <drawtype>0</drawtype> <color>00C8C8</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[finwait1]</key> </item> </graph_item> <graph_item> <sortorder>5</sortorder> <drawtype>0</drawtype> <color>C8C800</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[finwait2]</key> </item> </graph_item> <graph_item> <sortorder>6</sortorder> <drawtype>0</drawtype> <color>C8C8C8</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[lastack]</key> </item> </graph_item> <graph_item> <sortorder>7</sortorder> <drawtype>0</drawtype> <color>960000</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[listen]</key> </item> </graph_item> <graph_item> <sortorder>8</sortorder> <drawtype>0</drawtype> <color>009600</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[synrecv]</key> </item> </graph_item> <graph_item> <sortorder>9</sortorder> <drawtype>0</drawtype> <color>000096</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[synsent]</key> </item> </graph_item> <graph_item> <sortorder>10</sortorder> <drawtype>0</drawtype> <color>960096</color> <yaxisside>0</yaxisside> <calc_fnc>2</calc_fnc> <type>0</type> <item> <host>Template TCP Connection Status</host> <key>tcp.status[timewait]</key> </item> </graph_item> </graph_items> </graph> </graphs> </zabbix_export>