Logstash使用进阶篇
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.通过rsyslog收集haproxy日志到Elaticsearch集群
在CentOS 6.x及之前的版本称为”syslog”,CentOS 7.x之后更名为”rsyslog”,根据官方的介绍，rsyslog(2013年版本)可以达到每秒转发百万条日志级别，官方网址: http://www.rsyslog.com/。
我们既然学习了Elastic stack，再来学习rsyslog是否有多此一举的感觉呢？其实并不然，要知道在一些网络设备(比如路由器，防火墙，交换机，负载均衡器等)在出厂后很难安装一些软件，需要用其自带的工具进行管理。
接下来我们用rsyslog模拟收集负载均衡器(haproxy)的日志。
1>.配置logstash服务器
[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/rsyslog-to-elasticsearch.conf [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# cat /etc/logstash/conf.d/rsyslog-to-elasticsearch.conf input { syslog { port => 8888 type => "rsyslog-172.200.5.102" } } output { if [type] == "rsyslog-172.200.5.102" { elasticsearch { hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200","http://es103.yinzhengjie.com:9200"] index => "rsyslog-172.200.5.102-%{+YYYY.MM.dd}" } } } [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/rsyslog-to-elasticsearch.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-09 05:18:41.613 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [INFO ] 2020-06-09 05:18:46.004 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# systemctl start logstash.service [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 [::ffff:172.200.5.102]:9300 *:* LISTEN 0 128 [::]:22 [::]:* LISTEN 0 50 *:8888 *:* LISTEN 0 50 [::ffff:127.0.0.1]:9600 *:* LISTEN 0 128 [::ffff:172.200.5.102]:9200 *:* [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# tail -100f /var/log/logstash/logstash-plain.log [2020-06-09T05:19:44,538][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.8.9"} [2020-06-09T05:19:56,838][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} [2020-06-09T05:19:59,481][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es101.yinzhengjie.com:9200/, http://es102.yinzheng jie.com:9200/, http://es103.yinzhengjie.com:9200/]}}[2020-06-09T05:20:01,005][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es101.yinzhengjie.com:9200/"} [2020-06-09T05:20:01,385][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6} [2020-06-09T05:20:01,387][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6} [2020-06-09T05:20:01,971][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es102.yinzhengjie.com:9200/"} [2020-06-09T05:20:02,545][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es103.yinzhengjie.com:9200/"} [2020-06-09T05:20:03,503][INFO ][logstash.outputs.elasticsearch] Using default mapping template [2020-06-09T05:20:03,517][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es101.yinzhengjie.com:9200", "http:// es102.yinzhengjie.com:9200", "http://es103.yinzhengjie.com:9200"]}[2020-06-09T05:20:03,627][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_i nterval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}[2020-06-09T05:20:05,955][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x42e73542 run>"} [2020-06-09T05:20:05,979][INFO ][logstash.inputs.syslog ] Starting syslog udp listener {:address=>"0.0.0.0:8888"} [2020-06-09T05:20:05,987][INFO ][logstash.inputs.syslog ] Starting syslog tcp listener {:address=>"0.0.0.0:8888"} [2020-06-09T05:20:06,041][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2020-06-09T05:20:06,419][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
2>.Ubuntu系统安装并配置harproxy收集日志
[root@es103.yinzhengjie.com ~]# apt-get -y install haproxy Reading package lists... Done Building dependency tree Reading state information... Done Suggested packages: vim-haproxy haproxy-doc The following NEW packages will be installed: haproxy 0 upgraded, 1 newly installed, 0 to remove and 79 not upgraded. Need to get 1,117 kB of archives. After this operation, 2,374 kB of additional disk space will be used. Get:1 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 haproxy amd64 1.8.8-1ubuntu0.10 [1,117 kB] Fetched 1,117 kB in 0s (2,288 kB/s) Selecting previously unselected package haproxy. (Reading database ... 120486 files and directories currently installed.) Preparing to unpack .../haproxy_1.8.8-1ubuntu0.10_amd64.deb ... Unpacking haproxy (1.8.8-1ubuntu0.10) ... Setting up haproxy (1.8.8-1ubuntu0.10) ... Created symlink /etc/systemd/system/multi-user.target.wants/haproxy.service → /lib/systemd/system/haproxy.service. Processing triggers for ureadahead (0.100.0-21) ... Processing triggers for systemd (237-3ubuntu10.38) ... Processing triggers for man-db (2.8.3-2ubuntu0.1) ... Processing triggers for rsyslog (8.32.0-1ubuntu4) ... [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# vim /etc/haproxy/haproxy.cfg [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# cat /etc/haproxy/haproxy.cfg global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners stats timeout 30s user haproxy group haproxy daemon # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # Default ciphers to use on SSL-enabled listening sockets. # For more information, see ciphers(1SSL). This list is from: # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ # An alternative list with additional directives can be obtained from # https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS ssl-default-bind-options no-sslv3 defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http listen kibana bind 0.0.0.0:5601 mode http log global server 172.200.5.101 172.200.5.101:5601 check [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# vim /etc/rsyslog.d/49-haproxy.conf [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# cat /etc/rsyslog.d/49-haproxy.conf # Create an additional socket in haproxy's chroot in order to allow logging via # /dev/log to chroot'ed HAProxy processes $AddUnixListenSocket /var/lib/haproxy/dev/log # Send HAProxy messages to a dedicated logfile if $programname startswith 'haproxy' then /var/log/haproxy.log #使用TCP方式将harproxy的日志传递给logstash服务器 if $programname startswith 'haproxy' then @@172.200.5.102:8888 &~ [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# systemctl restart haproxy.service [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# systemctl restart rsyslog.service #修改上述两个配置文件后记得重启haproxy服务哟
[root@es103.yinzhengjie.com ~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 0.0.0.0:5601 0.0.0.0:* LISTEN 0 128 [::ffff:172.200.5.103]:9200 *:* LISTEN 0 128 [::ffff:172.200.5.103]:9300 *:* LISTEN 0 128 [::]:22 [::]:* [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]#
3>.CentOS系统安装并配置harproxy收集日志
[root@es201.yinzhengjie.com ~]# yum -y install haproxy Loaded plugins: fastestmirror Determining fastest mirrors * base: mirrors.bfsu.edu.cn * extras: mirrors.bfsu.edu.cn * updates: mirrors.bfsu.edu.cn base | 3.6 kB 00:00:00 extras | 2.9 kB 00:00:00 updates | 2.9 kB 00:00:00 (1/4): extras/7/x86_64/primary_db | 194 kB 00:00:00 (2/4): updates/7/x86_64/primary_db | 2.1 MB 00:00:00 (3/4): base/7/x86_64/primary_db | 6.1 MB 00:00:01 (4/4): base/7/x86_64/group_gz | 153 kB 00:00:01 Resolving Dependencies --> Running transaction check ---> Package haproxy.x86_64 0:1.5.18-9.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================= Package Arch Version Repository Size ============================================================================================================================================================================================= Installing: haproxy x86_64 1.5.18-9.el7 base 834 k Transaction Summary ============================================================================================================================================================================================= Install 1 Package Total download size: 834 k Installed size: 2.6 M Downloading packages: haproxy-1.5.18-9.el7.x86_64.rpm | 834 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : haproxy-1.5.18-9.el7.x86_64 1/1 Verifying : haproxy-1.5.18-9.el7.x86_64 1/1 Installed: haproxy.x86_64 0:1.5.18-9.el7 Complete! [root@es201.yinzhengjie.com ~]#
[root@es201.yinzhengjie.com ~]# vim /etc/haproxy/haproxy.cfg [root@es201.yinzhengjie.com ~]# [root@es201.yinzhengjie.com ~]# cat /etc/haproxy/haproxy.cfg #--------------------------------------------------------------------- # Example configuration for a possible web application. See the # full configuration options online. # # http://haproxy.1wt.eu/download/1.4/doc/configuration.txt # #--------------------------------------------------------------------- #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global # to have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 #--------------------------------------------------------------------- # main frontend which proxys to the backends #--------------------------------------------------------------------- frontend main *:5000 acl url_static path_beg -i /static /images /javascript /stylesheets acl url_static path_end -i .jpg .gif .png .css .js use_backend static if url_static default_backend app #--------------------------------------------------------------------- # static backend for serving up images, stylesheets and such #--------------------------------------------------------------------- backend static balance roundrobin server static 127.0.0.1:4331 check #--------------------------------------------------------------------- # round robin balancing between the various backends #--------------------------------------------------------------------- backend app balance roundrobin server app1 127.0.0.1:5001 check server app2 127.0.0.1:5002 check server app3 127.0.0.1:5003 check server app4 127.0.0.1:5004 check listen kibana bind 0.0.0.0:5601 mode http log global server 172.200.5.101 172.200.5.101:5601 check [root@es201.yinzhengjie.com ~]#
[root@es201.yinzhengjie.com ~]# vim /etc/rsyslog.conf [root@es201.yinzhengjie.com ~]# [root@es201.yinzhengjie.com ~]# cat /etc/rsyslog.conf # rsyslog configuration file # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html #### MODULES #### # The imjournal module bellow is now used as a message source instead of imuxsock. $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imjournal # provides access to the systemd journal #$ModLoad imklog # reads kernel messages (the same are read from journald) #$ModLoad immark # provides --MARK-- message capability # Provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 # Provides TCP syslog reception $ModLoad imtcp $InputTCPServerRun 514 #### GLOBAL DIRECTIVES #### # Where to place auxiliary files $WorkDirectory /var/lib/rsyslog # Use default timestamp format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # File syncing capability is disabled by default. This feature is usually not required, # not useful and an extreme performance hit #$ActionFileEnableSync on # Include all config files in /etc/rsyslog.d/ $IncludeConfig /etc/rsyslog.d/*.conf # Turn off message reception via local log socket; # local messages are retrieved through imjournal now. $OmitLocalLogging on # File to store the position in the journal $IMJournalStateFile imjournal.state #### RULES #### # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg :omusrmsg:* # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # ### begin forwarding rule ### # The statement between the begin ... end define a SINGLE forwarding # rule. They belong together, do NOT split them. If you create multiple # forwarding rules, duplicate the whole block! # Remote Logging (we use TCP for reliable delivery) # # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. #$ActionQueueFileName fwdRule1 # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinite retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host:514 # ### end of the forwarding rule ### # #Add by yinzhengjie #需要注意的是，开启上面的UDP,TCP功能，否则无法通过相关协议发送数据到logstash local2.* /var/log/haproxy.log local2.* @@172.200.5.102:8888 [root@es201.yinzhengjie.com ~]#
[root@es201.yinzhengjie.com ~]# systemctl restart haproxy.service [root@es201.yinzhengjie.com ~]# [root@es201.yinzhengjie.com ~]# systemctl restart rsyslog.service #同理，修改配置文件后，需要重启服务哟~ [root@es201.yinzhengjie.com ~]#
[root@es201.yinzhengjie.com ~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 3000 *:5000 *:* LISTEN 0 128 *:22 *:* LISTEN 0 3000 *:5601 *:* LISTEN 0 25 *:514 *:* LISTEN 0 128 :::22 :::* LISTEN 0 25 :::514 :::* [root@es201.yinzhengjie.com ~]# [root@es201.yinzhengjie.com ~]#
4>.在kibana界面种新建索引模式
5>.分别访问Ubuntu和CentOS的harpoxy的5601端口，观察日志采集情况
[root@es102.yinzhengjie.com ~]# tcpdump -i ens33 -c 2 -q -XX -vvv -nn tcp dst port 8888 tcpdump: listening on ens33, link-type EN10MB (Ethernet), capture size 262144 bytes 06:20:49.786369 IP (tos 0x0, ttl 64, id 9500, offset 0, flags [DF], proto TCP (6), length 521) 172.200.5.103.33674 > 172.200.5.102.8888: tcp 469 0x0000: 000c 2990 9cc1 000c 2906 42e6 0800 4500 ..).....).B...E. 0x0010: 0209 251c 4000 4006 af75 acc8 0567 acc8 ..%.@.@..u...g.. 0x0020: 0566 838a 22b8 5dbe a1e1 b3b7 bab5 8018 .f..".]......... 0x0030: 01f6 3fa4 0000 0101 080a e7d8 4d67 ea6d ..?.........Mg.m 0x0040: 9efc 3c31 3334 3e4a 756e 2020 3920 3036 ..<134>Jun..9.06 0x0050: 3a31 353a 3332 2065 7331 3033 2068 6170 :15:32.es103.hap 0x0060: 726f 7879 5b34 3832 355d 3a20 3137 322e roxy[4825]:.172. 0x0070: 3230 302e 302e 313a 3534 3335 3120 5b30 200.0.1:54351.[0 0x0080: 392f 4a75 6e2f 3230 3230 3a30 363a 3135 9/Jun/2020:06:15 0x0090: 3a33 322e 3134 315d 206b 6962 616e 6120 :32.141].kibana. 0x00a0: 6b69 6261 6e61 2f31 3732 2e32 3030 2e35 kibana/172.200.5 0x00b0: 2e31 3031 2030 2f30 2f31 2f39 2f31 3020 .101.0/0/1/9/10. 0x00c0: 3230 3020 3231 3530 202d 202d 202d 2d2d 200.2150.-.-.--- 0x00d0: 2d20 362f 362f 302f 302f 3020 302f 3020 -.6/6/0/0/0.0/0. 0x00e0: 2250 4f53 5420 2f65 6c61 7374 6963 7365 "POST./elasticse 0x00f0: 6172 6368 2f5f 6d73 6561 7263 683f 7265 arch/_msearch?re 0x0100: 7374 5f74 6f74 616c 5f68 6974 735f 6173 st_total_hits_as 0x0110: 5f69 6e74 3d74 7275 6526 6967 6e6f 7265 _int=true&ignore 0x0120: 5f74 6872 6f74 746c 6564 3d74 7275 6520 _throttled=true. 0x0130: 4854 5450 2f31 2e31 220a 3c31 3334 3e4a HTTP/1.1".<134>J 0x0140: 756e 2020 3920 3036 3a32 303a 3439 2065 un..9.06:20:49.e 0x0150: 7331 3033 2068 6170 726f 7879 5b34 3832 s103.haproxy[482 0x0160: 355d 3a20 3137 322e 3230 302e 302e 313a 5]:.172.200.0.1: 0x0170: 3534 3734 3220 5b30 392f 4a75 6e2f 3230 54742.[09/Jun/20 0x0180: 3230 3a30 363a 3230 3a34 392e 3737 355d 20:06:20:49.775] 0x0190: 206b 6962 616e 6120 6b69 6261 6e61 2f31 .kibana.kibana/1 0x01a0: 3732 2e32 3030 2e35 2e31 3031 2030 2f30 72.200.5.101.0/0 0x01b0: 2f31 2f32 2f34 2033 3034 2032 3939 202d /1/2/4.304.299.- 0x01c0: 202d 202d 2d2d 2d20 312f 312f 302f 302f .-.----.1/1/0/0/ 0x01d0: 3020 302f 3020 2247 4554 202f 7569 2f66 0.0/0."GET./ui/f 0x01e0: 6f6e 7473 2f6f 7065 6e5f 7361 6e73 2f6f onts/open_sans/o 0x01f0: 7065 6e5f 7361 6e73 5f76 3135 5f6c 6174 pen_sans_v15_lat 0x0200: 696e 5f36 3030 2e77 6f66 6632 2048 5454 in_600.woff2.HTT 0x0210: 502f 312e 3122 0a P/1.1". 06:20:51.043713 IP (tos 0x0, ttl 64, id 9501, offset 0, flags [DF], proto TCP (6), length 460) 172.200.5.103.33674 > 172.200.5.102.8888: tcp 408 0x0000: 000c 2990 9cc1 000c 2906 42e6 0800 4500 ..).....).B...E. 0x0010: 01cc 251d 4000 4006 afb1 acc8 0567 acc8 ..%.@.@......g.. 0x0020: 0566 838a 22b8 5dbe a3b6 b3b7 bab5 8018 .f..".]......... 0x0030: 01f6 0220 0000 0101 080a e7d8 5250 ea72 ............RP.r 0x0040: 77b8 3c31 3334 3e4a 756e 2020 3920 3036 w.<134>Jun..9.06 0x0050: 3a32 303a 3439 2065 7331 3033 2068 6170 :20:49.es103.hap 0x0060: 726f 7879 5b34 3832 355d 3a20 3137 322e roxy[4825]:.172. 0x0070: 3230 302e 302e 313a 3534 3734 3220 5b30 200.0.1:54742.[0 0x0080: 392f 4a75 6e2f 3230 3230 3a30 363a 3230 9/Jun/2020:06:20 0x0090: 3a34 392e 3737 355d 206b 6962 616e 6120 :49.775].kibana. 0x00a0: 6b69 6261 6e61 2f31 3732 2e32 3030 2e35 kibana/172.200.5 0x00b0: 2e31 3031 2030 2f30 2f31 2f32 2f34 2033 .101.0/0/1/2/4.3 0x00c0: 3034 2032 3939 202d 202d 202d 2d2d 2d20 04.299.-.-.----. 0x00d0: 312f 312f 302f 302f 3020 302f 3020 2247 1/1/0/0/0.0/0."G 0x00e0: 4554 202f 7569 2f66 6f6e 7473 2f6f 7065 ET./ui/fonts/ope 0x00f0: 6e5f 7361 6e73 2f6f 7065 6e5f 7361 6e73 n_sans/open_sans 0x0100: 5f76 3135 5f6c 6174 696e 5f36 3030 2e77 _v15_latin_600.w 0x0110: 6f66 6632 2048 5454 502f 312e 3122 0a3c off2.HTTP/1.1".< 0x0120: 3133 343e 4a75 6e20 2039 2030 363a 3230 134>Jun..9.06:20 0x0130: 3a35 3120 6573 3130 3320 6861 7072 6f78 :51.es103.haprox 0x0140: 795b 3438 3235 5d3a 2031 3732 2e32 3030 y[4825]:.172.200 0x0150: 2e30 2e31 3a35 3437 3432 205b 3039 2f4a .0.1:54742.[09/J 0x0160: 756e 2f32 3032 303a 3036 3a32 303a 3531 un/2020:06:20:51 0x0170: 2e30 3035 5d20 6b69 6261 6e61 206b 6962 .005].kibana.kib 0x0180: 616e 612f 3137 322e 3230 302e 352e 3130 ana/172.200.5.10 0x0190: 3120 302f 302f 312f 3238 2f33 3220 3230 1.0/0/1/28/32.20 0x01a0: 3020 3137 3933 3120 2d20 2d20 2d2d 2d2d 0.17931.-.-.---- 0x01b0: 2032 2f32 2f30 2f30 2f30 2030 2f30 2022 .2/2/0/0/0.0/0." 0x01c0: 4745 5420 2f61 7070 2f6b 6962 616e 6120 GET./app/kibana. 0x01d0: 4854 5450 2f31 2e31 220a HTTP/1.1". 2 packets captured 3 packets received by filter 0 packets dropped by kernel [root@es102.yinzhengjie.com ~]#
二.logstash结合redis收集ubuntu系统日志案例
1>.在ubuntu系统安装redis
[root@es103.yinzhengjie.com ~]# apt-get -y install redis Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: libjemalloc1 redis-server redis-tools Suggested packages: ruby-redis The following NEW packages will be installed: libjemalloc1 redis redis-server redis-tools 0 upgraded, 4 newly installed, 0 to remove and 79 not upgraded. Need to get 637 kB of archives. After this operation, 3,083 kB of additional disk space will be used. Get:1 http://mirrors.aliyun.com/ubuntu bionic/universe amd64 libjemalloc1 amd64 3.6.0-11 [82.4 kB] Get:2 http://mirrors.aliyun.com/ubuntu bionic-security/universe amd64 redis-tools amd64 5:4.0.9-1ubuntu0.2 [516 kB] Get:3 http://mirrors.aliyun.com/ubuntu bionic-security/universe amd64 redis-server amd64 5:4.0.9-1ubuntu0.2 [35.4 kB] Get:4 http://mirrors.aliyun.com/ubuntu bionic-security/universe amd64 redis all 5:4.0.9-1ubuntu0.2 [3,084 B] Fetched 637 kB in 1s (969 kB/s) Selecting previously unselected package libjemalloc1. (Reading database ... 120605 files and directories currently installed.) Preparing to unpack .../libjemalloc1_3.6.0-11_amd64.deb ... Unpacking libjemalloc1 (3.6.0-11) ... Selecting previously unselected package redis-tools. Preparing to unpack .../redis-tools_5%3a4.0.9-1ubuntu0.2_amd64.deb ... Unpacking redis-tools (5:4.0.9-1ubuntu0.2) ... Selecting previously unselected package redis-server. Preparing to unpack .../redis-server_5%3a4.0.9-1ubuntu0.2_amd64.deb ... Unpacking redis-server (5:4.0.9-1ubuntu0.2) ... Selecting previously unselected package redis. Preparing to unpack .../redis_5%3a4.0.9-1ubuntu0.2_all.deb ... Unpacking redis (5:4.0.9-1ubuntu0.2) ... Processing triggers for ureadahead (0.100.0-21) ... Setting up libjemalloc1 (3.6.0-11) ... Processing triggers for libc-bin (2.27-3ubuntu1) ... Processing triggers for systemd (237-3ubuntu10.38) ... Processing triggers for man-db (2.8.3-2ubuntu0.1) ... Setting up redis-tools (5:4.0.9-1ubuntu0.2) ... Setting up redis-server (5:4.0.9-1ubuntu0.2) ... Created symlink /etc/systemd/system/redis.service → /lib/systemd/system/redis-server.service. Created symlink /etc/systemd/system/multi-user.target.wants/redis-server.service → /lib/systemd/system/redis-server.service. Setting up redis (5:4.0.9-1ubuntu0.2) ... Processing triggers for systemd (237-3ubuntu10.38) ... Processing triggers for ureadahead (0.100.0-21) ... [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# egrep -v "^#|^$" /etc/redis/redis.conf bind 172.200.5.103 protected-mode yes port 6379 tcp-backlog 511 timeout 0 tcp-keepalive 300 daemonize yes supervised no pidfile /var/run/redis/redis-server.pid loglevel notice logfile /var/log/redis/redis-server.log databases 16 always-show-logo yes save 900 1 save 300 10 save 60 10000 stop-writes-on-bgsave-error yes rdbcompression yes rdbchecksum yes dbfilename dump.rdb dir /var/lib/redis slave-serve-stale-data yes slave-read-only yes repl-diskless-sync no repl-diskless-sync-delay 5 repl-disable-tcp-nodelay no slave-priority 100 requirepass yinzhengjie lazyfree-lazy-eviction no lazyfree-lazy-expire no lazyfree-lazy-server-del no slave-lazy-flush no appendonly no appendfilename "appendonly.aof" appendfsync everysec no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb aof-load-truncated yes aof-use-rdb-preamble no lua-time-limit 5000 slowlog-log-slower-than 10000 slowlog-max-len 128 latency-monitor-threshold 0 notify-keyspace-events "" hash-max-ziplist-entries 512 hash-max-ziplist-value 64 list-max-ziplist-size -2 list-compress-depth 0 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 hll-sparse-max-bytes 3000 activerehashing yes client-output-buffer-limit normal 0 0 0 client-output-buffer-limit slave 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60 hz 10 aof-rewrite-incremental-fsync yes [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# systemctl restart redis [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 0.0.0.0:5601 0.0.0.0:* LISTEN 0 128 172.200.5.103:6379 0.0.0.0:* LISTEN 0 128 [::ffff:172.200.5.103]:9200 *:* LISTEN 0 128 [::ffff:172.200.5.103]:9300 *:* LISTEN 0 128 [::]:22 [::]:* [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# redis-cli -h 172.200.5.103 #测试链接redis 172.200.5.103:6379> AUTH yinzhengjie #先输入密码进行验证 OK 172.200.5.103:6379> 172.200.5.103:6379> KEYS * (empty list or set) 172.200.5.103:6379> 172.200.5.103:6379> INFO #查看redis的信息 # Server redis_version:4.0.9 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:9435c3c2879311f3 redis_mode:standalone os:Linux 4.15.0-101-generic x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:7.4.0 process_id:6315 run_id:ab2ab759ecd21d90dd5461ff46c03262f20119fe tcp_port:6379 uptime_in_seconds:57 uptime_in_days:0 hz:10 lru_clock:14627002 executable:/usr/bin/redis-server config_file:/etc/redis/redis.conf # Clients connected_clients:1 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0 # Memory used_memory:841296 used_memory_human:821.58K used_memory_rss:3809280 used_memory_rss_human:3.63M used_memory_peak:841296 used_memory_peak_human:821.58K used_memory_peak_perc:100.12% used_memory_overhead:832142 used_memory_startup:782512 used_memory_dataset:9154 used_memory_dataset_perc:15.57% total_system_memory:4112252928 total_system_memory_human:3.83G used_memory_lua:37888 used_memory_lua_human:37.00K maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction mem_fragmentation_ratio:4.53 mem_allocator:jemalloc-3.6.0 active_defrag_running:0 lazyfree_pending_objects:0 # Persistence loading:0 rdb_changes_since_last_save:0 rdb_bgsave_in_progress:0 rdb_last_save_time:1591685249 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:0 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0 # Stats total_connections_received:1 total_commands_processed:2 instantaneous_ops_per_sec:0 total_net_input_bytes:84 total_net_output_bytes:43 instantaneous_input_kbps:0.00 instantaneous_output_kbps:0.00 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 evicted_keys:0 keyspace_hits:0 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:0 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0 # Replication role:master connected_slaves:0 master_replid:469148f0014cb7d764269c458593b5cc1155cb97 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0 # CPU used_cpu_sys:0.00 used_cpu_user:0.08 used_cpu_sys_children:0.00 used_cpu_user_children:0.00 # Cluster cluster_enabled:0 # Keyspace 172.200.5.103:6379>
2>.logstash写入数据到redis
[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/file-to-redis.conf [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# cat /etc/logstash/conf.d/file-to-redis.conf input { file { type => "syslog-172-200-5-102" path => "/var/log/syslog" start_position => "beginning" stat_interval => "3" } } output { if [type] == "syslog-172-200-5-102"{ redis { host => "172.200.5.103" port => "6379" password => "yinzhengjie" db => "1" key => "yinzhengjie-5-102" data_type => "list" } } } [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/file-to-redis.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-10 01:04:47.686 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [INFO ] 2020-06-10 01:04:51.673 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# systemctl restart logstash.service [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# ps -ef | grep logstash root 6003 5644 0 01:06 pts/1 00:00:00 tail -100f /var/log/logstash/logstash-plain.log root 6143 1 84 01:07 ? 00:00:50 /usr/local/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.a wt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.10.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.0.8.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.22.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/sharelogstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings /etc/logstashroot 6208 6084 0 01:08 pts/2 00:00:00 grep --color=auto logstash [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# redis-cli -h 172.200.5.103 172.200.5.103:6379> AUTH yinzhengjie OK 172.200.5.103:6379> 172.200.5.103:6379> SELECT 1 OK 172.200.5.103:6379[1]> 172.200.5.103:6379[1]> KEYS * 1) "yinzhengjie-5-102" 172.200.5.103:6379[1]>
3>.logstash从redis获取数据
[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/redis-to-elasticsearch.conf [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# cat /etc/logstash/conf.d/redis-to-elasticsearch.conf input { redis { host => "172.200.5.103" port => "6379" password => "yinzhengjie" db => "1" key => "yinzhengjie-5-102" data_type => "list" } } output { if [type] == "syslog-172-200-5-102" { elasticsearch { hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200","http://es103.yinzhengjie.com:9200"] index => "redis-syslog-172.200.5.102-%{+YYYY.MM.dd}" } } } [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis-to-elasticsearch.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-10 01:23:10.932 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [INFO ] 2020-06-10 01:23:15.150 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# ps -ef | grep logstash root 6143 1 6 01:07 ? 00:01:13 /usr/local/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.a wt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.10.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.0.8.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.22.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/sharelogstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings /etc/logstashroot 6256 6084 0 01:26 pts/2 00:00:00 grep --color=auto logstash [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# systemctl restart logstash.service [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# ps -ef | grep logstash root 6265 1 99 01:26 ? 00:00:02 /usr/local/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.a wt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.10.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.0.8.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.22.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/sharelogstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings /etc/logstashroot 6307 6084 0 01:26 pts/2 00:00:00 grep --color=auto logstash [root@es102.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# redis-cli -h 172.200.5.103 172.200.5.103:6379> AUTH yinzhengjie OK 172.200.5.103:6379> 172.200.5.103:6379> SELECT 1 OK 172.200.5.103:6379[1]> 172.200.5.103:6379[1]> KEYS * #重启logstsash之前，redis的数据未被消费，因此可以看到数据 1) "yinzhengjie-5-102" 172.200.5.103:6379[1]> 172.200.5.103:6379[1]> KEYS * #重启logstash之后，redis的数据被消费了，数据已经写入到elasticsearch集群啦，此时我们应该通过去kibana查看数据 (empty list or set) 172.200.5.103:6379[1]>
4>.通过kibana创建索引
5>.查看redis的数据是否成功写入elasticsearch集群
6>.编写监控redis的数据长度脚本
实际环境中，可能会出现当中堆积了大量的数据而logstash由于种种原因未能及时提取日志，此时会导致redis服务器的内存被大量使用，甚至会出现内存即将被使用完毕的情景。
因此监控redis的日志队列长度以及时处理很有必要，必要时需要扩容redis集群，但我个人比较喜欢使用kafka来替代redis，毕竟redis相对来是比较耗费内存的，而我们使用kafka就并不会过多的浪费内存。
[root@es102.yinzhengjie.com ~]# apt-get -y install python-pip Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: binutils binutils-common binutils-x86-64-linux-gnu build-essential cpp cpp-7 dpkg-dev fakeroot g++ g++-7 gcc gcc-7 gcc-7-base libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan4 libatomic1 libbinutils libc-dev-bin libc6-dev libcc1-0 libcilkrts5 libdpkg-perl libexpat1-dev libfakeroot libfile-fcntllock-perl libgcc-7-dev libgomp1 libisl19 libitm1 liblsan0 libmpc3 libmpx2 libpython-all-dev libpython-dev libpython-stdlib libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib libquadmath0 libstdc++-7-dev libtsan0 libubsan0 linux-libc-dev make manpages-dev python python-all python-all-dev python-asn1crypto python-cffi-backend python-crypto python-cryptography python-dbus python-dev python-enum34 python-gi python-idna python-ipaddress python-keyring python-keyrings.alt python-minimal python-pip-whl python-pkg-resources python-secretstorage python-setuptools python-six python-wheel python-xdg python2.7 python2.7-dev python2.7-minimal Suggested packages: binutils-doc cpp-doc gcc-7-locales debian-keyring g++-multilib g++-7-multilib gcc-7-doc libstdc++6-7-dbg gcc-multilib autoconf automake libtool flex bison gdb gcc-doc gcc-7-multilib libgcc1-dbg libgomp1-dbg libitm1-dbg libatomic1-dbg libasan4-dbg liblsan0-dbg libtsan0-dbg libubsan0-dbg libcilkrts5-dbg libmpx2-dbg libquadmath0-dbg glibc-doc bzr libstdc++-7-doc make-doc python-doc python-tk python-crypto-doc python-cryptography-doc python-cryptography-vectors python-dbus-dbg python-dbus-doc python-enum34-doc python-gi-cairo gnome-keyring libkf5wallet-bin gir1.2-gnomekeyring-1.0 python-fs python-gdata python-keyczar python-secretstorage-doc python-setuptools-doc python2.7-doc binfmt-support The following NEW packages will be installed: binutils binutils-common binutils-x86-64-linux-gnu build-essential cpp cpp-7 dpkg-dev fakeroot g++ g++-7 gcc gcc-7 gcc-7-base libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan4 libatomic1 libbinutils libc-dev-bin libc6-dev libcc1-0 libcilkrts5 libdpkg-perl libexpat1-dev libfakeroot libfile-fcntllock-perl libgcc-7-dev libgomp1 libisl19 libitm1 liblsan0 libmpc3 libmpx2 libpython-all-dev libpython-dev libpython-stdlib libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib libquadmath0 libstdc++-7-dev libtsan0 libubsan0 linux-libc-dev make manpages-dev python python-all python-all-dev python-asn1crypto python-cffi-backend python-crypto python-cryptography python-dbus python-dev python-enum34 python-gi python-idna python-ipaddress python-keyring python-keyrings.alt python-minimal python-pip python-pip-whl python-pkg-resources python-secretstorage python-setuptools python-six python-wheel python-xdg python2.7 python2.7-dev python2.7-minimal 0 upgraded, 75 newly installed, 0 to remove and 79 not upgraded. Need to get 80.1 MB of archives. After this operation, 239 MB of additional disk space will be used. Get:1 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libpython2.7-minimal amd64 2.7.17-1~18.04ubuntu1 [335 kB] Get:2 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 python2.7-minimal amd64 2.7.17-1~18.04ubuntu1 [1,294 kB] Get:3 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-minimal amd64 2.7.15~rc1-1 [28.1 kB] Get:4 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libpython2.7-stdlib amd64 2.7.17-1~18.04ubuntu1 [1,915 kB] Get:5 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 python2.7 amd64 2.7.17-1~18.04ubuntu1 [248 kB] Get:6 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libpython-stdlib amd64 2.7.15~rc1-1 [7,620 B] Get:7 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python amd64 2.7.15~rc1-1 [140 kB] Get:8 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 binutils-common amd64 2.30-21ubuntu1~18.04.3 [196 kB] Get:9 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libbinutils amd64 2.30-21ubuntu1~18.04.3 [488 kB] Get:10 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 binutils-x86-64-linux-gnu amd64 2.30-21ubuntu1~18.04.3 [1,839 kB] Get:11 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 binutils amd64 2.30-21ubuntu1~18.04.3 [3,388 B] Get:12 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libc-dev-bin amd64 2.27-3ubuntu1 [71.8 kB] Get:13 http://mirrors.aliyun.com/ubuntu bionic-proposed/main amd64 linux-libc-dev amd64 4.15.0-103.104 [1,001 kB] Get:14 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libc6-dev amd64 2.27-3ubuntu1 [2,587 kB] Get:15 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 gcc-7-base amd64 7.5.0-3ubuntu1~18.04 [18.3 kB] Get:16 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libisl19 amd64 0.19-1 [551 kB] Get:17 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libmpc3 amd64 1.1.0-1 [40.8 kB] Get:18 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 cpp-7 amd64 7.5.0-3ubuntu1~18.04 [8,591 kB] Get:19 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 cpp amd64 4:7.4.0-1ubuntu2.3 [27.7 kB] Get:20 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libcc1-0 amd64 8.4.0-1ubuntu1~18.04 [39.4 kB] Get:21 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libgomp1 amd64 8.4.0-1ubuntu1~18.04 [76.5 kB] Get:22 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libitm1 amd64 8.4.0-1ubuntu1~18.04 [27.9 kB] Get:23 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libatomic1 amd64 8.4.0-1ubuntu1~18.04 [9,192 B] Get:24 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libasan4 amd64 7.5.0-3ubuntu1~18.04 [358 kB] Get:25 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 liblsan0 amd64 8.4.0-1ubuntu1~18.04 [133 kB] Get:26 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libtsan0 amd64 8.4.0-1ubuntu1~18.04 [288 kB] Get:27 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libubsan0 amd64 7.5.0-3ubuntu1~18.04 [126 kB] Get:28 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libcilkrts5 amd64 7.5.0-3ubuntu1~18.04 [42.5 kB] Get:29 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libmpx2 amd64 8.4.0-1ubuntu1~18.04 [11.6 kB] Get:30 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libquadmath0 amd64 8.4.0-1ubuntu1~18.04 [134 kB] Get:31 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libgcc-7-dev amd64 7.5.0-3ubuntu1~18.04 [2,378 kB] Get:32 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 gcc-7 amd64 7.5.0-3ubuntu1~18.04 [9,381 kB] Get:33 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 gcc amd64 4:7.4.0-1ubuntu2.3 [5,184 B] Get:34 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libstdc++-7-dev amd64 7.5.0-3ubuntu1~18.04 [1,471 kB] Get:35 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 g++-7 amd64 7.5.0-3ubuntu1~18.04 [9,697 kB] Get:36 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 g++ amd64 4:7.4.0-1ubuntu2.3 [1,568 B] Get:37 http://mirrors.aliyun.com/ubuntu bionic/main amd64 make amd64 4.1-9.1ubuntu1 [154 kB] Get:38 http://mirrors.aliyun.com/ubuntu bionic-updates/main amd64 libdpkg-perl all 1.19.0.5ubuntu2.3 [211 kB] Get:39 http://mirrors.aliyun.com/ubuntu bionic-updates/main amd64 dpkg-dev all 1.19.0.5ubuntu2.3 [607 kB] Get:40 http://mirrors.aliyun.com/ubuntu bionic/main amd64 build-essential amd64 12.4ubuntu1 [4,758 B] Get:41 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libfakeroot amd64 1.22-2ubuntu1 [25.9 kB] Get:42 http://mirrors.aliyun.com/ubuntu bionic/main amd64 fakeroot amd64 1.22-2ubuntu1 [62.3 kB] Get:43 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libalgorithm-diff-perl all 1.19.03-1 [47.6 kB] Get:44 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libalgorithm-diff-xs-perl amd64 0.04-5 [11.1 kB] Get:45 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libalgorithm-merge-perl all 0.08-3 [12.0 kB] Get:46 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libexpat1-dev amd64 2.2.5-3ubuntu0.2 [122 kB] Get:47 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libfile-fcntllock-perl amd64 0.22-3build2 [33.2 kB] Get:48 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libpython2.7 amd64 2.7.17-1~18.04ubuntu1 [1,053 kB] Get:49 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libpython2.7-dev amd64 2.7.17-1~18.04ubuntu1 [28.3 MB] Get:50 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libpython-dev amd64 2.7.15~rc1-1 [7,684 B] Get:51 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libpython-all-dev amd64 2.7.15~rc1-1 [1,092 B] Get:52 http://mirrors.aliyun.com/ubuntu bionic/main amd64 manpages-dev all 4.15-1 [2,217 kB] Get:53 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-all amd64 2.7.15~rc1-1 [1,076 B] Get:54 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 python2.7-dev amd64 2.7.17-1~18.04ubuntu1 [279 kB] Get:55 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-dev amd64 2.7.15~rc1-1 [1,256 B] Get:56 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-all-dev amd64 2.7.15~rc1-1 [1,100 B] Get:57 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-asn1crypto all 0.24.0-1 [72.7 kB] Get:58 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-cffi-backend amd64 1.11.5-1 [63.4 kB] Get:59 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-crypto amd64 2.6.1-8ubuntu2 [244 kB] Get:60 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-enum34 all 1.1.6-2 [34.8 kB] Get:61 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-idna all 2.6-1 [32.4 kB] Get:62 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-ipaddress all 1.0.17-1 [18.2 kB] Get:63 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-six all 1.11.0-2 [11.3 kB] Get:64 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 python-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] Get:65 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-dbus amd64 1.2.6-1 [90.2 kB] Get:66 http://mirrors.aliyun.com/ubuntu bionic-updates/main amd64 python-gi amd64 3.26.1-2ubuntu1 [197 kB] Get:67 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-secretstorage all 2.3.1-2 [11.8 kB] Get:68 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-keyring all 10.6.0-1 [30.6 kB] Get:69 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-keyrings.alt all 3.0-1 [16.7 kB] Get:70 http://mirrors.aliyun.com/ubuntu bionic-updates/universe amd64 python-pip-whl all 9.0.1-2.3~ubuntu1.18.04.1 [1,653 kB] Get:71 http://mirrors.aliyun.com/ubuntu bionic-updates/universe amd64 python-pip all 9.0.1-2.3~ubuntu1.18.04.1 [151 kB] Get:72 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-pkg-resources all 39.0.1-2 [128 kB] Get:73 http://mirrors.aliyun.com/ubuntu bionic/main amd64 python-setuptools all 39.0.1-2 [329 kB] Get:74 http://mirrors.aliyun.com/ubuntu bionic/universe amd64 python-wheel all 0.30.0-0.2 [36.4 kB] Get:75 http://mirrors.aliyun.com/ubuntu bionic/universe amd64 python-xdg all 0.25-4ubuntu1 [31.3 kB] Fetched 80.1 MB in 39s (2,042 kB/s) Extracting templates from packages: 100% Selecting previously unselected package libpython2.7-minimal:amd64. (Reading database ... 119654 files and directories currently installed.) Preparing to unpack .../0-libpython2.7-minimal_2.7.17-1~18.04ubuntu1_amd64.deb ... Unpacking libpython2.7-minimal:amd64 (2.7.17-1~18.04ubuntu1) ... Selecting previously unselected package python2.7-minimal. Preparing to unpack .../1-python2.7-minimal_2.7.17-1~18.04ubuntu1_amd64.deb ... Unpacking python2.7-minimal (2.7.17-1~18.04ubuntu1) ... Selecting previously unselected package python-minimal. Preparing to unpack .../2-python-minimal_2.7.15~rc1-1_amd64.deb ... Unpacking python-minimal (2.7.15~rc1-1) ... Selecting previously unselected package libpython2.7-stdlib:amd64. Preparing to unpack .../3-libpython2.7-stdlib_2.7.17-1~18.04ubuntu1_amd64.deb ... Unpacking libpython2.7-stdlib:amd64 (2.7.17-1~18.04ubuntu1) ... Selecting previously unselected package python2.7. Preparing to unpack .../4-python2.7_2.7.17-1~18.04ubuntu1_amd64.deb ... Unpacking python2.7 (2.7.17-1~18.04ubuntu1) ... Selecting previously unselected package libpython-stdlib:amd64. Preparing to unpack .../5-libpython-stdlib_2.7.15~rc1-1_amd64.deb ... Unpacking libpython-stdlib:amd64 (2.7.15~rc1-1) ... Setting up libpython2.7-minimal:amd64 (2.7.17-1~18.04ubuntu1) ... Setting up python2.7-minimal (2.7.17-1~18.04ubuntu1) ... Linking and byte-compiling packages for runtime python2.7... Setting up python-minimal (2.7.15~rc1-1) ... Selecting previously unselected package python. (Reading database ... 120402 files and directories currently installed.) Preparing to unpack .../00-python_2.7.15~rc1-1_amd64.deb ... Unpacking python (2.7.15~rc1-1) ... Selecting previously unselected package binutils-common:amd64. Preparing to unpack .../01-binutils-common_2.30-21ubuntu1~18.04.3_amd64.deb ... Unpacking binutils-common:amd64 (2.30-21ubuntu1~18.04.3) ... Selecting previously unselected package libbinutils:amd64. Preparing to unpack .../02-libbinutils_2.30-21ubuntu1~18.04.3_amd64.deb ... Unpacking libbinutils:amd64 (2.30-21ubuntu1~18.04.3) ... Selecting previously unselected package binutils-x86-64-linux-gnu. Preparing to unpack .../03-binutils-x86-64-linux-gnu_2.30-21ubuntu1~18.04.3_amd64.deb ... Unpacking binutils-x86-64-linux-gnu (2.30-21ubuntu1~18.04.3) ... Selecting previously unselected package binutils. Preparing to unpack .../04-binutils_2.30-21ubuntu1~18.04.3_amd64.deb ... Unpacking binutils (2.30-21ubuntu1~18.04.3) ... Selecting previously unselected package libc-dev-bin. Preparing to unpack .../05-libc-dev-bin_2.27-3ubuntu1_amd64.deb ... Unpacking libc-dev-bin (2.27-3ubuntu1) ... Selecting previously unselected package linux-libc-dev:amd64. Preparing to unpack .../06-linux-libc-dev_4.15.0-103.104_amd64.deb ... Unpacking linux-libc-dev:amd64 (4.15.0-103.104) ... Selecting previously unselected package libc6-dev:amd64. Preparing to unpack .../07-libc6-dev_2.27-3ubuntu1_amd64.deb ... Unpacking libc6-dev:amd64 (2.27-3ubuntu1) ... Selecting previously unselected package gcc-7-base:amd64. Preparing to unpack .../08-gcc-7-base_7.5.0-3ubuntu1~18.04_amd64.deb ... Unpacking gcc-7-base:amd64 (7.5.0-3ubuntu1~18.04) ... Selecting previously unselected package libisl19:amd64. Preparing to unpack .../09-libisl19_0.19-1_amd64.deb ... Unpacking libisl19:amd64 (0.19-1) ... Selecting previously unselected package libmpc3:amd64. Preparing to unpack .../10-libmpc3_1.1.0-1_amd64.deb ... Unpacking libmpc3:amd64 (1.1.0-1) ... Selecting previously unselected package cpp-7. Preparing to unpack .../11-cpp-7_7.5.0-3ubuntu1~18.04_amd64.deb ... Unpacking cpp-7 (7.5.0-3ubuntu1~18.04) ... Selecting previously unselected package cpp. Preparing to unpack .../12-cpp_4%3a7.4.0-1ubuntu2.3_amd64.deb ... Unpacking cpp (4:7.4.0-1ubuntu2.3) ... Selecting previously unselected package libcc1-0:amd64. Preparing to unpack .../13-libcc1-0_8.4.0-1ubuntu1~18.04_amd64.deb ... Unpacking libcc1-0:amd64 (8.4.0-1ubuntu1~18.04) ... Selecting previously unselected package libgomp1:amd64. Preparing to unpack .../14-libgomp1_8.4.0-1ubuntu1~18.04_amd64.deb ... Unpacking libgomp1:amd64 (8.4.0-1ubuntu1~18.04) ... Selecting previously unselected package libitm1:amd64. Preparing to unpack .../15-libitm1_8.4.0-1ubuntu1~18.04_amd64.deb ... Unpacking libitm1:amd64 (8.4.0-1ubuntu1~18.04) ... Selecting previously unselected package libatomic1:amd64. Preparing to unpack .../16-libatomic1_8.4.0-1ubuntu1~18.04_amd64.deb ... Unpacking libatomic1:amd64 (8.4.0-1ubuntu1~18.04) ... Selecting previously unselected package libasan4:amd64. Preparing to unpack .../17-libasan4_7.5.0-3ubuntu1~18.04_amd64.deb ... Unpacking libasan4:amd64 (7.5.0-3ubuntu1~18.04) ... Selecting previously unselected package liblsan0:amd64. Preparing to unpack .../18-liblsan0_8.4.0-1ubuntu1~18.04_amd64.deb ... Unpacking liblsan0:amd64 (8.4.0-1ubuntu1~18.04) ... Selecting previously unselected package libtsan0:amd64. Preparing to unpack .../19-libtsan0_8.4.0-1ubuntu1~18.04_amd64.deb ... Unpacking libtsan0:amd64 (8.4.0-1ubuntu1~18.04) ... Selecting previously unselected package libubsan0:amd64. Preparing to unpack .../20-libubsan0_7.5.0-3ubuntu1~18.04_amd64.deb ... Unpacking libubsan0:amd64 (7.5.0-3ubuntu1~18.04) ... Selecting previously unselected package libcilkrts5:amd64. Preparing to unpack .../21-libcilkrts5_7.5.0-3ubuntu1~18.04_amd64.deb ... Unpacking libcilkrts5:amd64 (7.5.0-3ubuntu1~18.04) ... Selecting previously unselected package libmpx2:amd64. Preparing to unpack .../22-libmpx2_8.4.0-1ubuntu1~18.04_amd64.deb ... Unpacking libmpx2:amd64 (8.4.0-1ubuntu1~18.04) ... Selecting previously unselected package libquadmath0:amd64. Preparing to unpack .../23-libquadmath0_8.4.0-1ubuntu1~18.04_amd64.deb ... Unpacking libquadmath0:amd64 (8.4.0-1ubuntu1~18.04) ... Selecting previously unselected package libgcc-7-dev:amd64. Preparing to unpack .../24-libgcc-7-dev_7.5.0-3ubuntu1~18.04_amd64.deb ... Unpacking libgcc-7-dev:amd64 (7.5.0-3ubuntu1~18.04) ... Selecting previously unselected package gcc-7. Preparing to unpack .../25-gcc-7_7.5.0-3ubuntu1~18.04_amd64.deb ... Unpacking gcc-7 (7.5.0-3ubuntu1~18.04) ... Selecting previously unselected package gcc. Preparing to unpack .../26-gcc_4%3a7.4.0-1ubuntu2.3_amd64.deb ... Unpacking gcc (4:7.4.0-1ubuntu2.3) ... Selecting previously unselected package libstdc++-7-dev:amd64. Preparing to unpack .../27-libstdc++-7-dev_7.5.0-3ubuntu1~18.04_amd64.deb ... Unpacking libstdc++-7-dev:amd64 (7.5.0-3ubuntu1~18.04) ... Selecting previously unselected package g++-7. Preparing to unpack .../28-g++-7_7.5.0-3ubuntu1~18.04_amd64.deb ... Unpacking g++-7 (7.5.0-3ubuntu1~18.04) ... Selecting previously unselected package g++. Preparing to unpack .../29-g++_4%3a7.4.0-1ubuntu2.3_amd64.deb ... Unpacking g++ (4:7.4.0-1ubuntu2.3) ... Selecting previously unselected package make. Preparing to unpack .../30-make_4.1-9.1ubuntu1_amd64.deb ... Unpacking make (4.1-9.1ubuntu1) ... Selecting previously unselected package libdpkg-perl. Preparing to unpack .../31-libdpkg-perl_1.19.0.5ubuntu2.3_all.deb ... Unpacking libdpkg-perl (1.19.0.5ubuntu2.3) ... Selecting previously unselected package dpkg-dev. Preparing to unpack .../32-dpkg-dev_1.19.0.5ubuntu2.3_all.deb ... Unpacking dpkg-dev (1.19.0.5ubuntu2.3) ... Selecting previously unselected package build-essential. Preparing to unpack .../33-build-essential_12.4ubuntu1_amd64.deb ... Unpacking build-essential (12.4ubuntu1) ... Selecting previously unselected package libfakeroot:amd64. Preparing to unpack .../34-libfakeroot_1.22-2ubuntu1_amd64.deb ... Unpacking libfakeroot:amd64 (1.22-2ubuntu1) ... Selecting previously unselected package fakeroot. Preparing to unpack .../35-fakeroot_1.22-2ubuntu1_amd64.deb ... Unpacking fakeroot (1.22-2ubuntu1) ... Selecting previously unselected package libalgorithm-diff-perl. Preparing to unpack .../36-libalgorithm-diff-perl_1.19.03-1_all.deb ... Unpacking libalgorithm-diff-perl (1.19.03-1) ... Selecting previously unselected package libalgorithm-diff-xs-perl. Preparing to unpack .../37-libalgorithm-diff-xs-perl_0.04-5_amd64.deb ... Unpacking libalgorithm-diff-xs-perl (0.04-5) ... Selecting previously unselected package libalgorithm-merge-perl. Preparing to unpack .../38-libalgorithm-merge-perl_0.08-3_all.deb ... Unpacking libalgorithm-merge-perl (0.08-3) ... Selecting previously unselected package libexpat1-dev:amd64. Preparing to unpack .../39-libexpat1-dev_2.2.5-3ubuntu0.2_amd64.deb ... Unpacking libexpat1-dev:amd64 (2.2.5-3ubuntu0.2) ... Selecting previously unselected package libfile-fcntllock-perl. Preparing to unpack .../40-libfile-fcntllock-perl_0.22-3build2_amd64.deb ... Unpacking libfile-fcntllock-perl (0.22-3build2) ... Selecting previously unselected package libpython2.7:amd64. Preparing to unpack .../41-libpython2.7_2.7.17-1~18.04ubuntu1_amd64.deb ... Unpacking libpython2.7:amd64 (2.7.17-1~18.04ubuntu1) ... Selecting previously unselected package libpython2.7-dev:amd64. Preparing to unpack .../42-libpython2.7-dev_2.7.17-1~18.04ubuntu1_amd64.deb ... Unpacking libpython2.7-dev:amd64 (2.7.17-1~18.04ubuntu1) ... Selecting previously unselected package libpython-dev:amd64. Preparing to unpack .../43-libpython-dev_2.7.15~rc1-1_amd64.deb ... Unpacking libpython-dev:amd64 (2.7.15~rc1-1) ... Selecting previously unselected package libpython-all-dev:amd64. Preparing to unpack .../44-libpython-all-dev_2.7.15~rc1-1_amd64.deb ... Unpacking libpython-all-dev:amd64 (2.7.15~rc1-1) ... Selecting previously unselected package manpages-dev. Preparing to unpack .../45-manpages-dev_4.15-1_all.deb ... Unpacking manpages-dev (4.15-1) ... Selecting previously unselected package python-all. Preparing to unpack .../46-python-all_2.7.15~rc1-1_amd64.deb ... Unpacking python-all (2.7.15~rc1-1) ... Selecting previously unselected package python2.7-dev. Preparing to unpack .../47-python2.7-dev_2.7.17-1~18.04ubuntu1_amd64.deb ... Unpacking python2.7-dev (2.7.17-1~18.04ubuntu1) ... Selecting previously unselected package python-dev. Preparing to unpack .../48-python-dev_2.7.15~rc1-1_amd64.deb ... Unpacking python-dev (2.7.15~rc1-1) ... Selecting previously unselected package python-all-dev. Preparing to unpack .../49-python-all-dev_2.7.15~rc1-1_amd64.deb ... Unpacking python-all-dev (2.7.15~rc1-1) ... Selecting previously unselected package python-asn1crypto. Preparing to unpack .../50-python-asn1crypto_0.24.0-1_all.deb ... Unpacking python-asn1crypto (0.24.0-1) ... Selecting previously unselected package python-cffi-backend. Preparing to unpack .../51-python-cffi-backend_1.11.5-1_amd64.deb ... Unpacking python-cffi-backend (1.11.5-1) ... Selecting previously unselected package python-crypto. Preparing to unpack .../52-python-crypto_2.6.1-8ubuntu2_amd64.deb ... Unpacking python-crypto (2.6.1-8ubuntu2) ... Selecting previously unselected package python-enum34. Preparing to unpack .../53-python-enum34_1.1.6-2_all.deb ... Unpacking python-enum34 (1.1.6-2) ... Selecting previously unselected package python-idna. Preparing to unpack .../54-python-idna_2.6-1_all.deb ... Unpacking python-idna (2.6-1) ... Selecting previously unselected package python-ipaddress. Preparing to unpack .../55-python-ipaddress_1.0.17-1_all.deb ... Unpacking python-ipaddress (1.0.17-1) ... Selecting previously unselected package python-six. Preparing to unpack .../56-python-six_1.11.0-2_all.deb ... Unpacking python-six (1.11.0-2) ... Selecting previously unselected package python-cryptography. Preparing to unpack .../57-python-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... Unpacking python-cryptography (2.1.4-1ubuntu1.3) ... Selecting previously unselected package python-dbus. Preparing to unpack .../58-python-dbus_1.2.6-1_amd64.deb ... Unpacking python-dbus (1.2.6-1) ... Selecting previously unselected package python-gi. Preparing to unpack .../59-python-gi_3.26.1-2ubuntu1_amd64.deb ... Unpacking python-gi (3.26.1-2ubuntu1) ... Selecting previously unselected package python-secretstorage. Preparing to unpack .../60-python-secretstorage_2.3.1-2_all.deb ... Unpacking python-secretstorage (2.3.1-2) ... Selecting previously unselected package python-keyring. Preparing to unpack .../61-python-keyring_10.6.0-1_all.deb ... Unpacking python-keyring (10.6.0-1) ... Selecting previously unselected package python-keyrings.alt. Preparing to unpack .../62-python-keyrings.alt_3.0-1_all.deb ... Unpacking python-keyrings.alt (3.0-1) ... Selecting previously unselected package python-pip-whl. Preparing to unpack .../63-python-pip-whl_9.0.1-2.3~ubuntu1.18.04.1_all.deb ... Unpacking python-pip-whl (9.0.1-2.3~ubuntu1.18.04.1) ... Selecting previously unselected package python-pip. Preparing to unpack .../64-python-pip_9.0.1-2.3~ubuntu1.18.04.1_all.deb ... Unpacking python-pip (9.0.1-2.3~ubuntu1.18.04.1) ... Selecting previously unselected package python-pkg-resources. Preparing to unpack .../65-python-pkg-resources_39.0.1-2_all.deb ... Unpacking python-pkg-resources (39.0.1-2) ... Selecting previously unselected package python-setuptools. Preparing to unpack .../66-python-setuptools_39.0.1-2_all.deb ... Unpacking python-setuptools (39.0.1-2) ... Selecting previously unselected package python-wheel. Preparing to unpack .../67-python-wheel_0.30.0-0.2_all.deb ... Unpacking python-wheel (0.30.0-0.2) ... Selecting previously unselected package python-xdg. Preparing to unpack .../68-python-xdg_0.25-4ubuntu1_all.deb ... Unpacking python-xdg (0.25-4ubuntu1) ... Setting up libquadmath0:amd64 (8.4.0-1ubuntu1~18.04) ... Setting up libgomp1:amd64 (8.4.0-1ubuntu1~18.04) ... Setting up libatomic1:amd64 (8.4.0-1ubuntu1~18.04) ... Setting up python-pip-whl (9.0.1-2.3~ubuntu1.18.04.1) ... Setting up libcc1-0:amd64 (8.4.0-1ubuntu1~18.04) ... Setting up make (4.1-9.1ubuntu1) ... Processing triggers for mime-support (3.60ubuntu1) ... Setting up libtsan0:amd64 (8.4.0-1ubuntu1~18.04) ... Setting up linux-libc-dev:amd64 (4.15.0-103.104) ... Setting up libdpkg-perl (1.19.0.5ubuntu2.3) ... Setting up liblsan0:amd64 (8.4.0-1ubuntu1~18.04) ... Setting up gcc-7-base:amd64 (7.5.0-3ubuntu1~18.04) ... Setting up binutils-common:amd64 (2.30-21ubuntu1~18.04.3) ... Setting up libfile-fcntllock-perl (0.22-3build2) ... Setting up libmpx2:amd64 (8.4.0-1ubuntu1~18.04) ... Processing triggers for libc-bin (2.27-3ubuntu1) ... Setting up libfakeroot:amd64 (1.22-2ubuntu1) ... Setting up libalgorithm-diff-perl (1.19.03-1) ... Processing triggers for man-db (2.8.3-2ubuntu0.1) ... Setting up libmpc3:amd64 (1.1.0-1) ... Setting up libc-dev-bin (2.27-3ubuntu1) ... Setting up manpages-dev (4.15-1) ... Setting up libc6-dev:amd64 (2.27-3ubuntu1) ... Setting up libitm1:amd64 (8.4.0-1ubuntu1~18.04) ... Setting up libpython2.7-stdlib:amd64 (2.7.17-1~18.04ubuntu1) ... Setting up libisl19:amd64 (0.19-1) ... Setting up libasan4:amd64 (7.5.0-3ubuntu1~18.04) ... Setting up libbinutils:amd64 (2.30-21ubuntu1~18.04.3) ... Setting up libcilkrts5:amd64 (7.5.0-3ubuntu1~18.04) ... Setting up libubsan0:amd64 (7.5.0-3ubuntu1~18.04) ... Setting up python2.7 (2.7.17-1~18.04ubuntu1) ... Setting up fakeroot (1.22-2ubuntu1) ... update-alternatives: using /usr/bin/fakeroot-sysv to provide /usr/bin/fakeroot (fakeroot) in auto mode Setting up libgcc-7-dev:amd64 (7.5.0-3ubuntu1~18.04) ... Setting up cpp-7 (7.5.0-3ubuntu1~18.04) ... Setting up libstdc++-7-dev:amd64 (7.5.0-3ubuntu1~18.04) ... Setting up libpython-stdlib:amd64 (2.7.15~rc1-1) ... Setting up libalgorithm-merge-perl (0.08-3) ... Setting up libalgorithm-diff-xs-perl (0.04-5) ... Setting up libpython2.7:amd64 (2.7.17-1~18.04ubuntu1) ... Setting up libexpat1-dev:amd64 (2.2.5-3ubuntu0.2) ... Setting up libpython2.7-dev:amd64 (2.7.17-1~18.04ubuntu1) ... Setting up python2.7-dev (2.7.17-1~18.04ubuntu1) ... Setting up python (2.7.15~rc1-1) ... Setting up python-xdg (0.25-4ubuntu1) ... Setting up binutils-x86-64-linux-gnu (2.30-21ubuntu1~18.04.3) ... Setting up python-idna (2.6-1) ... Setting up cpp (4:7.4.0-1ubuntu2.3) ... Setting up libpython-dev:amd64 (2.7.15~rc1-1) ... Setting up python-asn1crypto (0.24.0-1) ... Setting up python-crypto (2.6.1-8ubuntu2) ... Setting up python-dev (2.7.15~rc1-1) ... Setting up python-wheel (0.30.0-0.2) ... Setting up libpython-all-dev:amd64 (2.7.15~rc1-1) ... Setting up python-pkg-resources (39.0.1-2) ... Setting up python-cffi-backend (1.11.5-1) ... Setting up python-gi (3.26.1-2ubuntu1) ... Setting up python-six (1.11.0-2) ... Setting up python-enum34 (1.1.6-2) ... Setting up binutils (2.30-21ubuntu1~18.04.3) ... Setting up python-dbus (1.2.6-1) ... Setting up python-ipaddress (1.0.17-1) ... Setting up python-pip (9.0.1-2.3~ubuntu1.18.04.1) ... Setting up python-all (2.7.15~rc1-1) ... Setting up python-setuptools (39.0.1-2) ... Setting up gcc-7 (7.5.0-3ubuntu1~18.04) ... Setting up g++-7 (7.5.0-3ubuntu1~18.04) ... Setting up gcc (4:7.4.0-1ubuntu2.3) ... Setting up python-keyrings.alt (3.0-1) ... Setting up dpkg-dev (1.19.0.5ubuntu2.3) ... Setting up python-all-dev (2.7.15~rc1-1) ... Setting up python-cryptography (2.1.4-1ubuntu1.3) ... Setting up g++ (4:7.4.0-1ubuntu2.3) ... update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode Setting up python-secretstorage (2.3.1-2) ... Setting up python-keyring (10.6.0-1) ... Setting up build-essential (12.4ubuntu1) ... Processing triggers for libc-bin (2.27-3ubuntu1) ... [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# pip install redis Collecting redis Downloading https://files.pythonhosted.org/packages/a7/7c/24fb0511df653cf1a5d938d8f5d19802a88cef255706fdda242ff97e91b7/redis-3.5.3-py2.py3-none-any.whl (72kB) 100% |████████████████████████████████| 81kB 14kB/s Installing collected packages: redis Successfully installed redis-3.5.3 [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# vim redis_monitor.py [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# cat redis_monitor.py #!/usr/bin/python3 #coding:utf-8 import redis def redis_data_length(): pool = redis.ConnectionPool(host="172.200.5.103",port=6379,db=1,password="yinzhengjie") conn = redis.Redis(connection_pool=pool) data = conn.llen("yinzhengjie-5-102") print(data) redis_data_length() [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# python2 redis_monitor.py #运行咱们的python脚本，注意哈，生产环境中可以根据该脚本的返回值来监控redis的延迟大小哟~ 99 [root@es102.yinzhengjie.com ~]# python2 redis_monitor.py 0 [root@es102.yinzhengjie.com ~]# python2 redis_monitor.py 99 [root@es102.yinzhengjie.com ~]# python2 redis_monitor.py 94 [root@es102.yinzhengjie.com ~]# python2 redis_monitor.py 97 [root@es102.yinzhengjie.com ~]# python2 redis_monitor.py 99 [root@es102.yinzhengjie.com ~]# python2 redis_monitor.py 99 [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# python2 redis_monitor.py 0 [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# python2 redis_monitor.py 50 [root@es102.yinzhengjie.com ~]#
三.logstash结合kafka使用案例
1>.博主推荐阅读
Kafka部署实战案例: https://www.cnblogs.com/yinzhengjie2020/p/13046902.html logstash结合kafka使用案例: https://www.elastic.co/guide/en/logstash/6.8/plugins-outputs-kafka.html https://www.elastic.co/guide/en/logstash/6.8/plugins-inputs-kafka.html
2>.手动创建topic
[root@kafka201.yinzhengjie.com ~]# kafka-topics.sh --bootstrap-server kafka201.yinzhengjie.com:9092 --list __consumer_offsets yinzhengjie-kafka [root@kafka201.yinzhengjie.com ~]#
[root@kafka201.yinzhengjie.com ~]# kafka-topics.sh --bootstrap-server kafka201.yinzhengjie.com:9092 --create --replication-factor 2 --partitions 10 --topic syslog-172-200-5-103 Created topic syslog-172-200-5-103. [root@kafka201.yinzhengjie.com ~]# [root@kafka201.yinzhengjie.com ~]# kafka-topics.sh --bootstrap-server kafka201.yinzhengjie.com:9092 --create --replication-factor 2 --partitions 10 --topic nginx-172-200-5-103 Created topic nginx-172-200-5-103. [root@kafka201.yinzhengjie.com ~]#
[root@kafka201.yinzhengjie.com ~]# kafka-topics.sh --bootstrap-server kafka201.yinzhengjie.com:9092 --list __consumer_offsets nginx-172-200-5-103 syslog-172-200-5-103 yinzhengjie-kafka [root@kafka201.yinzhengjie.com ~]#
3>.准备源数据
[root@es103.yinzhengjie.com ~]# ll /var/log/syslog -rw-r----- 1 syslog adm 481835 Jun 18 04:10 /var/log/syslog [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# chmod 644 /var/log/syslog [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# ll /var/log/syslog -rw-r--r-- 1 syslog adm 481835 Jun 18 04:10 /var/log/syslog [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# ll /var/log/nginx/access.log -rw-r----- 1 www-data adm 0 Jun 6 06:25 /var/log/nginx/access.log [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# chmod 644 /var/log/nginx/access.log [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# ll /var/log/nginx/access.log -rw-r--r-- 1 www-data adm 0 Jun 6 06:25 /var/log/nginx/access.log [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# systemctl start nginx [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 0.0.0.0:80 0.0.0.0:* LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 [::]:80 [::]:* LISTEN 0 128 [::]:22 [::]:* LISTEN 0 50 [::ffff:127.0.0.1]:9600 *:* [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# curl -I es103.yinzhengjie.com HTTP/1.1 200 OK Server: nginx/1.14.0 (Ubuntu) Date: Thu, 18 Jun 2020 04:15:20 GMT Content-Type: text/html; charset=utf-8 Content-Length: 612 Last-Modified: Fri, 05 Jun 2020 01:43:41 GMT Connection: keep-alive ETag: "5ed9a34d-264" Accept-Ranges: bytes [root@es103.yinzhengjie.com ~]#
4>.将源数据写入kafka集群
[root@es103.yinzhengjie.com ~]# vim /etc/logstash/conf.d/log-to-kafka.conf [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# cat /etc/logstash/conf.d/log-to-kafka.conf input { file { type => "syslog-172-200-5-103" path => "/var/log/syslog" start_position => "beginning" stat_interval => 3 } file { type => "nginx-172-200-5-103" path => "/var/log/nginx/access.log" start_position => "beginning" stat_interval => 3 codec => "json" } } output { if [type] == "syslog-172-200-5-103" { kafka { topic_id => "syslog-172-200-5-103" bootstrap_servers => "kafka201.yinzhengjie.com:9092,kafka202.yinzhengjie.com:9092,kafka203.yinzhengjie.com:9092" #注意哈，虽然syslog的日志不是json格式，但我们写入kafka集群时统一以json格式写入，这样便于从kafka取数据时以json进行解析。 codec => "json" } } if [type] == "nginx-172-200-5-103" { kafka { topic_id => "nginx-172-200-5-103" bootstrap_servers => "kafka201.yinzhengjie.com:9092,kafka202.yinzhengjie.com:9092,kafka203.yinzhengjie.com:9092" codec => "json" } } } [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# systemctl start logstash [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# systemctl status logstash ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: enabled) Active: active (running) since Thu 2020-06-18 04:12:44 UTC; 6s ago Main PID: 2617 (java) Tasks: 14 (limit: 2290) CGroup: /system.slice/logstash.service └─2617 /usr/local/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding Jun 18 04:12:44 es103.yinzhengjie.com systemd[1]: Started logstash. [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# tail -100f /var/log/logstash/logstash-plain.log sasl.login.refresh.window.jitter = 0.05 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT send.buffer.bytes = 131072 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm = https ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS transaction.timeout.ms = 60000 transactional.id = null value.serializer = class org.apache.kafka.common.serialization.StringSerializer [2020-06-18T04:13:11,515][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka version : 2.1.0 [2020-06-18T04:13:11,515][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka commitId : eec43959745f444f [2020-06-18T04:13:11,521][INFO ][org.apache.kafka.clients.producer.ProducerConfig] ProducerConfig values: acks = 1 batch.size = 16384 bootstrap.servers = [kafka201.yinzhengjie.com:9092, kafka202.yinzhengjie.com:9092, kafka203.yinzhengjie.com:9092] buffer.memory = 33554432 client.dns.lookup = default client.id = compression.type = none connections.max.idle.ms = 540000 delivery.timeout.ms = 120000 enable.idempotence = false interceptor.classes = [] key.serializer = class org.apache.kafka.common.serialization.StringSerializer linger.ms = 0 max.block.ms = 60000 max.in.flight.requests.per.connection = 5 max.request.size = 1048576 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner receive.buffer.bytes = 32768 reconnect.backoff.max.ms = 10 reconnect.backoff.ms = 10 request.timeout.ms = 30000 retries = 2147483647 retry.backoff.ms = 100 sasl.client.callback.handler.class = null sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT send.buffer.bytes = 131072 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm = https ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS transaction.timeout.ms = 60000 transactional.id = null value.serializer = class org.apache.kafka.common.serialization.StringSerializer [2020-06-18T04:13:11,528][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka version : 2.1.0 [2020-06-18T04:13:11,528][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka commitId : eec43959745f444f [2020-06-18T04:13:11,798][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_f 5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[2020-06-18T04:13:11,826][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_d 883144359d3b4f516b37dba51fab2a2", :path=>["/var/log/nginx/access.log"]}[2020-06-18T04:13:11,852][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x6a3df8ab run>"} [2020-06-18T04:13:11,949][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections [2020-06-18T04:13:11,950][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections [2020-06-18T04:13:11,967][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2020-06-18T04:13:12,364][WARN ][filewatch.tailmode.handlers.createinitial] failed to open /var/log/nginx/access.log: #<Errno::EACCES: Permission denied - /var/log/nginx/access.log>, ["org/ jruby/RubyIO.java:1236:in `sysopen'", "org/jruby/RubyFile.java:367:in `initialize'", "org/jruby/RubyIO.java:1155:in `open'"][2020-06-18T04:13:12,503][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2020-06-18T04:13:13,027][INFO ][org.apache.kafka.clients.Metadata] Cluster ID: aDBW0CNdQeC8lxYFd77sxg
[root@kafka201.yinzhengjie.com ~]# kafka-console-consumer.sh --bootstrap-server kafka201.yinzhengjie.com:9092 --topic syslog-172-200-5-103 ...... 2020-06-18T04:13:16.214Z es103.yinzhengjie.com Jun 18 04:13:11 es103 logstash[2617]: #011sasl.login.refresh.window.jitter = 0.05 2020-06-18T04:13:16.305Z es103.yinzhengjie.com Jun 18 04:13:11 es103 logstash[2617]: #011ssl.keystore.password = null 2020-06-18T04:13:16.307Z es103.yinzhengjie.com Jun 18 04:13:11 es103 logstash[2617]: #011transactional.id = null 2020-06-18T04:13:16.214Z es103.yinzhengjie.com Jun 18 04:13:11 es103 logstash[2617]: #011send.buffer.bytes = 131072 2020-06-18T04:13:16.306Z es103.yinzhengjie.com Jun 18 04:13:11 es103 logstash[2617]: #011ssl.provider = null 2020-06-18T04:13:16.307Z es103.yinzhengjie.com Jun 18 04:13:11 es103 logstash[2617]: [2020-06-18T04:13:11,528][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka commitId : eec439597 45f444f2020-06-18T04:13:16.214Z es103.yinzhengjie.com Jun 18 04:13:11 es103 logstash[2617]: #011ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] 2020-06-18T04:13:16.306Z es103.yinzhengjie.com Jun 18 04:13:11 es103 logstash[2617]: #011ssl.trustmanager.algorithm = PKIX 2020-06-18T04:13:16.308Z es103.yinzhengjie.com Jun 18 04:13:11 es103 logstash[2617]: [2020-06-18T04:13:11,826][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_d883144359d3b4f516b37dba51fab2a2", :path=>["/var/log/nginx/access.log"]}2020-06-18T04:13:16.214Z es103.yinzhengjie.com Jun 18 04:13:11 es103 logstash[2617]: #011ssl.cipher.suites = null 2020-06-18T04:13:16.306Z es103.yinzhengjie.com Jun 18 04:13:11 es103 logstash[2617]: #011ssl.secure.random.implementation = null 2020-06-18T04:13:16.308Z es103.yinzhengjie.com Jun 18 04:13:11 es103 logstash[2617]: [2020-06-18T04:13:11,798][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_f5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}2020-06-18T04:13:16.214Z es103.yinzhengjie.com Jun 18 04:13:11 es103 logstash[2617]: #011security.protocol = PLAINTEXT 2020-06-18T04:13:16.306Z es103.yinzhengjie.com Jun 18 04:13:11 es103 logstash[2617]: #011ssl.protocol = TLS 2020-06-18T04:13:16.307Z es103.yinzhengjie.com Jun 18 04:13:11 es103 logstash[2617]: [2020-06-18T04:13:11,528][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka version : 2.1.0 2020-06-18T04:13:19.320Z es103.yinzhengjie.com Jun 18 04:13:12 es103 logstash[2617]: [2020-06-18T04:13:12,364][WARN ][filewatch.tailmode.handlers.createinitial] failed to open /var/log/ngin x/access.log: #<Errno::EACCES: Permission denied - /var/log/nginx/access.log>, ["org/jruby/RubyIO.java:1236:in `sysopen'", "org/jruby/RubyFile.java:367:in `initialize'", "org/jruby/RubyIO.java:1155:in `open'"]2020-06-18T04:13:19.320Z es103.yinzhengjie.com Jun 18 04:13:13 es103 logstash[2617]: [2020-06-18T04:13:13,027][INFO ][org.apache.kafka.clients.Metadata] Cluster ID: aDBW0CNdQeC8lxYFd77sxg 2020-06-18T04:13:19.320Z es103.yinzhengjie.com Jun 18 04:13:12 es103 logstash[2617]: [2020-06-18T04:13:12,503][INFO ][logstash.agent ] Successfully started Logstash API endpoint { :port=>9600}2020-06-18T04:15:04.376Z es103.yinzhengjie.com Jun 18 04:15:03 es103 systemd[1]: Starting A high performance web server and a reverse proxy server... 2020-06-18T04:15:04.376Z es103.yinzhengjie.com Jun 18 04:15:04 es103 systemd[1]: Started A high performance web server and a reverse proxy server. 2020-06-18T04:15:22.386Z es103.yinzhengjie.com Jun 18 04:15:21 es103 logstash[2617]: [2020-06-18T04:15:21,752][INFO ][org.apache.kafka.clients.Metadata] Cluster ID: aDBW0CNdQeC8lxYFd77sxg 2020-06-18T04:17:04.435Z es103.yinzhengjie.com Jun 18 04:17:01 es103 CRON[2818]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
[root@kafka201.yinzhengjie.com ~]# kafka-console-consumer.sh --bootstrap-server kafka201.yinzhengjie.com:9092 --topic nginx-172-200-5-103 {"@timestamp":"2020-06-18T04:15:20.000Z","xff":"-","type":"nginx-172-200-5-103","responsetime":0.0,"upstreamtime":"-","http_user_agent":"curl/7.58.0","host":"172.200.5.103","domain":"es103. yinzhengjie.com","http_host":"es103.yinzhengjie.com","referer":"-","status":"200","upstreamhost":"-","@version":"1","path":"/var/log/nginx/access.log","clientip":"172.200.5.103","uri":"/index.nginx-debian.html","size":0,"tcp_xff":""}
5>.将kafka集群的数据写入到elasticsearch集群
[root@es103.yinzhengjie.com ~]# ll /etc/logstash/conf.d/ total 16 drwxrwxr-x 2 root root 4096 Jun 18 06:19 ./ drwxrwxr-x 3 root root 4096 Jun 18 03:22 ../ -rw-r--r-- 1 root root 955 Jun 18 06:19 kafka-to-elasticsearch.conf -rw-r--r-- 1 root root 1095 Jun 18 06:08 log-to-kafka.conf [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# systemctl restart logstash #编写好配置文件后记得重启logstash服务哟~ [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# vim /etc/logstash/conf.d/log-to-kafka.conf [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# cat /etc/logstash/conf.d/log-to-kafka.conf input { file { type => "syslog-172-200-5-103" path => "/var/log/syslog" start_position => "beginning" stat_interval => 3 } file { type => "nginx-172-200-5-103" path => "/var/log/nginx/access.log" start_position => "beginning" stat_interval => 3 codec => "json" } } output { if [type] == "syslog-172-200-5-103" { kafka { topic_id => "syslog-172-200-5-103" bootstrap_servers => "kafka201.yinzhengjie.com:9092,kafka202.yinzhengjie.com:9092,kafka203.yinzhengjie.com:9092" #注意哈，虽然syslog的日志不是json格式，但我们写入kafka集群时统一以json格式写入，这样便于从kafka取数据时以json进行解析。 codec => "json" } } if [type] == "nginx-172-200-5-103" { kafka { topic_id => "nginx-172-200-5-103" bootstrap_servers => "kafka201.yinzhengjie.com:9092,kafka202.yinzhengjie.com:9092,kafka203.yinzhengjie.com:9092" codec => "json" } } } [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# vim /etc/logstash/conf.d/kafka-to-elasticsearch.conf [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# cat /etc/logstash/conf.d/kafka-to-elasticsearch.conf input { kafka { #如果有多个topic需要进行消费则可以将其写在同一个数组中,当然你也可以拆开写哟~ topics => ["nginx-172-200-5-103","syslog-172-200-5-103"] bootstrap_servers => "kafka201.yinzhengjie.com:9092,kafka202.yinzhengjie.com:9092,kafka203.yinzhengjie.com:9092" codec => "json" group_id => "yinzhengjie-logstash" decorate_events => true consumer_threads => 3 } } output{ if [type] == "nginx-172-200-5-103" { elasticsearch { hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200"] index => "kafka-nginx-172.200.5.103-%{+YYYY.MM.dd}" } } if [type] == "syslog-172-200-5-103" { elasticsearch { hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200"] index => "kafka-syslog-172.200.5.103-%{+YYYY.ww}" } } } [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# tail -100f /var/log/logstash/logstash-plain.log ssl.truststore.password = null ssl.truststore.type = JKS value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer [2020-06-18T06:21:52,838][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka version : 2.1.0 [2020-06-18T06:21:52,838][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka commitId : eec43959745f444f [2020-06-18T06:21:52,839][INFO ][org.apache.kafka.clients.consumer.ConsumerConfig] ConsumerConfig values: auto.commit.interval.ms = 5000 auto.offset.reset = latest bootstrap.servers = [kafka201.yinzhengjie.com:9092, kafka202.yinzhengjie.com:9092, kafka203.yinzhengjie.com:9092] check.crcs = true client.dns.lookup = default client.id = logstash-2 connections.max.idle.ms = 540000 default.api.timeout.ms = 60000 enable.auto.commit = true exclude.internal.topics = true fetch.max.bytes = 52428800 fetch.max.wait.ms = 500 fetch.min.bytes = 1 group.id = yinzhengjie-logstash heartbeat.interval.ms = 3000 interceptor.classes = [] internal.leave.group.on.close = true isolation.level = read_uncommitted key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer max.partition.fetch.bytes = 1048576 max.poll.interval.ms = 300000 max.poll.records = 500 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor] receive.buffer.bytes = 65536 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 30000 retry.backoff.ms = 100 sasl.client.callback.handler.class = null sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT send.buffer.bytes = 131072 session.timeout.ms = 10000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm = https ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer [2020-06-18T06:21:52,843][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka version : 2.1.0 [2020-06-18T06:21:52,843][INFO ][org.apache.kafka.common.utils.AppInfoParser] Kafka commitId : eec43959745f444f [2020-06-18T06:21:52,846][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections [2020-06-18T06:21:52,854][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections [2020-06-18T06:21:53,050][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2020-06-18T06:21:53,115][INFO ][org.apache.kafka.clients.Metadata] Cluster ID: aDBW0CNdQeC8lxYFd77sxg [2020-06-18T06:21:53,121][INFO ][org.apache.kafka.clients.Metadata] Cluster ID: aDBW0CNdQeC8lxYFd77sxg [2020-06-18T06:21:53,124][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-0, groupId=yinzhengjie-logstash] Discovered group coordinator k afka203.yinzhengjie.com:9092 (id: 2147483444 rack: null)[2020-06-18T06:21:53,130][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-2, groupId=yinzhengjie-logstash] Discovered group coordinator k afka203.yinzhengjie.com:9092 (id: 2147483444 rack: null)[2020-06-18T06:21:53,126][INFO ][org.apache.kafka.clients.Metadata] Cluster ID: aDBW0CNdQeC8lxYFd77sxg [2020-06-18T06:21:53,138][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-1, groupId=yinzhengjie-logstash] Discovered group coordinator k afka203.yinzhengjie.com:9092 (id: 2147483444 rack: null)[2020-06-18T06:21:53,149][INFO ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] [Consumer clientId=logstash-0, groupId=yinzhengjie-logstash] Revoking previously assigned p artitions [][2020-06-18T06:21:53,149][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-0, groupId=yinzhengjie-logstash] (Re-)joining group [2020-06-18T06:21:53,150][INFO ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] [Consumer clientId=logstash-1, groupId=yinzhengjie-logstash] Revoking previously assigned p artitions [][2020-06-18T06:21:53,151][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-1, groupId=yinzhengjie-logstash] (Re-)joining group [2020-06-18T06:21:53,170][INFO ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] [Consumer clientId=logstash-2, groupId=yinzhengjie-logstash] Revoking previously assigned p artitions [][2020-06-18T06:21:53,170][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-2, groupId=yinzhengjie-logstash] (Re-)joining group [2020-06-18T06:21:53,672][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2020-06-18T06:21:54,740][INFO ][org.apache.kafka.clients.Metadata] Cluster ID: aDBW0CNdQeC8lxYFd77sxg [2020-06-18T06:21:56,184][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-1, groupId=yinzhengjie-logstash] Successfully joined group with generation 11[2020-06-18T06:21:56,184][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-0, groupId=yinzhengjie-logstash] Successfully joined group with generation 11[2020-06-18T06:21:56,185][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-2, groupId=yinzhengjie-logstash] Successfully joined group with generation 11[2020-06-18T06:21:56,191][INFO ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] [Consumer clientId=logstash-0, groupId=yinzhengjie-logstash] Setting newly assigned partiti ons [syslog-172-200-5-103-0, syslog-172-200-5-103-1, syslog-172-200-5-103-2, syslog-172-200-5-103-3, nginx-172-200-5-103-1, nginx-172-200-5-103-2, nginx-172-200-5-103-0, nginx-172-200-5-103-3][2020-06-18T06:21:56,192][INFO ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] [Consumer clientId=logstash-2, groupId=yinzhengjie-logstash] Setting newly assigned partiti ons [nginx-172-200-5-103-9, nginx-172-200-5-103-7, nginx-172-200-5-103-8, syslog-172-200-5-103-7, syslog-172-200-5-103-8, syslog-172-200-5-103-9][2020-06-18T06:21:56,191][INFO ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] [Consumer clientId=logstash-1, groupId=yinzhengjie-logstash] Setting newly assigned partiti ons [syslog-172-200-5-103-4, syslog-172-200-5-103-5, syslog-172-200-5-103-6, nginx-172-200-5-103-5, nginx-172-200-5-103-6, nginx-172-200-5-103-4][2020-06-18T06:22:08,609][INFO ][org.apache.kafka.clients.Metadata] Cluster ID: aDBW0CNdQeC8lxYFd77sxg [2020-06-18T06:25:05,848][WARN ][filewatch.tailmode.handlers.create] failed to open /var/log/nginx/access.log: #<Errno::EACCES: Permission denied - /var/log/nginx/access.log>, ["org/jruby/R ubyI
