ELK之部署 logstash

Logstash 是一个开源的数据收集引擎，可以水平伸缩，而且 logstash 整个 ELK
当中拥有最多插件的一个组件，其可以接收来自不同来源的数据并统一输出到指
定的且可以是多个不同目的地。

https://github.com/elastic/logstash
https://baike.baidu.com/item/Ruby/11419 #基于 ruby 开发

安装jdk8

[root@logstash1 ~]# apt install openjdk-8-jdk -y

使用dpkg安装

[root@logstash1 src]# dpkg -i logstash-7.12.1-amd64.deb

启动

[root@logstash1 src]# systemctl start logstash.service

查看状态

[root@logstash1 src]# systemctl status logstash.service

测试

[root@logstash1 src]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug }}'

123 # 手动输入
# 返回数据，即可
{
          "host" => "logstash1.example.local",
       "message" => "123",
      "@version" => "1",
    "@timestamp" => 2021-08-24T06:48:28.756Z
}

测试输出到 elasticsearch：

[root@logstash1 ~]# /usr/share/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch {hosts => ["172.31.2.101:9200"] index => "mytest-%{+YYYY.MM.dd}"                       }}'

收集日志

通过 logstash 收集日志

收集单个系统日志并输出至文件

[root@logstash1 ~]# vim /etc/logstash/conf.d/system.conf

input {
    stdin {
       type => stdin
   }
}
output {
    file {
       path => "/tmp/linux104.log"
   }
}

执行测试语法

[root@logstash1 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/system.conf -t

重启

[root@logstash1 ~]# systemctl restart logstash.service

或者执行

[root@logstash1 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/system.conf

查看

[root@logstash1 ~]# ll /tmp/linux104.log

-rw-r--r-- 1 root root 121 Aug 24 16:40 /tmp/linux104.log

写入es

[root@logstash1 ~]# vim /etc/logstash/conf.d/system.con

input {
    file {
       path => "/var/log/syslog"
   }
}

output {
    elasticsearch {
       hosts => ["172.31.2.102:9200","172.31.2.103:9200"]
       index => "long-system-syslog-%{+YYYY.MM.dd}"
   }
}

重启

[root@logstash1 ~]# systemctl restart logstash.service

或者执行

[root@logstash1 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/system.conf

把收集多个日志一起写入es集群

[root@logstash1 ~]# cat /etc/logstash/conf.d/es-muber-test.conf

input {
   file {
       path => "/var/log/syslog"
       start_position => "beginning"
       stat_interval => 3
       type => "syslog"
   }

   file {
     path => "/var/log/bootstrap.log"
     start_position => "beginning"
     stat_interval => 3
     type => "bootstrap"
   }
}

output {
    if [type] == "syslog"{
    elasticsearch {
       hosts => ["172.31.2.101:9200","172.31.2.102:9200"]
       index => "long-system-syslog-%{+YYYY.MM.dd}"
   }}

   if [type] == "bootstrap"{
     elasticsearch {
        hosts => ["172.31.2.101:9200"]
        index => "long-bootstrap-log-%{+YYYY.MM.dd}"
   }}
}

重启

[root@logstash1 ~]# systemctl restart logstash.service

或者执行

[root@logstash1 ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/es-muber-test.conf

把index索引添加到kibana