logstash部署及项目日志输出到ES

目录

• logstash简介
• 安装logstash
  • logstash的基本语法
  • 测试标准输入输出
  • 测试输出到文件
  • 测试输出到ES
    • 指定配置文件启动
    • 配置文件内容
    • 后台运行脚本
    • 参考

logstash简介

logstash是一个收集日志的组件可以水平伸缩，而且logstash是整个ELK当中拥有最多插件的一个组件，其可以接收来自不同源的数据并统一输入到指定的且可以是不同目的地。

logstash收集日志基本流程: input-->codec-->filter-->codec-->output

1.input:从哪里收集日志。

2.filter:发出去前进行过滤

3.output:输出至Elasticsearch或Redis消息队列

4.codec:输出至前台，方便边实践边测试

5.数据量不大日志按照月来进行收集

其主要优势是含有丰富的输入和输出格式的支持，其配置格式主要是以下情况：

input { stdin {} } output { elasticsearch { hosts => ["192.168.56.11:9200"] index => "logstash-test-%{+YYYY.MM.dd}" } }

• input 代表输入源，stdin代表控制台输入
• output 代表输出源

详细配置语法可自行查看博文或者官网

安装logstash

环境准备：关闭防火墙和Selinux，并且安装java环境
logstash下载地址：https://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.rpm
[root@linux-node1 ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.rpm
[root@linux-node1 ~]# yum install -y logstash-6.0.0.rpm 
[root@linux-node1 ~]# rpm -ql logstash
#node2节点安装logstash
[root@linux-node2 ~]# yum install -y logstash-6.0.0.rpm 
[root@linux-node1 ~]# ll /etc/logstash/conf.d/     #logstash的主配置目录
总用量 0

logstash的基本语法

input {
        指定输入
}
output {
        指定输出
}

测试标准输入输出

[root@linux-node1 ~]# /usr/share/logstash/bin/logstash -e 'input { stdin {} } output { stdout { codec => rubydebug} }'      #标准输入输出
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
The stdin plugin is now waiting for input:
hello  #输入
{
      "@version" => "1",              #@version时间版本号，一个事件就是一个ruby对象
          "host" => "linux-node1",       #host标记事件发生在哪里
    "@timestamp" => 2017-12-08T14:56:25.395Z,      #@timestamp，用来标记当前事件发生的时间
       "message" => "hello"       #消息的具体内容
}

测试输出到文件

[root@linux-node1 ~]# /usr/share/logstash/bin/logstash -e 'input { stdin {} } output { file { path => "/tmp/test-%{+YYYY.MM.dd}.log"} }'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
hello
[root@linux-node1 ~]# cat /tmp/test-2017.12.09.log 
{"@version":"1","host":"linux-node1","@timestamp":"2017-12-09T08:23:14.896Z","message":"hello"}
开启gzip压缩输出
[root@linux-node1 ~]# /usr/share/logstash/bin/logstash -e 'input { stdin {} } outpu{ file { path => "/tmp/test-%{+YYYY.MM.dd}.log.tar.gz" gzip => true } }'
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
hello
[root@linux-node1 ~]# ll /tmp/test-2017.12.09.log.tar.gz 
-rw-r--r-- 1 root root 105 12月  9 16:26 /tmp/test-2017.12.09.log.tar.gz

测试输出到ES

/usr/share/logstash/bin/logstash -e 'input { stdin {} } output { elasticsearch { hosts => ["192.168.56.11:9200"] index => "logstash-test-%{+YYYY.MM.dd}" } }'

指定配置文件启动

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf -t

配置文件内容

input {
        file{
                path => "/export/logs/gateway/gateway-provider.%{+YYYY-MM-dd}"
                type => "elasticsearch-java-log"
                start_position => "beginning"
                stat_interval => "2"
                codec => multiline {
                        pattern => "^["    #以"["开头进行正则匹配
                        negate => "true"  #正则匹配成功
                        what => "previous"  #和前面的内容进行合并
                }
        }
}
output {
        if [type] == "elasticsearch-java-log" {
                elasticsearch {
                        hosts => ["10.159.42.37:9200"]
                        index => "gateway-log-%{+YYYY.MM.dd}"
                }
        }
}

后台运行脚本

nohup /usr/local/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf  -w 8 -b 1000 > /dev/null 2>&1 &

参考

https://blog.51cto.com/jinlong/2055424

https://blog.51cto.com/jinlong/2055024

https://blog.51cto.com/jinlong/2056598

https://yq.aliyun.com/articles/604138

https://blog.csdn.net/ljx1528/article/details/100031330

https://my.oschina.net/wangmengjun/blog/861636

https://blog.csdn.net/weixin_34306593/article/details/93020544