实战一：filebeat收集日志到redis再由logstash转存到elasticsearch主机
框架图：
环境准备：
A主机：elasticsearch/kibana IP地址：192.168.7.100
B主机：logstash IP地址：192.168.7.102
C主机：filebeat/nginx IP地址：192.168.7.103
D主机： redis IP地址： 192.168.7.104
1、filebeat收集系统和nginx日志到redis主机
1.1、安装redis服务，并修改配置
1、安装redis服务
# yum install redis -y
2、修改redis配置文件，修改监听地址和密码
[root@web1 ~]# vim /etc/redis.conf bind 0.0.0.0 requirepasswd 123456
3、启动redis服务
# systemctl start redis
1.2、修改filebeat主机配置，将日志存到redis服务器上
1、修改filebeat主机配置文件，将日志存储到redis服务器上
[root@filebate tmp]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/messages
fields:
host: "192.168.7.103"
type: "filebeat-syslog-7-103"
app: "syslog"
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
fields:
host: "192.168.7.103"
type: "filebeat-nginx-accesslog-7-103"
app: "nginx"
output.redis:
hosts: ["192.168.7.104"] # 写入redis服务器主机IP地址
port: 6379 # redis监听的端口号
password: "123456" # redis密码
key: "filebeat-log-7-103" # 自定义的key
db: 0 # 选择默认的数据库
timeout: 5 #超时时长，可以修改再大点
2、查看filebeat的关键信息
[root@filebate tmp]# grep -v "#" /etc/filebeat/filebeat.yml | grep -v "^$"
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/messages
fields:
host: "192.168.7.103"
type: "filebeat-syslog-7-103"
app: "syslog"
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
fields:
host: "192.168.7.103"
type: "filebeat-nginx-accesslog-7-103"
app: "nginx"
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
output.redis:
hosts: ["192.168.7.104"]
port: 6379
password: "123456"
key: "filebeat-log-7-103"
db: 0
timeout: 5
3、启动filebeat服务
# systemctl restart filebeat
2、在redis主机上测试验证数据
1、登陆redis客户端查看数据，此时可以看到对应的key值已经到达，说明数据可以到达redis服务器。
[root@web1 ~]# redis-cli -h 192.168.7.104 192.168.7.104:6379> auth 123456 OK 192.168.7.104:6379> KEYS * 1) "filebeat-log-7-103" 192.168.7.104:6379>
3、在logstash收集redis服务器的日志
1、修改logstash配置文件，收集redis日志
[root@logstash conf.d]# vim logstash-to-es.conf
input {
redis {
host => "192.168.7.104" # redis主机的IP地址
port => "6379" # 端口
db => "0" # 与filebeat对应的数据库
password => "123456" #密码
data_type => "list" # 日志类型
key => "filebeat-log-7-103" # 与filebeat对应的key值
codec => "json"
}
}
output {
if [fields][app] == "syslog" { # 与filebeat主机的app类型一致
elasticsearch {
hosts => ["192.168.7.100:9200"] # 日志转到elasticsearch主机上
index => "logstash-syslog-7-103-%{+YYYY.MM.dd}"
}}
if [fields][app] == "nginx" { # 与filebeat主机的app类型一致
elasticsearch {
hosts => ["192.168.7.100:9200"]
index => "logstash-nginx-accesslog-7-103-%{+YYYY.MM.dd}"
}}
}
检查语法是否存在问题，如果不存在问题就启动服务
[root@logstash conf.d]# logstash -f logstash-to-es.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-03-16 10:05:05.487 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK # 检查语法正确 [INFO ] 2020-03-16 10:05:16.597 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
2、重启logstash服务器
# systemctl restart logstash
3、查看head插件收集的日志名称，此时就可以看到日志提取到的信息
4、在kibana网页上创建索引
1、在kibana网页上创建Nginx日志索引，同理系统日志也可以这样创建
2、在discover查看提取到的nginx日志数据
3、查看收集到的系统日志
实战二：logstash结合filebeat收集到redis日志，并转存到elasticsearch主机
框架图：
环境准备：
这里没有太多测试主机，都是以单机形式测试，生产环境可以按上面的部署
A主机：elasticsearch/kibana IP地址：192.168.7.100
B主机：logstash-A IP地址：192.168.7.102
C主机：filebeat/nginx IP地址：192.168.7.103
D主机： redis IP地址： 192.168.7.104
E主机： logstash-B IP地址：192.168.7.101
1、安装并配置filebeat主机
1、安装filebeat包，这里需要在官网上下载包
[root@filebeat-1 ~]# yum install filebeat-6.8.1-x86_64.rpm -y
2、修改filebeat配置文件，将日志由filebeat传递到第一个logstash主机上，如果有多个filebeat对多个logstash主机进行转存日志，可以在output.logstash配置段，写入不同的logstash主机的IP地址
[root@filebate ~]# grep -v "#" /etc/filebeat/filebeat.yml | grep -v "^$"
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/messages
fields:
host: "192.168.7.103"
type: "filebeat-syslog-7-103"
app: "syslog"
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
fields:
host: "192.168.7.103" # 指定本机的IP地址
type: "filebeat-nginx-accesslog-7-103"
app: "nginx"
output.logstash:
hosts: ["192.168.7.101:5044"] # 写到指定的logstash服务器上,如果有多个filebeat主机传递到不同logstash主机时，可以在另一个filebeat主机上写上另一个logstash主机的IP地址
enabled: true # 是否传递到logstash服务器，默认是开启
work: 1 # 工作线程
compression_level: 3 # 压缩等级
3、重启filebeat服务
# systemctl restart filebeat
2、修改logstash-B主机，将日志存储到redis服务器上
1、在/etc/logstash/conf.d/目录下创建一个存储到redis日志的配置文件，如果有多个filebeat、logstash和redis，可以分别对redis主机进行存储日志，减少logstash压力
[root@logstash-1 conf.d]# cat filebeat-to-logstash.conf
input {
beats {
host => "192.168.7.101" # logstash主机的IP地址，如果还有其他logstash主机转存到redis主机上，可以在另一台logstash主机上写入对应本机的IP地址，分担logstash主机的压力
port => 5044 # 端口号
codec => "json"
}
}
output {
if [fields][app] == "syslog" {
redis {
host => "192.168.7.104" # 存储到redis服务器地址
port => "6379"
db => "0"
data_type => "list"
password => "123456"
key => "filebeat-syslog-7-103" #定义不同日志的key，方便区分
codec => "json"
}}
if [fields][app] == "nginx" {
redis {
host => "192.168.7.104"
port => "6379"
db => "0"
data_type => "list"
password => "123456"
key => "filebeat-nginx-log-7-103" # 定义不同的key，方便分析
codec => "json"
}
}
}
2、对logstash主机进行测试
[root@logstash-1 conf.d]# logstash -f filebeat-to-logstash.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-03-16 11:23:31.687 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK # 测试配置文件正常
重新启动logstash服务
# systemctl restart logstash
3、此时在redis主机上可以看到两个key值，说明logstash主机已经将日志存到redis主机上
[root@web1 ~]# redis-cli -h 192.168.7.104 192.168.7.104:6379> auth 123456 OK 192.168.7.104:6379> KEYS * 1) "filebeat-nginx-log-7-103" 2) "filebeat-syslog-7-103"
3、在logstash-A主机上配置提取redis的日志并转存到elasticsearch主机上
1、在logstash主机的/etc/logstash/conf.d目录下创建一个提取redis主机的日志
[root@logstash conf.d]# cat logstash-to-es.conf
input {
redis {
host => "192.168.7.104" # redis主机的IP地址
port => "6379"
db => "0"
password => "123456"
data_type => "list"
key => "filebeat-syslog-7-103" # 写入对应的filebeat的key值
codec => "json"
}
redis {
host => "192.168.7.104" # redis主机的IP地址
port => "6379"
db => "0"
password => "123456"
data_type => "list"
key => "filebeat-nginx-log-7-103" # 针对filebeat写入的key值，
codec => "json"
}
}
output {
if [fields][app] == "syslog" { # 对应filebeat主机的app类型
elasticsearch {
hosts => ["192.168.7.100:9200"] # elasticsearch主机IP地址
index => "logstash-syslog-7-103-%{+YYYY.MM.dd}"
}}
if [fields][app] == "nginx" { # 对应filebeat主机的app类型
elasticsearch {
hosts => ["192.168.7.100:9200"]
index => "logstash-nginx-accesslog-7-103-%{+YYYY.MM.dd}"
}}
}
2、测试logstash主机的配置文件
[root@logstash conf.d]# logstash -f logstash-to-es.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-03-16 11:31:30.943 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK
3、重启logstash主机服务
# systemctl restart logstash
4、在head插件查看获取到的系统日志和nginx日志
4、在kibana创建索引，查看收集到的日志信息
1、创建Nginx索引，系统日志索引同理
2、查看创建的索引信息
