先贴一下我的BELK架构
1、Download and install the Public Signing Key:
# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
2、You may need to install the apt-transport-https package on Debian before proceeding:
# aptitude install -y apt-transport-https
3、Save the repository definition to /etc/apt/sources.list.d/elastic-5.x.list:
# echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-5.x.list
4、Run aptitude update, and the repository is ready for use. For example, you can install Filebeat by running:
# aptitude update
# aptitude install -y filebeat
5、To configure the Beat to start automatically during boot, run:
# update-rc.d filebeat defaults 95 10
6、为nginx添加json日志格式
# vim /usr/local/nginx/conf/nginx.conf
log_format json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",'
'"remote_user":"$remote_user",'
'"request":"$request",'
'"http_user_agent":"$http_user_agent",'
'"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"url":"$uri",'
'"domain":"$host",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"status":"$status"}';
access_log /var/log/nginx/access.log json;
7、重载nginx服务
# /usr/local/nginx/sbin/nginx -t
# /usr/local/nginx/sbin/nginx -s reload
8、修改filebeat的配置文件。如果同一台机器上要收集多个日志文件，而且每个日志要输出到不同的索引，那么可以把每个prospector单独定义一个document_type，然后在logstash上通过 if 判断输出到不同的索引。
# vim /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
paths:
- /usr/local/nginx/logs/zixun.oupeng.com.access.log
document_type: zixun-nginx-access
- input_type: log
paths:
- /usr/local/nginx/logs/water.oupeng.com.access.log
document_type: water-nginx-access
- input_type: log
paths:
- /usr/local/nginx/logs/nav.oupeng.com.access.log
document_type: nav-nginx-access
- input_type: log
paths:
- /usr/local/nginx/logs/wood.oupeng.com.access.log
document_type: wood-nginx-access
- input_type: log
paths:
- /usr/local/nginx/logs/redir.oupeng.com.access.log
document_type: redir-nginx-access
- input_type: log
paths:
- /usr/local/nginx/logs/default.access.log
document_type: default-nginx-access
- input_type: log
paths:
- /usr/local/nginx/logs/kibana.oupeng.com.access.log
document_type: kibana-nginx-access
output.logstash:
hosts: ["192.168.3.56:5044","192.168.3.49:5044","192.168.3.57:5044"]
loadbalance: true
9、启动filebeat服务
启动之前可以测试一下配置是否正确
# filebeat.sh --help
-configtest：Test configuration and exit.
-e：Log to stderr and disable syslog/file output
# filebeat.sh -configtest -e
2017/07/09 17:36:59.623072 beat.go:285: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2017/07/09 17:36:59.623123 beat.go:186: INFO Setup Beat: filebeat; Version: 5.5.0
2017/07/09 17:36:59.623211 logstash.go:90: INFO Max Retries set to: 3
2017/07/09 17:36:59.623218 metrics.go:23: INFO Metrics logging every 30s
2017/07/09 17:36:59.623493 outputs.go:108: INFO Activated logstash as output plugin.
2017/07/09 17:36:59.623683 publish.go:295: INFO Publisher name: uy05-09
2017/07/09 17:36:59.625146 async.go:63: INFO Flush Interval set to: 1s
2017/07/09 17:36:59.625176 async.go:64: INFO Max Bulk Size set to: 2048
Config OK
# /etc/init.d/filebeat start
10、编写logstash pipeline配置文件。这里通过 if 判断将不同type的日志输出到不同的索引。
# vim /etc/logstash/conf.d/nginx.conf
input {
beats {
port => 5044
codec => "json"
}
}
output {
if [type] == "zixun-nginx-access" {
elasticsearch {
hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
index => "zixun-nginx-access-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
template_overwrite => true
user => elastic
password => Monkey
}}
if [type] == "water-nginx-access" {
elasticsearch {
hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
index => "water-nginx-access-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
template_overwrite => true
user => elastic
password => Monkey
}}
if [type] == "nav-nginx-access" {
elasticsearch {
hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
index => "nav-nginx-access-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
template_overwrite => true
user => elastic
password => Monkey
}}
if [type] == "wood-nginx-access" {
elasticsearch {
hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
index => "wood-nginx-access-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
template_overwrite => true
user => elastic
password => Monkey
}}
if [type] == "redir-nginx-access" {
elasticsearch {
hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
index => "redir-nginx-access-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
template_overwrite => true
user => elastic
password => Monkey
}}
if [type] == "default-nginx-access" {
elasticsearch {
hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
index => "default-nginx-access-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
template_overwrite => true
user => elastic
password => Monkey
}}
if [type] == "zx-opgirl-cn-nginx-access" {
elasticsearch {
hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
index => "zx-opgirl-cn-nginx-access-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
template_overwrite => true
user => elastic
password => Monkey
}}
if [type] == "www-oupeng-nginx-access" {
elasticsearch {
hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
index => "www-oupeng-nginx-access-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
template_overwrite => true
user => elastic
password => Monkey
}}
if [type] == "kibana-nginx-access" {
elasticsearch {
hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
index => "kibana-nginx-access-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
template_overwrite => true
user => elastic
password => Monkey
}}
}
11、启动logstash
# nohup logstash -f /etc/logstash/conf.d/nginx.conf &
12、在kiaban上添加索引并绘图，绘图方法参考上一篇
