最近经理开会说公司要安装ELK日志管理让我们搭建ELK，下面是我搭建步骤和流程，用三台机测试机器搭建的。
软件包我都 给你们放/usr/local/src/elk目录下
安装目录都放在/usr/local/下
数据都放在/data0/elk/目录下
日志都放在/data0/logs/elk目录下
一，安装Elasticsearch
1，安装JAVA环境
（1）下载jdk：
http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
（2）查看系统是否安装低版本
java -version
#如果低于8请卸载
yum remove -y java
yum groupremove -y java
（3）rpm安装jdk
rpm -ivh jdk-8u161-linux-x64.rpm
echo "JAVA_HOME=/usr/java/jdk1.8.0_161
JRE_HOME=/usr/java/jdkjdk1.8.0_161/jre
PATH=${PATH}:${JAVA_HOME}/bin:${JRE_HOME}/bin
CLASSPATH=:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar:${JRE_HOME}/lib
export JAVA_HOME JRE_HOME PATH CLASSPATH" >> /etc/profile.d/java.sh
source /etc/profile.d/java.sh
2，安装ELasticsearch
（1）安装elasticsearch:
cd /usr/local/src
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.tar.gz
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.tar.gz.sha512
#shasum -a 512 -c elasticsearch-6.2.3.tar.gz.sha512
sha512sum -c elasticsearch-6.2.3.tar.gz.sha512
tar -xzf elasticsearch-6.2.3.tar.gz
mv elasticsearch-6.2.3 /usr/local/
注意：Centos6不支持SecComp，而ES6默认bootstrap.system_call_filter为true，需要禁用。
禁用方法：在elasticsearch.yml中配置bootstrap.system_call_filter为false，注意要在Memory下面:
取消bootstrap.memory_lock的注释，添加bootstrap.system_call_filter 配置
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
（2）配置优化elasticsrach
1）#jvm启动参数优化：/usr/local/elasticsearch-6.2.3/config/jvm.options
-Xms1g
-Xmx1g
#初始和最大最好一致，避免GC
#Xms代表总的堆空间的初始大小
#Xmx表示总的堆空间的最大大小
2）elasticsearch配置文件/usr/local/elasticsearch-6.2.3/config/elasticsearch.yml
#集群名称
cluster.name: sinashow-elk
#节点名称
node.name: sinashow-elk-192-168-9-176
#锁定内存
bootstrap.memory_lock: true
#通信绑定地址和http端口
network.host: 192.168.9.176
http.port: 9200
#数据和日志存放目录
logs: /data0/es
data: /data0/es
discovery.zen.ping.unicast.hosts: ["192.168.9.176"]
#增加参数，使head插件可以访问es
http.cors.enabled: true
http.cors.allow-origin: "*"
3）系统设置更改：
echo "es soft nofile 65536
es hard nofile 131072
es soft memlock unlimited
es hard memlock unlimited
es soft nproc 4096
es hard nproc 4096" >> /etc/security/limits.conf
sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
#查看 GET _nodes/stats/process?filter_path=**.max_file_descriptors
禁用swap分区：
swapoff -a
更改/etc/fstab
#查看GET _nodes?filter_path=**.mlockall
（3）安装head插件
1）安装node
cd /usr/local/src/elk
wget https://npm.taobao.org/mirrors/node/latest-v4.x/node-v4.4.7-linux-x64.tar.gz
tar -zxvf node-v4.4.7-linux-x64.tar.gz
#配置环境变量
# vim /etc/profile.d/node.sh
export NODE_HOME=/usr/local/src/elk/node-v4.4.7-linux-x64
export PATH=${PATH}:${NODE_HOME}/bin
export NODE_PATH=${NODE_HOME}/lib/node_modules
#加载环境变量
source /etc/profile.d/node.sh
2）安装grunt
npm install -g cnpm --registry=https://registry.npm.taobao.org
npm install -g grunt
npm install -g grunt-cli --registry=https://registry.npm.taobao.org --no-proxy
node -v
npm -v
grunt -version
3）下载head插件
wget https://github.com/mobz/elasticsearch-head/archive/master.zip
unzip master.zip
npm install
#npm install -g cnpm --registry=https://registry.npm.taobao.org
#cnpm install
4）修改Gruntfile.js
connect: {
server: {
options: {
port: 9100,
base: '.',
keepalive: true,
hostname: '*'
5）修改连接地址_site/app.js
this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://192.168.9.176:9200";
（4）启Elasticsearch并测试
groupadd es
useradd es -g es
passwd es
chown -R es.es /usr/lcoal/elasticsearch-6.2.3
mkdir -p /data0/es
chown -R es.es /usr/local/elasticsearch-6.2.3/
chown -R es.es /data0/es/
su es
cd /usr/lcoal/elasticsearch-6.2.3
./bin/elasticsearch
# curl -i "http://localhost:9200/"
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 435
{
"name" : "fS0IUJs",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "NmnxM5hkSVKFDId9-pC-CA",
"version" : {
"number" : "6.2.2",
"build_hash" : "10b1edd",
"build_date" : "2018-02-16T19:01:30.685723Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
（5）启动head插件
/usr/local/src/elk/elasticsearch-head-master
nohup grunt server >/dev/null 2>&1 &
二，安装kibana环境
1，下载安装kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.3-linux-x86_64.tar.gz
sha1sum kibana-6.2.3-linux-x86_64.tar.gz
tar -xzf kibana-6.2.3-linux-x86_64.tar.gz
cd kibana-6.2.3-linux-x86_64/
2，配置kibana /usr/local/kibana-6.2.3-linux-x86_64/config/kibana.yml
#监听端口
server.port: 5601
#监听地址
server.host: "192.168.9.176"
#服务名称
server.name: "sinashow-kibana"
#连接Elasticsearch地址
elasticsearch.url: "http://192.168.9.176:9200"
#超时时间
elasticsearch.pingTimeout: 5000
elasticsearch.requestTimeout: 60000
3，启动
cd /usr/local/kibana-6.2.3-linux-x86_64
nohup ./bin/kibana >dev/null 2>&1 &
4，测试
curl -I "http://192.168.9.176:5601"
三，kibana和head插件配置nginx代理访问控制
1，nginx环境安装
（1）安装pcre-devel，Nginx支持HTTP Rewrite
yum -y install gcc gcc-c++ autoconf automake zlib zlib-devel
pcre pcre-devel openssl bzip2-devel libxml2 libxml2-devel curl-devel
libjpeg libjpeg-devel libpng libpng-devel openssl-devel libevent libevent-devel
（2）安装Nginx
# 首先添加用户nginx，实现以之运行nginx服务进程：
groupadd -r nginx
useradd -r -g nginx -s /sbin/nologin nginx
# 接着开始编译和安装：
wget http://nginx.org/download/nginx-1.12.2.tar.gz
tar xf nginx-1.12.2.tar.gz
cd nginx-1.12.2
./configure
--prefix=/usr/local/nginx
--user=nginx
--group=nginx
--with-http_ssl_module
--with-http_flv_module
--with-file-aio
--with-http_stub_status_module
--with-http_gzip_static_module
--with-pcre
--with-stream
--with-http_mp4_module
make && make install
（3）Nginx日志切割
0 0 * * * cd /data0/logs/ && /bin/sh cut_nginx_log.sh >/dev/null 2>&1
# mkdir -p /data0/logs/nginx
# cat /data0/logs/cut_nginx_log.sh
#!/bin/bash
logs_path="/data0/logs/nginx"
YDATE=`date +"%Y"`
MDATE=`date +"%m"`
DDATE=`date -d"1 day ago" +"%d"`
HDATE=`date +"%H"`
mkdir -p ${logs_path}/${YDATE}/${MDATE}/${DDATE}/
yesterday=`date -d "yesterday" +"%Y%m%d"`
for logname in `find ${logs_path} -maxdepth 1 -name "*.log"`
do
domain=`echo $logname | sed "s#/data0/logs/nginx/##"`
mv ${logs_path}/$domain
${logs_path}/${YDATE}/${MDATE}/${DDATE}/$domain\_${yesterday}.log
done
/usr/local/nginx/sbin/nginx -s reload
find ${logs_path} -name "*.log" -mtime +15 -exec rm -fr {} ;
2，Kibana配置nginx代理访问控制：
（1）nginx主配置文件
# cat /usr/local/nginx/conf/nginx.conf
user nginx;
worker_processes 8;
worker_rlimit_nofile 102400;
error_log /data0/logs/nginx/error.log notice;
pid /data0/logs/nginx/nginx.pid;
events {
use epoll;
worker_connections 65535;
}
http {
include mime.types;
default_type application/octet-stream;
log_format nginx_log '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
client_max_body_size 50m;
client_body_buffer_size 256k;
client_header_timeout 120s;
client_body_timeout 120s;
send_timeout 1m;
sendfile on;
keepalive_timeout 120;
proxy_ignore_client_abort on;
proxy_connect_timeout 600;
proxy_read_timeout 600;
proxy_send_timeout 600;
proxy_buffer_size 32k;
proxy_buffers 8 128k;
proxy_busy_buffers_size 128k;
proxy_temp_file_write_size 512k;
gzip on;
server {
listen 80 default_server;
server_name _;
return 404;
}
include vhosts/*.conf;
}
(2）/usr/local/nginx/conf/vhosts/es-head.sinashow.com.conf
upstream es-head.sinashow.com {
#server 127.0.0.1:9200 max_fails=3 fail_timeout=1000s;
server 192.168.9.176:9100 max_fails=3 fail_timeout=1000s;
ip_hash;
}
server {
listen 80;
server_name es-head.sinashow.com;
location / {
root html;
index index.html index.htm;
proxy_pass http://es-head.sinashow.com;
allow 111.198.228.124;
deny all;
auth_basic "closed site";
auth_basic_user_file /usr/local/nginx/conf/.htpasswd;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 512k;
access_log /data0/logs/nginx/es-head.sinashow.com.log nginx_log;
}
}
(3）/usr/local/nginx/conf/vhosts/es-kibana.sinashow.com.conf
upstream es-kibana.sinashow.com {
#server 127.0.0.1:9200 max_fails=3 fail_timeout=1000s;
server 192.168.9.176:5601 max_fails=3 fail_timeout=1000s;
ip_hash;
}
server {
listen 80;
server_name es-kibana.sinashow.com;
location / {
root html;
index index.html index.htm;
proxy_pass http://es-kibana.sinashow.com;
allow 111.198.228.124;
deny all;
auth_basic "closed site";
auth_basic_user_file /usr/local/nginx/conf/.htpasswd;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 512k;
access_log /data0/logs/nginx/es-kibana.sinashow.com.log nginx_log;
}
}
(4）创建nginx访问控制账号
yum install -y httpd
htpasswd -c /usr/local/nginx/conf/.htpasswd esuser
3，启动nginx并测试
#检测
/usr/local/nginx/sbin/nginx -t
#启动
/usr/local/nginx/sbin/nginx
#停止
/usr/local/nginx/sbin/nginx -s stop
#重载
/usr/local/nginx/sbin/nginx -s reload
测试：
http:// es-kibana.sinashow.com
http:// es-head.sinashow.com
四，安装Logstash环境
1，安装logstash
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.3.tar.gz
tar xf logstash-6.2.3.tar.gz
cd logstash-6.2.3
./bin/logstash -e 'input { stdin { } } output { stdout {} }'
2，配置logstash /usr/local/logstash-6.2.3/config/logstash.yml
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => "192.168.9.176:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
3，启动logstash
cd /usr/local/logstash-6.2.3/
五，安装Filebeat环境
1，rpm包安装
curl -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.3-x86_64.rpm
rpm -vi filebeat-6.2.3-x86_64.rpm
2，配置filebeat /etc/filebeat/filebeat.yml
filebeat.prospectors:
- type: log
enabled: true
paths:
- /var/log/*.log
output.logstash:
hosts: ["192.168.9.176:5044"]
3，启动filebeat
systemctl start filebeat.service
总结：
集群配置 需要修改
cluster.name 、node.name 、network.host: 0.0.0.0【此处默认localhost】、http.port: 9200
只要集群名相同，且机器处于同一局域网同一网段，es会自动去发现其他的节点
注意：elasticsearch.yml中设置项中冒号右边要加一个空格
小伙伴们在搭建的时候会遇到很多错误，例如修改需要系统参数，还有bootstrap checks faild错误
解决办法=>
在配置文件里面添加（）下面两行代码:
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
报错：max number of threads [1024] for user [lish] likely too low, increase to at least [2048] 解决：切换到root用户，进入limits.d目录下修改配置文件。 vi /etc/security/limits.d/90-nproc.conf
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144] 解决：切换到root用户修改配置sysctl.conf vi /etc/sysctl.conf 添加下面配置： vm.max_map_count=655360 并执行命令： sysctl -p
好了就总结这些了