packetbeat部署
ELK的部署过程可参考我的另一篇文章：点击前往
本次采用的是rpm包安装，rpm包安装源使用的则是清华大学：rpm包下载链接https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/yum/7.5.2/
安装配置
# 安装
yum install libpcap https://mirrors.tuna.tsinghua.edu.cn/elasticstack/7.x/yum/7.5.2/packetbeat-7.5.2-x86_64.rpm -y
# 修改配置
cat /etc/packetbeat/packetbeat.yml
#============================== Network device ================================
# Select the network interface to sniff the data. On Linux, you can use the
# "any" keyword to sniff on all connected interfaces.
# 监控网卡设备，默认是所有网卡
packetbeat.interfaces.device: any
#================================== Flows =====================================
# Set `enabled: false` or comment out all options to disable flows reporting.
packetbeat.flows:
# Set network flow timeout. Flow is killed if no packet is received before being
# timed out.
timeout: 30s
# Configure reporting period. If set to -1, only killed flows will be reported
period: 10s
#========================== Transaction protocols =============================
packetbeat.protocols:
#- type: icmp
# Enable ICMPv4 and ICMPv6 monitoring. Default: false
# enabled: true
#- type: amqp
# Configure the ports where to listen for AMQP traffic. You can disable
# the AMQP protocol by commenting out the list of ports.
# ports: [5672]
#- type: cassandra
#Cassandra port for traffic monitoring.
# ports: [9042]
#- type: dhcpv4
# Configure the DHCP for IPv4 ports.
# ports: [67, 68]
#- type: dns
# Configure the ports where to listen for DNS traffic. You can disable
# the DNS protocol by commenting out the list of ports.
# ports: [53]
#- type: http
# Configure the ports where to listen for HTTP traffic. You can disable
# the HTTP protocol by commenting out the list of ports.
# ports: [80, 8080, 8000, 5000, 8002]
#- type: memcache
# Configure the ports where to listen for memcache traffic. You can disable
# the Memcache protocol by commenting out the list of ports.
# ports: [11211]
# 根据你实际的端口修改端口号，其他的用户到就注释了，或者添加 enable：false 参数关机了就行，版本不一样可能关闭的参数不同，具体参数自行前往官网查询
- type: mysql
# Configure the ports where to listen for MySQL traffic. You can disable
# the MySQL protocol by commenting out the list of ports.
ports: [3306,3307]
#- type: pgsql
# Configure the ports where to listen for Pgsql traffic. You can disable
# the Pgsql protocol by commenting out the list of ports.
# ports: [5432]
#- type: redis
# Configure the ports where to listen for Redis traffic. You can disable
# the Redis protocol by commenting out the list of ports.
# ports: [6379]
#- type: thrift
# Configure the ports where to listen for Thrift-RPC traffic. You can disable
# the Thrift-RPC protocol by commenting out the list of ports.
# ports: [9090]
#- type: mongodb
# Configure the ports where to listen for MongoDB traffic. You can disable
# the MongoDB protocol by commenting out the list of ports.
# ports: [27017]
#- type: nfs
# Configure the ports where to listen for NFS traffic. You can disable
# the NFS protocol by commenting out the list of ports.
# ports: [2049]
- type: tls
# Configure the ports where to listen for TLS traffic. You can disable
# the TLS protocol by commenting out the list of ports.
ports:
- 443 # HTTPS
- 993 # IMAPS
- 995 # POP3S
- 5223 # XMPP over SSL
- 8443
- 8883 # Secure MQTT
- 9243 # Elasticsearch
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 1
#index.codec: best_compression
#_source.enabled: false
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
# 设置kibana地址，如果是本机的话就使用默认的就行,如果没有配置对会影响后面模板的导入
host: "localhost:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
# 这是es地址，跟kibana一样，如果是本机就保持默认即可，如果配置错了会影响后面模板的导入
hosts: ["localhost:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
# 日志输出还支持本地文件输出，或者输出到logstash等等，具体的可参考官方文档。建议刚开始可采用本地文件的方式输出，方便测试
#================================ Processors =====================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
#================================ Logging =====================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: error
logging.to_files: true
logging.files:
path: /var/log/packetbeat
name: error.log
keepfiles: 7
permissions: 0644
# 以上我中文注释的地方是必改地方，稍微注意一下，每天贴出来的或者没有中文注释的可选择性的修改。
导入模板
# 百度有很多是采用接口的方式导入的模板，我这采用的则是原生命令的方式，因接口的方式是个坑，整了好久一堆报错
# 导入索引模板
packetbeat setup --index-management
# 导入可视化图标
packetbeat setup --dashboards
启动测试
# 启动
systemctl start packetbeat
如果是直接输出到es中的话，这时候我们前往kibana查看是否已经产生索引文件
因为我们之前已经导入了索引模板，所以这时候直接查看估计就已经有数据了，如下图：
索引生命周期
上面这一波操作，packetbeat就已经安装完毕了。这里要提一下的就是索引的生命周期功能，什么叫索引生命周期呢？我的理解就是我们可以自定义策略来达到自定义删除，自定义定义每个索引的大小等等。当然这个功能也可通过脚本调动api接口来实现，但既然它提供了，那就可用直接拿来用，就没必要另外写脚本了。
因我们是导入过索引模板的，所以索引生命周期也是直接就有了的。我们只需启动，并配置即可。
我们可以配置一个索引的大小，如果索引大于这个大小那么则会另起一个索引并重命名;最大存在时间，这个我理解的则是在这个时间内所有的数据都往这个索引里写，超过这个时间则另起一个索引。
温阶段与冷阶段因无业务需求所暂且没用过，官方介绍文档
删除阶段启动，选择时间范围，我的理解则是 这个索引超过你设定的时间后则删除这个索引，经测试发现，这个索引是非正在存储数据的索引。比如索引 log-0001 昨日创建的，今日创建了 log-0002，我设定的是1天删除，那么今日则删除 log-0001索引
官方也提供了一个专门管理索引的工具，curator工具介绍