Logstash使用快速入门
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.部署Logstash
博主推荐阅读: https://www.cnblogs.com/yinzhengjie2020/p/13022403.html
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash --help #查看logstash脚本的帮助信息 WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Usage: bin/logstash [OPTIONS] Options: -n, --node.name NAME Specify the name of this logstash instance, if no value is given it will default to the current hostname. (default: "es103.yinzhengjie.com") -f, --path.config CONFIG_PATH Load the logstash config from a specific file or directory. If a directory is given, all files in that directory will be concatenated in lexicographical order and then parsed as a single config file. You can also specify wildcards (globs) and any matched files will be loaded in the order described above. -e, --config.string CONFIG_STRING Use the given string as the configuration data. Same syntax as the config file. If no input is specified, then the following is used as the default input: "input { stdin { type => stdin } }" and if no output is specified, then the following is used as the default output: "output { stdout { codec => rubydebug } }" If you wish to use both defaults, please use the empty string for the '-e' flag. (default: nil) --field-reference-parser MODE Use the given MODE when parsing field references. The field reference parser is used to expand field references in your pipeline configs, and will be becoming more strict to better handle illegal and ambbiguous inputs in a future release of Logstash. Available MODEs are: - `LEGACY`: parse with the legacy parser, which is known to handle ambiguous- and illegal-syntax in surprising ways; warnings will not be emitted. - `COMPAT`: warn once for each distinct ambiguous- or illegal-syntax input, but continue to expand field references with the legacy parser. - `STRICT`: parse in a strict manner; when given ambiguous- or illegal-syntax input, raises a runtime exception that should be handled by the calling plugin. The MODE can also be set with `config.field_reference.parser` (default: "COMPAT") --modules MODULES Load Logstash modules. Modules can be defined using multiple instances '--modules module1 --modules module2', or comma-separated syntax '--modules=module1,module2' Cannot be used in conjunction with '-e' or '-f' Use of '--modules' will override modules declared in the 'logstash.yml' file. -M, --modules.variable MODULES_VARIABLE Load variables for module template. Multiple instances of '-M' or '--modules.variable' are supported. Ignored if '--modules' flag is not used. Should be in the format of '-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"' as in '-M "example.var.filter.mutate.fieldname=fieldvalue"' --setup Load index template into Elasticsearch, and saved searches, index-pattern, visualizations, and dashboards into Kibana when running modules. (default: false) --cloud.id CLOUD_ID Sets the elasticsearch and kibana host settings for module connections in Elastic Cloud. Your Elastic Cloud User interface or the Cloud support team should provide this. Add an optional label prefix '<label>:' to help you identify multiple cloud.ids. e.g. 'staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy' --cloud.auth CLOUD_AUTH Sets the elasticsearch and kibana username and password for module connections in Elastic Cloud e.g. 'username:<password>' --pipeline.id ID Sets the ID of the pipeline. (default: "main") -w, --pipeline.workers COUNT Sets the number of pipeline workers to run. (default: 2) --java-execution Use Java execution engine. (default: false) -b, --pipeline.batch.size SIZE Size of batches the pipeline is to work in. (default: 125) -u, --pipeline.batch.delay DELAY_IN_MS When creating pipeline batches, how long to wait while polling for the next event. (default: 50) --pipeline.unsafe_shutdown Force logstash to exit during shutdown even if there are still inflight events in memory. By default, logstash will refuse to quit until all received events have been pushed to the outputs. (default: false) --path.data PATH This should point to a writable directory. Logstash will use this directory whenever it needs to store data. Plugins will also have access to this path. (default: "/usr/share/logstash/data") -p, --path.plugins PATH A path of where to find plugins. This flag can be given multiple times to include multiple paths. Plugins are expected to be in a specific directory hierarchy: 'PATH/logstash/TYPE/NAME.rb' where TYPE is 'inputs' 'filters', 'outputs' or 'codecs' and NAME is the name of the plugin. (default: []) -l, --path.logs PATH Write logstash internal logs to the given file. Without this flag, logstash will emit logs to standard output. (default: "/usr/share/logstash/logs") --log.level LEVEL Set the log level for logstash. Possible values are: - fatal - error - warn - info - debug - trace (default: "info") --config.debug Print the compiled config ruby code out as a debug log (you must also have --log.level=debug enabled). WARNING: This will include any 'password' options passed to plugin configs as plaintext, and may result in plaintext passwords appearing in your logs! (default: false) -i, --interactive SHELL Drop to shell instead of running as normal. Valid shells are "irb" and "pry" -V, --version Emit the version of logstash and its friends, then exit. -t, --config.test_and_exit Check configuration for valid syntax and then exit. (default: false) -r, --config.reload.automatic Monitor configuration changes and reload whenever it is changed. NOTE: use SIGHUP to manually reload the config (default: false) --config.reload.interval RELOAD_INTERVAL How frequently to poll the configuration location for changes, in seconds. (default: 3000000000) --http.host HTTP_HOST Web API binding host (default: "127.0.0.1") --http.port HTTP_PORT Web API http port (default: 9600..9700) --log.format FORMAT Specify if Logstash should write its own logs in JSON form (one event per line) or in plain text (using Ruby's Object#inspect) (default: "plain") --path.settings SETTINGS_DIR Directory containing logstash.yml file. This can also be set through the LS_SETTINGS_DIR environment variable. (default: "/usr/share/logstash/config") --verbose Set the log level to info. DEPRECATED: use --log.level=info instead. --debug Set the log level to debug. DEPRECATED: use --log.level=debug instead. --quiet Set the log level to info. DEPRECATED: use --log.level=info instead. -h, --help print help [root@es103.yinzhengjie.com ~]#
二.input(file) ---> output(stdout)案例
1>.编写配置文件并检查语法是否错误
[root@es103.yinzhengjie.com ~]# vim /etc/logstash/conf.d/file-stdout.conf [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# cat /etc/logstash/conf.d/file-stdout.conf input { file { type => "syslog" path => "/var/log/syslog" start_position => "beginning" stat_interval => 3 } } output { stdout { codec => "rubydebug" } } [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/file-stdout.conf -t #检查配置文件 WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-05 00:27:36.243 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK #注意哈，如果出现了"Configuration OK"说明配置文件语法正确 [INFO ] 2020-06-05 00:27:40.363 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [root@es103.yinzhengjie.com ~]#
2>.以root身份启动logstash任务
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/file-stdout.conf #以root身份启动logstash进程 WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-05 00:35:22.337 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2020-06-05 00:35:22.347 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.9"} [INFO ] 2020-06-05 00:35:27.742 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batc h.delay"=>50}[INFO ] 2020-06-05 00:35:28.106 [[main]-pipeline-manager] file - No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/fil e/.sincedb_f5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[INFO ] 2020-06-05 00:35:28.151 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x3dddf5cf run>"} [INFO ] 2020-06-05 00:35:28.229 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelin es=>[]}[INFO ] 2020-06-05 00:35:28.244 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections [INFO ] 2020-06-05 00:35:28.706 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600} ...... #如下所示，"/var/log/syslog"的每一行都被抽取成JSON格式发送给stdout。 { "@timestamp" => 2020-06-05T00:33:24.840Z, "type" => "syslog", "message" => "Jun 5 00:11:42 es103 systemd[3705]: Reached target Basic System.", "host" => "es103.yinzhengjie.com", "path" => "/var/log/syslog", "@version" => "1" } { "@timestamp" => 2020-06-05T00:33:24.841Z, "type" => "syslog", "message" => "Jun 5 00:11:42 es103 systemd[3705]: Startup finished in 319ms.", "host" => "es103.yinzhengjie.com", "path" => "/var/log/syslog", "@version" => "1" } { "@timestamp" => 2020-06-05T00:33:24.841Z, "type" => "syslog", "message" => "Jun 5 00:17:01 es103 CRON[3934]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)", "host" => "es103.yinzhengjie.com", "path" => "/var/log/syslog", "@version" => "1" } ......
3>.启动logstash任务后查看生成的隐藏文件信息
[root@es103.yinzhengjie.com ~]# ll /usr/share/logstash/data/plugins/inputs/file/ #当我们启动一个input插件为file的logstash任务时，会在该目录生成相应的配置信息。 total 12 drwxr-xr-x 2 root root 4096 Jun 5 00:36 ./ drwxr-xr-x 3 root root 4096 Jun 5 00:33 ../ -rw-r--r-- 1 root root 58 Jun 5 00:36 .sincedb_f5fdf6ea0ea92860c6a6b2b354bfcbbc [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# cat /usr/share/logstash/data/plugins/inputs/file/.sincedb_f5fdf6ea0ea92860c6a6b2b354bfcbbc #注意观察该隐藏文件的内容 5245734 0 2050 1418255 1591317366.7913418 /var/log/syslog [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# ls -il /var/log/syslog #不难发现该inode节点信息和logstash的隐藏文件记录的是一致的。 5245734 -rw-r----- 1 syslog adm 1418255 Jun 5 00:18 /var/log/syslog [root@es103.yinzhengjie.com ~]#
三.input(file) ---> output(elasticsearch)案例
1>.编写配置文件并检查语法是否错误
[root@es103.yinzhengjie.com ~]# vim /etc/logstash/conf.d/file-elasticsearch.conf [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# cat /etc/logstash/conf.d/file-elasticsearch.conf input { file { type => "syslog" path => "/var/log/syslog" start_position => "beginning" stat_interval => 3 } } output { elasticsearch { hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200","http://es103.yinzhengjie.com:9200"] index => "syslog-172.200.5.103-%{+YYYY.MM.dd}" } } [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/file-elasticsearch.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-05 00:52:51.381 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [INFO ] 2020-06-05 00:52:55.289 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [root@es103.yinzhengjie.com ~]#
2>.输入源文件无权限访问导致logstash任务运行失败案例
[root@es103.yinzhengjie.com ~]# ll /var/log/logstash/ total 8 drwxrwxr-x 2 logstash root 4096 May 4 18:27 ./ drwxrwxr-x 11 root syslog 4096 Jun 4 02:46 ../ [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# ll /var/log/syslog -rw-r----- 1 syslog adm 1418255 Jun 5 00:18 /var/log/syslog #不难发现，该文件对于其它用户(logstatsh)是没有访问权限的 [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# systemctl start logstash.service [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# ll /var/log/logstash/ total 16 drwxrwxr-x 2 logstash root 4096 Jun 5 00:58 ./ drwxrwxr-x 11 root syslog 4096 Jun 4 02:46 ../ -rw-r--r-- 1 logstash logstash 5285 Jun 5 01:03 logstash-plain.log #一般情况下，我们通过查看该文件就可以看到logstash的日志信息，如果任务失败在该文件可以找到原因 -rw-r--r-- 1 logstash logstash 0 Jun 5 00:58 logstash-slowlog-plain.log [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# tail -10f /var/log/logstash/logstash-plain.log #相信你不难从WARN日志级别找到"Permission denied"的关键词。 [2020-06-05T00:58:01,397][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"} [2020-06-05T00:58:01,422][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"} [2020-06-05T00:58:01,777][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.8.9"} [2020-06-05T00:58:01,798][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"d8171294-9203-4745-ab19-e671d626ac67", :path=>"/var/lib/logstash/uuid "}[2020-06-05T00:58:07,849][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} [2020-06-05T00:58:08,338][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es101.yinzhengjie.com:9200/, http://es102.yinzheng jie.com:9200/, http://es103.yinzhengjie.com:9200/]}}[2020-06-05T00:58:08,651][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es101.yinzhengjie.com:9200/"} [2020-06-05T00:58:08,722][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6} [2020-06-05T00:58:08,725][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6} [2020-06-05T00:58:08,732][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es102.yinzhengjie.com:9200/"} [2020-06-05T00:58:08,755][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es103.yinzhengjie.com:9200/"} [2020-06-05T00:58:08,796][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es101.yinzhengjie.com:9200", "http:// es102.yinzhengjie.com:9200", "http://es103.yinzhengjie.com:9200"]}[2020-06-05T00:58:08,815][INFO ][logstash.outputs.elasticsearch] Using default mapping template [2020-06-05T00:58:08,832][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_i nterval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}[2020-06-05T00:58:09,033][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_f 5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[2020-06-05T00:58:09,054][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_f 5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[2020-06-05T00:58:09,083][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x55c99cf3 run>"} [2020-06-05T00:58:09,138][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections [2020-06-05T00:58:09,146][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections [2020-06-05T00:58:09,175][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2020-06-05T00:58:09,556][WARN ][filewatch.tailmode.handlers.createinitial] failed to open /var/log/syslog: #<Errno::EACCES: Permission denied - /var/log/syslog>, ["org/jruby/RubyIO.java:12 36:in `sysopen'", "org/jruby/RubyFile.java:367:in `initialize'", "org/jruby/RubyIO.java:1155:in `open'"] [2020-06-05T00:58:09,567][WARN ][filewatch.tailmode.handlers.createinitial] failed to open /var/log/syslog: #<Errno::EACCES: Permission denied - /var/log/syslog>, ["org/jruby/RubyIO.java:12 36:in `sysopen'", "org/jruby/RubyFile.java:367:in `initialize'", "org/jruby/RubyIO.java:1155:in `open'"] [2020-06-05T00:58:09,613][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} ......
3>.上一步骤问题解决方案
[root@es103.yinzhengjie.com ~]# ll /var/log/syslog -rw-r----- 1 syslog adm 1425400 Jun 5 01:08 /var/log/syslog [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# chmod 644 /var/log/syslog #修改日志的权限，使得"logstash"用户可以访问 [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# ll /var/log/syslog -rw-r--r-- 1 syslog adm 1425400 Jun 5 01:08 /var/log/syslog [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# systemctl restart logstash.service #将源文件的权限配置为644后，需要重启服务，Logstash就回去加载"/etc/logstash/conf.d"目录下的所有配置文件并运行相应的job。
[root@es103.yinzhengjie.com ~]# tail -10f /var/log/logstash/logstash-plain.log #再次查看日志就可以看到正常的logstash收集数据的信息啦~ [2020-06-05T01:13:54,918][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es101.yinzhengjie.com:9200", "http:// es102.yinzhengjie.com:9200", "http://es103.yinzhengjie.com:9200"]}[2020-06-05T01:13:54,942][INFO ][logstash.outputs.elasticsearch] Using default mapping template [2020-06-05T01:13:54,992][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_i nterval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}[2020-06-05T01:13:55,226][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_f 5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[2020-06-05T01:13:55,256][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_f 5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[2020-06-05T01:13:55,291][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2bea6da8 run>"} [2020-06-05T01:13:55,333][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections [2020-06-05T01:13:55,335][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections [2020-06-05T01:13:55,406][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2020-06-05T01:13:55,913][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
四.多个输入源和多个输出源(多日志if语句使用)案例
1>.编写配置文件并检查语法是否错误
[root@es103.yinzhengjie.com ~]# vim /etc/logstash/conf.d/multiple-file-elasticsearch.conf [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# cat /etc/logstash/conf.d/multiple-file-elasticsearch.conf input { file { type => "syslog" path => "/var/log/syslog" start_position => "beginning" stat_interval => 3 } file { type => "nginx-log" path => "/var/log/nginx/access.log" start_position => "beginning" stat_interval => 3 } } output { if [type] == "syslog" { elasticsearch { hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200","http://es103.yinzhengjie.com:9200"] index => "syslog-172.200.5.103-%{+YYYY.MM.dd}" } file { path => "/tmp/syslog.txt" } } if [type] == "nginx-log" { elasticsearch { hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200","http://es103.yinzhengjie.com:9200"] index => "nginx-log-172.200.5.103-%{+YYYY.MM.dd}" } } } [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/multiple-file-elasticsearch.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-05 01:47:16.051 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [INFO ] 2020-06-05 01:47:29.814 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [root@es103.yinzhengjie.com ~]#
2>.安装Nginx服务并产生测试数据
[root@es103.yinzhengjie.com ~]# apt-get install nginx Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libjbig0 libjpeg-turbo8 libjpeg8 libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream libtiff5 libwebp6 libxpm4 nginx-common nginx-core Suggested packages: libgd-tools fcgiwrap nginx-doc ssl-cert The following NEW packages will be installed: fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libjbig0 libjpeg-turbo8 libjpeg8 libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream libtiff5 libwebp6 libxpm4 nginx nginx-common nginx-core 0 upgraded, 18 newly installed, 0 to remove and 79 not upgraded. Need to get 2,462 kB of archives. After this operation, 8,210 kB of additional disk space will be used. Do you want to continue? [Y/n] y Get:1 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libjpeg-turbo8 amd64 1.5.2-0ubuntu5.18.04.3 [110 kB] Get:2 http://mirrors.aliyun.com/ubuntu bionic/main amd64 fonts-dejavu-core all 2.37-1 [1,041 kB] Get:3 http://mirrors.aliyun.com/ubuntu bionic/main amd64 fontconfig-config all 2.12.6-0ubuntu2 [55.8 kB] Get:4 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libfontconfig1 amd64 2.12.6-0ubuntu2 [137 kB] Get:5 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libjpeg8 amd64 8c-2ubuntu8 [2,194 B] Get:6 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libjbig0 amd64 2.1-3.1build1 [26.7 kB] Get:7 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libtiff5 amd64 4.0.9-5ubuntu0.3 [153 kB] Get:8 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libwebp6 amd64 0.6.1-2 [185 kB] Get:9 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libxpm4 amd64 1:3.5.12-1 [34.0 kB] Get:10 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libgd3 amd64 2.2.5-4ubuntu0.4 [119 kB] Get:11 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 nginx-common all 1.14.0-0ubuntu1.7 [37.4 kB] Get:12 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-http-geoip amd64 1.14.0-0ubuntu1.7 [11.2 kB] Get:13 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-http-image-filter amd64 1.14.0-0ubuntu1.7 [14.6 kB] Get:14 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-http-xslt-filter amd64 1.14.0-0ubuntu1.7 [13.0 kB] Get:15 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-mail amd64 1.14.0-0ubuntu1.7 [41.8 kB] Get:16 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-stream amd64 1.14.0-0ubuntu1.7 [63.7 kB] Get:17 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 nginx-core amd64 1.14.0-0ubuntu1.7 [413 kB] Get:18 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 nginx all 1.14.0-0ubuntu1.7 [3,596 B] Fetched 2,462 kB in 4s (635 kB/s) Preconfiguring packages ... Selecting previously unselected package libjpeg-turbo8:amd64. (Reading database ... 119418 files and directories currently installed.) Preparing to unpack .../00-libjpeg-turbo8_1.5.2-0ubuntu5.18.04.3_amd64.deb ... Unpacking libjpeg-turbo8:amd64 (1.5.2-0ubuntu5.18.04.3) ... Selecting previously unselected package fonts-dejavu-core. Preparing to unpack .../01-fonts-dejavu-core_2.37-1_all.deb ... Unpacking fonts-dejavu-core (2.37-1) ... Selecting previously unselected package fontconfig-config. Preparing to unpack .../02-fontconfig-config_2.12.6-0ubuntu2_all.deb ... Unpacking fontconfig-config (2.12.6-0ubuntu2) ... Selecting previously unselected package libfontconfig1:amd64. Preparing to unpack .../03-libfontconfig1_2.12.6-0ubuntu2_amd64.deb ... Unpacking libfontconfig1:amd64 (2.12.6-0ubuntu2) ... Selecting previously unselected package libjpeg8:amd64. Preparing to unpack .../04-libjpeg8_8c-2ubuntu8_amd64.deb ... Unpacking libjpeg8:amd64 (8c-2ubuntu8) ... Selecting previously unselected package libjbig0:amd64. Preparing to unpack .../05-libjbig0_2.1-3.1build1_amd64.deb ... Unpacking libjbig0:amd64 (2.1-3.1build1) ... Selecting previously unselected package libtiff5:amd64. Preparing to unpack .../06-libtiff5_4.0.9-5ubuntu0.3_amd64.deb ... Unpacking libtiff5:amd64 (4.0.9-5ubuntu0.3) ... Selecting previously unselected package libwebp6:amd64. Preparing to unpack .../07-libwebp6_0.6.1-2_amd64.deb ... Unpacking libwebp6:amd64 (0.6.1-2) ... Selecting previously unselected package libxpm4:amd64. Preparing to unpack .../08-libxpm4_1%3a3.5.12-1_amd64.deb ... Unpacking libxpm4:amd64 (1:3.5.12-1) ... Selecting previously unselected package libgd3:amd64. Preparing to unpack .../09-libgd3_2.2.5-4ubuntu0.4_amd64.deb ... Unpacking libgd3:amd64 (2.2.5-4ubuntu0.4) ... Selecting previously unselected package nginx-common. Preparing to unpack .../10-nginx-common_1.14.0-0ubuntu1.7_all.deb ... Unpacking nginx-common (1.14.0-0ubuntu1.7) ... Selecting previously unselected package libnginx-mod-http-geoip. Preparing to unpack .../11-libnginx-mod-http-geoip_1.14.0-0ubuntu1.7_amd64.deb ... Unpacking libnginx-mod-http-geoip (1.14.0-0ubuntu1.7) ... Selecting previously unselected package libnginx-mod-http-image-filter. Preparing to unpack .../12-libnginx-mod-http-image-filter_1.14.0-0ubuntu1.7_amd64.deb ... Unpacking libnginx-mod-http-image-filter (1.14.0-0ubuntu1.7) ... Selecting previously unselected package libnginx-mod-http-xslt-filter. Preparing to unpack .../13-libnginx-mod-http-xslt-filter_1.14.0-0ubuntu1.7_amd64.deb ... Unpacking libnginx-mod-http-xslt-filter (1.14.0-0ubuntu1.7) ... Selecting previously unselected package libnginx-mod-mail. Preparing to unpack .../14-libnginx-mod-mail_1.14.0-0ubuntu1.7_amd64.deb ... Unpacking libnginx-mod-mail (1.14.0-0ubuntu1.7) ... Selecting previously unselected package libnginx-mod-stream. Preparing to unpack .../15-libnginx-mod-stream_1.14.0-0ubuntu1.7_amd64.deb ... Unpacking libnginx-mod-stream (1.14.0-0ubuntu1.7) ... Selecting previously unselected package nginx-core. Preparing to unpack .../16-nginx-core_1.14.0-0ubuntu1.7_amd64.deb ... Unpacking nginx-core (1.14.0-0ubuntu1.7) ... Selecting previously unselected package nginx. Preparing to unpack .../17-nginx_1.14.0-0ubuntu1.7_all.deb ... Unpacking nginx (1.14.0-0ubuntu1.7) ... Processing triggers for ufw (0.36-0ubuntu0.18.04.1) ... Processing triggers for ureadahead (0.100.0-21) ... Setting up libjbig0:amd64 (2.1-3.1build1) ... Setting up fonts-dejavu-core (2.37-1) ... Setting up nginx-common (1.14.0-0ubuntu1.7) ... Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /lib/systemd/system/nginx.service. Setting up libjpeg-turbo8:amd64 (1.5.2-0ubuntu5.18.04.3) ... Processing triggers for libc-bin (2.27-3ubuntu1) ... Processing triggers for systemd (237-3ubuntu10.38) ... Setting up libnginx-mod-mail (1.14.0-0ubuntu1.7) ... Setting up libxpm4:amd64 (1:3.5.12-1) ... Processing triggers for man-db (2.8.3-2ubuntu0.1) ... Setting up libnginx-mod-http-xslt-filter (1.14.0-0ubuntu1.7) ... Setting up libnginx-mod-http-geoip (1.14.0-0ubuntu1.7) ... Setting up libwebp6:amd64 (0.6.1-2) ... Setting up libjpeg8:amd64 (8c-2ubuntu8) ... Setting up fontconfig-config (2.12.6-0ubuntu2) ... Setting up libnginx-mod-stream (1.14.0-0ubuntu1.7) ... Setting up libtiff5:amd64 (4.0.9-5ubuntu0.3) ... Setting up libfontconfig1:amd64 (2.12.6-0ubuntu2) ... Setting up libgd3:amd64 (2.2.5-4ubuntu0.4) ... Setting up libnginx-mod-http-image-filter (1.14.0-0ubuntu1.7) ... Setting up nginx-core (1.14.0-0ubuntu1.7) ... Setting up nginx (1.14.0-0ubuntu1.7) ... Processing triggers for ureadahead (0.100.0-21) ... Processing triggers for ufw (0.36-0ubuntu0.18.04.1) ... Processing triggers for libc-bin (2.27-3ubuntu1) ... [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# systemctl start nginx [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 0.0.0.0:80 0.0.0.0:* LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 [::]:80 [::]:* LISTEN 0 128 [::ffff:172.200.5.103]:9200 *:* LISTEN 0 128 [::ffff:172.200.5.103]:9300 *:* LISTEN 0 128 [::]:22 [::]:* LISTEN 0 50 [::ffff:127.0.0.1]:9600 *:* [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# systemctl status nginx ● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2020-06-05 01:43:50 UTC; 8min ago Docs: man:nginx(8) Process: 6237 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Process: 6226 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Main PID: 6240 (nginx) Tasks: 3 (limit: 4632) CGroup: /system.slice/nginx.service ├─6240 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; ├─6242 nginx: worker process └─6243 nginx: worker process Jun 05 01:43:49 es103.yinzhengjie.com systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 05 01:43:50 es103.yinzhengjie.com systemd[1]: Started A high performance web server and a reverse proxy server. [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# systemctl enable nginx Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable nginx [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# cat /var/log/nginx/access.log 172.200.0.1 - - [05/Jun/2020:01:54:33 +0000] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36 "172.200.0.1 - - [05/Jun/2020:01:54:33 +0000] "GET /favicon.ico HTTP/1.1" 404 209 "http://es103.yinzhengjie.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like G ecko) Chrome/83.0.4103.61 Safari/537.36"172.200.5.103 - - [05/Jun/2020:01:55:07 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.58.0" [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# ll /var/log/nginx/access.log -rw-r----- 1 www-data adm 511 Jun 5 01:55 /var/log/nginx/access.log [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# chmod 644 /var/log/nginx/access.log [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# ll /var/log/nginx/access.log -rw-r--r-- 1 www-data adm 511 Jun 5 01:55 /var/log/nginx/access.log [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]#
3>.重启Elasticsearch服务
[root@es103.yinzhengjie.com ~]# systemctl restart logstash #重启Logstash服务，使得配置文件生效。
[root@es103.yinzhengjie.com ~]# tail -100f /var/log/logstash/logstash-plain.log ...... [2020-06-05T02:09:03,939][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} [2020-06-05T02:09:04,383][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es101.yinzhengjie.com:9200/, http://es102.yinzheng jie.com:9200/, http://es103.yinzhengjie.com:9200/]}}[2020-06-05T02:09:04,580][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es101.yinzhengjie.com:9200/"} [2020-06-05T02:09:04,631][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6} [2020-06-05T02:09:04,635][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6} [2020-06-05T02:09:04,641][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es102.yinzhengjie.com:9200/"} [2020-06-05T02:09:04,653][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es103.yinzhengjie.com:9200/"} [2020-06-05T02:09:04,691][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es101.yinzhengjie.com:9200", "http:// es102.yinzhengjie.com:9200", "http://es103.yinzhengjie.com:9200"]}[2020-06-05T02:09:04,719][INFO ][logstash.outputs.elasticsearch] Using default mapping template [2020-06-05T02:09:04,733][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es101.yinzhengjie.com:9200/, http://es102.yinzheng jie.com:9200/, http://es103.yinzhengjie.com:9200/]}}[2020-06-05T02:09:04,743][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es101.yinzhengjie.com:9200/"} [2020-06-05T02:09:04,748][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6} [2020-06-05T02:09:04,749][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6} [2020-06-05T02:09:04,753][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es102.yinzhengjie.com:9200/"} [2020-06-05T02:09:04,761][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_i nterval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}[2020-06-05T02:09:04,787][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es103.yinzhengjie.com:9200/"} [2020-06-05T02:09:04,796][INFO ][logstash.outputs.elasticsearch] Using default mapping template [2020-06-05T02:09:04,799][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_i nterval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}[2020-06-05T02:09:04,799][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es101.yinzhengjie.com:9200", "http:// es102.yinzhengjie.com:9200", "http://es103.yinzhengjie.com:9200"]}[2020-06-05T02:09:04,848][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://es101.yinzhengjie.com:9200/, http://es102.yinzheng jie.com:9200/, http://es103.yinzhengjie.com:9200/]}}[2020-06-05T02:09:04,854][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es101.yinzhengjie.com:9200/"} [2020-06-05T02:09:04,859][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6} [2020-06-05T02:09:04,860][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6} [2020-06-05T02:09:04,864][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es102.yinzhengjie.com:9200/"} [2020-06-05T02:09:04,880][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://es103.yinzhengjie.com:9200/"} [2020-06-05T02:09:04,887][INFO ][logstash.outputs.elasticsearch] Using default mapping template [2020-06-05T02:09:04,889][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_i nterval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}[2020-06-05T02:09:04,891][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://es101.yinzhengjie.com:9200", "http:// es102.yinzhengjie.com:9200", "http://es103.yinzhengjie.com:9200"]}[2020-06-05T02:09:05,228][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_f 5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[2020-06-05T02:09:05,251][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_f 5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[2020-06-05T02:09:05,256][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_f 5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}[2020-06-05T02:09:05,263][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/var/lib/logstash/plugins/inputs/file/.sincedb_d 883144359d3b4f516b37dba51fab2a2", :path=>["/var/log/nginx/access.log"]}[2020-06-05T02:09:05,304][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x3875545b run>"} [2020-06-05T02:09:05,331][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections [2020-06-05T02:09:05,331][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections [2020-06-05T02:09:05,331][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections [2020-06-05T02:09:05,363][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections [2020-06-05T02:09:05,473][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2020-06-05T02:09:06,734][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2020-06-05T02:09:07,267][INFO ][logstash.outputs.file ] Opening file {:path=>"/tmp/syslog.txt"}
4>.在Kibana界面上添加索引
关于添加索引的步骤我之前有演示过，这里就不罗嗦了，直接上图。 博主推荐阅读: https://www.cnblogs.com/yinzhengjie2020/p/13022403.html
5>.查看"/tmp/syslog.txt"是否生成
[root@es103.yinzhengjie.com ~]# ll -h /tmp/syslog.txt #很明显，该文件已经存在啦~ -rw-r--r-- 1 logstash logstash 646M Jun 5 02:31 /tmp/syslog.txt [root@es103.yinzhengjie.com ~]#
五.收集tomcat日志案例
1>.安装tomcat
[root@es102.yinzhengjie.com ~]# wget https://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-8/v8.5.55/bin/apache-tomcat-8.5.55.tar.gz --2020-06-05 04:33:38-- https://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-8/v8.5.55/bin/apache-tomcat-8.5.55.tar.gz Resolving mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)... 101.6.8.193, 2402:f000:1:408:8100::1 Connecting to mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)|101.6.8.193|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 10371538 (9.9M) [application/x-gzip] Saving to: ‘apache-tomcat-8.5.55.tar.gz’ apache-tomcat-8.5.55.tar.gz 100%[====================================================================================================>] 9.89M 2.93MB/s in 3.4s 2020-06-05 04:33:41 (2.93 MB/s) - ‘apache-tomcat-8.5.55.tar.gz’ saved [10371538/10371538] [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# ls apache-tomcat-8.5.55.tar.gz [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# tar -zxf apache-tomcat-8.5.55.tar.gz -C /yinzhengjie/softwares/ [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# ll /yinzhengjie/softwares/ total 0 drwxr-xr-x 4 root root 54 Jun 5 04:34 ./ drwxr-xr-x 4 root root 35 Jun 3 02:53 ../ drwxr-xr-x 9 root root 220 Jun 5 04:34 apache-tomcat-8.5.55/ drwxr-xr-x 7 uucp 143 245 Dec 15 2018 jdk1.8.0_201/ [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# /yinzhengjie/softwares/apache-tomcat-8.5.55/bin/catalina.sh start Using CATALINA_BASE: /yinzhengjie/softwares/apache-tomcat-8.5.55 Using CATALINA_HOME: /yinzhengjie/softwares/apache-tomcat-8.5.55 Using CATALINA_TMPDIR: /yinzhengjie/softwares/apache-tomcat-8.5.55/temp Using JRE_HOME: /yinzhengjie/softwares/jdk1.8.0_201/jre Using CLASSPATH: /yinzhengjie/softwares/apache-tomcat-8.5.55/bin/bootstrap.jar:/yinzhengjie/softwares/apache-tomcat-8.5.55/bin/tomcat-juli.jar Tomcat started. [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 100 *:8080 *:* LISTEN 0 128 [::ffff:172.200.5.102]:9200 *:* LISTEN 0 128 [::ffff:172.200.5.102]:9300 *:* LISTEN 0 128 [::]:22 [::]:* [root@es102.yinzhengjie.com ~]#
1 [root@es102.yinzhengjie.com ~]# cat /yinzhengjie/softwares/apache-tomcat-8.5.55/conf/server.xml 2 <?xml version="1.0" encoding="UTF-8"?> 3 <!-- 4 Licensed to the Apache Software Foundation (ASF) under one or more 5 contributor license agreements. See the NOTICE file distributed with 6 this work for additional information regarding copyright ownership. 7 The ASF licenses this file to You under the Apache License, Version 2.0 8 (the "License"); you may not use this file except in compliance with 9 the License. You may obtain a copy of the License at 10 11 http://www.apache.org/licenses/LICENSE-2.0 12 13 Unless required by applicable law or agreed to in writing, software 14 distributed under the License is distributed on an "AS IS" BASIS, 15 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 See the License for the specific language governing permissions and 17 limitations under the License. 18 --> 19 <!-- Note: A "Server" is not itself a "Container", so you may not 20 define subcomponents such as "Valves" at this level. 21 Documentation at /docs/config/server.html 22 --> 23 <Server port="8005" shutdown="SHUTDOWN"> 24 <Listener className="org.apache.catalina.startup.VersionLoggerListener" /> 25 <!-- Security listener. Documentation at /docs/config/listeners.html 26 <Listener className="org.apache.catalina.security.SecurityListener" /> 27 --> 28 <!--APR library loader. Documentation at /docs/apr.html --> 29 <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" /> 30 <!-- Prevent memory leaks due to use of particular java/javax APIs--> 31 <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" /> 32 <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" /> 33 <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" /> 34 35 <!-- Global JNDI resources 36 Documentation at /docs/jndi-resources-howto.html 37 --> 38 <GlobalNamingResources> 39 <!-- Editable user database that can also be used by 40 UserDatabaseRealm to authenticate users 41 --> 42 <Resource name="UserDatabase" auth="Container" 43 type="org.apache.catalina.UserDatabase" 44 description="User database that can be updated and saved" 45 factory="org.apache.catalina.users.MemoryUserDatabaseFactory" 46 pathname="conf/tomcat-users.xml" /> 47 </GlobalNamingResources> 48 49 <!-- A "Service" is a collection of one or more "Connectors" that share 50 a single "Container" Note: A "Service" is not itself a "Container", 51 so you may not define subcomponents such as "Valves" at this level. 52 Documentation at /docs/config/service.html 53 --> 54 <Service name="Catalina"> 55 56 <!--The connectors can use a shared executor, you can define one or more named thread pools--> 57 <!-- 58 <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" 59 maxThreads="150" minSpareThreads="4"/> 60 --> 61 62 63 <!-- A "Connector" represents an endpoint by which requests are received 64 and responses are returned. Documentation at : 65 Java HTTP Connector: /docs/config/http.html 66 Java AJP Connector: /docs/config/ajp.html 67 APR (HTTP/AJP) Connector: /docs/apr.html 68 Define a non-SSL/TLS HTTP/1.1 Connector on port 8080 69 --> 70 <Connector port="8080" protocol="HTTP/1.1" 71 connectionTimeout="20000" 72 redirectPort="8443" /> 73 <!-- A "Connector" using the shared thread pool--> 74 <!-- 75 <Connector executor="tomcatThreadPool" 76 port="8080" protocol="HTTP/1.1" 77 connectionTimeout="20000" 78 redirectPort="8443" /> 79 --> 80 <!-- Define an SSL/TLS HTTP/1.1 Connector on port 8443 81 This connector uses the NIO implementation. The default 82 SSLImplementation will depend on the presence of the APR/native 83 library and the useOpenSSL attribute of the 84 AprLifecycleListener. 85 Either JSSE or OpenSSL style configuration may be used regardless of 86 the SSLImplementation selected. JSSE style configuration is used below. 87 --> 88 <!-- 89 <Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol" 90 maxThreads="150" SSLEnabled="true"> 91 <SSLHostConfig> 92 <Certificate certificateKeystoreFile="conf/localhost-rsa.jks" 93 type="RSA" /> 94 </SSLHostConfig> 95 </Connector> 96 --> 97 <!-- Define an SSL/TLS HTTP/1.1 Connector on port 8443 with HTTP/2 98 This connector uses the APR/native implementation which always uses 99 OpenSSL for TLS. 100 Either JSSE or OpenSSL style configuration may be used. OpenSSL style 101 configuration is used below. 102 --> 103 <!-- 104 <Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol" 105 maxThreads="150" SSLEnabled="true" > 106 <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" /> 107 <SSLHostConfig> 108 <Certificate certificateKeyFile="conf/localhost-rsa-key.pem" 109 certificateFile="conf/localhost-rsa-cert.pem" 110 certificateChainFile="conf/localhost-rsa-chain.pem" 111 type="RSA" /> 112 </SSLHostConfig> 113 </Connector> 114 --> 115 116 <!-- Define an AJP 1.3 Connector on port 8009 --> 117 <!-- 118 <Connector protocol="AJP/1.3" 119 address="::1" 120 port="8009" 121 redirectPort="8443" /> 122 --> 123 124 <!-- An Engine represents the entry point (within Catalina) that processes 125 every request. The Engine implementation for Tomcat stand alone 126 analyzes the HTTP headers included with the request, and passes them 127 on to the appropriate Host (virtual host). 128 Documentation at /docs/config/engine.html --> 129 130 <!-- You should set jvmRoute to support load-balancing via AJP ie : 131 <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1"> 132 --> 133 <Engine name="Catalina" defaultHost="localhost"> 134 135 <!--For clustering, please take a look at documentation at: 136 /docs/cluster-howto.html (simple how to) 137 /docs/config/cluster.html (reference documentation) --> 138 <!-- 139 <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/> 140 --> 141 142 <!-- Use the LockOutRealm to prevent attempts to guess user passwords 143 via a brute-force attack --> 144 <Realm className="org.apache.catalina.realm.LockOutRealm"> 145 <!-- This Realm uses the UserDatabase configured in the global JNDI 146 resources under the key "UserDatabase". Any edits 147 that are performed against this UserDatabase are immediately 148 available for use by the Realm. --> 149 <Realm className="org.apache.catalina.realm.UserDatabaseRealm" 150 resourceName="UserDatabase"/> 151 </Realm> 152 153 <Host name="localhost" appBase="webapps" 154 unpackWARs="true" autoDeploy="true"> 155 156 <!-- SingleSignOn valve, share authentication between web applications 157 Documentation at: /docs/config/valve.html --> 158 <!-- 159 <Valve className="org.apache.catalina.authenticator.SingleSignOn" /> 160 --> 161 162 <!-- Access log processes all example. 163 Documentation at: /docs/config/valve.html 164 Note: The pattern used is equivalent to using pattern="common" --> 165 <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" 166 prefix="localhost_access_log" suffix=".txt" 167 pattern="%h %l %u %t "%r" %s %b" /> 168 169 </Host> 170 </Engine> 171 </Service> 172 </Server> 173 [root@es102.yinzhengjie.com ~]#
1 [root@es102.yinzhengjie.com ~]# vim /yinzhengjie/softwares/apache-tomcat-8.5.55/conf/server.xml #将tomcat日志转换为JSON格式 2 [root@es102.yinzhengjie.com ~]# 3 [root@es102.yinzhengjie.com ~]# cat /yinzhengjie/softwares/apache-tomcat-8.5.55/conf/server.xml 4 <?xml version="1.0" encoding="UTF-8"?> 5 <!-- 6 Licensed to the Apache Software Foundation (ASF) under one or more 7 contributor license agreements. See the NOTICE file distributed with 8 this work for additional information regarding copyright ownership. 9 The ASF licenses this file to You under the Apache License, Version 2.0 10 (the "License"); you may not use this file except in compliance with 11 the License. You may obtain a copy of the License at 12 13 http://www.apache.org/licenses/LICENSE-2.0 14 15 Unless required by applicable law or agreed to in writing, software 16 distributed under the License is distributed on an "AS IS" BASIS, 17 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 18 See the License for the specific language governing permissions and 19 limitations under the License. 20 --> 21 <!-- Note: A "Server" is not itself a "Container", so you may not 22 define subcomponents such as "Valves" at this level. 23 Documentation at /docs/config/server.html 24 --> 25 <Server port="8005" shutdown="SHUTDOWN"> 26 <Listener className="org.apache.catalina.startup.VersionLoggerListener" /> 27 <!-- Security listener. Documentation at /docs/config/listeners.html 28 <Listener className="org.apache.catalina.security.SecurityListener" /> 29 --> 30 <!--APR library loader. Documentation at /docs/apr.html --> 31 <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" /> 32 <!-- Prevent memory leaks due to use of particular java/javax APIs--> 33 <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" /> 34 <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" /> 35 <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" /> 36 37 <!-- Global JNDI resources 38 Documentation at /docs/jndi-resources-howto.html 39 --> 40 <GlobalNamingResources> 41 <!-- Editable user database that can also be used by 42 UserDatabaseRealm to authenticate users 43 --> 44 <Resource name="UserDatabase" auth="Container" 45 type="org.apache.catalina.UserDatabase" 46 description="User database that can be updated and saved" 47 factory="org.apache.catalina.users.MemoryUserDatabaseFactory" 48 pathname="conf/tomcat-users.xml" /> 49 </GlobalNamingResources> 50 51 <!-- A "Service" is a collection of one or more "Connectors" that share 52 a single "Container" Note: A "Service" is not itself a "Container", 53 so you may not define subcomponents such as "Valves" at this level. 54 Documentation at /docs/config/service.html 55 --> 56 <Service name="Catalina"> 57 58 <!--The connectors can use a shared executor, you can define one or more named thread pools--> 59 <!-- 60 <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" 61 maxThreads="150" minSpareThreads="4"/> 62 --> 63 64 65 <!-- A "Connector" represents an endpoint by which requests are received 66 and responses are returned. Documentation at : 67 Java HTTP Connector: /docs/config/http.html 68 Java AJP Connector: /docs/config/ajp.html 69 APR (HTTP/AJP) Connector: /docs/apr.html 70 Define a non-SSL/TLS HTTP/1.1 Connector on port 8080 71 --> 72 <Connector port="8080" protocol="HTTP/1.1" 73 connectionTimeout="20000" 74 redirectPort="8443" /> 75 <!-- A "Connector" using the shared thread pool--> 76 <!-- 77 <Connector executor="tomcatThreadPool" 78 port="8080" protocol="HTTP/1.1" 79 connectionTimeout="20000" 80 redirectPort="8443" /> 81 --> 82 <!-- Define an SSL/TLS HTTP/1.1 Connector on port 8443 83 This connector uses the NIO implementation. The default 84 SSLImplementation will depend on the presence of the APR/native 85 library and the useOpenSSL attribute of the 86 AprLifecycleListener. 87 Either JSSE or OpenSSL style configuration may be used regardless of 88 the SSLImplementation selected. JSSE style configuration is used below. 89 --> 90 <!-- 91 <Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol" 92 maxThreads="150" SSLEnabled="true"> 93 <SSLHostConfig> 94 <Certificate certificateKeystoreFile="conf/localhost-rsa.jks" 95 type="RSA" /> 96 </SSLHostConfig> 97 </Connector> 98 --> 99 <!-- Define an SSL/TLS HTTP/1.1 Connector on port 8443 with HTTP/2 100 This connector uses the APR/native implementation which always uses 101 OpenSSL for TLS. 102 Either JSSE or OpenSSL style configuration may be used. OpenSSL style 103 configuration is used below. 104 --> 105 <!-- 106 <Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol" 107 maxThreads="150" SSLEnabled="true" > 108 <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" /> 109 <SSLHostConfig> 110 <Certificate certificateKeyFile="conf/localhost-rsa-key.pem" 111 certificateFile="conf/localhost-rsa-cert.pem" 112 certificateChainFile="conf/localhost-rsa-chain.pem" 113 type="RSA" /> 114 </SSLHostConfig> 115 </Connector> 116 --> 117 118 <!-- Define an AJP 1.3 Connector on port 8009 --> 119 <!-- 120 <Connector protocol="AJP/1.3" 121 address="::1" 122 port="8009" 123 redirectPort="8443" /> 124 --> 125 126 <!-- An Engine represents the entry point (within Catalina) that processes 127 every request. The Engine implementation for Tomcat stand alone 128 analyzes the HTTP headers included with the request, and passes them 129 on to the appropriate Host (virtual host). 130 Documentation at /docs/config/engine.html --> 131 132 <!-- You should set jvmRoute to support load-balancing via AJP ie : 133 <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1"> 134 --> 135 <Engine name="Catalina" defaultHost="localhost"> 136 137 <!--For clustering, please take a look at documentation at: 138 /docs/cluster-howto.html (simple how to) 139 /docs/config/cluster.html (reference documentation) --> 140 <!-- 141 <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/> 142 --> 143 144 <!-- Use the LockOutRealm to prevent attempts to guess user passwords 145 via a brute-force attack --> 146 <Realm className="org.apache.catalina.realm.LockOutRealm"> 147 <!-- This Realm uses the UserDatabase configured in the global JNDI 148 resources under the key "UserDatabase". Any edits 149 that are performed against this UserDatabase are immediately 150 available for use by the Realm. --> 151 <Realm className="org.apache.catalina.realm.UserDatabaseRealm" 152 resourceName="UserDatabase"/> 153 </Realm> 154 155 <Host name="localhost" appBase="webapps" 156 unpackWARs="true" autoDeploy="true"> 157 158 <!-- SingleSignOn valve, share authentication between web applications 159 Documentation at: /docs/config/valve.html --> 160 <!-- 161 <Valve className="org.apache.catalina.authenticator.SingleSignOn" /> 162 --> 163 164 <!-- Access log processes all example. 165 Documentation at: /docs/config/valve.html 166 Note: The pattern used is equivalent to using pattern="common" --> 167 <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" 168 prefix="es102.yinzhengjie.com_access_log" suffix=".log" 169 pattern="{"client":"%h", "client user":"%l", "authenticated":"%u", "access time":"%t", "method":"%r", "status":"%s", "send bytes":"%b", "Query?string":"%q", "partner":"%{Referer}i", "Agent version":"%{User-Agent}i"}" /> 170 </Host> 171 </Engine> 172 </Service> 173 </Server> 174 [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# rm -f /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/* #重启tomcat服务之前记得删除之前的访问日志哟~(重启会自动生成新的文件) [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# ll /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/ total 0 drwxr-x--- 2 root root 6 Jun 6 03:13 ./ drwxr-xr-x 9 root root 220 Jun 5 04:34 ../ [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 100 *:8080 *:* LISTEN 0 128 [::ffff:172.200.5.102]:9200 *:* LISTEN 0 128 [::ffff:172.200.5.102]:9300 *:* LISTEN 0 128 [::]:22 [::]:* LISTEN 0 1 [::ffff:127.0.0.1]:8005 *:* [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# /yinzhengjie/softwares/apache-tomcat-8.5.55/bin/catalina.sh stop Using CATALINA_BASE: /yinzhengjie/softwares/apache-tomcat-8.5.55 Using CATALINA_HOME: /yinzhengjie/softwares/apache-tomcat-8.5.55 Using CATALINA_TMPDIR: /yinzhengjie/softwares/apache-tomcat-8.5.55/temp Using JRE_HOME: /yinzhengjie/softwares/jdk1.8.0_201/jre Using CLASSPATH: /yinzhengjie/softwares/apache-tomcat-8.5.55/bin/bootstrap.jar:/yinzhengjie/softwares/apache-tomcat-8.5.55/bin/tomcat-juli.jar [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 [::ffff:172.200.5.102]:9200 *:* LISTEN 0 128 [::ffff:172.200.5.102]:9300 *:* LISTEN 0 128 [::]:22 [::]:* [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 [::ffff:172.200.5.102]:9200 *:* LISTEN 0 128 [::ffff:172.200.5.102]:9300 *:* LISTEN 0 128 [::]:22 [::]:* [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# /yinzhengjie/softwares/apache-tomcat-8.5.55/bin/catalina.sh start Using CATALINA_BASE: /yinzhengjie/softwares/apache-tomcat-8.5.55 Using CATALINA_HOME: /yinzhengjie/softwares/apache-tomcat-8.5.55 Using CATALINA_TMPDIR: /yinzhengjie/softwares/apache-tomcat-8.5.55/temp Using JRE_HOME: /yinzhengjie/softwares/jdk1.8.0_201/jre Using CLASSPATH: /yinzhengjie/softwares/apache-tomcat-8.5.55/bin/bootstrap.jar:/yinzhengjie/softwares/apache-tomcat-8.5.55/bin/tomcat-juli.jar Tomcat started. [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# ss -ntl State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 100 *:8080 *:* LISTEN 0 128 [::ffff:172.200.5.102]:9200 *:* LISTEN 0 128 [::ffff:172.200.5.102]:9300 *:* LISTEN 0 128 [::]:22 [::]:* LISTEN 0 1 [::ffff:127.0.0.1]:8005 *:* [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# ll /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/ total 24 drwxr-x--- 2 root root 209 Jun 6 03:17 ./ drwxr-xr-x 9 root root 220 Jun 5 04:34 ../ -rw-r----- 1 root root 6395 Jun 6 03:17 catalina.2020-06-06.log -rw-r----- 1 root root 6395 Jun 6 03:17 catalina.out -rw-r----- 1 root root 762 Jun 6 03:18 es102.yinzhengjie.com_access_log.2020-06-06.log -rw-r----- 1 root root 0 Jun 6 03:16 host-manager.2020-06-06.log -rw-r----- 1 root root 459 Jun 6 03:17 localhost.2020-06-06.log -rw-r----- 1 root root 0 Jun 6 03:16 manager.2020-06-06.log [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# tail -10f /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log {"client":"172.200.0.1", "client user":"-", "authenticated":"-", "access time":"[06/Jun/2020:03:18:19 +0000]", "method":"GET / HTTP/1.1", "status":"200", "send bytes":"11215", "Query?string":"", "partner":"-", "Agent version":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"}{"client":"172.200.0.1", "client user":"-", "authenticated":"-", "access time":"[06/Jun/2020:03:18:19 +0000]", "method":"GET /favicon.ico HTTP/1.1", "status":"200", "send bytes" :"21630", "Query?string":"", "partner":"http://es102.yinzhengjie.com:8080/", "Agent version":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"}
[root@es102.yinzhengjie.com ~]# ll /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log -rw-r----- 1 root root 762 Jun 6 03:18 /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# chmod 644 /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# ll /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log -rw-r--r-- 1 root root 762 Jun 6 03:18 /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# ll /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/ -d drwxr-x--- 2 root root 209 Jun 6 03:17 /yinzhengjie/softwares/apache-tomcat-8.5.55/logs// [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# chmod o+x /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/ [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# ll /yinzhengjie/softwares/apache-tomcat-8.5.55/logs/ -d drwxr-x--x 2 root root 209 Jun 6 03:17 /yinzhengjie/softwares/apache-tomcat-8.5.55/logs// [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]#
2>.编写配置文件并检查语法是否错误
[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/java-elasticsearch.conf [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# cat /etc/logstash/conf.d/java-elasticsearch.conf input { file { type => "java-log" path => "/var/log/logstash/logstash-plain.log" start_position => "beginning" stat_interval => 3 } file { type => "tomcat-access-log" path => "/yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.*.log" start_position => "beginning" stat_interval => 3 codec => "json" } } output { if [type] == "java-log" { elasticsearch { hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200","http://es103.yinzhengjie.com:9200"] index => "java-log-172.200.5.102-%{+YYYY.MM.dd}" } } if [type] == "tomcat-access-log" { elasticsearch { hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200","http://es103.yinzhengjie.com:9200"] index => "tomcat-access-log-172.200.5.102-%{+YYYY.MM.dd}" } file { path => "/tmp/tomcat-access-log" } } } [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/java-elasticsearch.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-06 04:50:06.980 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [INFO ] 2020-06-06 04:50:11.656 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [root@es102.yinzhengjie.com ~]#
3>.启动logstash服务
[root@es102.yinzhengjie.com ~]# vim /etc/systemd/system/logstash.service [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# cat /etc/systemd/system/logstash.service [Unit] Description=logstash [Service] Type=simple User=root Group=root # Load env vars from /etc/default/ and /etc/sysconfig/ if they exist. # Prefixing the path with '-' makes it try to load, but if the file doesn't # exist, it continues onward. EnvironmentFile=-/etc/default/logstash EnvironmentFile=-/etc/sysconfig/logstash ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" Restart=always WorkingDirectory=/ Nice=19 LimitNOFILE=16384 [Install] WantedBy=multi-user.target [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# systemctl daemon-reload #使得上述配置生效 [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# systemctl restart logstash.service #重启logstash服务 [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# ll /tmp/tomcat-access-log -rw-r--r-- 1 root root 1152 Jun 6 05:42 /tmp/tomcat-access-log [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# cat /tmp/tomcat-access-log {"access time":"[06/Jun/2020:03:18:19 +0000]","@timestamp":"2020-06-06T05:42:17.263Z","method":"GET / HTTP/1.1","type":"tomcat-access-log","send bytes":"11215","Agent version":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36","@version":"1","client":"172.200.0.1","client user":"-","partner":"-","path":"/yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log","Query?string":"","host":"es102.yinzhengjie.com","status":"200","authenticated":"-"}{"access time":"[06/Jun/2020:03:18:19 +0000]","@timestamp":"2020-06-06T05:42:19.224Z","method":"GET /favicon.ico HTTP/1.1","type":"tomcat-access-log","send bytes":"21630","Agent version":"M ozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36","@version":"1","client":"172.200.0.1","client user":"-","partner":"http://es102.yinzhengjie.com:8080/","path":"/yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log","Query?string":"","host":"es102.yinzhengjie.com","status":"200","authenticated":"-"}[root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]#
4>.在kibana查看写入Elasticsearch集群的JSON日志
5>.统计日志的IP地址个数
[root@es102.yinzhengjie.com ~]# ll /tmp/tomcat-access-log -rw-r--r-- 1 root root 1152 Jun 6 05:42 /tmp/tomcat-access-log [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# cat /tmp/tomcat-access-log {"access time":"[06/Jun/2020:03:18:19 +0000]","@timestamp":"2020-06-06T05:42:17.263Z","method":"GET / HTTP/1.1","type":"tomcat-access-log","send bytes":"11215","Agent version":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36","@version":"1","client":"172.200.0.1","client user":"-","partner":"-","path":"/yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log","Query?string":"","host":"es102.yinzhengjie.com","status":"200","authenticated":"-"}{"access time":"[06/Jun/2020:03:18:19 +0000]","@timestamp":"2020-06-06T05:42:19.224Z","method":"GET /favicon.ico HTTP/1.1","type":"tomcat-access-log","send bytes":"21630","Agent version":"M ozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36","@version":"1","client":"172.200.0.1","client user":"-","partner":"http://es102.yinzhengjie.com:8080/","path":"/yinzhengjie/softwares/apache-tomcat-8.5.55/logs/es102.yinzhengjie.com_access_log.2020-06-06.log","Query?string":"","host":"es102.yinzhengjie.com","status":"200","authenticated":"-"}[root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# cat log.py #!/usr/bin/python3 #__conding__:uft-8 status_200=[] status_404=[] with open("/tmp/tomcat-access-log") as f: for line in f.readlines(): line = eval(line) print(line.get("client")) if line.get("status") == "200": status_200.append(line.get) elif line.get(status) == "404": status_404.append(line.get) else: print("状态码错误") print("状态码200的有: ",len(status_200)) print("状态码404的有: ",len(status_404)) [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# python3 log.py 172.200.0.1 172.200.0.1 状态码200的有: 2 状态码404的有: 0 [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]#
六.日志的多行合并案例
关于多行合并官方也有相应的案例说明，我这里就不再赘述了，感兴趣的小伙伴可自行查看官网。 博主推荐阅读: https://www.elastic.co/guide/en/logstash/6.8/plugins-codecs-multiline.html
1>.编写配置文件
[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/stdin-stdout.conf [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# cat /etc/logstash/conf.d/stdin-stdout.conf input { stdin { codec => multiline { pattern => "^[" negate => "true" what => "previous" } } } output { stdout { codec => "rubydebug" } } [root@es102.yinzhengjie.com ~]#
2>.检查配置语法是否错误
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/stdin-stdout.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-06 04:29:38.082 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [INFO ] 2020-06-06 04:29:42.272 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [root@es102.yinzhengjie.com ~]#
3>.测试多行合并
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/stdin-stdout.conf WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-06 04:19:29.466 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2020-06-06 04:19:29.479 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.9"} [INFO ] 2020-06-06 04:19:29.505 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"79da8bcd-0a33-4dab-a25a-df89c5387e12", :path=>"/usr/share/logstash/dat a/uuid"}[INFO ] 2020-06-06 04:19:34.517 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batc h.delay"=>50}[INFO ] 2020-06-06 04:19:34.635 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7c854c5f run>"} The stdin plugin is now waiting for input: [INFO ] 2020-06-06 04:19:34.718 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelin es=>[]}[INFO ] 2020-06-06 04:19:35.779 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9601} yinzhengjie 2020 blog[https://www.cnblogs.com/yinzhengjie/] bigdata [2020520 /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated { "@version" => "1", "@timestamp" => 2020-06-06T04:20:41.993Z, "message" => "yinzhengjie 2020 blog[https://www.cnblogs.com/yinzhengjie/] bigdata", "tags" => [ [0] "multiline" ], "host" => "es102.yinzhengjie.com" } yinzhengjie&jasonYin[op] LOL Python Golang [6666 { "@version" => "1", "@timestamp" => 2020-06-06T04:24:05.829Z, "message" => "[2020520 yinzhengjie&jasonYin[op] LOL Python Golang", "tags" => [ [0] "multiline" ], "host" => "es102.yinzhengjie.com" }
七.收集nginx日志
1>.安装nginx服务
[root@es102.yinzhengjie.com ~]# apt-get -y install nginx Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libjbig0 libjpeg-turbo8 libjpeg8 libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream libtiff5 libwebp6 libxpm4 nginx-common nginx-core Suggested packages: libgd-tools fcgiwrap nginx-doc ssl-cert The following NEW packages will be installed: fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libjbig0 libjpeg-turbo8 libjpeg8 libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream libtiff5 libwebp6 libxpm4 nginx nginx-common nginx-core 0 upgraded, 18 newly installed, 0 to remove and 79 not upgraded. Need to get 2,462 kB of archives. After this operation, 8,210 kB of additional disk space will be used. Get:1 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libjpeg-turbo8 amd64 1.5.2-0ubuntu5.18.04.3 [110 kB] Get:2 http://mirrors.aliyun.com/ubuntu bionic/main amd64 fonts-dejavu-core all 2.37-1 [1,041 kB] Get:3 http://mirrors.aliyun.com/ubuntu bionic/main amd64 fontconfig-config all 2.12.6-0ubuntu2 [55.8 kB] Get:4 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libfontconfig1 amd64 2.12.6-0ubuntu2 [137 kB] Get:5 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libjpeg8 amd64 8c-2ubuntu8 [2,194 B] Get:6 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libjbig0 amd64 2.1-3.1build1 [26.7 kB] Get:7 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libtiff5 amd64 4.0.9-5ubuntu0.3 [153 kB] Get:8 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libwebp6 amd64 0.6.1-2 [185 kB] Get:9 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libxpm4 amd64 1:3.5.12-1 [34.0 kB] Get:10 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libgd3 amd64 2.2.5-4ubuntu0.4 [119 kB] Get:11 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 nginx-common all 1.14.0-0ubuntu1.7 [37.4 kB] Get:12 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-http-geoip amd64 1.14.0-0ubuntu1.7 [11.2 kB] Get:13 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-http-image-filter amd64 1.14.0-0ubuntu1.7 [14.6 kB] Get:14 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-http-xslt-filter amd64 1.14.0-0ubuntu1.7 [13.0 kB] Get:15 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-mail amd64 1.14.0-0ubuntu1.7 [41.8 kB] Get:16 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 libnginx-mod-stream amd64 1.14.0-0ubuntu1.7 [63.7 kB] Get:17 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 nginx-core amd64 1.14.0-0ubuntu1.7 [413 kB] Get:18 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 nginx all 1.14.0-0ubuntu1.7 [3,596 B] Fetched 2,462 kB in 2s (1,168 kB/s) Preconfiguring packages ... Selecting previously unselected package libjpeg-turbo8:amd64. (Reading database ... 119418 files and directories currently installed.) Preparing to unpack .../00-libjpeg-turbo8_1.5.2-0ubuntu5.18.04.3_amd64.deb ... Unpacking libjpeg-turbo8:amd64 (1.5.2-0ubuntu5.18.04.3) ... Selecting previously unselected package fonts-dejavu-core. Preparing to unpack .../01-fonts-dejavu-core_2.37-1_all.deb ... Unpacking fonts-dejavu-core (2.37-1) ... Selecting previously unselected package fontconfig-config. Preparing to unpack .../02-fontconfig-config_2.12.6-0ubuntu2_all.deb ... Unpacking fontconfig-config (2.12.6-0ubuntu2) ... Selecting previously unselected package libfontconfig1:amd64. Preparing to unpack .../03-libfontconfig1_2.12.6-0ubuntu2_amd64.deb ... Unpacking libfontconfig1:amd64 (2.12.6-0ubuntu2) ... Selecting previously unselected package libjpeg8:amd64. Preparing to unpack .../04-libjpeg8_8c-2ubuntu8_amd64.deb ... Unpacking libjpeg8:amd64 (8c-2ubuntu8) ... Selecting previously unselected package libjbig0:amd64. Preparing to unpack .../05-libjbig0_2.1-3.1build1_amd64.deb ... Unpacking libjbig0:amd64 (2.1-3.1build1) ... Selecting previously unselected package libtiff5:amd64. Preparing to unpack .../06-libtiff5_4.0.9-5ubuntu0.3_amd64.deb ... Unpacking libtiff5:amd64 (4.0.9-5ubuntu0.3) ... Selecting previously unselected package libwebp6:amd64. Preparing to unpack .../07-libwebp6_0.6.1-2_amd64.deb ... Unpacking libwebp6:amd64 (0.6.1-2) ... Selecting previously unselected package libxpm4:amd64. Preparing to unpack .../08-libxpm4_1%3a3.5.12-1_amd64.deb ... Unpacking libxpm4:amd64 (1:3.5.12-1) ... Selecting previously unselected package libgd3:amd64. Preparing to unpack .../09-libgd3_2.2.5-4ubuntu0.4_amd64.deb ... Unpacking libgd3:amd64 (2.2.5-4ubuntu0.4) ... Selecting previously unselected package nginx-common. Preparing to unpack .../10-nginx-common_1.14.0-0ubuntu1.7_all.deb ... Unpacking nginx-common (1.14.0-0ubuntu1.7) ... Selecting previously unselected package libnginx-mod-http-geoip. Preparing to unpack .../11-libnginx-mod-http-geoip_1.14.0-0ubuntu1.7_amd64.deb ... Unpacking libnginx-mod-http-geoip (1.14.0-0ubuntu1.7) ... Selecting previously unselected package libnginx-mod-http-image-filter. Preparing to unpack .../12-libnginx-mod-http-image-filter_1.14.0-0ubuntu1.7_amd64.deb ... Unpacking libnginx-mod-http-image-filter (1.14.0-0ubuntu1.7) ... Selecting previously unselected package libnginx-mod-http-xslt-filter. Preparing to unpack .../13-libnginx-mod-http-xslt-filter_1.14.0-0ubuntu1.7_amd64.deb ... Unpacking libnginx-mod-http-xslt-filter (1.14.0-0ubuntu1.7) ... Selecting previously unselected package libnginx-mod-mail. Preparing to unpack .../14-libnginx-mod-mail_1.14.0-0ubuntu1.7_amd64.deb ... Unpacking libnginx-mod-mail (1.14.0-0ubuntu1.7) ... Selecting previously unselected package libnginx-mod-stream. Preparing to unpack .../15-libnginx-mod-stream_1.14.0-0ubuntu1.7_amd64.deb ... Unpacking libnginx-mod-stream (1.14.0-0ubuntu1.7) ... Selecting previously unselected package nginx-core. Preparing to unpack .../16-nginx-core_1.14.0-0ubuntu1.7_amd64.deb ... Unpacking nginx-core (1.14.0-0ubuntu1.7) ... Selecting previously unselected package nginx. Preparing to unpack .../17-nginx_1.14.0-0ubuntu1.7_all.deb ... Unpacking nginx (1.14.0-0ubuntu1.7) ... Processing triggers for ufw (0.36-0ubuntu0.18.04.1) ... Processing triggers for ureadahead (0.100.0-21) ... Setting up libjbig0:amd64 (2.1-3.1build1) ... Setting up fonts-dejavu-core (2.37-1) ... Setting up nginx-common (1.14.0-0ubuntu1.7) ... Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /lib/systemd/system/nginx.service. Setting up libjpeg-turbo8:amd64 (1.5.2-0ubuntu5.18.04.3) ... Processing triggers for libc-bin (2.27-3ubuntu1) ... Processing triggers for systemd (237-3ubuntu10.38) ... Setting up libnginx-mod-mail (1.14.0-0ubuntu1.7) ... Setting up libxpm4:amd64 (1:3.5.12-1) ... Processing triggers for man-db (2.8.3-2ubuntu0.1) ... Setting up libnginx-mod-http-xslt-filter (1.14.0-0ubuntu1.7) ... Setting up libnginx-mod-http-geoip (1.14.0-0ubuntu1.7) ... Setting up libwebp6:amd64 (0.6.1-2) ... Setting up libjpeg8:amd64 (8c-2ubuntu8) ... Setting up fontconfig-config (2.12.6-0ubuntu2) ... Setting up libnginx-mod-stream (1.14.0-0ubuntu1.7) ... Setting up libtiff5:amd64 (4.0.9-5ubuntu0.3) ... Setting up libfontconfig1:amd64 (2.12.6-0ubuntu2) ... Setting up libgd3:amd64 (2.2.5-4ubuntu0.4) ... Setting up libnginx-mod-http-image-filter (1.14.0-0ubuntu1.7) ... Setting up nginx-core (1.14.0-0ubuntu1.7) ... Setting up nginx (1.14.0-0ubuntu1.7) ... Processing triggers for ureadahead (0.100.0-21) ... Processing triggers for ufw (0.36-0ubuntu0.18.04.1) ... Processing triggers for libc-bin (2.27-3ubuntu1) ... [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# vim /etc/nginx/nginx.conf [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# cat /etc/nginx/nginx.conf worker_processes 4; worker_cpu_affinity 00000001 00000010 00000100 00001000; events { worker_connections 100000; use epoll; accept_mutex on; multi_accept on; } http { include mime.types; default_type text/html; charset utf-8; log_format my_access_json '{"@timestamp":"$time_iso8601",' '"host":"$server_addr",' '"clientip":"$remote_addr",' '"size":$body_bytes_sent,' '"responsetime":$request_time,' '"upstreamtime":"$upstream_response_time",' '"upstreamhost":"$upstream_addr",' '"http_host":"$host",' '"uri":"$uri",' '"domain":"$host",' '"xff":"$http_x_forwarded_for",' '"referer":"$http_referer",' '"tcp_xff":"$proxy_protocol_addr",' '"http_user_agent":"$http_user_agent",' '"status":"$status"}'; access_log /var/log/nginx/access.log my_access_json; error_log /var/log/nginx/error.log; gzip on; include /etc/nginx/conf.d/*.conf; include /etc/nginx/sites-enabled/*; } [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# systemctl start nginx #启动Nginx服务 [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# systemctl enable nginx #将nginx设置为开机自启动 Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable nginx [root@es102.yinzhengjie.com ~]#
2>.编写配置文件并检查语法是否错误
[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/nginx-elasticsearch.conf [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# cat /etc/logstash/conf.d/nginx-elasticsearch.conf input { file { type => "nginx-access-log" path => "/var/log/nginx/access.log" start_position => "beginning" stat_interval => 3 codec => "json" } } output { if [type] == "nginx-access-log" { elasticsearch { hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200","http://es103.yinzhengjie.com:9200"] index => "nginx-access-log-172.200.5.102-%{+YYYY.MM.dd}" } } } [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx-elasticsearch.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-06 07:57:30.291 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [INFO ] 2020-06-06 07:57:35.681 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [root@es102.yinzhengjie.com ~]#
3>.启动logstash服务
[root@es102.yinzhengjie.com ~]# systemctl start logstash
4>.在Kibana页面中添加索引
5>.查看nginx的日志
八.收集TCP日志
1>.编写配置文件并检查语法是否错误
[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/tcp-elasticsearch.conf [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# cat /etc/logstash/conf.d/tcp-elasticsearch.conf input { tcp { port => "8888" codec => "json" } } output { stdout { codec => "rubydebug" } } [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# vim /etc/logstash/conf.d/tcp-elasticsearch.conf [root@es102.yinzhengjie.com ~]# [root@es102.yinzhengjie.com ~]# cat /etc/logstash/conf.d/tcp-elasticsearch.conf input { tcp { port => "8888" codec => "json" } } output { elasticsearch { hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200","http://es103.yinzhengjie.com:9200"] index => "tcp-log-172.200.5.102-%{+YYYY.MM.dd}" } } [root@es102.yinzhengjie.com ~]#
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tcp-elasticsearch.conf -t WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-06 08:25:30.447 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified Configuration OK [INFO ] 2020-06-06 08:25:34.496 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [root@es102.yinzhengjie.com ~]#
2>.传数测试数据
[root@es102.yinzhengjie.com ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tcp-elasticsearch.conf WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN ] 2020-06-06 08:26:02.135 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO ] 2020-06-06 08:26:02.150 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.9"} [INFO ] 2020-06-06 08:26:07.011 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batc h.delay"=>50}[INFO ] 2020-06-06 08:26:07.155 [[main]-pipeline-manager] tcp - Automatically switching from json to json_lines codec {:plugin=>"tcp"} [INFO ] 2020-06-06 08:26:07.498 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xfcc6877 run>"} [INFO ] 2020-06-06 08:26:07.564 [[main]<tcp] tcp - Starting tcp input listener {:address=>"0.0.0.0:8888", :ssl_enable=>"false"} [INFO ] 2020-06-06 08:26:07.612 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelin es=>[]}[INFO ] 2020-06-06 08:26:08.000 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600} /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated { "@timestamp" => 2020-06-06T08:34:26.207Z, "port" => 57610, "password" => "123", "@version" => "1", "host" => "es103.yinzhengjie.com", "username" => "yinzhengjie" } { "@timestamp" => 2020-06-06T08:36:02.000Z, "port" => 57614, "password" => "666", "@version" => "1", "host" => "es103.yinzhengjie.com", "username" => "yinzhengjie" }
[root@es103.yinzhengjie.com ~]# apt-get -y install nmap Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: libblas3 liblinear3 liblua5.3-0 Suggested packages: liblinear-tools liblinear-dev ndiff The following NEW packages will be installed: libblas3 liblinear3 liblua5.3-0 nmap 0 upgraded, 4 newly installed, 0 to remove and 79 not upgraded. Need to get 5,467 kB of archives. After this operation, 25.0 MB of additional disk space will be used. Get:1 http://mirrors.aliyun.com/ubuntu bionic/main amd64 libblas3 amd64 3.7.1-4ubuntu1 [140 kB] Get:2 http://mirrors.aliyun.com/ubuntu bionic/main amd64 liblinear3 amd64 2.1.0+dfsg-2 [39.3 kB] Get:3 http://mirrors.aliyun.com/ubuntu bionic-security/main amd64 liblua5.3-0 amd64 5.3.3-1ubuntu0.18.04.1 [115 kB] Get:4 http://mirrors.aliyun.com/ubuntu bionic/main amd64 nmap amd64 7.60-1ubuntu5 [5,174 kB] Fetched 5,467 kB in 1s (4,379 kB/s) Selecting previously unselected package libblas3:amd64. (Reading database ... 119654 files and directories currently installed.) Preparing to unpack .../libblas3_3.7.1-4ubuntu1_amd64.deb ... Unpacking libblas3:amd64 (3.7.1-4ubuntu1) ... Selecting previously unselected package liblinear3:amd64. Preparing to unpack .../liblinear3_2.1.0+dfsg-2_amd64.deb ... Unpacking liblinear3:amd64 (2.1.0+dfsg-2) ... Selecting previously unselected package liblua5.3-0:amd64. Preparing to unpack .../liblua5.3-0_5.3.3-1ubuntu0.18.04.1_amd64.deb ... Unpacking liblua5.3-0:amd64 (5.3.3-1ubuntu0.18.04.1) ... Selecting previously unselected package nmap. Preparing to unpack .../nmap_7.60-1ubuntu5_amd64.deb ... Unpacking nmap (7.60-1ubuntu5) ... Setting up libblas3:amd64 (3.7.1-4ubuntu1) ... update-alternatives: using /usr/lib/x86_64-linux-gnu/blas/libblas.so.3 to provide /usr/lib/x86_64-linux-gnu/libblas.so.3 (libblas.so.3-x86_64-linux-gnu) in auto mode Processing triggers for libc-bin (2.27-3ubuntu1) ... Processing triggers for man-db (2.8.3-2ubuntu0.1) ... Setting up liblinear3:amd64 (2.1.0+dfsg-2) ... Setting up liblua5.3-0:amd64 (5.3.3-1ubuntu0.18.04.1) ... Setting up nmap (7.60-1ubuntu5) ... Processing triggers for libc-bin (2.27-3ubuntu1) ... [root@es103.yinzhengjie.com ~]#
[root@es103.yinzhengjie.com ~]# echo "{"username":"yinzhengjie","password":"123"}" | nc -q 1 es102.yinzhengjie.com 8888 #nc命令默认是安装的，可以直接使用 [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# [root@es103.yinzhengjie.com ~]# echo "{"username":"yinzhengjie","password":"666"}" | ncat es102.yinzhengjie.com 8888 #需要安装nmap工具才能使用ncat命令哟~ [root@es103.yinzhengjie.com ~]#
3>.传输文件到logstash服务器
[root@es103.yinzhengjie.com ~]# cat /var/log/syslog | ncat es102.yinzhengjie.com 8888
