1、版本介绍
Elasticsearch：
版本：2.4.0
Logstash：
版本：2.4.0
所需要的安装文件，到官网下载即可。
还需要对应的数据库JDBC，这里使用的是mysql-connector-java-5.1.39.jar
2、Logstash安装配置
Logstash直接下载后解压即可，主要是配置文件的内容编写。
安装logstash-input-jdbc的Logstash插件，用来进行mysql、oracle等的数据同步。
[zsz@VS-zsz logstash-2.4.0]$ bin/plugin install logstash-input-jdbc
The use of bin/plugin is deprecated and will be removed in a feature release. Please use bin/logstash-plugin.
Validating logstash-input-jdbc
Installing logstash-input-jdbc
Installation successful
The use of bin/plugin is deprecated and will be removed in a feature release. Please use bin/logstash-plugin.
Validating logstash-input-jdbc
Installing logstash-input-jdbc
Installation successful
配置文件（自行指定文件名，这里命名为logstash-mysql.conf ）：
[zsz@VS-zsz conf]$ vi logstash-mysql.conf
input {
jdbc {
jdbc_driver_library => "/usr/local/logstash-2.4.0/mysql-connector-java-5.1.39.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://192.168.****:3306/******?characterEncoding=UTF-8&useSSL=false"
jdbc_user => "*****"
jdbc_password => "*********"
statement => "SELECT * FROM news limit 0,1"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
schedule => "* * * * *"
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => "192.168.****" //所有同步elasticsearch的集群信息
index => "myindex"
}
}
jdbc {
jdbc_driver_library => "/usr/local/logstash-2.4.0/mysql-connector-java-5.1.39.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://192.168.****:3306/******?characterEncoding=UTF-8&useSSL=false"
jdbc_user => "*****"
jdbc_password => "*********"
statement => "SELECT * FROM news limit 0,1"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
schedule => "* * * * *"
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => "192.168.****" //所有同步elasticsearch的集群信息
index => "myindex"
}
}
启动Logstash
[zsz@VS-zsz conf]$ /usr/local/logstash-2.4.0/bin/logstash -f /usr/local/logstash-2.4.0/conf/logstash-mysql.conf
Settings: Default pipeline workers: 2
Pipeline main started
{
"pk" => 8,
"user_pk" => 11,
"media_pk" => 5,
"topic_pk" => 0
}
{
"pk" => 8,
"user_pk" => 11,
"media_pk" => 5,
"topic_pk" => 0
}
Pipeline main started
{
"pk" => 8,
"user_pk" => 11,
"media_pk" => 5,
"topic_pk" => 0
}
{
"pk" => 8,
"user_pk" => 11,
"media_pk" => 5,
"topic_pk" => 0
}
........
这个进程会一直执行下去，因为设置的schedule => "* * * * *"（每分钟执行一次），如果想结束进程需要kill掉进程。
查看elasticsearch是否同步数据成功
[root@VS-zsz conf]# curl '192.168.31.79:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open test 5 1 0 0 1.5kb 795b
green open myindex 5 1 494 0 924.7kb 457.6kb
[root@VS-zsz conf]# curl '192.168.31.78:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open test 5 1 0 0 1.5kb 795b
green open myindex 5 1 494 0 925kb 457.8kb
[root@VS-zsz conf]# curl '192.168.31.79:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open test 5 1 0 0 1.5kb 795b
green open myindex 5 1 494 0 925kb 457.8kb
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open test 5 1 0 0 1.5kb 795b
green open myindex 5 1 494 0 924.7kb 457.6kb
[root@VS-zsz conf]# curl '192.168.31.78:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open test 5 1 0 0 1.5kb 795b
green open myindex 5 1 494 0 925kb 457.8kb
[root@VS-zsz conf]# curl '192.168.31.79:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open test 5 1 0 0 1.5kb 795b
green open myindex 5 1 494 0 925kb 457.8kb
说明数据成功导入，而且在设置了定时任务的情况下， myindex索引的容量不断增加。
5、常见错误：
（1）Pipeline aborted due to error {:exception=>"LogStash::ConfigurationError", :backtrace=>..................stopping pipeline {:id=>"main"}
原因：logstash-mysql.conf 文件配置错误，对于>=2.*的版本， elasticsearch 的参数名应为hosts，如果设置为host则会报错。此处应该是可以配置多个数据源。