Elasticsearch 与 Mongodb 数据同步问题

1、mongo-connector工具

首先安装python环境

wget http://www.python.org/ftp/python/3.0.1/Python-3.0.1.tgz

tar -zxvf Python-3.0.1.tgz

cd Python-3.0.1

./configure

make all

make install

看来新装的版本生效了，做个软连接应该就OK。

ln -s /usr/local/bin/python3.0 /usr/bin/python

python -V

安装pip

$ wget https://bootstrap.pypa.io/get-pip.py

$ python get-pip.py

$ pip -V　　#查看pip版本

安装mongo-connector

如果是 Elasticsearch 5.x

pip install 'mongo-connector[elastic5]'

[Reference] https://github.com/mongodb-labs/mongo-connector

执行命令：

mongo-connector -m 192.9.8.204:27017 -t 192.9.8.223:9200 -d elastic2_doc_manager -n wymlib.ym_literature

-m :mongodb地址，-t：目的地址，-d : Used to specify the path to each doc manager file that will be used. -n:指定数据库

2、elasticsearch-river-mongodb

该工具只支持 elasticsearch 1.7.3 及以下版本

bin/plugin --install com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb/2.0.9

mongodb-river同步信息属性配置完整版：

 1 $ curl -XPUT "localhost:9200/_river/${es.river.name}/_meta" -d '
 2 {
 3   "type": "mongodb",
 4   "mongodb": { 
 5     "servers":
 6     [
 7       { "host": ${mongo.instance1.host}, "port": ${mongo.instance1.port} },
 8       { "host": ${mongo.instance2.host}, "port": ${mongo.instance2.port} }
 9     ],
10     "options": { 
11       "secondary_read_preference" : true, 
12       "drop_collection": ${mongo.drop.collection}, 
13       "exclude_fields": ${mongo.exclude.fields},
14       "include_fields": ${mongo.include.fields},
15       "include_collection": ${mongo.include.collection},
16       "import_all_collections": ${mongo.import.all.collections},
17       "initial_timestamp": {
18         "script_type": ${mongo.initial.timestamp.script.type},
19         "script": ${mongo.initial.timestamp.script}
20       },
21       "skip_initial_import" : ${mongo.skip.initial.import},
22       "store_statistics" : ${mongo.store.statistics},
23     },
24     "credentials":
25     [
26       { "db": "local", "user": ${mongo.local.user}, "password": ${mongo.local.password} },
27       { "db": "admin", "user": ${mongo.db.user}, "password": ${mongo.db.password} }
28     ],
29     "db": ${mongo.db.name}, 
30     "collection": ${mongo.collection.name}, 
31     "gridfs": ${mongo.is.gridfs.collection},
32     "filter": ${mongo.filter}
33   }, 
34   "index": { 
35     "name": ${es.index.name}, 
36     "throttle_size": ${es.throttle.size},
37     "bulk_size": ${es.bulk.size},
38     "type": ${es.type.name}
39     "bulk": {
40       "actions": ${es.bulk.actions},
41       "size": ${es.bulk.size},
42       "concurrent_requests": ${es.bulk.concurrent.requests},
43       "flush_interval": ${es.bulk.flush.interval}
44     }
45   }
46 }'

[Reference] https://github.com/richardwilly98/elasticsearch-river-mongodb