CentOS7.4+MongoBD3.6.4集群(Shard)部署以及大数据量入库

前言

mongodb支持自动分片，集群自动的切分数据，做负载均衡。避免上面的分片管理难度。mongodb分片是将集合切合成小块，分散到若干片里面，每个片负责所有数据的一部分。这些块对应用程序来说是透明的，不需要知道哪些数据分布到哪些片上，甚至不在乎是否有做过分片，应用程序连接mongos进程，mongos知道数据和片的对应关系，将客户端请求转发到正确的片上，如果请求有了响应，mongos将结果收集起来返回给客户端程序。

分片适用场景：

1）服务器磁盘不够用

2）单个mongod不能满足日益频繁写请求

3）将大量数据存放于内存中提高性能

      建立分片需要三种角色：

1.shard server
      保存实际数据容器。每个shard可以是单个的mongod实例，也可以是复制集，即使片内又多台服务器，只能有一个主服务器，其他的保存相同数据的副本。为了实现每个shard内部的auto-failover，强烈建议为每个shard配置一组Replica Set。

2.config server

      为了将一个特定的collection 存储在多个shard 中，需要为该collection 指定一个shardkey，shardkey 可以决定该条记录属于哪个chunk。Config Servers 就是用来存储：所有shard 节点的配置信息、每个chunk 的shardkey 范围、chunk 在各shard 的分布情况、该集群中所有DB 和collection 的sharding 配置信息。

3.route server
      集群前端路由，路由所有请求，然后将结果聚合。客户端由此接入，询问config server需要到哪些shard上查询或保存数据，再连接到相应的shard进行操作，最后将结果返回给客户端。客户端只需要将原先发送给mongod的请求原封不动的发给mongos(即route server)即可，不必知道数据分布在哪个shard上。

shard key：设置分片时，需要从集合中选一个键，作为数据拆分的依据，这个键就是shard key。shard key的选择决定了插入操作在片之间的分布。shard key保证足够的不一致性，数据才能更好的分布到多台服务器上。同时保持块在一个合理的规模是非常重要的，这样数据平衡和移动块不会消耗大量的资源。

1. 参考说明

参考文档：

https://www.cnblogs.com/ityouknow/p/7566682.html

https://docs.mongodb.com/manual/tutorial/deploy-shard-cluster/

2. 安装环境说明

2.1. 环境说明

	服务器
主机名	server1.smartmap.com	server2.smartmap.com	server3.smartmap.com	server4.smartmap.com
IP	192.168.1.31	192.168.1.32	192.168.1.33	192.168.1.34
Subnet mask	255.255.255.0	255.255.255.0	255.255.255.0	255.255.255.0
Gateway	192.168.1.1	192.168.1.1	192.168.1.1	192.168.1.1
DNS	218.30.19.50 61.134.1.5	218.30.19.50 61.134.1.5	218.30.19.50 61.134.1.5	218.30.19.50 61.134.1.5
	运行服务
	Config	Config	Config	Route
	Shard1	Shard1	Shard1	Shard2
	Shard2	Shard2	Shard3	Shard3
	Shard3	Shard4	Shard4	Shard4
	Route

服务	端口
Route	20000
Config	21000
Shard1	27001
Shard2	27002
Shard3	27003
Shard4	27004

2.2. 安装基础软件

[root@server1~]# yum install unzip wget ntp

2.3. 安装与配置NTP

[root@server1 ~]# yum update

[root@server1 ~]# yum install unzip wget ntp

[root@server1 ~]# systemctl is-enabled ntpd

disabled

[root@server1 ~]# systemctl enable ntpd

Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.

[root@server1 ~]# systemctl start ntpd

[root@server1 ~]# ntpdate -u 2.asia.pool.ntp.org

2.4. 添加用户

[root@server1 ~]# useradd mongodb

[root@server1 ~]# passwd mongodb

[root@server1 ~]# chmod u+w /etc/sudoers

[root@server1 ~]#

[root@server1 ~]# vi /etc/sudoers

添加如下内容：

mongodb ALL=(ALL) NOPASSWD: ALL

3. 安装

3.1. MongoDB下载

https://www.mongodb.com/download-center?jmp=nav#community

[root@server1 ~]# mkdir /opt/mongodb

[root@server1 ~]# chown -R mongodb:mongodb /opt/mongodb/

3.2. MongoDB解压

[root@server1 ~]# su – mongodb

[mongodb@server1 mongodb]$ cd /opt/mongodb

[mongodb@server1 mongodb]$ tar -zxvf mongodb-linux-x86_64-rhel70-3.6.4.tgz

[mongodb@server1 mongodb]$ mv mongodb-linux-x86_64-rhel70-3.6.4 mongodb-app

3.3. 创建相关目录

在所有的服务器上建立conf、mongos、config、shard1、shard2、shard3、shard4、目录，因为mongos不存储数据，只需要建立日志文件目录即可

[mongodb@server1 mongodb]$ mkdir -p /opt/mongodb/mongodb-app/conf

[mongodb@server1 mongodb]$ mkdir -p /opt/mongodb/mongodb-app/data/mongos/log

[mongodb@server1 mongodb]$ mkdir -p /opt/mongodb/mongodb-app/data/config/data

[mongodb@server1 mongodb]$ mkdir -p /opt/mongodb/mongodb-app/data/config/log

[mongodb@server1 mongodb]$ mkdir -p /opt/mongodb/mongodb-app/data/shard1/data

[mongodb@server1 mongodb]$ mkdir -p /opt/mongodb/mongodb-app/data/shard1/log

[mongodb@server1 mongodb]$ mkdir -p /opt/mongodb/mongodb-app/data/shard2/data

[mongodb@server1 mongodb]$ mkdir -p /opt/mongodb/mongodb-app/data/shard2/log

[mongodb@server1 mongodb]$ mkdir -p /opt/mongodb/mongodb-app/data/shard3/data

[mongodb@server1 mongodb]$ mkdir -p /opt/mongodb/mongodb-app/data/shard3/log

[mongodb@server1 mongodb]$ mkdir -p /opt/mongodb/mongodb-app/data/shard4/data

[mongodb@server1 mongodb]$ mkdir -p /opt/mongodb/mongodb-app/data/shard4/log

3.4. 复制到其它节点

[mongodb@server2 ~]$ scp -r mongodb@192.168.1.31:/opt/mongodb/mongodb-app /opt/mongodb/

[mongodb@server3 ~]$ scp -r mongodb@192.168.1.31:/opt/mongodb/mongodb-app /opt/mongodb/

[mongodb@server4 ~]$ scp -r mongodb@192.168.1.31:/opt/mongodb/mongodb-app /opt/mongodb/

3.5. 环境变量

[mongodb@server1 mongodb]$ sudo vi /etc/profile

export MONGODB_HOME=/opt/mongodb/mongodb-app

export PATH=$MONGODB_HOME/bin:$PATH

[mongodb@server1 mongodb]$ source /etc/profile

[mongodb@server1 mongodb]$ mongod -v

4. 配置

4.1. config server配置服务器

4.1.1. 创建配置文件

服务器31，32，33上配置以下内容

[mongodb@server1 mongodb]$ vi /opt/mongodb/mongodb-app/conf/config.conf

## content

systemLog:

destination: file

logAppend: true

path: /opt/mongodb/mongodb-app/data/config/log/config.log

# Where and how to store data.

storage:

dbPath: /opt/mongodb/mongodb-app/data/config/data

journal:

enabled: true

# how the process runs

processManagement:

fork: true

pidFilePath: /opt/mongodb/mongodb-app/data/config/log/configsrv.pid

# network interfaces

net:

port: 21000

bindIp: 0.0.0.0

#operationProfiling:

replication:

replSetName: config

sharding:

clusterRole: configsvr

4.1.2. 启动三台服务器的config server

[mongodb@server1 mongodb]$ mongod --config /opt/mongodb/mongodb-app/conf/config.conf

about to fork child process, waiting until server is ready for connections.

forked process: 8072

child process started successfully, parent exiting

[mongodb@server1 mongodb]$

[mongodb@server1 mongodb]$ mongod --shutdown --config /opt/mongodb/mongodb-app/conf/config.conf

4.1.3. 登录任意一台配置服务器，初始化配置副本集

[mongodb@server1 conf]$ mongo 192.168.1.31:21000

MongoDB shell version v3.6.4

connecting to: mongodb://192.168.1.31:21000/test

MongoDB server version: 3.6.4

> use admin;

switched to db admin

> config = {

_id : "config",

members : [

{_id : 0, host : "192.168.1.31:21000" },

{_id : 1, host : "192.168.1.32:21000" },

{_id : 2, host : "192.168.1.33:21000" }

]

}

> rs.initiate(config);

config:SECONDARY> rs.status();

mongo 192.168.1.31:21000

#切换数据库

use admin;

#定义配置变量

config = {

_id : "config",

members : [

{_id : 0, host : "192.168.1.31:21000" },

{_id : 1, host : "192.168.1.32:21000" },

{_id : 2, host : "192.168.1.33:21000" }

]

}

#初始化副本集

rs.initiate(config);

#查看分区状态

rs.status();

4.2. shard server配置服务器

4.2.1. shard1配置

4.2.1.1.创建配置文件

服务器31，32，33上配置以下内容

[mongodb@server1 mongodb]$ vi /opt/mongodb/mongodb-app/conf/shard1.conf

# shard1 config

systemLog:

destination: file

logAppend: true

path: /opt/mongodb/mongodb-app/data/shard1/log/shard1.log

# Where and how to store data.

storage:

dbPath: /opt/mongodb/mongodb-app/data/shard1/data

journal:

enabled: true

wiredTiger:

engineConfig:

cacheSizeGB: 20

# how the process runs

processManagement:

fork: true

pidFilePath: /opt/mongodb/mongodb-app/data/shard1/log/shard1.pid

# network interfaces

net:

port: 27001

bindIp: 0.0.0.0

#operationProfiling:

replication:

replSetName: shard1

sharding:

clusterRole: shardsvr

4.2.1.2.启动三台服务器的shard1 server

[mongodb@server1 mongodb]$ mongod --config /opt/mongodb/mongodb-app/conf/shard1.conf

about to fork child process, waiting until server is ready for connections.

forked process: 8420

child process started successfully, parent exiting

[mongodb@server1 mongodb]$

[mongodb@server1 mongodb]$ mongod --shutdown --config /opt/mongodb/mongodb-app/conf/shard1.conf

killing process with pid: 8420

4.2.1.3.登录任意一台配置服务器，初始化配置副本集

[mongodb@server1 mongodb]$ mongo 192.168.1.31:27001

MongoDB shell version v3.6.4

connecting to: mongodb://192.168.1.31:27001/test

MongoDB server version: 3.6.4

> use admin;

switched to db admin

> config = {

_id : "shard1",

members : [

{_id : 0, host : "192.168.1.31:27001" },

{_id : 1, host : "192.168.1.32:27001" },

{_id : 2, host : "192.168.1.33:27001" }

]

}

> rs.initiate(config);

{ "ok" : 1 }

shard1:SECONDARY> rs.status();

4.2.2. shard2配置

4.2.2.1.创建配置文件

服务器34，31，32上配置以下内容

[mongodb@server4 mongodb]$ vi /opt/mongodb/mongodb-app/conf/shard2.conf

# shard2 config

systemLog:

destination: file

logAppend: true

path: /opt/mongodb/mongodb-app/data/shard2/log/shard2.log

# Where and how to store data.

storage:

dbPath: /opt/mongodb/mongodb-app/data/shard2/data

journal:

enabled: true

wiredTiger:

engineConfig:

cacheSizeGB: 20

# how the process runs

processManagement:

fork: true

pidFilePath: /opt/mongodb/mongodb-app/data/shard2/log/shard2.pid

# network interfaces

net:

port: 27002

bindIp: 0.0.0.0

#operationProfiling:

replication:

replSetName: shard2

sharding:

clusterRole: shardsvr

4.2.2.2.启动三台服务器的shard2 server

[mongodb@server4 mongodb]$ mongod --config /opt/mongodb/mongodb-app/conf/shard2.conf

about to fork child process, waiting until server is ready for connections.

forked process: 8025

child process started successfully, parent exiting

[mongodb@server4 mongodb]$

[mongodb@server1 mongodb]$ mongod --shutdown --config /opt/mongodb/mongodb-app/conf/shard2.conf

killing process with pid: 8420

4.2.2.3.登录任意一台配置服务器，初始化配置副本集

[mongodb@server4 mongodb]$ mongo 192.168.1.34:27002

MongoDB shell version v3.6.4

connecting to: mongodb://192.168.1.34:27002/test

MongoDB server version: 3.6.4

Welcome to the MongoDB shell.

> use admin;

switched to db admin

> config = {

_id : "shard2",

members : [

{_id : 0, host : "192.168.1.34:27002" },

{_id : 1, host : "192.168.1.31:27002" },

{_id : 2, host : "192.168.1.32:27002" }

]

}

> rs.initiate(config);

{ "ok" : 1 }

shard2:OTHER> rs.status();

4.2.3. shard3配置

4.2.3.1.创建配置文件

服务器33，34，31上配置以下内容

[mongodb@server3 mongodb]$ vi /opt/mongodb/mongodb-app/conf/shard3.conf

# shard3 config

systemLog:

destination: file

logAppend: true

path: /opt/mongodb/mongodb-app/data/shard3/log/shard3.log

# Where and how to store data.

storage:

dbPath: /opt/mongodb/mongodb-app/data/shard3/data

journal:

enabled: true

wiredTiger:

engineConfig:

cacheSizeGB: 20

# how the process runs

processManagement:

fork: true

pidFilePath: /opt/mongodb/mongodb-app/data/shard3/log/shard3.pid

# network interfaces

net:

port: 27003

bindIp: 0.0.0.0

#operationProfiling:

replication:

replSetName: shard3

sharding:

clusterRole: shardsvr

4.2.3.2.启动三台服务器的shard3 server

[mongodb@server3 mongodb]$ mongod --config /opt/mongodb/mongodb-app/conf/shard3.conf

about to fork child process, waiting until server is ready for connections.

forked process: 8179

child process started successfully, parent exiting

[mongodb@server3 mongodb]$

[mongodb@server1 mongodb]$ mongod --shutdown --config /opt/mongodb/mongodb-app/conf/shard3.conf

killing process with pid: 8420

4.2.3.3.登录任意一台配置服务器，初始化配置副本集

[mongodb@server3 mongodb]$ mongo 192.168.1.33:27003

MongoDB shell version v3.6.4

connecting to: mongodb://192.168.1.33:27003/test

MongoDB server version: 3.6.4

Welcome to the MongoDB shell.

> use admin

switched to db admin

> config = {

_id : "shard3",

members : [

{_id : 0, host : "192.168.1.33:27003" },

{_id : 1, host : "192.168.1.34:27003" },

{_id : 2, host : "192.168.1.31:27003" }

]

}

> rs.initiate(config);

{ "ok" : 1 }

shard3:OTHER> rs.status();

4.2.4. shard4配置

4.2.4.1.创建配置文件

服务器32，33，34上配置以下内容

[mongodb@server2 mongodb]$ vi /opt/mongodb/mongodb-app/conf/shard4.conf

# shard4 config

systemLog:

destination: file

logAppend: true

path: /opt/mongodb/mongodb-app/data/shard4/log/shard4.log

# Where and how to store data.

storage:

dbPath: /opt/mongodb/mongodb-app/data/shard4/data

journal:

enabled: true

wiredTiger:

engineConfig:

cacheSizeGB: 20

# how the process runs

processManagement:

fork: true

pidFilePath: /opt/mongodb/mongodb-app/data/shard4/log/shard4.pid

# network interfaces

net:

port: 27004

bindIp: 0.0.0.0

#operationProfiling:

replication:

replSetName: shard4

sharding:

clusterRole: shardsvr

4.2.4.2.启动三台服务器的shard4 server

[mongodb@server2 mongodb]$ mongod --config /opt/mongodb/mongodb-app/conf/shard4.conf

about to fork child process, waiting until server is ready for connections.

forked process: 8436

child process started successfully, parent exiting

[mongodb@server4 mongodb]$ mongod --shutdown --config /opt/mongodb/mongodb-app/conf/shard4.conf

killing process with pid: 8238

4.2.4.3.登录任意一台配置服务器，初始化配置副本集

[mongodb@server2 mongodb]$ mongo 192.168.1.32:27004

MongoDB shell version v3.6.4

connecting to: mongodb://192.168.1.32:27004/test

MongoDB server version: 3.6.4

Welcome to the MongoDB shell.

For interactive help, type "help".

> use admin

switched to db admin

> config = {

_id : "shard4",

members : [

{_id : 0, host : "192.168.1.32:27004" },

{_id : 1, host : "192.168.1.33:27004" },

{_id : 2, host : "192.168.1.34:27004" }

]

}

> rs.initiate(config);

{ "ok" : 1 }

shard4:OTHER> rs.status();

4.3. 配置路由服务器 mongos

4.3.1. 创建配置文件

服务器31，34上配置以下内容

[mongodb@server1 mongodb]$ vi /opt/mongodb/mongodb-app/conf/mongos.conf

systemLog:

destination: file

logAppend: true

path: /opt/mongodb/mongodb-app/data/mongos/log/mongos.log

processManagement:

fork: true

# pidFilePath: /opt/mongodb/mongodb-app/data/mongos/log/mongos.pid

# network interfaces

net:

port: 20000

bindIp: 0.0.0.0

# 监听的配置服务器,只能有1个或者3个 config为配置服务器的副本集名字

sharding:

configDB: config/192.168.1.31:21000,192.168.1.32:21000,192.168.1.33:21000

4.3.2. 启动三台服务器的config server

注意：先启动配置服务器和分片服务器,后启动路由实例

[mongodb@server1 conf]$ mongos --config /opt/mongodb/mongodb-app/conf/mongos.conf

about to fork child process, waiting until server is ready for connections.

forked process: 9869

child process started successfully, parent exiting

4.4. 启用分片

目前搭建了mongodb配置服务器、路由服务器，各个分片服务器，不过应用程序连接到mongos路由服务器并不能使用分片机制，还需要在程序里设置分片配置，让分片生效。

登陆任意一台mongos

[mongodb@server1 data]$ mongo 192.168.1.31:20000/admin

MongoDB shell version v3.6.4

connecting to: mongodb://192.168.1.31:20000/admin

MongoDB server version: 3.6.4

mongos> use admin;

switched to db admin

mongos> sh.addShard("shard1/192.168.1.31:27001,192.168.1.32:27001,192.168.1.33:27001")

mongos> sh.addShard("shard2/192.168.1.34:27002,192.168.1.31:27002,192.168.1.32:27002")

mongos> sh.addShard("shard3/192.168.1.33:27003,192.168.1.34:27003,192.168.1.31:27003")

mongos> sh.addShard("shard4/192.168.1.32:27004,192.168.1.33:27004,192.168.1.34:27004")

mongos> sh.status()

--- Sharding Status ---

sharding version: {

"_id" : 1,

"minCompatibleVersion" : 5,

"currentVersion" : 6,

"clusterId" : ObjectId("5b00d641da35619896a78891")

}

shards:

{ "_id" : "shard1", "host" : "shard1/192.168.1.31:27001,192.168.1.32:27001,192.168.1.33:27001", "state" : 1 }

{ "_id" : "shard2", "host" : "shard2/192.168.1.31:27002,192.168.1.32:27002,192.168.1.34:27002", "state" : 1 }

{ "_id" : "shard3", "host" : "shard3/192.168.1.31:27003,192.168.1.33:27003,192.168.1.34:27003", "state" : 1 }

{ "_id" : "shard4", "host" : "shard4/192.168.1.32:27004,192.168.1.33:27004,192.168.1.34:27004", "state" : 1 }

active mongoses:

"3.6.4" : 2

autosplit:

Currently enabled: yes

balancer:

Currently enabled: yes

Currently running: no

Failed balancer rounds in last 5 attempts: 0

Migration Results for the last 24 hours:

No recent migrations

databases:

{ "_id" : "config", "primary" : "config", "partitioned" : true }

mongos>

mongos> sh.getBalancerState()

true

mongos> sh.startBalancer()

{

"ok" : 1,

"$clusterTime" : {

"clusterTime" : Timestamp(1526784919, 4),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

"operationTime" : Timestamp(1526784919, 4)

}

5. 验证

5.1. 列出shard

[mongodb@server1 data]$ mongo 192.168.1.31:20000/admin

MongoDB shell version v3.6.4

connecting to: mongodb://192.168.1.31:20000/admin

MongoDB server version: 3.6.4

mongos> use admin;

switched to db admin

mongos> db.runCommand({ listshards:1 }); #列出 shard 个数

{

"shards" : [

{

"_id" : "shard1",

"host" : "shard1/192.168.1.31:27001,192.168.1.32:27001,192.168.1.33:27001",

"state" : 1

{

"_id" : "shard2",

"host" : "shard2/192.168.1.31:27002,192.168.1.32:27002,192.168.1.34:27002",

"state" : 1

{

"_id" : "shard3",

"host" : "shard3/192.168.1.31:27003,192.168.1.33:27003,192.168.1.34:27003",

"state" : 1

{

"_id" : "shard4",

"host" : "shard4/192.168.1.32:27004,192.168.1.33:27004,192.168.1.34:27004",

"state" : 1

}

"ok" : 1,

"$clusterTime" : {

"clusterTime" : Timestamp(1526786005, 2),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

"operationTime" : Timestamp(1526786005, 2)

}

5.2. 创建 Shard功能的数据库与表

[mongodb@server1 data]$ mongo 192.168.1.31:20000/admin

MongoDB shell version v3.6.4

connecting to: mongodb://192.168.1.31:20000/admin

MongoDB server version: 3.6.4

mongos> use admin;

switched to db admin

mongos> db.runCommand({ enablesharding: "RHY" });

{

"ok" : 1,

"$clusterTime" : {

"clusterTime" : Timestamp(1526791888, 10),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

"operationTime" : Timestamp(1526791888, 10)

}

mongos> db.runCommand({ shardcollection: "RHY.ST_RIVER_R", key: {STCD: 1}, unique: false })

{

"collectionsharded" : "RHY.ST_RIVER_R",

"collectionUUID" : UUID("370a8a62-cd5d-488a-b752-6fedba5da507"),

"ok" : 1,

"$clusterTime" : {

"clusterTime" : Timestamp(1526791904, 14),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

}

"operationTime" : Timestamp(1526791904, 14)

}

mongos>

5.3. 向单个MongoDB中导入CSV数据

[root@server1 bin]# ./mongoimport -d RHY -c ST_STBPRP_B --type csv --headerline --file /opt/mongodb/mydata/ST_STBPRP_B.CSV

2018-05-20T16:18:37.849+0800 connected to: localhost

2018-05-20T16:18:38.748+0800 imported 27276 documents

5.4. 向Shard中导入数据

5.4.1. 安装Python包

pip install pymongo

5.4.2. Python访问MongoDB的Shard集群

import sys

import json

import pymongo

import datetime

from pymongo import MongoClient

client = MongoClient('mongodb://192.168.1.31:20000,192.168.1.34:20000')

db = client.RHY

collection = db.ST_RIVER_R

f = open("D:/bigdata/st_river_r.CSV")

line = f.readline()

print(line)

fieldNames = line.split(',')

# STCD,TM,Z,Q,XSA,XSAVV,XSMXV,FLWCHRCD,WPTN,MSQMT,MSAMT,MSVMT

line = f.readline()

count = 0

records = []

insertCount = 0

while line:

count = count + 1

fieldValues = line.split(',')

iflen(fieldValues) == 12or fieldValues[0].strip() != '':

insertObj = {}

STCD = fieldValues[0]

insertObj['STCD'] = STCD

TM = fieldValues[1]

if TM.strip() != '':

TM = datetime.datetime.strptime(TM, '%Y-%m-%d %H:%M:%S')

insertObj['TM'] = TM

Z = fieldValues[2]

if Z.strip() != '':

Z = float(Z)

insertObj['Z'] = Z

Q = fieldValues[3]

if Q.strip() != '':

Q = float(Q)

insertObj['Q'] = Q

# XSA

XSA = fieldValues[4]

if XSA.strip() != '':

XSA = float(XSA)

insertObj['XSA'] = XSA

# XSAVV

XSAVV = fieldValues[5]

if XSAVV.strip() != '':

XSAVV = float(XSAVV)

insertObj['XSAVV'] = XSAVV

XSMXV = fieldValues[6]

if XSMXV.strip() != '':

XSMXV = float(XSMXV)

insertObj['XSMXV'] = XSMXV

FLWCHRCD = fieldValues[7]

if FLWCHRCD.strip() != '':

insertObj['FLWCHRCD'] = FLWCHRCD

WPTN = fieldValues[8]

if WPTN.strip() != '':

insertObj['WPTN'] = WPTN

MSQMT = fieldValues[9]

if MSQMT.strip() != '':

insertObj['MSQMT'] = MSQMT

MSAMT = fieldValues[10]

if MSAMT.strip() != '':

insertObj['MSAMT'] = MSAMT

MSVMT = fieldValues[11]

if MSVMT.strip() != '':

insertObj['MSVMT'] = MSVMT

# collection.insert_one(insertObj)

# collection.insert_many(new_posts)

records.append(insertObj)

iflen(records) == 1000:

insertCount = insertCount + 1

if count > 1451000:

collection.insert_many(records)

print(str(count) + ' ' + str(insertCount))

print(count)

records = []

else:

print(line)

line = f.readline()

f.close()

client.close()

5.5. 从单个MongoDB中备份数据

[root@server1 bin]# ./mongodump -h 127.0.0.1:27017 -d RHY -o /opt/mongodb/mydata/dump

5.6. 向Shard中恢复数据

[mongodb@server1 bin]$ ./mongorestore -h 192.168.1.31 --port 20000 -d RHY /opt/mongodb/mydata/dump/RHY

2018-05-20T17:15:20.762+0800 the --db and --collection args should only be used when restoring from a BSON file. Other uses are deprecated and will not exist in the future; use --nsInclude instead

2018-05-20T17:15:20.763+0800 building a list of collections to restore from /opt/mongodb/mydata/dump/RHY dir

2018-05-20T17:15:20.767+0800 reading metadata for RHY.ST_RIVER_R from /opt/mongodb/mydata/dump/RHY/ST_RIVER_R.metadata.json

2018-05-20T17:15:20.767+0800 reading metadata for RHY.ST_STBPRP_B from /opt/mongodb/mydata/dump/RHY/ST_STBPRP_B.metadata.json

2018-05-20T17:15:20.779+0800 restoring RHY.ST_RIVER_R from /opt/mongodb/mydata/dump/RHY/ST_RIVER_R.bson

2018-05-20T17:15:20.779+0800 restoring RHY.ST_STBPRP_B from /opt/mongodb/mydata/dump/RHY/ST_STBPRP_B.bson

2018-05-20T17:15:23.772+0800 [####....................] RHY.ST_STBPRP_B 2.61MB/13.6MB (19.1%)

2018-05-20T17:15:23.772+0800 [........................] RHY.ST_RIVER_R 849KB/2.89GB (0.0%)

2018-05-20T17:15:23.772+0800

2018-05-20T17:15:26.759+0800 [##########..............] RHY.ST_STBPRP_B 6.04MB/13.6MB (44.4%)

2018-05-20T17:15:26.759+0800 [........................] RHY.ST_RIVER_R 2.15MB/2.89GB (0.1%)

2018-05-20T17:15:26.759+0800

2018-05-20T17:15:29.759+0800 [#####################...] RHY.ST_STBPRP_B 12.0MB/13.6MB (87.9%)

2018-05-20T17:15:29.759+0800 [........................] RHY.ST_RIVER_R 4.63MB/2.89GB (0.2%)

2018-05-20T17:15:29.759+0800

2018-05-20T17:15:31.447+0800 [########################] RHY.ST_STBPRP_B 13.6MB/13.6MB (100.0%)

2018-05-20T17:15:31.447+0800 no indexes to restore

2018-05-20T17:15:31.447+0800 finished restoring RHY.ST_STBPRP_B (27276 documents)

2018-05-20T17:15:32.758+0800 [........................] RHY.ST_RIVER_R 6.79MB/2.89GB (0.2%)

2018-05-20T17:15:35.758+0800 [........................] RHY.ST_RIVER_R 8.61MB/2.89GB (0.3%)

2018-05-20T17:15:38.758+0800 [........................] RHY.ST_RIVER_R 11.9MB/2.89GB (0.4%)

2018-05-20T17:15:41.758+0800 [........................] RHY.ST_RIVER_R 15.7MB/2.89GB (0.5%)

[mongodb@server4 conf]$ mongo 192.168.1.31:20000

MongoDB shell version v3.6.4

connecting to: mongodb://192.168.1.31:20000/test

MongoDB server version: 3.6.4

Server has startup warnings:

2018-05-20T17:01:28.017+0800 I CONTROL [main]

2018-05-20T17:01:28.018+0800 I CONTROL [main] ** WARNING: Access control is not enabled for the database.

2018-05-20T17:01:28.018+0800 I CONTROL [main] ** Read and write access to data and configuration is unrestricted.

2018-05-20T17:01:28.018+0800 I CONTROL [main]

mongos> use RHY

switched to db RHY

mongos> db.ST_RIVER_R.count()

1589817

mongos> db.ST_RIVER_R.findOne()

{

"_id" : ObjectId("5b012fdee4a39884b75e4880"),

"STCD" : 60812000,

"TM" : "2011-04-13 02:00:00",

"Z" : 405,

"Q" : 60,

"XSA" : "",

"XSAVV" : "",

"XSMXV" : "",

"FLWCHRCD" : "",

"WPTN" : 4,

"MSQMT" : 1,

"MSAMT" : "",

"MSVMT" : ""

}

mongos> db.ST_RIVER_R.find({"STCD":60812000})

{ "_id" : ObjectId("5b012fdee4a39884b75e4880"), "STCD" : 60812000, "TM" : "2011-04-13 02:00:00", "Z" : 405, "Q" : 60, "XSA" : "", "XSAVV" : "", "XSMXV" : "", "FLWCHRCD" : "", "WPTN" : 4, "MSQMT" : 1, "MSAMT" : "", "MSVMT" : "" }

{ "_id" : ObjectId("5b012fdee4a39884b75e48ad"), "STCD" : 60812000, "TM" : "2011-04-19 02:00:00", "Z" : 377.71, "Q" : 13.3, "XSA" : "", "XSAVV" : "", "XSMXV" : "", "FLWCHRCD" : "", "WPTN" : 6, "MSQMT" : 1, "MSAMT" : "", "MSVMT" : "" }