一、Flannel网络简介
Flannel是一种基于overlay网络的跨主机容器网络解决方案，也就是将TCP数据包封装在另一种网络包里面进行路由转发和通信，Flannel是CoreOS开发,专门用于docker多机互联的一个工具,让集群中的不同节点主机创建的容器都具有全集群唯一的虚拟ip地址，Flannel使用go语言编写
二、Flannel实现原理
2.1、原理说明
1、Flannel为每个host分配一个subnet，容器从这个subnet中分配IP，这些IP可以在host间路由，容器间无需使用nat和端口映射即可实现跨主机通信
2、每个subnet都是从一个更大的IP池中划分的，flannel会在每个主机上运行一个叫flanneld的agent，其职责就是从池子中分配subnet
3、Flannel使用etcd存放网络配置、已分配 的subnet、host的IP等信息
4、Flannel数据包在主机间转发是由backend实现的，目前已经支持UDP、VxLAN、host-gw、AWS VPC和GCE路由等多种backend
2.2、数据转发流程
1、容器直接使用目标容器的ip访问，默认通过容器内部的eth0发送出去。
2、报文通过veth pair被发送到vethXXX。
3、ethXXX是直接连接到虚拟交换机docker0的，报文通过虚拟bridge docker0发送出去。
4、查找路由表，外部容器ip的报文都会转发到flannel0虚拟网卡，这是一个P2P的虚拟网卡，然后报文就被转发到监听在另一端的flanneld。
5、flanneld通过etcd维护了各个节点之间的路由表，把原来的报文UDP封装一层，通过配置的iface发送出去。
6、报文通过主机之间的网络找到目标主机。
7、报文继续往上，到传输层，交给监听在8285端口的flanneld程序处理。
8、数据被解包，然后发送给flannel0虚拟网卡。
9、查找路由表，发现对应容器的报文要交给docker0。
10、docker0找到连到自己的容器，把报文发送过去。
三、部署etcd集群
3.1、环境准备
节点名称
|
IP地址
|
安装软件
|
node1
|
192.168.0.115
|
etcd
|
node2
|
192.168.0.116
|
etcd
|
node3
|
192.168.0.117
|
etcd
|
3.2、安装etcd
3.3、配置etcd
# cp /etc/etcd/etcd.conf{,_bak}
【注释：每个ETCD_NAME必须不同，绿色部分的ip为当前宿主机的ip】
# grep -v '^#' /etc/etcd/etcd.conf
ETCD_NAME="node1"
ETCD_DATA_DIR="/var/lib/etcd/node1.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.0.115:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.0.115:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.115:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.0.115:2379"
ETCD_INITIAL_CLUSTER="node1=http://192.168.0.115:2380,node2=http://192.168.0.116:2380,node3=http://192.168.0.117:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
3.4、修改etcd启动文件
# cp /usr/lib/systemd/system/etcd.service{,_bak}
# cat /usr/lib/systemd/system/etcd.service
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=etcd
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd
--name="${ETCD_NAME}"
--data-dir="${ETCD_DATA_DIR}"
--listen-peer-urls="${ETCD_LISTEN_PEER_URLS}"
--listen-client-urls="${ETCD_LISTEN_CLIENT_URLS}"
--initial-advertise-peer-urls="${ETCD_INITIAL_ADVERTISE_PEER_URLS}"
--advertise-client-urls="${ETCD_ADVERTISE_CLIENT_URLS}"
--initial-cluster="${ETCD_INITIAL_CLUSTER}"
--initial-cluster-token="${ETCD_INITIAL_CLUSTER_TOKEN}"
--initial-cluster-state="${ETCD_INITIAL_CLUSTER_STATE}""
Restart=on-failure
LimitNOFILE=65536
3.5、启动etcd服务
【注释：另外两台服务器，操作也如上】
# systemctl start etcd.service
3.6、检测etcd集群状态，至此etcd安装完成
# 查看cluster状态
# etcdctl cluster-health
member 3e398d43ae9c8720 is healthy: got healthy result from http://192.168.0.116:2379
member 65368524050cc2e8 is healthy: got healthy result from http://192.168.0.115:2379
member d8ff06c8c9b413da is healthy: got healthy result from http://192.168.0.117:2379
cluster is healthy
# 列出etcd服务状态，从列出信息可以看出，目前是node2为主节点。
# etcdctl member list
3e398d43ae9c8720: name=node2 peerURLs=http://192.168.0.116:2380 clientURLs=http://192.168.0.116:2379 isLeader=true
65368524050cc2e8: name=node1 peerURLs=http://192.168.0.115:2380 clientURLs=http://192.168.0.115:2379 isLeader=false
d8ff06c8c9b413da: name=node3 peerURLs=http://192.168.0.117:2380 clientURLs=http://192.168.0.117:2379 isLeader=false
3.7、添加flannel网络配置信息到etcd
【注释： 此（flannel_use）目录自己可以定义，但是此处设置的目录必须与flannel配置文件中FLANNEL_ETCD_PREFIX="/flannel_use/network"配置保持一致，flannel启动程序只认带“config”的key，否则会报错Not a directory (/flannel_use/network)】
# 固定配置方式
# etcdctl set /flannel_use/network/config '{"Network":"10.10.0.0/16"}'
四、部署flannel
4.1、安装flannel
4.2、修改flannel配置文件
# cp /etc/sysconfig/flanneld{,_bak}
# cat /etc/sysconfig/flanneld
# Flanneld configuration options
# etcd url location. Point this to the server where etcd runs
FLANNEL_ETCD_ENDPOINTS="http://192.168.0.115:2379,http://192.168.0.116:2379,http://192.168.0.117:2379"
# etcd config key. This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_PREFIX="/flannel_use/network"
# Any additional options that you want to pass
#FLANNEL_OPTIONS=""
4.3、启动flannel
# systemctl start flanneld
# systemctl status flanneld
● flanneld.service - Flanneld overlay address etcd agent
Loaded: loaded (/usr/lib/systemd/system/flanneld.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2019-12-23 15:01:07 CST; 4s ago
Process: 1828 ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker (code=exited, status=0/SUCCESS)
Main PID: 1817 (flanneld)
Memory: 18.8M
CGroup: /system.slice/flanneld.service
└─1817 /usr/bin/flanneld -etcd-endpoints=http://192.168.0.115:2379,http://192.168.0.116:2379,http://192.168.0.117:2379 -etcd-prefix=/flannel_use/network
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.365994 1817 main.go:132] Installing signal handlers
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.366705 1817 manager.go:136] Determining IP address of default interface
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.366916 1817 manager.go:149] Using interface with name eth0 and address 192.168.0.109
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.366933 1817 manager.go:166] Defaulting external address to interface address (192.168.0.109)
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.375600 1817 local_manager.go:179] Picking subnet in range 10.10.1.0 ... 10.10.255.0
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.383110 1817 manager.go:250] Lease acquired: 10.10.88.0/24
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.383333 1817 network.go:98] Watching for new subnet leases
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.388324 1817 network.go:191] Subnet added: 10.10.65.0/24
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.388344 1817 network.go:191] Subnet added: 10.10.50.0/24
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 systemd[1]: Started Flanneld overlay address etcd agent.
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:16:3e:2c:36:be brd ff:ff:ff:ff:ff:ff
inet 192.168.0.109/24 brd 192.168.0.255 scope global dynamic eth0
valid_lft 314756444sec preferred_lft 314756444sec
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:30:1b:b8:fd brd ff:ff:ff:ff:ff:ff
inet 10.10.0.1/24 brd 10.10.0.255 scope global docker0
valid_lft forever preferred_lft forever
4: flannel0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UNKNOWN qlen 500
link/none
inet 10.10.88.0/16 scope global flannel0
valid_lft forever preferred_lft forever
4.4、注释
启动Flannel后，一定要记得重启docker，这样Flannel配置分配的ip才能生效，即docker0虚拟网卡的ip会变成上面flannel设定的ip段
4.5、修改docker启动/配置文件使用flannel网络
# cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd --insecure-registry=172.17.29.74 -H fd:// --containerd=/run/containerd/containerd.sock $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3
# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
# cat /etc/docker/daemon.json
{
"registry-mirrors": ["https://registry.docker-cn.com"]
}
4.6、重启docker
# systemctl daemon-reload
# systemctl restart docker
4.7、查看docker是否使用flannel网络
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:16:3e:2c:36:be brd ff:ff:ff:ff:ff:ff
inet 192.168.0.109/24 brd 192.168.0.255 scope global dynamic eth0
valid_lft 314756133sec preferred_lft 314756133sec
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
link/ether 02:42:30:1b:b8:fd brd ff:ff:ff:ff:ff:ff
inet 10.10.88.1/24 brd 10.10.88.255 scope global docker0
valid_lft forever preferred_lft forever
4: flannel0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UNKNOWN qlen 500
link/none
inet 10.10.88.0/16 scope global flannel0
valid_lft forever preferred_lft forever
4.8、如果容器无法联通，是由于flannel.0网卡和docker0网卡通过iptables的forward转发，所以需确保如下设置
1、核中的forward功能开启(立即生效，重启后效果不再)
# echo "1" > /proc/sys/net/ipv4/ip_forward
2、包不会被iptables的forward规则拦截
# iptables -P FORWARD ACCEPT