问题一：两台debian-8机器，debian-phy作为manager node, debian-vm作为worker node. 部署swarm时，debian-vm加入集群失败，提示CA证书问题。
解决办法：
1. manager node时间比worker node时间快30分钟，修改时间一致后，worker node加入集群依然提示失败； 2. 重启manager node上docker服务，worker node加入集群成功。
问题二： 创建多replicas的服务时，manager node上服务运行正常，worker node上服务运行失败，提示：
Error: Failed to find a load balancer IP to use for network
解决办法： 发现manager node上docker version 为: docker-ce-17.04.1, worker node上docker version为 docker-ce-18.06.1，将manager node上docker升级到docker-ce-18.06.1后问题解决.
问题三： SUSE12SP2上安装docker-17.04，docker swarm部署时发现两个问题, 描述如下， 暂未解决(debian上验证无问题)。
1. 创建service时, --publish 参数暴露的端口不能被物理机网络访问，但可以通过容器和物理机间的172网段访问； 2. 创建多个不同的service，在跨节点机器上，服务发现不可用(ping server 不通), 但是通过IP地址可以互相访问。
节点版本信息如下：
1 linux # cat /etc/SuSE-release 2 SUSE Linux Enterprise Server 12 (x86_64) 3 VERSION = 12 4 PATCHLEVEL = 2 5 # This file is deprecated and will be removed in a future service pack or release. 6 # Please check /etc/os-release for details about this release. 7 linux # 8 9 10 linux # docker version 11 Client: 12 Version: 17.04.0-ce 13 API version: 1.28 14 Go version: go1.7.5 15 Git commit: 78d1802 16 Built: Tue May 30 18:21:18 2017 17 OS/Arch: linux/amd64 18 19 Server: 20 Version: 17.04.0-ce 21 API version: 1.28 (minimum version 1.12) 22 Go version: go1.7.5 23 Git commit: 78d1802 24 Built: Tue May 30 18:21:18 2017 25 OS/Arch: linux/amd64 26 Experimental: false 27 linux:/app/original/worker # 28 29 30 linux # docker service create --replicas 1 --name server -e APP_PORT=5000 --network docker-net --publish 5000:5000 env/server:v0.1
日志记录：
1 linux:/app/env # docker swarm init --advertise-addr 10.9.23.241 --listen-addr 10.9.23.241 2 Swarm initialized: current node (pcrsf5o2corbm6ol3dlmgtjtt) is now a manager. 3 4 To add a worker to this swarm, run the following command: 5 6 docker swarm join 7 --token SWMTKN-1-25isp458n3vftu7cj3p9gul68pe291hn58ekswq9ox8m52e6x9-5dk5aw452oe7ismwekz942xaq 8 10.9.23.241:2377 9 10 To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions. 11 12 linux:/app/env # docker node ls 13 ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS 14 nsf5u8othzfaw26121v06a44e SUSE12-29161 Ready Active 15 pcrsf5o2corbm6ol3dlmgtjtt * linux Ready Active Leader 16 linux:/app/env # docker network ls 17 NETWORK ID NAME DRIVER SCOPE 18 6aa823f9ce29 bridge bridge local 19 0d741bdf766c docker_gwbridge bridge local 20 c9767a06fa1c host host local 21 pyx9mde4js3o ingress overlay swarm 22 393c7ca2630e none null local 23 linux:/app/env # docker service create --replicas 1 --name server -e APP_PORT=5000 --network docker-net --publish 5000:5000 env/server:v0.1 24 image env/server:v0.1 could not be accessed on a registry to record 25 its digest. Each node will access env/server:v0.1 independently, 26 possibly leading to different nodes running different 27 versions of the image. 28 29 tqhnrmfln4v8m5z858py9f8gv 30 linux:/app/env # docker service ls 31 ID NAME MODE REPLICAS IMAGE 32 tqhnrmfln4v8 server replicated 1/1 env/server:v0.1 33 linux:/app/env # docker service ps server 34 ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 35 5fa4bkyjfitt server.1 env/server:v0.1 linux Running Running 34 seconds ago 36 linux:/app/env # docker ps -a 37 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 38 c00de9e5d3d1 env/server:v0.1 "python server.py" 39 seconds ago Up 37 seconds server.1.5fa4bkyjfittr95stqk0fcjcy 39 linux:/app/env #
1 version: "3" 2 services: 3 server: 4 image: env/server:v0.1 5 build: 6 context: /app/env/server 7 dockerfile: Dockerfile 8 labels: 9 description: "tcp server test script" 10 ports: 11 - "3000:3000" 12 restart: always 13 environment: 14 - APP_PORT=${APP_PORT-3000} 15 networks: 16 - overlay 17 deploy: 18 replicas: 1 19 20 worker: 21 image: env/worker:v0.1 22 build: /app/env/worker 23 labels: 24 description: "tcp client test script" 25 links: 26 - server 27 environment: 28 - APP_PORT=${APP_PORT-3000} 29 - APP_DOMAIN=server 30 networks: 31 - overlay 32 depends_on: 33 - server 34 deploy: 35 replicas: 3 36 restart_policy: 37 condition: on-failure 38 39 worker_test: 40 image: env/worker_test:v0.1 41 networks: 42 - overlay 43 deploy: 44 replicas: 3 45 46 networks: 47 overlay:
1 # 创建信息 2 linux:/app/env # echo $APP_PORT 3 4 linux:/app/env # 5 linux:/app/env # docker stack deploy -c docker-compose.yml env 6 Ignoring unsupported options: build, links, restart 7 8 Creating network env_overlay 9 Creating service env_worker 10 Creating service env_worker_test 11 Creating service env_server 12 List the tasks in the stack 13 linux:/app/env # docker stack ps env 14 ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 15 l70oopv2pxka env_server.1 env/server:v0.1 linux Running Running 4 seconds ago 16 qqxa9agnmu0h env_worker_test.1 env/worker_test:v0.1 SUSE12-29161 Running Starting less than a second ago 17 9c09mhhyyf7l env_worker.1 env/worker:v0.1 SUSE12-29161 Running Starting less than a second ago 18 6nq3u3yuogan env_worker_test.2 env/worker_test:v0.1 SUSE12-29161 Running Starting less than a second ago 19 yandlm9mo5cz env_worker.2 env/worker:v0.1 SUSE12-29161 Running Starting less than a second ago 20 pwfjesuiry5k env_worker_test.3 env/worker_test:v0.1 linux Running Running 5 seconds ago 21 i9w2ukpzjuqm env_worker.3 env/worker:v0.1 linux Running Running 5 seconds ago 22 linux:/app/env # docker service ls 23 ID NAME MODE REPLICAS IMAGE 24 kowpw9auo1fh env_worker replicated 3/3 env/worker:v0.1 25 thi3akvbsnwv env_server replicated 1/1 env/server:v0.1 26 w1dqusnruk6d env_worker_test replicated 3/3 env/worker_test:v0.1 27 linux:/app/env # docker service inspect env_server --pretty 28 29 ID: thi3akvbsnwv8gflf535vw9l1 30 Name: env_server 31 Labels: 32 com.docker.stack.namespace=env 33 Service Mode: Replicated 34 Replicas: 1 35 Placement: 36 ContainerSpec: 37 Image: env/server:v0.1 38 Env: APP_PORT=3000 39 Resources: 40 Networks: vz00o7esbvhtkfbx5ac1us2es 41 Endpoint Mode: vip 42 Ports: 43 PublishedPort = 3000 44 Protocol = tcp 45 TargetPort = 3000 46 PublishMode = ingress 47 linux:/app/env # 48 49 50 # 同一节点上容器通过服务名或者IP地址可以ping通 51 linux:/app/env # docker ps 52 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 53 6ac0525f2cd0 env/server:v0.1 "python server.py" 2 minutes ago Up 2 minutes env_server.1.l70oopv2pxkahpzsqhpat92k0 54 304f4c3c2b94 env/worker_test:v0.1 "python worker.py" 2 minutes ago Up 2 minutes env_worker_test.3.pwfjesuiry5kv8tdz5irohxw8 55 d55a9aac49bf env/worker:v0.1 "python worker.py" 2 minutes ago Up 2 minutes env_worker.3.i9w2ukpzjuqmgkwcrymjm8az1 56 linux:/app/env # docker exec d55a9aac49bf ping -c 2 server 57 PING server (10.0.0.10) 56(84) bytes of data. 58 64 bytes from 10.0.0.10: icmp_seq=1 ttl=64 time=0.082 ms 59 64 bytes from 10.0.0.10: icmp_seq=2 ttl=64 time=0.053 ms 60 61 --- server ping statistics --- 62 2 packets transmitted, 2 received, 0% packet loss, time 10010ms 63 rtt min/avg/max/mdev = 0.053/0.067/0.082/0.016 ms 64 linux:/app/env # 65 66 # 不同节点上容器根据IP地址可以互相ping通，但不能通过服务发现ping通 67 SUSE12-29161:/home/jason # docker ps 68 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 69 60a36b02635b env/worker:v0.1 "python worker.py" 6 minutes ago Up 6 minutes env_worker.2.yandlm9mo5czanzxemwu7kfba 70 e64012631aa2 env/worker:v0.1 "python worker.py" 6 minutes ago Up 6 minutes env_worker.1.9c09mhhyyf7ld1b04a2qqwzih 71 4751d586e3f9 env/worker_test:v0.1 "python worker.py" 6 minutes ago Up 6 minutes env_worker_test.1.qqxa9agnmu0hg8x4f9fsf8vbz 72 98d80f439180 env/worker_test:v0.1 "python worker.py" 6 minutes ago Up 6 minutes env_worker_test.2.6nq3u3yuoganvb72b92sb2rdu 73 SUSE12-29161:/home/jason # docker exec -it 4751d586e3f9 /bin/bash 74 root@4751d586e3f9:/env/worker# ping -c 2 server 75 PING server (10.0.0.10) 56(84) bytes of data. 76 From 4751d586e3f9 (10.0.0.7) icmp_seq=1 Destination Host Unreachable 77 From 4751d586e3f9 (10.0.0.7) icmp_seq=2 Destination Host Unreachable 78 79 --- server ping statistics --- 80 2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1007ms 81 pipe 2 82 root@4751d586e3f9:/env/worker# ping -c 2 10.9.23.241 83 PING 10.9.23.241 (10.9.23.241) 56(84) bytes of data. 84 64 bytes from 10.9.23.241: icmp_seq=1 ttl=60 time=0.348 ms 85 64 bytes from 10.9.23.241: icmp_seq=2 ttl=60 time=0.257 ms 86 87 --- 10.9.23.241 ping statistics --- 88 2 packets transmitted, 2 received, 0% packet loss, time 999ms 89 rtt min/avg/max/mdev = 0.257/0.302/0.348/0.048 ms 90 root@4751d586e3f9:/env/worker# ^C 91 root@4751d586e3f9:/env/worker# ping -c 2 10.0.0.11 92 PING 10.0.0.11 (10.0.0.11) 56(84) bytes of data. 93 64 bytes from 10.0.0.11: icmp_seq=1 ttl=64 time=0.516 ms 94 64 bytes from 10.0.0.11: icmp_seq=2 ttl=64 time=0.336 ms 95 96 --- 10.0.0.11 ping statistics --- 97 2 packets transmitted, 2 received, 0% packet loss, time 999ms 98 rtt min/avg/max/mdev = 0.336/0.426/0.516/0.090 ms 99 root@4751d586e3f9:/env/worker#