- 单节点etcd publish error
正在愉快的进行jenkins流程，突然发现etcd连接不上去了。重新reboot后发现日志publish error
Oct 31 10:22:42 k8s-master etcd: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://etcd:2379,http://etcd:4001 Oct 31 10:22:42 k8s-master etcd: recognized environment variable ETCD_NAME, but unused: shadowed by corresponding flag Oct 31 10:22:42 k8s-master etcd: recognized environment variable ETCD_DATA_DIR, but unused: shadowed by corresponding flag Oct 31 10:22:42 k8s-master etcd: recognized environment variable ETCD_LISTEN_CLIENT_URLS, but unused: shadowed by corresponding flag Oct 31 10:22:42 k8s-master etcd: etcd Version: 3.1.3 Oct 31 10:22:42 k8s-master etcd: Git SHA: 21fdcc6 Oct 31 10:22:42 k8s-master etcd: Go Version: go1.7.4 Oct 31 10:22:42 k8s-master etcd: Go OS/Arch: linux/amd64 Oct 31 10:22:42 k8s-master etcd: setting maximum number of CPUs to 1, total number of available CPUs is 1 Oct 31 10:22:42 k8s-master etcd: the server is already initialized as member before, starting as etcd member... Oct 31 10:22:42 k8s-master etcd: listening for peers on http://localhost:2380 Oct 31 10:22:42 k8s-master etcd: listening for client requests on 0.0.0.0:2379 Oct 31 10:22:42 k8s-master etcd: listening for client requests on 0.0.0.0:4001 Oct 31 10:22:42 k8s-master etcd: recovered store from snapshot at index 210021 Oct 31 10:22:42 k8s-master etcd: name = master Oct 31 10:22:42 k8s-master etcd: data dir = /var/lib/etcd/default.etcd Oct 31 10:22:42 k8s-master etcd: member dir = /var/lib/etcd/default.etcd/member Oct 31 10:22:42 k8s-master etcd: heartbeat = 100ms Oct 31 10:22:42 k8s-master etcd: election = 1000ms Oct 31 10:22:42 k8s-master etcd: snapshot count = 10000 Oct 31 10:22:42 k8s-master etcd: advertise client URLs = http://etcd:2379,http://etcd:4001 Oct 31 10:22:42 k8s-master etcd: ignored file 0000000000000001-0000000000012700.wal.broken in wal Oct 31 10:22:42 k8s-master etcd: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 215587 Oct 31 10:22:42 k8s-master etcd: 8e9e05c52164694d became follower at term 19 Oct 31 10:22:42 k8s-master etcd: newRaft 8e9e05c52164694d [peers: [8e9e05c52164694d], term: 19, commit: 215587, applied: 210021, lastindex: 215587, lastterm: 19] Oct 31 10:22:42 k8s-master etcd: enabled capabilities for version 3.1 Oct 31 10:22:42 k8s-master etcd: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32 from store Oct 31 10:22:42 k8s-master etcd: set the cluster version to 3.1 from store Oct 31 10:22:42 k8s-master etcd: starting server... [version: 3.1.3, cluster version: 3.1] Oct 31 10:22:49 k8s-master etcd: publish error: etcdserver: request timed out Oct 31 10:22:56 k8s-master etcd: publish error: etcdserver: request timed out Oct 31 10:23:03 k8s-master etcd: publish error: etcdserver: request timed out Oct 31 10:23:10 k8s-master etcd: publish error: etcdserver: request timed out Oct 31 10:23:17 k8s-master etcd: publish error: etcdserver: request timed out Oct 31 10:23:24 k8s-master etcd: publish error: etcdserver: request timed out
查了一下，原因可能是因为主机压力比较大没有在5s内将信息写回照成不一致。
解决办法是删除data dir: /var/lib/etcd/default.etcd 中的数据，然后启动成功
不好的地方是，所有的信息基本全丢，flanneld信息又需要再写一次
etcdctl mk /atomic.io/network/config '{ "Network": "10.0.0.0/16" }'
各种pv,pvc,pod啥信息也没有了。。。。。
- Docker Service无法启动
在一台nodes节点上因为连接不上etcd导致启动失败，因为flannel组件需要连接etcd获取网段，docker服务依赖于flanned Service
[Unit] Description=Flanneld overlay address etcd agent After=network.target After=network-online.target Wants=network-online.target After=etcd.service Before=docker.service
分析命令主要包括
systemctl list-unit-files 列出所有可用的Unit systemctl list-units 列出所有正在运行的Unit systemctl --failed 列出所有失败单元 systemctl mask httpd.service 禁用服务 systemctl unmask httpd.service systemctl kill httpd 杀死服务
解决办法，使用systemctl unmask flanneld.service禁止flanneld服务，然后删除
/usr/lib/systemd/system/flanneld.service
/etc/systemd/system/docker.service.requires/flanneld.service，
使用systemctl daemon-reload重新加载服务配置文件，最后systemctl start docker.service，
发现docker启动成功了