k8s二进制部署环境DNS、harbor仓库、etcd环境准备02
1.2 安装方式选择
- Minikube 预览使用,仅供学习
- 二进制安装(生产首选,新手推荐)
- kubeadmin安装
简单,用k8s跑k8s自己,熟手推荐
新手不推荐的原因是容易知其然不知其所以然
出问题后找不到解决办法
2 部署准备
2.1 准备工作
准备5台2C/2g/50g虚拟机,网络10.11.0.0/24
预装centos7.6,做完基础优化
安装部署bind9,部署自建DNS系统
准备自签证书环境
安装部署docker和harbor仓库
机器列表
|
主机名 |
IP地址 |
用途 |
|
server01 |
10.11.0.206 |
proxy1 |
|
server02 |
10.11.0.207 |
proxy2,master1 |
|
server03 |
10.11.0.208 |
Master2 |
|
server04 |
10.11.0.209 |
Etcd03 |
|
server05 |
10.11.0.210 |
运维主机 |
Server01
# getenforce
Disabled
# more /etc/sysconfig/network-scripts/ifcfg-eth0
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
NAME=eth0
DEVICE=eth0
ONBOOT=yes
IPADDR=10.11.0.207
GATEWAY=10.11.0.254
NETWROK=10.11.0.0
DNS1=10.11.0.206
DNS2=8.8.8.8
# yum install wget net-tools telnet tree nmap sysstat lrzsz dos2unix -y
2.2 部署DNS服务bind9
2.2.1 安装配置DNS服务
在0.206上部署bind的DNS服务
yum install bind bind-utils -y
修改并校验配置文件
# more /etc/named.conf options { listen-on port 53 { 10.11.0.206; }; directory "/var/named"; dump-file "/var/named/data/cache_dump.db"; statistics-file "/var/named/data/named_stats.txt"; memstatistics-file "/var/named/data/named_mem_stats.txt"; recursing-file "/var/named/data/named.recursing"; secroots-file "/var/named/data/named.secroots"; allow-query { any; }; forwarders { 192.168.10.254; }; #上一层DNS地址(网关或公网DNS) recursion yes; dnssec-enable no; dnssec-validation no; /* Path to ISC DLV key */ bindkeys-file "/etc/named.root.key"; managed-keys-directory "/var/named/dynamic"; pid-file "/run/named/named.pid"; session-keyfile "/run/named/session.key"; }; logging { channel default_debug { file "data/named.run"; severity dynamic; }; }; zone "." IN { type hint; file "named.ca"; }; include "/etc/named.rfc1912.zones"; include "/etc/named.root.key";
# 配置检测
# named-checkconf
2.2.2 增加自定义域和对于配置
在域配置中增加自定义域
[root@server01 named]# cat /etc/named.rfc1912.zones
# 添加自定义主机域
zone "host.com" IN { type master; file "host.com.zone"; allow-update { 10.11.0.206; }; }; # 添加自定义业务域 zone "chinasoft.com" IN { type master; file "chinasoft.com.zone"; allow-update { 10.11.0.206; }; };
host.com和chinasoft.com都是我们自定义的域名,一般用host.com做为主机域
chinasoft.com为业务域,业务不同可以配置多个
为自定义域host.com创建配置文件
# more host.com.zone $ORIGIN host.com. $TTL 600 ; 10 minutes @ IN SOA dns.host.com. dnsadmin.host.com. ( 2020081701 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum(1 day) ) NS dns.host.com. $TTL 60 ; 1 minute dns A 10.11.0.206 server01 A 10.11.0.206 server02 A 10.11.0.207 server03 A 10.11.0.208 server04 A 10.11.0.209 server05 A 10.11.0.210
为自定义域chinasoft.com创建配置文件
# more chinasoft.com.zone
$ORIGIN chinasoft.com. $TTL 600 ; 10 minutes @ IN SOA dns.chinasoft.com. dnsadmin.chinasoft.com. ( 2020081812 ; serial 10800 ; refresh (3 hours) 900 ; retry (15 minutes) 604800 ; expire (1 week) 86400 ; minimum(1 day) ) NS dns.chinasoft.com. $TTL 60 ; 1 minute dns A 10.11.0.206
host.com域用于主机之间通信,所以要先增加上所有主机
chinasoft.com域用于后面的业务解析用,因此不需要先添加主机
2.2.3 启动并验证DNS服务
再次检查配置并启动dns服务
# named-checkconf
# systemctl start named
# ss -lntup|grep 53
# 验证结果
[root@server01 named]# dig -t A server01.host.com @10.11.0.206 +short
10.11.0.206
[root@server01 named]# dig -t A server02.host.com @10.11.0.206 +short
10.11.0.207
2.2.4 所有主机修改网络配置
5台K8S主机都需要按如下方式修改网络配置
# 修改dns并添加搜索域
sed -i 's#^DNS.*#DNS1=10.11.0.206#g' /etc/sysconfig/network-scripts/ifcfg-eth0
echo "search=host.com" >>/etc/sysconfig/network-scripts/ifcfg-eth0
systemctl restart network
# 检查DNS配置
~]# cat /etc/resolv.conf
# Generated by NetworkManager
search host.com
nameserver 10.11.0.206
~]# dig -t A server01.host.com +short
10.11.0.206
# 一定记得检查dns配置文件中是否有search信息
windows宿主机也要改
wmnet8网卡更改DNS：10.11.0.206
# ping通才行,否则检查
ping server01.host.com
2.3 自签发证书环境准备
操作在0.210这个运维机上完成
2.3.1 下载安装cfssl
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/bin/cfssl-json
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/bin/cfssl-certinfo
chmod +x /usr/bin/cfssl*
2.3.2 生成ca证书文件
mkdir /opt/certs # more /opt/certs/ca-csr.json { "CN": "chinasoft", "hosts": [ ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "beijing", "L": "beijing", "O": "chinasoft", "OU": "ops" } ], "ca": { "expiry": "175200h" } }
CN: Common Name，浏览器使用该字段验证网站是否合法，一般写的是域名。非常重要。浏览器使用该字段验证网站是否合法
C: Country，
国家
ST: State，州，省
L: Locality，地区，城市
O: Organization Name，组织名称，公司名称
OU: Organization Unit Name，组织单位名称，公司部门
2.3.3 生成ca证书
cd /opt/certs
cfssl gencert -initca ca-csr.json | cfssl-json -bare ca
certs]# ll
-rw-r--r-- 1 root root 1005 Aug 17 21:34 ca.csr
-rw-r--r-- 1 root root 233 Aug 17 21:33 ca-csr.json
-rw------- 1 root root 1675 Aug 17 21:34 ca-key.pem
-rw-r--r-- 1 root root 1363 Aug 17 21:34 ca.pem
2.4 docker环境准备
2.4.1 安装并配置docker
curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
mkdir /etc/docker/
# more /etc/docker/daemon.json
{ "graph": "/data/docker", "storage-driver": "overlay2", "insecure-registries": ["registry.access.redhat.com","quay.io","harbor.chinasoft.com"], "registry-mirrors": ["https://q2gr04ke.mirror.aliyuncs.com"], "bip": "172.7.207.1/24", "exec-opts": ["native.cgroupdriver=systemd"], "live-restore": true }
注意:bip要根据宿主机ip变化
server01.host.com bip 172.7.207.1/24
server02.host.com bip 172.7.208.1/24
server05.host.com bip 172.7.210.1/24
2.4.2 启动docker
mkdir -p /data/docker
systemctl start docker
systemctl enable docker
docker --version
2.5 部署harbor私有仓库
下载地址:https://github.com/goharbor/harbor/releases/download/v1.8.5/harbor-offline-installer-v1.8.5.tgz
2.5.1 下载并解压
tar xf harbor-offline-installer-v1.8.5.tgz -C /opt/
cd /opt/
mv harbor/ harbor-v1.8.5
ln -s /opt/harbor-v1.8.5/ /opt/harbor
2.5.2 编辑配置文件
[root@server05 opt]# vi /opt/harbor/harbor.yml
# 以下是修改项,手动在配置文件中更改
hostname: harbor.chinasoft.com http: port: 180 harbor_admin_password:harbor123456 data_volume: /data/harbor log: level: info rotate_count: 50 rotate_size:200M location: /data/harbor/logs # mkdir -p /data/harbor/logs
2.5.3 使用docker-compose启动harbor
# cd /opt/harbor/
yum install docker-compose -y
sh /opt/harbor/install.sh
docker-compose ps
docker ps -a
2.5.4 使用dns解析harbor
在0.206 DNS服务上操作
# vi /var/named/chinasoft.com.zone
2020032002 ; serial #每次修改DNS解析后,都要滚动此ID
harbor A 10.11.0.210
# systemctl restart named
# dig -t A harbor.chinasoft.com +short
10.11.0.210
2.5.5 使用nginx反向代理harbor
回到0.210运维机上操作
# yum install nginx -y [root@server05 opt]# more /etc/nginx/conf.d/harbor.chinasoft.com.conf server { listen 80; server_name harbor.chinasoft.com; client_max_body_size 1000m; location / { proxy_pass http://127.0.0.1:180; } } # nginx -t # systemctl start nginx # systemctl enable nginx
浏览器输入：harbor.chinasoft.com
用户名：admin 密码：harbor123456
新建项目：public 访问级别：公开
2.5.6 提前准备pauser/nginx基础镜像
pauser镜像是k8s启动pod时,预先用来创建相关资源(如名称空间)的
nginx镜像是k8s部署好以后,我们测试pod创建所用的
docker login harbor.chinasoft.com -uadmin -pharbor123456
docker pull kubernetes/pause
docker pull nginx:1.17.9
docker tag kubernetes/pause:latest harbor.chinasoft.com/public/pause:latest
docker tag nginx:1.17.9 harbor.chinasoft.com/public/nginx:v1.17.9
docker push harbor.chinasoft.com/public/pause:latest
docker push harbor.chinasoft.com/public/nginx:v1.17.9
2.6 准备nginx文件服务
创建一个nginx虚拟主机,用来提供文件访问访问,主要依赖nginx的autoindex属性
2.6.1 创建文件访问
在0.206上
# 创建配置
# vim /etc/nginx/conf.d/k8s-yaml.chinasoft.com.conf server { listen 80; server_name k8s-yaml.chinasoft.com; location / { autoindex on; default_type text/plain; root /data/k8s-yaml; } }
# 启动nginx
mkdir -p /data/k8s-yaml/coredns
nginx -t
nginx -s reload
2.6.2 添加域名解析
在0.206的bind9域名服务器上,增加DNS记录
vi /var/named/chinasoft.com.zone
# 在最后添加一条解析记录
k8s-yaml A 10.11.0.210
# 同时滚动serial为
@ IN SOA dns.chinasoft.com. dnsadmin.chinasoft.com. ( 202009261634 ; serial
重启服务并验证:
systemctl restart named
[root@hdss7-11 ~]# dig -t A k8s-yaml.chinasoft.com +short
10.11.0.210
3 部署master节点-etcd服务
3.1 部署etcd集群
分别在207/208/209 上安装ectd服务,206节点作为备选节点
3.1.1 创建生成CA证书的JSON配置文件
在0.210上操作
一个配置里面包含了server端,clinet端和双向(peer)通信所需要的配置,后面创建证书的时候会传入不同的参数调用不同的配置
# vim /opt/certs/ca-config.json { "signing": { "default": { "expiry": "175200h" }, "profiles": { "server": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "server auth" ] }, "client": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "client auth" ] }, "peer": { "expiry": "175200h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } }
证书时间统一为10年,不怕过期
证书类型
client certificate：客户端使用，用于服务端认证客户端,例如etcdctl、etcd
proxy、fleetctl、docker客户端
server certificate：服务端使用，客户端以此验证服务端身份,例如docker服务端、kube-apiserver
peer certificate：双向证书，用于etcd集群成员间通信
3.1.3.创建生成自签发请求(csr)的json配置文件
注意:
需要将所有可能用来部署etcd的机器,都加入到hosts列表中
否则后期重新加入不在列表中的机器,需要更换所有etcd服务的证书
# cat /opt/certs/etcd-peer-csr.json { "CN": "k8s-etcd", "hosts": [ "10.11.0.206", "10.11.0.207", "10.11.0.208", "10.11.0.209" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "beijing", "L": "beijing", "O": "zq", "OU": "ops" } ] }
3.1.4.生成etcd证书文件
cd /opt/certs/ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json |cfssl-json -bare etcd-peer
[root@server05certs]# ll total 36 -rw-r--r-- 1 root root 837 Apr 19 15:35 ca-config.json -rw-r--r-- 1 root root 989 Apr 16 20:53 ca.csr -rw-r--r-- 1 root root 324 Apr 16 20:52 ca-csr.json -rw------- 1 root root 1679 Apr 16 20:53 ca-key.pem -rw-r--r-- 1 root root 1330 Apr 16 20:53 ca.pem -rw-r--r-- 1 root root 1062 Apr 19 15:35 etcd-peer.csr -rw-r--r-- 1 root root 363 Apr 19 15:35 etcd-peer-csr.json -rw------- 1 root root 1679 Apr 19 15:35 etcd-peer-key.pem -rw-r--r-- 1 root root 1419 Apr 19 15:35 etcd-peer.pem
3.2 安装启动etcd集群
以0.207做为演示,另外2台机器大同小异,不相同的配置都会特别说明
3.2.1 创建etcd用户和安装软件
etcd地址:https://github.com/etcd-io/etcd/tags
建议使用3.1版本,更高版本有问题
useradd -s /sbin/nologin -M etcd wget https://github.com/etcd-io/etcd/archive/v3.1.20.tar.gz tar xf etcd-v3.1.20-linux-amd64.tar.gz -C /usr/local/ cd /usr/local/ mv etcd-v3.1.20-linux-amd64/ etcd-v3.1.20 ln -s /usr/local/etcd-v3.1.20/ /usr/local/etcd
3.2.2 创建目录，拷贝证书文件
创建证书目录、数据目录、日志目录
mkdir -p /usr/local/etcd/certs /data/etcd /data/logs/etcd-server chown -R etcd.etcd /usr/local/etcd-v3.1.20/ chown -R etcd.etcd /data/etcd/ chown -R etcd.etcd /data/logs/etcd-server/
拷贝生成的证书文件
cd /opt/etcd/certs scp server05:/opt/certs/ca.pem . scp server05:/opt/certs/etcd-peer.pem . scp server05:/opt/certs/etcd-peer-key.pem . chown -R etcd.etcd /opt/etcd/certs
也可以先创建一个NFS,直接从NFS中拷贝
3.2.3 创建etcd服务启动脚本
参数说明: https://blog.csdn.net/kmhysoft/article/details/71106995
[root@server02 docker]# cat /usr/local/etcd/etcd-server-startup.sh #!/bin/sh ./etcd --name etcd-server-0-207 --data-dir /data/etcd/etcd-server --listen-peer-urls https://10.11.0.207:2380 --listen-client-urls https://10.11.0.207:2379,http://127.0.0.1:2379 --quota-backend-bytes 8000000000 --initial-advertise-peer-urls https://10.11.0.207:2380 --advertise-client-urls https://10.11.0.207:2379,http://127.0.0.1:2379 --initial-cluster etcd-server-0-207=https://10.11.0.207:2380,etcd-server-0-208=https://10.11.0.208:2380,etcd-server-0-209=https://10.11.0.209:2380 --ca-file ./certs/ca.pem --cert-file ./certs/etcd-peer.pem --key-file ./certs/etcd-peer-key.pem --client-cert-auth --trusted-ca-file ./certs/ca.pem --peer-ca-file ./certs/ca.pem --peer-cert-file ./certs/etcd-peer.pem --peer-key-file ./certs/etcd-peer-key.pem --peer-client-cert-auth --peer-trusted-ca-file ./certs/ca.pem --log-output stdout # chmod +x /usr/local/etcd/etcd-server-startup.sh
注意:以上启动脚本,有几个配置项在每个服务器都有所不同
--name #节点名字
--listen-peer-urls #监听其他节点所用的地址
--listen-client-urls #监听etcd客户端的地址
--initial-advertise-peer-urls #与其他节点交互信息的地址
--advertise-client-urls #与etcd客户端交互信息的地址
3.2.4 使用supervisor启动etcd
安装supervisor软件
yum install supervisor -y
systemctl start supervisord
systemctl enable supervisord
创建supervisor管理etcd的配置文件
配置说明参考: https://www.jianshu.com/p/53b5737534e8
# cat /etc/supervisord.d/etcd-server.ini [program:etcd-server] ; 显示的程序名,类型my.cnf,可以有多个 command=sh /usr/local/etcd/etcd-server-startup.sh numprocs=1 ; 启动进程数 (def 1) directory=/opt/etcd ; 启动命令前切换的目录 (def no cwd) autostart=true ; 是否自启 (default: true) autorestart=true ; 是否自动重启 (default: true) startsecs=30 ; 服务运行多久判断为成功(def. 1) startretries=3 ; 启动重试次数 (default 3) exitcodes=0,2 ; 退出状态码 (default 0,2) stopsignal=QUIT ; 退出信号 (default TERM) stopwaitsecs=10 ; 退出延迟时间 (default 10) user=etcd ; 运行用户 redirect_stderr=true ; 是否重定向错误输出到标准输出(def false) stdout_logfile=/data/logs/etcd-server/etcd.stdout.log stdout_logfile_maxbytes=64MB ; 日志文件大小 (default 50MB) stdout_logfile_backups=4 ; 日志文件滚动个数 (default 10) stdout_capture_maxbytes=1MB ; 设定capture管道的大小(default 0) ;子进程还有子进程,需要添加这个参数,避免产生孤儿进程 killasgroup=true stopasgroup=true
启动etcd服务并检查
supervisorctl update
supervisorctl status
netstat -lntup|grep etcd
3.2.5 部署启动集群其他机器
略
3.2.6 检查集群状态
[root@server02 docker]# /usr/local/etcd/etcdctl cluster-health member 18147702e67ecd43 is healthy: got healthy result from http://127.0.0.1:2379 member 75bbe2490207be71 is healthy: got healthy result from http://127.0.0.1:2379 member 7c790ae61abbf138 is healthy: got healthy result from http://127.0.0.1:2379 cluster is healthy [root@server02 docker]# /usr/local/etcd/etcdctl member list 18147702e67ecd43: name=etcd-server-0-208 peerURLs=https://10.11.0.208:2380 clientURLs=http://127.0.0.1:2379,https://10.11.0.208:2379 isLeader=true 75bbe2490207be71: name=etcd-server-0-207 peerURLs=https://10.11.0.207:2380 clientURLs=http://127.0.0.1:2379,https://10.11.0.207:2379 isLeader=false 7c790ae61abbf138: name=etcd-server-0-209 peerURLs=https://10.11.0.209:2380 clientURLs=http://127.0.0.1:2379,https://10.11.0.209:2379 isLeader=false