环境说明
| kubernetes版本 | nginx-ingress-controller版本 | 使用端口情况 |
|---|---|---|
| 1.18.18 | 0.45.0 | 80、443、8443 |
官方说明：
下载所需的 yaml 文件
mkdir ~/ingress && cd ~/ingress
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.45.0/deploy/static/provider/baremetal/deploy.yaml
修改配置文件
这里演示的是高可用的 nginx-ingress-controller 服务。
# 在 ingress-nginx-controller 容器的 deploy.spec 添加 replicas: 2
spec:
replicas: 2
将原本的 nodeport 修改成 clusterIP
# 在 ingress-nginx-controller service的 svc.spec 注释掉 type: NodePort
spec:
# type: NodePort
将容器端口映射到宿主机
# 在 ingress-nginx-controller 容器的 deploy.spec.template.spec 添加 hostNetwork: true
spec:
hostNetwork: true
# 在 ingress-nginx-controller 容器的 deploy.spec.template.spec.containers.ports 添加 hostPost 字段
ports:
- name: http
containerPort: 80 # 添加的字段
hostPort: 80
protocol: TCP
- name: https
containerPort: 443 # 添加的字段
hostPort: 443
protocol: TCP
- name: webhook
containerPort: 8443
protocol: TCP
修改DNS的策略
# 在 ingress-nginx-controller 容器的 deploy.spec.template.spec 修改 dnsPolicy
spec:
dnsPolicy: ClusterFirstWithHostNet
修改下载镜像路径
# 在 ingress-nginx-controller 容器的 deploy.spec.template.spec.containers 修改 image 字段
containers:
- name: controller
image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v0.45.0
指定 pod 调度特定节点
# 节点添加标签
kubectl label node k8s-node02 kubernetes.io/ingress=nginx
kubectl label node k8s-node03 kubernetes.io/ingress=nginx
# 在 ingress-nginx-controller 容器的 deploy.spec.template.spec 修改 nodeSelector
nodeSelector:
kubernetes.io/ingress: nginx
# 在 ingress-nginx-controller 容器的 deploy.spec.template.spec 添加 affinity
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
topologyKey: kubernetes.io/hostname
启动服务
$ kubectl apply -f deploy.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
$ kubectl -n ingress-nginx get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx-admission-create-tm6hb 0/1 Completed 0 21s 20.0.85.198 k8s-node01 <none> <none>
ingress-nginx-admission-patch-64bgc 0/1 Completed 1 21s 20.0.32.136 k8s-master01 <none> <none>
ingress-nginx-controller-656cf6c7fd-lw9dx 1/1 Running 0 21s 192.168.32.138 k8s-node03 <none> <none>
ingress-nginx-controller-656cf6c7fd-ncsrz 1/1 Running 0 21s 192.168.32.137 k8s-node02 <none> <none>
验证
创建nginx应用
cat > nginx.yaml <<-EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
app: my-nginx
template:
metadata:
labels:
app: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
resources:
limits:
memory: "200Mi"
cpu: "500m"
requests:
memory: "100Mi"
cpu: "100m"
ports:
- name: web
containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: my-nginx
ports:
- port: 80
targetPort: web
EOF
$ kubectl apply -f nginx.yaml
deployment.apps/my-nginx created
service/nginx-service created
$ kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
my-nginx-759cf4d696-vkj4q 1/1 Running 0 4m10s 20.0.85.199 k8s-node01 <none> <none>
创建ingress资源
$ cat > nginx-ingress.yaml <<-EOF
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress
labels:
name: nginx-ingress
spec:
backend:
serviceName: nginx-service
servicePort: 80
rules:
- host: www.ecloud.com
http:
paths:
- path: /
backend:
serviceName: nginx-service
servicePort: 80
EOF
$ kubectl apply -f nginx-ingress.yaml
ingress.extensions/nginx-ingress created
$ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
nginx-ingress <none> www.ecloud.com 192.168.32.137,192.168.32.138 80 21s
使用域名访问
$ echo '192.168.32.137 www.ecloud.com' >> /etc/hosts
$ curl www.ecloud.com
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
可以通过
keepalived+LVS高可用，使用VIP做域名解析。这里就不实现了。
sysctl 调优
# 临时临时
kubectl patch deployment -n ingress-nginx nginx-ingress-controller
--patch="$(curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/customization/sysctl/patch.json)"
# 永久生效
# 在 ingress-nginx-controller 容器的 deploy.spec.template.spec 添加 initContainers
initContainers:
- name: sysctl
image: alpine:3.13
securityContext:
privileged: true
command: ["sh", "-c", "sysctl -w net.core.somaxconn=32768; sysctl -w net.ipv4.ip_local_port_range='32768 65535'"]
变化：
- 积压队列设置net.core.somaxconn从128到32768
- 临时端口设置net.ipv4.ip_local_port_range从32768 60999到32768 65535(符合端口规划)
附加iptables规则
iptables -t filter -I INPUT -p tcp -m multiport --dport 80,443,8443 -m comment --comment "nginx ingress controller ports" -j ACCEPT