NGINX Plus Ingress Controller介绍
NGINX Ingress Controller是NGINX公司开发的可用于k8s充当Ingress Controller的产品，有NGINX和NGINX Plus两个版本。NGINX是开源免费版，而NGINX Plus是商业版本。
NGINX Plus Ingress Controller除了支持基于http内容的7层路由、TLS/SSL卸载等标准的k8s Ingress功能外，还提供了其它诸多的扩展功能。
而除了k8s原生定义的Ingress之外，NGINX Plus Ingress Controller还通过CRD定义了功能更丰富，配置更简单的7层负载均衡资源，VirtualServer和VirtualServerRoute（简称VS和VSR），用于对Ingress资源做补充和替换。例如通过VS的Split功能，就能很方便的在k8s中实现按比例的灰度发布。
Exposing the NGINX Ingress Controller
Once the base configuration is in place, the next step is to expose the NGINX Ingress Controller to the outside world to allow it to start receiving connections. This could be through a load-balancer like on AWS, GCP, or Azure. The other option when deploying on your own infrastructure, or a cloud provider with less capabilities, is to create a service with a NodePort to allow access to the Ingress Controller.
LoadBalancer
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/cloud/deploy.yamlNodePort
Using the NGINX-provided service-nodeport.yaml file, which is located in GitHub, will define a service that runs on ports 80 and 443. It can be applied using a single command line, as done before.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/baremetal/deploy.yaml
Validate the NGINX Ingress Controller
The final step is to make sure the Ingress controller is running.
❯ kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx ingress-nginx-admission-create-wb4rm 0/1 Completed 0 17m
ingress-nginx ingress-nginx-admission-patch-dqsnv 0/1 Completed 2 17m
ingress-nginx ingress-nginx-controller-74fd5565fb-lw6nq 1/1 Running 0 17m
❯ kubectl get services ingress-nginx-controller --namespace=ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.21.1.110 10.0.0.3 80:32495/TCP,443:30703/TCP 17mroot@ubuntu:~# kubectl get ingress NAME CLASS HOSTS ADDRESS PORTS AGE example-ingress <none> ubuntu 80 22h root@ubuntu:~# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 34d web2 NodePort 10.98.248.91 <none> 8097:31480/TCP 28h web3 NodePort 10.106.191.94 <none> 8097:30753/TCP 28h root@ubuntu:~#
host namepsace 并没有监听80端口
[root@centos7 my_nginx_ingress]# netstat -lpn | grep 80 tcp 0 0 10.10.16.251:2380 0.0.0.0:* LISTEN 18279/etcd tcp 0 0 0.0.0.0:31480 0.0.0.0:* LISTEN 21108/kube-proxy
pod namespace
[root@centos7 my_nginx_ingress]# kubectl get pods ingress-nginx-controller-577fbcd469-bm5nb -n ingress-nginx -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ingress-nginx-controller-577fbcd469-bm5nb 1/1 Running 3 65m 10.244.129.171 centos7 <none> <none> [root@centos7 my_nginx_ingress]#
bash-5.1$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 4: eth0@if134: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1480 qdisc noqueue state UP link/ether 6a:32:c6:ec:24:43 brd ff:ff:ff:ff:ff:ff inet 10.244.129.171/32 brd 10.244.129.171 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::6832:c6ff:feec:2443/64 scope link valid_lft forever preferred_lft forever bash-5.1$ netstat -lpn | grep ':80' | wc -l netstat: can't scan /proc - are you root? 256 bash-5.1$
访问不了 http://10.244.129.171:80/web2
root@ubuntu:~# wget http://10.244.129.171:80/web2 --2021-08-05 16:19:43-- http://10.244.129.171/web2 Connecting to 10.244.129.171:80... connected. HTTP request sent, awaiting response... 404 Not Found 2021-08-05 16:19:46 ERROR 404: Not Found. root@ubuntu:~# ping 10.244.129.171 PING 10.244.129.171 (10.244.129.171) 56(84) bytes of data. 64 bytes from 10.244.129.171: icmp_seq=1 ttl=63 time=1381 ms 64 bytes from 10.244.129.171: icmp_seq=2 ttl=63 time=1011 ms 64 bytes from 10.244.129.171: icmp_seq=3 ttl=63 time=659 ms ^C --- 10.244.129.171 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2017ms rtt min/avg/max/mdev = 659.121/1017.308/1381.155/294.797 ms, pipe 2 ^Croot@ubuntu:~#
root@ubuntu:~# kubectl describe ingress example-ingress Name: example-ingress Namespace: default Address: Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) Rules: Host Path Backends ---- ---- -------- ubuntu /web2 web2:8097 (10.244.41.1:80) /web3 web3:8097 (10.244.41.2:80) Annotations: ingress.kubernetes.io/rewrite-target: / Events: <none> root@ubuntu:~#
可以访问nginx
root@cloud:~# telnet 10.244.129.171 80 Trying 10.244.129.171... Connected to 10.244.129.171. Escape character is '^]'. ^CConnection closed by foreign host. root@cloud:~#
其他的访问不了
root@cloud:~# curl -I -H "Host: example-ingress.mydomain.com" http://10.244.129.171:80/web2 -k HTTP/1.1 404 Not Found Date: Thu, 05 Aug 2021 09:11:05 GMT Content-Type: text/html Content-Length: 153 Connection: keep-alive root@cloud:~# curl http://10.244.129.171:80/web2 -kL <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx</center> </body> </html> root@cloud:~# curl http://10.244.129.171:80/web2 -k <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx</center> </body> </html> root@cloud:~#
kubectl exec -it ingress-nginx-controller-577fbcd469-bm5nb -n ingress-nginx -- cat /etc/nginx/nginx.conf
root@cloud:~# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller LoadBalancer 10.101.13.73 <pending> 80:31719/TCP,443:30561/TCP 170m ingress-nginx-controller-admission ClusterIP 10.106.105.156 <none> 443/TCP 170m root@cloud:~#
Error from server (NotFound): pods "ingress-nginx-controller-577fbcd469-bm5nb" not found [root@centos7 ~]# kubectl exec -it ingress-nginx-controller-577fbcd469-bm5nb -n ingress-nginx -- bash bash-5.1$ ls fastcgi.conf fastcgi_params.default koi-win mime.types.default nginx.conf owasp-modsecurity-crs template win-utf fastcgi.conf.default geoip lua modsecurity nginx.conf.default scgi_params uwsgi_params fastcgi_params koi-utf mime.types modules opentracing.json scgi_params.default uwsgi_params.default bash-5.1$ cd / bash-5.1$ ls bin etc lib nginx-ingress-controller root srv usr dbg home media opt run sys var dev ingress-controller mnt proc sbin tmp wait-shutdown bash-5.1$ ./dbg conf | grep example-ingress set $ingress_name "example-ingress"; set $ingress_name "example-ingress"; set $ingress_name "example-ingress"; set $ingress_name "example-ingress"; set $ingress_name "example-ingress";
bash-5.1$ ./dbg conf | grep example-ingress set $ingress_name "example-ingress"; set $ingress_name "example-ingress"; set $ingress_name "example-ingress"; set $ingress_name "example-ingress"; set $ingress_name "example-ingress"; bash-5.1$ ./dbg conf | grep example-ingress -A 10 -B 10 set $proxy_upstream_name "-"; ssl_certificate_by_lua_block { certificate.call() } location /web3/ { set $namespace "default"; set $ingress_name "example-ingress"; set $service_name "web3"; set $service_port "8097"; set $location_path "/web3"; set $global_rate_limit_exceeding n; rewrite_by_lua_block { lua_ingress.rewrite({ force_ssl_redirect = false, ssl_redirect = true, force_no_ssl_redirect = false, -- proxy_pass http://upstream_balancer; proxy_redirect off; } location = /web3 { set $namespace "default"; set $ingress_name "example-ingress"; set $service_name "web3"; set $service_port "8097"; set $location_path "/web3"; set $global_rate_limit_exceeding n; rewrite_by_lua_block { lua_ingress.rewrite({ force_ssl_redirect = false, ssl_redirect = true, force_no_ssl_redirect = false, -- proxy_pass http://upstream_balancer; proxy_redirect off; } location /web2/ { set $namespace "default"; set $ingress_name "example-ingress"; set $service_name "web2"; set $service_port "8097"; set $location_path "/web2"; set $global_rate_limit_exceeding n; rewrite_by_lua_block { lua_ingress.rewrite({ force_ssl_redirect = false, ssl_redirect = true, force_no_ssl_redirect = false, -- proxy_pass http://upstream_balancer; proxy_redirect off; } location = /web2 { set $namespace "default"; set $ingress_name "example-ingress"; set $service_name "web2"; set $service_port "8097"; set $location_path "/web2"; set $global_rate_limit_exceeding n; rewrite_by_lua_block { lua_ingress.rewrite({ force_ssl_redirect = false, ssl_redirect = true, force_no_ssl_redirect = false, -- proxy_pass http://upstream_balancer; proxy_redirect off; } location / { set $namespace "default"; set $ingress_name "example-ingress"; set $service_name ""; set $service_port ""; set $location_path "/"; set $global_rate_limit_exceeding n; rewrite_by_lua_block { lua_ingress.rewrite({ force_ssl_redirect = false, ssl_redirect = true, force_no_ssl_redirect = false, bash-5.1$
service
root@ubuntu:~# iptables -t nat -S | grep 8097 -A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.98.248.91/32 -p tcp -m comment --comment "default/web2: cluster IP" -m tcp --dport 8097 -j KUBE-MARK-MASQ -A KUBE-SERVICES -d 10.98.248.91/32 -p tcp -m comment --comment "default/web2: cluster IP" -m tcp --dport 8097 -j KUBE-SVC-7UGYW7HVUL5B72AK -A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.106.191.94/32 -p tcp -m comment --comment "default/web3: cluster IP" -m tcp --dport 8097 -j KUBE-MARK-MASQ -A KUBE-SERVICES -d 10.106.191.94/32 -p tcp -m comment --comment "default/web3: cluster IP" -m tcp --dport 8097 -j KUBE-SVC-CV3FAXXTZCDKDMUO root@ubuntu:~# wget http://10.98.248.91:8097 --2021-08-05 16:10:11-- http://10.98.248.91:8097/ Connecting to 10.98.248.91:8097... connected. HTTP request sent, awaiting response... 200 OK Length: 612 [text/html] Saving to: ‘index.html.2’ index.html.2 100%[====================================================================================================================================================>] 612 --.-KB/s in 0s 2021-08-05 16:10:11 (84.8 MB/s) - ‘index.html.2’ saved [612/612] root@ubuntu:~#
ingrss 改成
root@ubuntu:~/nginx_ingress# cat web-ingress-lb.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: web-ingress-lb namespace: default annotations: kubernetes.io/ingress.class: nginx spec: rules: - host: web3.mydomain.com http: paths: - backend: serviceName: web3 servicePort: 8097 - host: web2.mydomain.com http: paths: - backend: serviceName: web2 servicePort: 8097
service 不采用nodeport
root@ubuntu:~/nginx_ingress# cat web-deployment-v2.yaml apiVersion: apps/v1 kind: Deployment metadata: name: web2 namespace: default spec: selector: matchLabels: run: web2 template: metadata: labels: run: web2 spec: containers: - image: nginx imagePullPolicy: IfNotPresent name: web2 ports: - containerPort: 80 protocol: TCP nodeSelector: kubernetes.io/hostname: cloud root@ubuntu:~/nginx_ingress# cat web-deployment-v3.yaml apiVersion: apps/v1 kind: Deployment metadata: name: web3 namespace: default spec: selector: matchLabels: run: web3 template: metadata: labels: run: web3 spec: containers: - image: nginx imagePullPolicy: IfNotPresent name: web3 ports: - containerPort: 80 protocol: TCP nodeSelector: kubernetes.io/hostname: cloud root@ubuntu:~/nginx_ingress# cat web-service-v2.yaml apiVersion: v1 kind: Service metadata: name: web2 namespace: default spec: ports: - port: 8097 protocol: TCP targetPort: 80 selector: run: web2 #type: NodePort root@ubuntu:~/nginx_ingress# cat web-service-v3.yaml apiVersion: v1 kind: Service metadata: name: web3 namespace: default spec: ports: - port: 8097 protocol: TCP targetPort: 80 selector: run: web3 #type: NodePort root@ubuntu:~/nginx_ingress#
root@ubuntu:~/nginx_ingress# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE apache-svc ClusterIP 10.111.63.105 <none> 80/TCP 14h kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 35d nginx-svc ClusterIP 10.103.182.145 <none> 80/TCP 14h web2 ClusterIP 10.99.87.66 <none> 8097/TCP 14h web3 ClusterIP 10.107.70.171 <none> 8097/TCP 14h root@ubuntu:~/nginx_ingress#
root@ubuntu:~/nginx_ingress# kubectl get ing NAME CLASS HOSTS ADDRESS PORTS AGE example-ingress <none> ubuntu.com 80 21m micro-ingress <none> nginx.mydomain.com,apache.mydomain.com 80 14h web-ingress <none> web.mydomain.com 80 14m web-ingress-lb <none> web3.mydomain.com,web2.mydomain.com 80 9m47s root@ubuntu:~/nginx_ingress# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller LoadBalancer 10.109.135.148 <pending> 80:31324/TCP,443:31274/TCP 14h ingress-nginx-controller-admission ClusterIP 10.107.93.85 <none> 443/TCP 14h root@ubuntu:~/nginx_ingress#
可以访问
root@cloud:~# curl -I -H "Host: web2.mydomain.com" http://10.109.135.148 HTTP/1.1 200 OK Date: Fri, 06 Aug 2021 02:19:29 GMT Content-Type: text/html Content-Length: 612 Connection: keep-alive Last-Modified: Tue, 25 May 2021 12:28:56 GMT ETag: "60aced88-264" Accept-Ranges: bytes root@cloud:~# curl -I -H "Host: web3.mydomain.com" http://10.109.135.148 HTTP/1.1 200 OK Date: Fri, 06 Aug 2021 02:19:34 GMT Content-Type: text/html Content-Length: 612 Connection: keep-alive Last-Modified: Tue, 25 May 2021 12:28:56 GMT ETag: "60aced88-264" Accept-Ranges: bytes root@cloud:~#
demo2
root@ubuntu:~/nginx_ingress# cat apache-app-svc.yaml apiVersion: apps/v1 kind: Deployment metadata: name: apache-app spec: replicas: 2 selector: matchLabels: app: apache-app template: metadata: labels: app: apache-app spec: containers: - name: apache-app image: httpd:latest ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: apache-svc labels: spec: ports: - port: 80 targetPort: 80 protocol: TCP selector: app: apache-app root@ubuntu:~/nginx_ingress# cat nginx-app-svc.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-app spec: replicas: 2 selector: matchLabels: app: nginx-app template: metadata: labels: app: nginx-app spec: containers: - name: nginx-app image: nginx:latest ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx-svc labels: spec: ports: - port: 80 targetPort: 80 protocol: TCP selector: app: nginx-app root@ubuntu:~/nginx_ingress# cat micro-ingress.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: micro-ingress namespace: default annotations: kubernetes.io/ingress.class: nginx spec: rules: - host: nginx.mydomain.com http: paths: - backend: serviceName: nginx-svc servicePort: 80 - host: apache.mydomain.com http: paths: - backend: serviceName: apache-svc servicePort: 80 root@ubuntu:~/nginx_ingress#
root@ubuntu:~/nginx_ingress# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller LoadBalancer 10.109.135.148 <pending> 80:31324/TCP,443:31274/TCP 14h ingress-nginx-controller-admission ClusterIP 10.107.93.85 <none> 443/TCP 14h root@ubuntu:~/nginx_ingress#
root@ubuntu:~/nginx_ingress# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE apache-svc ClusterIP 10.111.63.105 <none> 80/TCP 14h kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 35d nginx-svc ClusterIP 10.103.182.145 <none> 80/TCP 14h web2 ClusterIP 10.99.87.66 <none> 8097/TCP 15h web3 ClusterIP 10.107.70.171 <none> 8097/TCP 15h root@ubuntu:~/nginx_ingress#
root@cloud:~# curl -I -H "Host: apache.mydomain.com" http://10.109.135.148 HTTP/1.1 200 OK Date: Fri, 06 Aug 2021 02:24:56 GMT Content-Type: text/html Content-Length: 45 Connection: keep-alive Last-Modified: Mon, 11 Jun 2007 18:53:14 GMT ETag: "2d-432a5e4a73a80" Accept-Ranges: bytes root@cloud:~# curl -I -H "Host: nginx.mydomain.com" http://10.109.135.148 HTTP/1.1 200 OK Date: Fri, 06 Aug 2021 02:25:06 GMT Content-Type: text/html Content-Length: 612 Connection: keep-alive Last-Modified: Tue, 06 Jul 2021 14:59:17 GMT ETag: "60e46fc5-264" Accept-Ranges: bytes root@cloud:~#
root@cloud:~# kubectl exec -it ingress-nginx-controller-7478b9dbb5-6qk65 -n ingress-nginx -- /dbg backends all [ { "name": "default-apache-svc-80", "service": { "metadata": { "creationTimestamp": null }, "spec": { "ports": [ { "protocol": "TCP", "port": 80, "targetPort": 80 } ], "selector": { "app": "apache-app" }, "clusterIP": "10.111.63.105", "type": "ClusterIP", "sessionAffinity": "None" }, "status": { "loadBalancer": {} } }, "port": 80, "sslPassthrough": false, "endpoints": [ { "address": "10.244.129.184", "port": "80" }, { "address": "10.244.243.197", "port": "80" } ], "sessionAffinityConfig": { "name": "", "mode": "", "cookieSessionAffinity": { "name": "" } }, "upstreamHashByConfig": { "upstream-hash-by-subset-size": 3 }, "noServer": false, "trafficShapingPolicy": { "weight": 0, "header": "", "headerValue": "", "headerPattern": "", "cookie": "" } }, { "name": "default-nginx-svc-80", "service": { "metadata": { "creationTimestamp": null }, "spec": { "ports": [ { "protocol": "TCP", "port": 80, "targetPort": 80 } ], "selector": { "app": "nginx-app" }, "clusterIP": "10.103.182.145", "type": "ClusterIP", "sessionAffinity": "None" }, "status": { "loadBalancer": {} } }, "port": 80, "sslPassthrough": false, "endpoints": [ { "address": "10.244.129.179", "port": "80" }, { "address": "10.244.243.195", "port": "80" } ], "sessionAffinityConfig": { "name": "", "mode": "", "cookieSessionAffinity": { "name": "" } }, "upstreamHashByConfig": { "upstream-hash-by-subset-size": 3 }, "noServer": false, "trafficShapingPolicy": { "weight": 0, "header": "", "headerValue": "", "headerPattern": "", "cookie": "" } }, { "name": "default-web2-8097", "service": { "metadata": { "creationTimestamp": null }, "spec": { "ports": [ { "protocol": "TCP", "port": 8097, "targetPort": 80 } ], "selector": { "run": "web2" }, "clusterIP": "10.99.87.66", "type": "ClusterIP", "sessionAffinity": "None" }, "status": { "loadBalancer": {} } }, "port": 8097, "sslPassthrough": false, "endpoints": [ { "address": "10.244.41.1", "port": "80" } ], "sessionAffinityConfig": { "name": "", "mode": "", "cookieSessionAffinity": { "name": "" } }, "upstreamHashByConfig": { "upstream-hash-by-subset-size": 3 }, "noServer": false, "trafficShapingPolicy": { "weight": 0, "header": "", "headerValue": "", "headerPattern": "", "cookie": "" } }, { "name": "default-web3-8097", "service": { "metadata": { "creationTimestamp": null }, "spec": { "ports": [ { "protocol": "TCP", "port": 8097, "targetPort": 80 } ], "selector": { "run": "web3" }, "clusterIP": "10.107.70.171", "type": "ClusterIP", "sessionAffinity": "None" }, "status": { "loadBalancer": {} } }, "port": 8097, "sslPassthrough": false, "endpoints": [ { "address": "10.244.41.2", "port": "80" } ], "sessionAffinityConfig": { "name": "", "mode": "", "cookieSessionAffinity": { "name": "" } }, "upstreamHashByConfig": { "upstream-hash-by-subset-size": 3 }, "noServer": false, "trafficShapingPolicy": { "weight": 0, "header": "", "headerValue": "", "headerPattern": "", "cookie": "" } }, { "name": "upstream-default-backend", "port": 0, "sslPassthrough": false, "endpoints": [ { "address": "127.0.0.1", "port": "8181" } ], "sessionAffinityConfig": { "name": "", "mode": "", "cookieSessionAffinity": { "name": "" } }, "upstreamHashByConfig": {}, "noServer": false, "trafficShapingPolicy": { "weight": 0, "header": "", "headerValue": "", "headerPattern": "", "cookie": "" } } ] root@cloud:~#
configureDynamically
(dlv) b configureDynamically Breakpoint 1 set at 0xef07e0 for k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).configureDynamically() k8s.io/ingress-nginx/internal/ingress/controller/nginx.go:843 (dlv) c > k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).configureDynamically() k8s.io/ingress-nginx/internal/ingress/controller/nginx.go:843 (hits goroutine(342):1 total:1) (PC: 0xef07e0) Warning: debugging optimized function (dlv) bt 0 0x0000000000ef07e0 in k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).configureDynamically at k8s.io/ingress-nginx/internal/ingress/controller/nginx.go:843 1 0x0000000000ef4f20 in k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).syncIngress.func1 at k8s.io/ingress-nginx/internal/ingress/controller/controller.go:188 2 0x0000000000c5949c in k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection at k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:211 3 0x0000000000c59c10 in k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff at k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:399 4 0x0000000000eda258 in k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).syncIngress at k8s.io/ingress-nginx/internal/ingress/controller/controller.go:187 5 0x0000000000ef6e78 in k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).syncIngress-fm at k8s.io/ingress-nginx/internal/ingress/controller/controller.go:128 6 0x0000000000ed1b54 in k8s.io/ingress-nginx/internal/task.(*Queue).worker at k8s.io/ingress-nginx/internal/task/queue.go:129 7 0x0000000000ed2408 in k8s.io/ingress-nginx/internal/task.(*Queue).worker-fm at k8s.io/ingress-nginx/internal/task/queue.go:109 8 0x0000000000c5a254 in k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1 at k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155 9 0x0000000000c59334 in k8s.io/apimachinery/pkg/util/wait.BackoffUntil at k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156 10 0x0000000000c592a8 in k8s.io/apimachinery/pkg/util/wait.JitterUntil at k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133 11 0x0000000000ed1120 in k8s.io/apimachinery/pkg/util/wait.Until at k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:90 12 0x0000000000ed1120 in k8s.io/ingress-nginx/internal/task.(*Queue).Run at k8s.io/ingress-nginx/internal/task/queue.go:61 13 0x0000000000074194 in runtime.goexit at runtime/asm_arm64.s:1148
(dlv) b ingress-nginx/internal/ingress/controller/controller.go:188 Breakpoint 1 set at 0xef4f14 for k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).syncIngress.func1() k8s.io/ingress-nginx/internal/ingress/controller/controller.go:188 (dlv) c > k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).syncIngress.func1() k8s.io/ingress-nginx/internal/ingress/controller/controller.go:188 (hits goroutine(321):1 total:1) (PC: 0xef4f14) Warning: debugging optimized function (dlv) p pcfg *k8s.io/ingress-nginx/internal/ingress.Configuration { Backends: []*k8s.io/ingress-nginx/internal/ingress.Backend len: 5, cap: 5, [ *(*"k8s.io/ingress-nginx/internal/ingress.Backend")(0x4001001500), *(*"k8s.io/ingress-nginx/internal/ingress.Backend")(0x4001001380), *(*"k8s.io/ingress-nginx/internal/ingress.Backend")(0x4001001800), *(*"k8s.io/ingress-nginx/internal/ingress.Backend")(0x4001001680), *(*"k8s.io/ingress-nginx/internal/ingress.Backend")(0x4001001200), ], Servers: []*k8s.io/ingress-nginx/internal/ingress.Server len: 7, cap: 7, [ *(*"k8s.io/ingress-nginx/internal/ingress.Server")(0x4000269000), *(*"k8s.io/ingress-nginx/internal/ingress.Server")(0x4000ab6800), *(*"k8s.io/ingress-nginx/internal/ingress.Server")(0x4000ab6600), *(*"k8s.io/ingress-nginx/internal/ingress.Server")(0x4000ab7000), *(*"k8s.io/ingress-nginx/internal/ingress.Server")(0x4000ab6a00), *(*"k8s.io/ingress-nginx/internal/ingress.Server")(0x4000ab6e00), *(*"k8s.io/ingress-nginx/internal/ingress.Server")(0x4000ab6c00), ], TCPEndpoints: []k8s.io/ingress-nginx/internal/ingress.L4Service len: 0, cap: 0, [], UDPEndpoints: []k8s.io/ingress-nginx/internal/ingress.L4Service len: 0, cap: 0, [], PassthroughBackends: []*k8s.io/ingress-nginx/internal/ingress.SSLPassthroughBackend len: 0, cap: 0, nil, BackendConfigChecksum: "3401804793957086141", ConfigurationChecksum: "8128718577782006139", DefaultSSLCertificate: *k8s.io/ingress-nginx/internal/ingress.SSLCert { Name: "", Namespace: "", Certificate: *(*"crypto/x509.Certificate")(0x4000322580), CACertificate: []*crypto/x509.Certificate len: 0, cap: 0, nil, CAFileName: "", CASHA: "", CRLFileName: "", CRLSHA: "", PemFileName: "/etc/ingress-controller/ssl/default-fake-certificate.pem", PemSHA: "81b07e1879b354e742c32b8fa86c3e61d755c68b", CN: []string len: 2, cap: 2, [ "Kubernetes Ingress Controller Fake Certificate", "ingress.local", ], ExpireTime: (*time.Time)(0x400045e1a8), PemCertKey: "-----BEGIN CERTIFICATE----- MIIDcDCCAligAwIBAgIRAM7vQVegndPi6GSO...+2869 more", UID: "00000000-0000-0000-0000-000000000000",},} (dlv) p pcfg.Servers []*k8s.io/ingress-nginx/internal/ingress.Server len: 7, cap: 7, [ *{ Hostname: "_", SSLPassthrough: false, SSLCert: *(*"k8s.io/ingress-nginx/internal/ingress.SSLCert")(0x400045e0f0), Locations: []*k8s.io/ingress-nginx/internal/ingress.Location len: 1, cap: 1, [ *(*"k8s.io/ingress-nginx/internal/ingress.Location")(0x4000ec4000), ], Aliases: []string len: 0, cap: 0, nil, RedirectFromToWWW: false, CertificateAuth: (*"k8s.io/ingress-nginx/internal/ingress/annotations/authtls.Config")(0x4000269058), ProxySSL: (*"k8s.io/ingress-nginx/internal/ingress/annotations/proxyssl.Config")(0x40002690f8), ServerSnippet: "", SSLCiphers: "", SSLPreferServerCiphers: "", AuthTLSError: "",}, *{ Hostname: "apache.mydomain.com", SSLPassthrough: false, SSLCert: *k8s.io/ingress-nginx/internal/ingress.SSLCert nil, Locations: []*k8s.io/ingress-nginx/internal/ingress.Location len: 1, cap: 1, [ *(*"k8s.io/ingress-nginx/internal/ingress.Location")(0x4000ec4e00), ], Aliases: []string len: 0, cap: 0, [], RedirectFromToWWW: false, CertificateAuth: (*"k8s.io/ingress-nginx/internal/ingress/annotations/authtls.Config")(0x4000ab6858), ProxySSL: (*"k8s.io/ingress-nginx/internal/ingress/annotations/proxyssl.Config")(0x4000ab68f8), ServerSnippet: "", SSLCiphers: "", SSLPreferServerCiphers: "", AuthTLSError: "",}, *{ Hostname: "nginx.mydomain.com", SSLPassthrough: false, SSLCert: *k8s.io/ingress-nginx/internal/ingress.SSLCert nil, Locations: []*k8s.io/ingress-nginx/internal/ingress.Location len: 1, cap: 1, [ *(*"k8s.io/ingress-nginx/internal/ingress.Location")(0x4000ec4700), ], Aliases: []string len: 0, cap: 0, [], RedirectFromToWWW: false, CertificateAuth: (*"k8s.io/ingress-nginx/internal/ingress/annotations/authtls.Config")(0x4000ab6658), ProxySSL: (*"k8s.io/ingress-nginx/internal/ingress/annotations/proxyssl.Config")(0x4000ab66f8), ServerSnippet: "", SSLCiphers: "", SSLPreferServerCiphers: "", AuthTLSError: "",}, *{ Hostname: "ubuntu.com", SSLPassthrough: false, SSLCert: *k8s.io/ingress-nginx/internal/ingress.SSLCert nil, Locations: []*k8s.io/ingress-nginx/internal/ingress.Location len: 5, cap: 8, [ *(*"k8s.io/ingress-nginx/internal/ingress.Location")(0x4000ecc700), *(*"k8s.io/ingress-nginx/internal/ingress.Location")(0x4000ecce00), *(*"k8s.io/ingress-nginx/internal/ingress.Location")(0x4000ecc000), *(*"k8s.io/ingress-nginx/internal/ingress.Location")(0x4000ecd500), *(*"k8s.io/ingress-nginx/internal/ingress.Location")(0x4000ec6a00), ], Aliases: []string len: 0, cap: 0, [], RedirectFromToWWW: false, CertificateAuth: (*"k8s.io/ingress-nginx/internal/ingress/annotations/authtls.Config")(0x4000ab7058), ProxySSL: (*"k8s.io/ingress-nginx/internal/ingress/annotations/proxyssl.Config")(0x4000ab70f8), ServerSnippet: "", SSLCiphers: "", SSLPreferServerCiphers: "", AuthTLSError: "",}, *{ Hostname: "web.mydomain.com", SSLPassthrough: false, SSLCert: *k8s.io/ingress-nginx/internal/ingress.SSLCert nil, Locations: []*k8s.io/ingress-nginx/internal/ingress.Location len: 5, cap: 8, [ *(*"k8s.io/ingress-nginx/internal/ingress.Location")(0x4000ec7100), *(*"k8s.io/ingress-nginx/internal/ingress.Location")(0x4000ecdc00), *(*"k8s.io/ingress-nginx/internal/ingress.Location")(0x4000ec7800), *(*"k8s.io/ingress-nginx/internal/ingress.Location")(0x4000ece300), *(*"k8s.io/ingress-nginx/internal/ingress.Location")(0x4000ec5500), ], Aliases: []string len: 0, cap: 0, [], RedirectFromToWWW: false, CertificateAuth: (*"k8s.io/ingress-nginx/internal/ingress/annotations/authtls.Config")(0x4000ab6a58), ProxySSL: (*"k8s.io/ingress-nginx/internal/ingress/annotations/proxyssl.Config")(0x4000ab6af8), ServerSnippet: "", SSLCiphers: "", SSLPreferServerCiphers: "", AuthTLSError: "",}, *{ Hostname: "web2.mydomain.com", SSLPassthrough: false, SSLCert: *k8s.io/ingress-nginx/internal/ingress.SSLCert nil, Locations: []*k8s.io/ingress-nginx/internal/ingress.Location len: 1, cap: 1, [ *(*"k8s.io/ingress-nginx/internal/ingress.Location")(0x4000ec6300), ], Aliases: []string len: 0, cap: 0, [], RedirectFromToWWW: false, CertificateAuth: (*"k8s.io/ingress-nginx/internal/ingress/annotations/authtls.Config")(0x4000ab6e58), ProxySSL: (*"k8s.io/ingress-nginx/internal/ingress/annotations/proxyssl.Config")(0x4000ab6ef8), ServerSnippet: "", SSLCiphers: "", SSLPreferServerCiphers: "", AuthTLSError: "",}, *{ Hostname: "web3.mydomain.com", SSLPassthrough: false, SSLCert: *k8s.io/ingress-nginx/internal/ingress.SSLCert nil, Locations: []*k8s.io/ingress-nginx/internal/ingress.Location len: 1, cap: 1, [ *(*"k8s.io/ingress-nginx/internal/ingress.Location")(0x4000ec5c00), ], Aliases: []string len: 0, cap: 0, [], RedirectFromToWWW: false, CertificateAuth: (*"k8s.io/ingress-nginx/internal/ingress/annotations/authtls.Config")(0x4000ab6c58), ProxySSL: (*"k8s.io/ingress-nginx/internal/ingress/annotations/proxyssl.Config")(0x4000ab6cf8), ServerSnippet: "", SSLCiphers: "", SSLPreferServerCiphers: "", AuthTLSError: "",}, ] (dlv) p pcfg.Backends []*k8s.io/ingress-nginx/internal/ingress.Backend len: 5, cap: 5, [ *{ Name: "default-apache-svc-80", Service: *(*"k8s.io/api/core/v1.Service")(0x4000966e40), Port: (*"k8s.io/apimachinery/pkg/util/intstr.IntOrString")(0x4001001518), SSLPassthrough: false, Endpoints: []k8s.io/ingress-nginx/internal/ingress.Endpoint len: 2, cap: 2, [ (*"k8s.io/ingress-nginx/internal/ingress.Endpoint")(0x40001915e0), (*"k8s.io/ingress-nginx/internal/ingress.Endpoint")(0x4000191608), ], SessionAffinity: (*"k8s.io/ingress-nginx/internal/ingress.SessionAffinityConfig")(0x4001001558), UpstreamHashBy: (*"k8s.io/ingress-nginx/internal/ingress.UpstreamHashByConfig")(0x40010015d8), LoadBalancing: "", NoServer: false, TrafficShapingPolicy: (*"k8s.io/ingress-nginx/internal/ingress.TrafficShapingPolicy")(0x4001001610), AlternativeBackends: []string len: 0, cap: 0, nil,}, *{ Name: "default-nginx-svc-80", Service: *(*"k8s.io/api/core/v1.Service")(0x4000966be0), Port: (*"k8s.io/apimachinery/pkg/util/intstr.IntOrString")(0x4001001398), SSLPassthrough: false, Endpoints: []k8s.io/ingress-nginx/internal/ingress.Endpoint len: 2, cap: 2, [ (*"k8s.io/ingress-nginx/internal/ingress.Endpoint")(0x4000191540), (*"k8s.io/ingress-nginx/internal/ingress.Endpoint")(0x4000191568), ], SessionAffinity: (*"k8s.io/ingress-nginx/internal/ingress.SessionAffinityConfig")(0x40010013d8), UpstreamHashBy: (*"k8s.io/ingress-nginx/internal/ingress.UpstreamHashByConfig")(0x4001001458), LoadBalancing: "", NoServer: false, TrafficShapingPolicy: (*"k8s.io/ingress-nginx/internal/ingress.TrafficShapingPolicy")(0x4001001490), AlternativeBackends: []string len: 0, cap: 0, nil,}, *{ Name: "default-web2-8097", Service: *(*"k8s.io/api/core/v1.Service")(0x40009664c0), Port: (*"k8s.io/apimachinery/pkg/util/intstr.IntOrString")(0x4001001818), SSLPassthrough: false, Endpoints: []k8s.io/ingress-nginx/internal/ingress.Endpoint len: 1, cap: 1, [ (*"k8s.io/ingress-nginx/internal/ingress.Endpoint")(0x40005431d0), ], SessionAffinity: (*"k8s.io/ingress-nginx/internal/ingress.SessionAffinityConfig")(0x4001001858), UpstreamHashBy: (*"k8s.io/ingress-nginx/internal/ingress.UpstreamHashByConfig")(0x40010018d8), LoadBalancing: "", NoServer: false, TrafficShapingPolicy: (*"k8s.io/ingress-nginx/internal/ingress.TrafficShapingPolicy")(0x4001001910), AlternativeBackends: []string len: 0, cap: 0, nil,}, *{ Name: "default-web3-8097", Service: *(*"k8s.io/api/core/v1.Service")(0x4000966720), Port: (*"k8s.io/apimachinery/pkg/util/intstr.IntOrString")(0x4001001698), SSLPassthrough: false, Endpoints: []k8s.io/ingress-nginx/internal/ingress.Endpoint len: 1, cap: 1, [ (*"k8s.io/ingress-nginx/internal/ingress.Endpoint")(0x4000543140), ], SessionAffinity: (*"k8s.io/ingress-nginx/internal/ingress.SessionAffinityConfig")(0x40010016d8), UpstreamHashBy: (*"k8s.io/ingress-nginx/internal/ingress.UpstreamHashByConfig")(0x4001001758), LoadBalancing: "", NoServer: false, TrafficShapingPolicy: (*"k8s.io/ingress-nginx/internal/ingress.TrafficShapingPolicy")(0x4001001790), AlternativeBackends: []string len: 0, cap: 0, nil,}, *{ Name: "upstream-default-backend", Service: *k8s.io/api/core/v1.Service nil, Port: (*"k8s.io/apimachinery/pkg/util/intstr.IntOrString")(0x4001001218), SSLPassthrough: false, Endpoints: []k8s.io/ingress-nginx/internal/ingress.Endpoint len: 1, cap: 1, [ (*"k8s.io/ingress-nginx/internal/ingress.Endpoint")(0x4000542f60), ], SessionAffinity: (*"k8s.io/ingress-nginx/internal/ingress.SessionAffinityConfig")(0x4001001258), UpstreamHashBy: (*"k8s.io/ingress-nginx/internal/ingress.UpstreamHashByConfig")(0x40010012d8), LoadBalancing: "", NoServer: false, TrafficShapingPolicy: (*"k8s.io/ingress-nginx/internal/ingress.TrafficShapingPolicy")(0x4001001310), AlternativeBackends: []string len: 0, cap: 0, nil,}, ] (dlv)
(dlv) bt 0 0x0000000000d75fe0 in k8s.io/ingress-nginx/internal/nginx.NewPostStatusRequest at k8s.io/ingress-nginx/internal/nginx/main.go:81 1 0x0000000000ef2048 in k8s.io/ingress-nginx/internal/ingress/controller.configureCertificates at k8s.io/ingress-nginx/internal/ingress/controller/nginx.go:1015 2 0x0000000000ef0974 in k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).configureDynamically at k8s.io/ingress-nginx/internal/ingress/controller/nginx.go:862 3 0x0000000000ef4f20 in k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).syncIngress.func1 at k8s.io/ingress-nginx/internal/ingress/controller/controller.go:188 4 0x0000000000c5949c in k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection at k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:211 5 0x0000000000c59c10 in k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff at k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:399 6 0x0000000000eda258 in k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).syncIngress at k8s.io/ingress-nginx/internal/ingress/controller/controller.go:187 7 0x0000000000ef6e78 in k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).syncIngress-fm at k8s.io/ingress-nginx/internal/ingress/controller/controller.go:128 8 0x0000000000ed1b54 in k8s.io/ingress-nginx/internal/task.(*Queue).worker at k8s.io/ingress-nginx/internal/task/queue.go:129 9 0x0000000000ed2408 in k8s.io/ingress-nginx/internal/task.(*Queue).worker-fm at k8s.io/ingress-nginx/internal/task/queue.go:109 10 0x0000000000c5a254 in k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1 at k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155 11 0x0000000000c59334 in k8s.io/apimachinery/pkg/util/wait.BackoffUntil at k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156 12 0x0000000000c592a8 in k8s.io/apimachinery/pkg/util/wait.JitterUntil at k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133 13 0x0000000000ed1120 in k8s.io/apimachinery/pkg/util/wait.Until at k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:90 14 0x0000000000ed1120 in k8s.io/ingress-nginx/internal/task.(*Queue).Run at k8s.io/ingress-nginx/internal/task/queue.go:61 15 0x0000000000074194 in runtime.goexit at runtime/asm_arm64.s:1148 (dlv) c
NewGetStatusRequest
root@cloud:~# kubectl exec -it ingress-nginx-controller-7478b9dbb5-6qk65 -n ingress-nginx -- netstat -pan | grep 10246 tcp 0 0 127.0.0.1:10246 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:10246 127.0.0.1:41092 TIME_WAIT - tcp 0 0 127.0.0.1:10246 127.0.0.1:41210 TIME_WAIT - tcp 0 0 127.0.0.1:10246 127.0.0.1:41240 TIME_WAIT - tcp 0 0 127.0.0.1:10246 127.0.0.1:41120 TIME_WAIT - tcp 0 0 127.0.0.1:10246 127.0.0.1:41180 TIME_WAIT - tcp 0 0 127.0.0.1:10246 127.0.0.1:41152 TIME_WAIT - root@cloud:~#
# default server, used for NGINX healthcheck and access to nginx stats server { listen 127.0.0.1:10246; set $proxy_upstream_name "internal"; keepalive_timeout 0; gzip off; access_log off; location /healthz { return 200; } -- error("require failed: " .. tostring(res)) else configuration = res end ok, res = pcall(require, "tcp_udp_configuration") if not ok then error("require failed: " .. tostring(res)) else tcp_udp_configuration = res tcp_udp_configuration.prohibited_localhost_port = '10246' end ok, res = pcall(require, "tcp_udp_balancer") if not ok then error("require failed: " .. tostring(res)) else tcp_udp_balancer = res end }
root@cloud:~# kubectl exec -it ingress-nginx-controller-7478b9dbb5-6qk65 -n ingress-nginx -- curl http://127.0.0.1:10246/configuration/backends -o backend.json % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2887 0 2887 0 0 1064k 0 --:--:-- --:--:-- --:--:-- 1409k root@cloud:~# ls arm calicoctl container-support-utils google-cloud-sdk.tar.gz ingress-nginx-controller.v0.47.0.tar libseccomp qemu5.0 start.sh virtio-fs articles calico.yaml controller101 gov ing.text LICENSE qemu-system subcommands workloads bazel client.c controller.v0.48.1.debug.tar gov.c iso mycontainer rbac-kdd.yaml test_container zircon boost_1_69_0 cloud.yaml delve gvisor kata.1.9 ndn README.md trojan-go zircon2 boost_1_69_0.tar.gz cmd gcc-7.5.0 haokan keepalived.conf ndn-cxx runtime ubuntu.yaml zircon3 boost_1_70_0.tar.gz config gcc-7.5.0.tar.gz hello.txt kernel.5.5.19 onlyGvisor share udpclient c++ containerd1.3 gcc-tools hyper kube-flannel.yml OWNERS start_hp.sh udpclient.c root@cloud:~# kubectl get pods ingress-nginx-controller-7478b9dbb5-6qk65 -n ingress-nginx -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ingress-nginx-controller-7478b9dbb5-6qk65 1/1 Running 3 15h 10.244.41.5 cloud <none> <none> root@cloud:~# curl http://10.244.41.5:10246/configuration/backends -o backend.json % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to 10.244.41.5 port 10246: Connection refused root@cloud:~# kubectl exec -it ingress-nginx-controller-7478b9dbb5-6qk65 -n ingress-nginx -- curl http://127.0.0.1:10246/configuration/backends [{"name":"default-apache-svc-80","service":{"metadata":{"creationTimestamp":null},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":80}],"selector":{"app":"apache-app"},"clusterIP":"10.111.63.105","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":80,"sslPassthrough":false,"endpoints":[{"address":"10.244.129.184","port":"80"},{"address":"10.244.243.197","port":"80"}],"sessionAffinityConfig":{"name":"","mode":"","cookieSessionAffinity":{"name":""}},"upstreamHashByConfig":{"upstream-hash-by-subset-size":3},"noServer":false,"trafficShapingPolicy":{"weight":0,"header":"","headerValue":"","headerPattern":"","cookie":""}},{"name":"default-nginx-svc-80","service":{"metadata":{"creationTimestamp":null},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":80}],"selector":{"app":"nginx-app"},"clusterIP":"10.103.182.145","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":80,"sslPassthrough":false,"endpoints":[{"address":"10.244.129.179","port":"80"},{"address":"10.244.243.195","port":"80"}],"sessionAffinityConfig":{"name":"","mode":"","cookieSessionAffinity":{"name":""}},"upstreamHashByConfig":{"upstream-hash-by-subset-size":3},"noServer":false,"trafficShapingPolicy":{"weight":0,"header":"","headerValue":"","headerPattern":"","cookie":""}},{"name":"default-web2-8097","service":{"metadata":{"creationTimestamp":null},"spec":{"ports":[{"protocol":"TCP","port":8097,"targetPort":80}],"selector":{"run":"web2"},"clusterIP":"10.99.87.66","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":8097,"sslPassthrough":false,"endpoints":[{"address":"10.244.41.1","port":"80"}],"sessionAffinityConfig":{"name":"","mode":"","cookieSessionAffinity":{"name":""}},"upstreamHashByConfig":{"upstream-hash-by-subset-size":3},"noServer":false,"trafficShapingPolicy":{"weight":0,"header":"","headerValue":"","headerPattern":"","cookie":""}},{"name":"default-web3-8097","service":{"metadata":{"creationTimestamp":null},"spec":{"ports":[{"protocol":"TCP","port":8097,"targetPort":80}],"selector":{"run":"web3"},"clusterIP":"10.107.70.171","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":8097,"sslPassthrough":false,"endpoints":[{"address":"10.244.41.2","port":"80"}],"sessionAffinityConfig":{"name":"","mode":"","cookieSessionAffinity":{"name":""}},"upstreamHashByConfig":{"upstream-hash-by-subset-size":3},"noServer":false,"trafficShapingPolicy":{"weight":0,"header":"","headerValue":"","headerPattern":"","cookie":""}},{"name":"upstream-default-backend","port":0,"sslPassthrough":false,"endpoints":[{"address":"127.0.0.1","port":"8181"}],"sessionAffinityConfig":{"name":"","mode":"","cookieSessionAffinity":{"name":""}},"upstreamHashByConfig":{},"noServer":false,"trafficShapingPolicy":{"weight":0,"header":"","headerValue":"","headerPattern":"","cookie":""}}]root@cloud:~# root@cloud:~# root@cloud:~# kubectl exec -it ingress-nginx-controller-7478b9dbb5-6qk65 -n ingress-nginx -- bash bash-5.1$ ls backend.json fastcgi_params koi-utf mime.types modules opentracing.json scgi_params.default uwsgi_params.default fastcgi.conf fastcgi_params.default koi-win mime.types.default nginx.conf owasp-modsecurity-crs template win-utf fastcgi.conf.default geoip lua modsecurity nginx.conf.default scgi_params uwsgi_params bash-5.1$ curl http://127.0.0.1:10246/configuration/backends -o backend.json % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2887 0 2887 0 0 1510k 0 --:--:-- --:--:-- --:--:-- 2819k bash-5.1$ cat backend.json [{"name":"default-apache-svc-80","service":{"metadata":{"creationTimestamp":null},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":80}],"selector":{"app":"apache-app"},"clusterIP":"10.111.63.105","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":80,"sslPassthrough":false,"endpoints":[{"address":"10.244.129.184","port":"80"},{"address":"10.244.243.197","port":"80"}],"sessionAffinityConfig":{"name":"","mode":"","cookieSessionAffinity":{"name":""}},"upstreamHashByConfig":{"upstream-hash-by-subset-size":3},"noServer":false,"trafficShapingPolicy":{"weight":0,"header":"","headerValue":"","headerPattern":"","cookie":""}},{"name":"default-nginx-svc-80","service":{"metadata":{"creationTimestamp":null},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":80}],"selector":{"app":"nginx-app"},"clusterIP":"10.103.182.145","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":80,"sslPassthrough":false,"endpoints":[{"address":"10.244.129.179","port":"80"},{"address":"10.244.243.195","port":"80"}],"sessionAffinityConfig":{"name":"","mode":"","cookieSessionAffinity":{"name":""}},"upstreamHashByConfig":{"upstream-hash-by-subset-size":3},"noServer":false,"trafficShapingPolicy":{"weight":0,"header":"","headerValue":"","headerPattern":"","cookie":""}},{"name":"default-web2-8097","service":{"metadata":{"creationTimestamp":null},"spec":{"ports":[{"protocol":"TCP","port":8097,"targetPort":80}],"selector":{"run":"web2"},"clusterIP":"10.99.87.66","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":8097,"sslPassthrough":false,"endpoints":[{"address":"10.244.41.1","port":"80"}],"sessionAffinityConfig":{"name":"","mode":"","cookieSessionAffinity":{"name":""}},"upstreamHashByConfig":{"upstream-hash-by-subset-size":3},"noServer":false,"trafficShapingPolicy":{"weight":0,"header":"","headerValue":"","headerPattern":"","cookie":""}},{"name":"default-web3-8097","service":{"metadata":{"creationTimestamp":null},"spec":{"ports":[{"protocol":"TCP","port":8097,"targetPort":80}],"selector":{"run":"web3"},"clusterIP":"10.107.70.171","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":8097,"sslPassthrough":false,"endpoints":[{"address":"10.244.41.2","port":"80"}],"sessionAffinityConfig":{"name":"","mode":"","cookieSessionAffinity":{"name":""}},"upstreamHashByConfig":{"upstream-hash-by-subset-size":3},"noServer":false,"trafficShapingPolicy":{"weight":0,"header":"","headerValue":"","headerPattern":"","cookie":""}},{"name":"upstream-default-backend","port":0,"sslPassthrough":false,"endpoints":[{"address":"127.0.0.1","port":"8181"}],"sessionAffinityConfig":{"name":"","mode":"","cookieSessionAffinity":{"name":""}},"upstreamHashByConfig":{},"noServer":false,"trafficShapingPolicy":{"weight":0,"header":"","headerValue":"","headerPattern":"","cookie":""}}]bash-5.1$ bash-5.1$ bash-5.1$
OpenResty balancer_by_lua 配置指令
upstream upstream_balancer { ### Attention!!! # # We no longer create "upstream" section for every backend. # Backends are handled dynamically using Lua. If you would like to debug # and see what backends ingress-nginx has in its memory you can # install our kubectl plugin https://kubernetes.github.io/ingress-nginx/kubectl-plugin. # Once you have the plugin you can use "kubectl ingress-nginx backends" command to # inspect current backends. # ### -- } lua_add_variable $proxy_upstream_name; log_format log_stream '[$remote_addr] [$time_local] $protocol $status $bytes_sent $bytes_received $session_time'; access_log /var/log/nginx/access.log log_stream ; error_log /var/log/nginx/error.log notice; upstream upstream_balancer { server 0.0.0.1:1234; # placeholder balancer_by_lua_block { tcp_udp_balancer.balance() } }
openresty
[root@centos7 ~]# openresty -V nginx version: openresty/1.19.3.2 built by gcc 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC) built with OpenSSL 1.1.1i 8 Dec 2020 (running with OpenSSL 1.1.1k 25 Mar 2021) TLS SNI support enabled configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.19 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../ngx_stream_lua-0.0.9 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --with-cc='ccache gcc -fdiagnostics-color=always' --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module
mkdir ~/work cd ~/work mkdir logs/ conf/
tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN 87749/nginx: master [root@centos7 work]# cat conf/nginx.conf worker_processes 1; error_log logs/error.log; events { worker_connections 1024; } http { server { listen 8888; location / { default_type text/html; content_by_lua ' ngx.say("<p>hello, world</p>") '; } } }
[root@centos7 conf]# ls /usr/local/openresty/nginx/sbin nginx [root@centos7 conf]# PATH=/usr/local/openresty/nginx/sbin:$PATH [root@centos7 conf]# export PATH [root@centos7 conf]#
[root@centos7 conf]# ls /usr/local/openresty/nginx/sbin nginx [root@centos7 conf]# PATH=/usr/local/openresty/nginx/sbin:$PATH [root@centos7 conf]# export PATH [root@centos7 conf]# env | grep open PATH=/usr/local/openresty/nginx/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin [root@centos7 conf]# ls /usr/local/openresty/nginx/ conf html logs sbin [root@centos7 conf]# ls /usr/local/openresty/ bin COPYRIGHT luajit lualib nginx openssl111 pcre site zlib [root@centos7 conf]# nginx -V nginx version: openresty/1.19.3.2 built by gcc 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC) built with OpenSSL 1.1.1i 8 Dec 2020 (running with OpenSSL 1.1.1k 25 Mar 2021) TLS SNI support enabled configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.19 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../ngx_stream_lua-0.0.9 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --with-cc='ccache gcc -fdiagnostics-color=always' --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module [root@centos7 conf]#
[root@centos7 conf]# nginx -p `pwd`/ -c conf/nginx.conf nginx: [alert] could not open error log file: open() "/root/work/conf/logs/error.log" failed (2: No such file or directory) 2021/08/06 03:05:15 [emerg] 87416#87416: open() "/root/work/conf/conf/nginx.conf" failed (2: No such file or directory) [root@centos7 conf]# cd .. [root@centos7 work]# nginx -p `pwd`/ -c conf/nginx.conf [root@centos7 work]# ps -ef | grep nginx
[root@centos7 work]# netstat -pan | grep 8888 tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN 87749/nginx: master [root@centos7 work]#
[root@centos7 work]# curl http://localhost:8888/ <p>hello, world</p> You have new mail in /var/spool/mail/root [root@centos7 work]#
--启动 nginx --停止 nginx -s stop --重启 nginx -s reload --检验nginx配置是否正确 nginx -t ———————————————— 2
balancer_by_lua_block
[root@centos7 work]# cat conf/nginx.conf worker_processes 1; error_log logs/error.log; events { worker_connections 1024; } http{ upstream backend{ server 0.0.0.0; balancer_by_lua_block { local balancer = require "ngx.balancer" local host = {"10.99.87.66", "10.107.70.171"} local backend = "" local port = ngx.var.server_port local remote_ip = ngx.var.remote_addr local key = remote_ip..port local hash = ngx.crc32_long(key); hash = (hash % 2) + 1 backend = host[hash] ngx.log(ngx.DEBUG, "ip_hash=", ngx.var.remote_addr, " hash=", hash, " up=", backend, ":", port) local ok, err = balancer.set_current_peer(backend, port) if not ok then ngx.log(ngx.ERR, "failed to set the current peer: ", err) return ngx.exit(500) end ngx.log(ngx.DEBUG, "current peer ", backend, ":", port) } } server { listen 8889; listen 8097; listen 7777; server_name *.x.com; location / { proxy_pass http://backend; } } }
[root@centos7 work]# ps -elf | grep 8097 0 S root 15608 48416 0 80 0 - 1729 pipe_w 03:40 pts/0 00:00:00 grep --color=auto 8097 [root@centos7 work]# netstat -pan | grep 8889 tcp 0 0 0.0.0.0:8889 0.0.0.0:* LISTEN 118653/nginx: maste [root@centos7 work]# netstat -pan | grep 8097 tcp 0 0 0.0.0.0:8097 0.0.0.0:* LISTEN 118653/nginx: maste [root@centos7 work]# netstat -pan | grep 7777 tcp 0 0 0.0.0.0:7777 0.0.0.0:* LISTEN 118653/nginx: maste [root@centos7 work]#
[root@centos7 work]# nginx -p `pwd`/ -c conf/nginx.conf nginx: [emerg] "upstream" directive is not allowed here in /root/work/conf/nginx.conf:1
[root@centos7 work]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE apache-svc ClusterIP 10.111.63.105 <none> 80/TCP 20h kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 35d nginx-svc ClusterIP 10.103.182.145 <none> 80/TCP 20h web2 ClusterIP 10.99.87.66 <none> 8097/TCP 20h web3 ClusterIP 10.107.70.171 <none> 8097/TCP 20h [root@centos7 work]# curl http://localhost:8097/ <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html>