Calico Overlay网络
在Calico Overlay网络中有两种模式可选（仅支持IPV4地址）
- IP-in-IP （使用BGP实现）
- Vxlan （不使用BGP实现）
两种模式均支持如下参数
- Always: 永远进行 IPIP 封装(默认)
- CrossSubnet: 只在跨网段时才进行 IPIP 封装，适合有 Kubernetes 节点在其他网段的情况，属于中肯友好方案
- Never: 从不进行 IPIP 封装，适合确认所有 Kubernetes 节点都在同一个网段下的情况（配置此参数就开启了BGP模式）
在默认情况下，默认的 ipPool 启用了 IPIP 封装(至少通过官方安装文档安装的 Calico 是这样)，并且封装模式为 Always；这也就意味着任何时候都会在原报文上封装新 IP 地址，在这种情况下将外部流量路由到 RR 节点，RR 节点再转发进行 IPIP 封装时，可能出现网络无法联通的情况(没仔细追查，网络渣，猜测是 Pod 那边得到的源 IP 不对导致的)；此时我们应当调整 IPIP 封装策略为 CrossSubnet
导出 ipPool 配置
1
|
calicoctl get ippool default-ipv4-ippool -o yaml > ippool.yaml
|
修改 ipipMode 值为 CrossSubnet
1
|
apiVersion: projectcalico.org/v3
|
重新使用 calicoctl apply -f ippool.yaml 应用既可
- name: CALICO_IPV4POOL_IPIP value: "Always"
root@ubuntu:~# ./calicoctl node status Calico process is running. IPv4 BGP status +--------------+-------------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-------------------+-------+----------+-------------+ | 10.10.16.47 | node-to-node mesh | up | 08:24:07 | Established | | 10.10.16.81 | node-to-node mesh | up | 10:59:49 | Established | +--------------+-------------------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. root@ubuntu:~# ./calicoctl get node NAME bogon cloud ubuntu root@ubuntu:~#
改为
- name: CALICO_IPV4POOL_IPIP value: "Never"
root@ubuntu:~# ./calicoctl get node NAME bogon cloud ubuntu root@ubuntu:~# ./calicoctl node status Calico process is running. None of the BGP backend processes (BIRD or GoBGP) are running. root@ubuntu:~# ./calicoctl node status Calico process is running. IPv4 BGP status +--------------+-------------------+-------+----------+--------------------------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-------------------+-------+----------+--------------------------------+ | 10.10.16.81 | node-to-node mesh | up | 02:14:59 | Established | | 10.10.16.47 | node-to-node mesh | start | 02:15:02 | Active Socket: Connection | | | | | | closed | +--------------+-------------------+-------+----------+--------------------------------+ IPv6 BGP status No IPv6 peers found. root@ubuntu:~#
root@ubuntu:~# ./calicoctl node status Calico process is running. IPv4 BGP status +--------------+-------------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-------------------+-------+----------+-------------+ | 10.10.16.81 | node-to-node mesh | up | 02:15:22 | Established | | 10.10.16.47 | node-to-node mesh | up | 02:15:10 | Established | +--------------+-------------------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. root@ubuntu:~#
root@ubuntu:~# kubectl describe pod calico-node-dlq4c -n kube-system | grep CALICO_IPV4POOL_IPIP
CALICO_IPV4POOL_IPIP: Never
root@ubuntu:~#
创建pod之前
root@ubuntu:~# ip a sh tunl0 9: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 inet 10.244.243.192/32 scope global tunl0 valid_lft forever preferred_lft forever root@ubuntu:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.10.16.254 0.0.0.0 UG 0 0 0 enahisic2i0 10.10.16.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i0 10.244.2.0 10.10.16.47 255.255.255.0 UG 0 0 0 tunl0 10.244.29.0 10.10.16.81 255.255.255.192 UG 0 0 0 tunl0 10.244.41.0 10.10.16.47 255.255.255.192 UG 0 0 0 tunl0 10.244.243.192 0.0.0.0 255.255.255.192 U 0 0 0 * 10.244.243.194 0.0.0.0 255.255.255.255 UH 0 0 0 cali0d27bc8b0f7 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 root@ubuntu:~#
root@ubuntu:~# kubectl apply -f web-anti-affinity.yaml deployment.apps/web-nginx created root@ubuntu:~# kubectl get pods NAME READY STATUS RESTARTS AGE web-nginx-7bdc6b976b-7454h 1/1 Running 0 8s web-nginx-7bdc6b976b-cml8v 1/1 Running 0 8s web-nginx-7bdc6b976b-p7m2b 1/1 Running 0 8s root@ubuntu:~# cat web-anti-affinity.yaml apiVersion: apps/v1 kind: Deployment metadata: name: web-nginx spec: selector: matchLabels: app: web-nginx replicas: 3 template: metadata: labels: app: web-nginx spec: affinity: #pod 反亲和性, 打散 web-nginx 各个副本 podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - web-nginx topologyKey: "kubernetes.io/hostname" containers: - image: nginx imagePullPolicy: IfNotPresent name: web2-worker ports: - containerPort: 80 protocol: TCP root@ubuntu:~#
[root@bogon ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.10.16.254 0.0.0.0 UG 0 0 0 enahisic2i0 10.2.0.1 0.0.0.0 255.255.255.255 UH 0 0 0 v-cali-peer 10.10.16.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i0 10.10.34.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i2 10.10.102.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i1 10.244.2.0 10.10.16.47 255.255.255.0 UG 0 0 0 tunl0 10.244.29.0 0.0.0.0 255.255.255.192 U 0 0 0 * 10.244.29.1 0.0.0.0 255.255.255.255 UH 0 0 0 cali2e486421e22 10.244.29.4 0.0.0.0 255.255.255.255 UH 0 0 0 calibe3388252a1 10.244.41.0 10.10.16.47 255.255.255.192 UG 0 0 0 tunl0 10.244.243.192 10.10.16.82 255.255.255.192 UG 0 0 0 tunl0 14.14.18.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i3.310 172.16.100.0 0.0.0.0 255.255.255.0 U 0 0 0 brqf1411bad-10 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 172.168.104.0 0.0.0.0 255.255.255.0 U 0 0 0 enah2i3.1022 192.168.33.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i1 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
bogon pod执行ping
[root@bogon ~]# docker inspect 07b1320ef14c | grep -i pid "Pid": 57801, "PidMode": "", "PidsLimit": null, [root@bogon ~]# nsenter -n --target 57801 [root@bogon ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 4: eth0@if103: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether fa:34:76:a4:9d:f8 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.29.4/32 brd 10.244.29.4 scope global eth0 valid_lft forever preferred_lft forever [root@bogon ~]# ping 10.244.243.197 PING 10.244.243.197 (10.244.243.197) 56(84) bytes of data. 64 bytes from 10.244.243.197: icmp_seq=1 ttl=62 time=0.493 ms 64 bytes from 10.244.243.197: icmp_seq=2 ttl=62 time=0.219 ms 64 bytes from 10.244.243.197: icmp_seq=3 ttl=62 time=0.222 ms 64 bytes from 10.244.243.197: icmp_seq=4 ttl=62 time=0.194 ms 64 bytes from 10.244.243.197: icmp_seq=5 ttl=62 time=0.211 ms 64 bytes from 10.244.243.197: icmp_seq=6 ttl=62 time=0.234 ms 64 bytes from 10.244.243.197: icmp_seq=7 ttl=62 time=0.256 ms 64 bytes from 10.244.243.197: icmp_seq=8 ttl=62 time=0.218 ms 64 bytes from 10.244.243.197: icmp_seq=9 ttl=62 time=0.234 ms 64 bytes from 10.244.243.197: icmp_seq=10 ttl=62 time=0.304 ms 64 bytes from 10.244.243.197: icmp_seq=11 ttl=62 time=0.256 ms 64 bytes from 10.244.243.197: icmp_seq=12 ttl=62 time=0.230 ms 64 bytes from 10.244.243.197: icmp_seq=13 ttl=62 time=0.229 ms 64 bytes from 10.244.243.197: icmp_seq=14 ttl=62 time=0.196 ms 64 bytes from 10.244.243.197: icmp_seq=15 ttl=62 time=0.221 ms 64 bytes from 10.244.243.197: icmp_seq=16 ttl=62 time=0.234 ms
ubuntu抓包，ipip
root@ubuntu:~# tcpdump -i enahisic2i0 'ip proto 4' -env tcpdump: listening on enahisic2i0, link-type EN10MB (Ethernet), capture size 262144 bytes 10:27:46.524781 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 118: (tos 0x0, ttl 63, id 49710, offset 0, flags [DF], proto IPIP (4), length 104) 10.10.16.81 > 10.10.16.82: (tos 0x0, ttl 63, id 63551, offset 0, flags [DF], proto ICMP (1), length 84) 10.244.29.4 > 10.244.243.197: ICMP echo request, id 675, seq 39, length 64 10:27:46.524933 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 118: (tos 0x0, ttl 63, id 7959, offset 0, flags [none], proto IPIP (4), length 104) 10.10.16.82 > 10.10.16.81: (tos 0x0, ttl 63, id 33908, offset 0, flags [none], proto ICMP (1), length 84) 10.244.243.197 > 10.244.29.4: ICMP echo reply, id 675, seq 39, length 64 10:27:47.564786 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 118: (tos 0x0, ttl 63, id 49764, offset 0, flags [DF], proto IPIP (4), length 104) 10.10.16.81 > 10.10.16.82: (tos 0x0, ttl 63, id 63635, offset 0, flags [DF], proto ICMP (1), length 84) 10.244.29.4 > 10.244.243.197: ICMP echo request, id 675, seq 40, length 64 10:27:47.564899 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 118: (tos 0x0, ttl 63, id 8050, offset 0, flags [none], proto IPIP (4), length 104) 10.10.16.82 > 10.10.16.81: (tos 0x0, ttl 63, id 33952, offset 0, flags [none], proto ICMP (1), length 84) 10.244.243.197 > 10.244.29.4: ICMP echo reply, id 675, seq 40, length 64 10:27:48.604753 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 118: (tos 0x0, ttl 63, id 49853, offset 0, flags [DF], proto IPIP (4), length 104) 10.10.16.81 > 10.10.16.82: (tos 0x0, ttl 63, id 63699, offset 0, flags [DF], proto ICMP (1), length 84) 10.244.29.4 > 10.244.243.197: ICMP echo request, id 675, seq 41, length 64 10:27:48.604847 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 118: (tos 0x0, ttl 63, id 8082, offset 0, flags [none], proto IPIP (4), length 104) 10.10.16.82 > 10.10.16.81: (tos 0x0, ttl 63, id 34074, offset 0, flags [none], proto ICMP (1), length 84) 10.244.243.197 > 10.244.29.4: ICMP echo reply, id 675, seq 41, length 64 ^C 6 packets captured 6 packets received by filter 0 packets dropped by kernel root@ubuntu:~#
修改calico的网络模式IPIP变为BGP
需要修改calico的网络模式： 修改calico目录下面target目录下的calico.yaml加入以下内容（红色部分为修改的内容，上面IPIP方便定位）：
如果需要将calico网络模式从IPIP模式修改为bgp模式需要修改以下地方
1、添加BGp模式
修改calico的yaml配置文件，搜索IPIP，添加下面两行---METHOD（bgp模式），或者直接修改calico.yaml配置文件
[root@k8s-master1 target]# kubectl edit ds calico-node -n kube-system
- name: CALICO_IPV4POOL_IPIP
value: Always
- name: CALICO_AUTODETECTION_METHOD
value: interface=eth0
2、修改ippool，需要把ipipMode从Always修改成为Never（执行下面命令修改ipipMode）
[root@k8s-master1 target]# kubectl edit ippool
ipipMode: Never
修改之后
root@ubuntu:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.10.16.254 0.0.0.0 UG 0 0 0 enahisic2i0 10.10.16.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i0 10.244.2.0 10.10.16.47 255.255.255.0 UG 0 0 0 enahisic2i0 10.244.29.0 10.10.16.81 255.255.255.192 UG 0 0 0 enahisic2i0 10.244.41.0 10.10.16.47 255.255.255.192 UG 0 0 0 enahisic2i0 10.244.243.192 0.0.0.0 255.255.255.192 U 0 0 0 * 10.244.243.194 0.0.0.0 255.255.255.255 UH 0 0 0 cali0d27bc8b0f7 10.244.243.197 0.0.0.0 255.255.255.255 UH 0 0 0 cali090a0c3c4a2 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 root@ubuntu:~#
root@ubuntu:~# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-nginx-7bdc6b976b-7454h 1/1 Running 0 31m 10.244.29.4 bogon <none> <none> web-nginx-7bdc6b976b-cml8v 1/1 Running 0 31m 10.244.41.3 cloud <none> <none> web-nginx-7bdc6b976b-p7m2b 1/1 Running 0 31m 10.244.243.197 ubuntu <none> <none> root@ubuntu:~#
bogon路由
[root@bogon ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.10.16.254 0.0.0.0 UG 0 0 0 enahisic2i0 10.2.0.1 0.0.0.0 255.255.255.255 UH 0 0 0 v-cali-peer 10.10.16.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i0 10.10.34.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i2 10.10.102.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i1 10.244.2.0 10.10.16.47 255.255.255.0 UG 0 0 0 enahisic2i0 10.244.29.0 0.0.0.0 255.255.255.192 U 0 0 0 * 10.244.29.1 0.0.0.0 255.255.255.255 UH 0 0 0 cali2e486421e22 10.244.29.4 0.0.0.0 255.255.255.255 UH 0 0 0 calibe3388252a1 10.244.41.0 10.10.16.47 255.255.255.192 UG 0 0 0 enahisic2i0 10.244.243.192 10.10.16.82 255.255.255.192 UG 0 0 0 enahisic2i0 14.14.18.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i3.310 172.16.100.0 0.0.0.0 255.255.255.0 U 0 0 0 brqf1411bad-10 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 172.168.104.0 0.0.0.0 255.255.255.0 U 0 0 0 enah2i3.1022 192.168.33.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i1 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 You have mail in /var/spool/mail/root [root@bogon ~]#
mac 都一样
[root@bogon ~]# ip a sh cali2e486421e22 100: cali2e486421e22@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 6 inet6 fe80::ecee:eeff:feee:eeee/64 scope link valid_lft forever preferred_lft forever [root@bogon ~]# ip a sh calibe3388252a1 103: calibe3388252a1@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 7 inet6 fe80::ecee:eeff:feee:eeee/64 scope link valid_lft forever preferred_lft forever [root@bogon ~]#
bogon pod
[root@bogon ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 169.254.1.1 0.0.0.0 UG 0 0 0 eth0 169.254.1.1 0.0.0.0 255.255.255.255 UH 0 0 0 eth0 [root@bogon ~]# ip n 169.254.1.1 dev eth0 lladdr ee:ee:ee:ee:ee:ee STALE 10.10.16.81 dev eth0 lladdr ee:ee:ee:ee:ee:ee STALE [root@bogon ~]
bogon pod ping
[root@bogon ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 4: eth0@if103: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether fa:34:76:a4:9d:f8 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.29.4/32 brd 10.244.29.4 scope global eth0 valid_lft forever preferred_lft forever [root@bogon ~]# ping 10.244.243.197 PING 10.244.243.197 (10.244.243.197) 56(84) bytes of data. 64 bytes from 10.244.243.197: icmp_seq=1 ttl=62 time=0.307 ms 64 bytes from 10.244.243.197: icmp_seq=2 ttl=62 time=0.184 ms 64 bytes from 10.244.243.197: icmp_seq=3 ttl=62 time=0.225 ms 64 bytes from 10.244.243.197: icmp_seq=4 ttl=62 time=0.208 ms 64 bytes from 10.244.243.197: icmp_seq=5 ttl=62 time=0.188 ms 64 bytes from 10.244.243.197: icmp_seq=6 ttl=62 time=0.172 ms 64 bytes from 10.244.243.197: icmp_seq=7 ttl=62 time=0.188 ms 64 bytes from 10.244.243.197: icmp_seq=8 ttl=62 time=0.197 ms 64 bytes from 10.244.243.197: icmp_seq=9 ttl=62 time=0.186 ms 64 bytes from 10.244.243.197: icmp_seq=10 ttl=62 time=0.232 ms 64 bytes from 10.244.243.197: icmp_seq=11 ttl=62 time=0.172 ms 64 bytes from 10.244.243.197: icmp_seq=12 ttl=62 time=0.191 ms 64 bytes from 10.244.243.197: icmp_seq=13 ttl=62 time=0.148 ms 64 bytes from 10.244.243.197: icmp_seq=14 ttl=62 time=0.182 ms 64 bytes from 10.244.243.197: icmp_seq=15 ttl=62 time=0.171 ms 64 bytes from 10.244.243.197: icmp_seq=16 ttl=62 time=0.185 ms 64 bytes from 10.244.243.197: icmp_seq=17 ttl=62 time=0.193 ms 64 bytes from 10.244.243.197: icmp_seq=18 ttl=62 time=0.186 ms 64 bytes from 10.244.243.197: icmp_seq=19 ttl=62 time=0.168 ms 64 bytes from 10.244.243.197: icmp_seq=20 ttl=62 time=0.176 ms 64 bytes from 10.244.243.197: icmp_seq=21 ttl=62 time=0.182 ms 64 bytes from 10.244.243.197: icmp_seq=22 ttl=62 time=0.196 ms 64 bytes from 10.244.243.197: icmp_seq=23 ttl=62 time=0.172 ms 64 bytes from 10.244.243.197: icmp_seq=24 ttl=62 time=0.193 ms 64 bytes from 10.244.243.197: icmp_seq=25 ttl=62 time=0.183 ms 64 bytes from 10.244.243.197: icmp_seq=26 ttl=62 time=0.176 ms 64 bytes from 10.244.243.197: icmp_seq=27 ttl=62 time=0.176 ms 64 bytes from 10.244.243.197: icmp_seq=28 ttl=62 time=0.197 ms ^C --- 10.244.243.197 ping statistics --- 28 packets transmitted, 28 received, 0% packet loss, time 28104ms rtt min/avg/max/mdev = 0.148/0.190/0.307/0.031 ms [root@bogon ~]#
pod有ipip tunnel设备
[root@bogon ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 4: eth0@if103: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether fa:34:76:a4:9d:f8 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.29.4/32 brd 10.244.29.4 scope global eth0 valid_lft forever preferred_lft forever You have mail in /var/spool/mail/root [root@bogon ~]#
root@cloud:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 4: eth0@if44: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether f2:a1:42:ff:d4:86 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.41.3/32 brd 10.244.41.3 scope global eth0 valid_lft forever preferred_lft forever root@cloud:~#
ubuntu tcpdump
抓不到ipip报文
root@ubuntu:~# tcpdump -i enahisic2i0 'ip proto 4' -env tcpdump: listening on enahisic2i0, link-type EN10MB (Ethernet), capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel
root@ubuntu:~# tcpdump -i enahisic2i0 host 10.244.29.4 -env tcpdump: listening on enahisic2i0, link-type EN10MB (Ethernet), capture size 262144 bytes 10:57:52.764990 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 44888, offset 0, flags [DF], proto ICMP (1), length 84) 10.244.29.4 > 10.244.243.197: ICMP echo request, id 40823, seq 5, length 64 10:57:52.765062 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 21011, offset 0, flags [none], proto ICMP (1), length 84) 10.244.243.197 > 10.244.29.4: ICMP echo reply, id 40823, seq 5, length 64 10:57:53.804988 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 44952, offset 0, flags [DF], proto ICMP (1), length 84) 10.244.29.4 > 10.244.243.197: ICMP echo request, id 40823, seq 6, length 64 10:57:53.805037 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 21214, offset 0, flags [none], proto ICMP (1), length 84) 10.244.243.197 > 10.244.29.4: ICMP echo reply, id 40823, seq 6, length 64 10:57:54.844991 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 44997, offset 0, flags [DF], proto ICMP (1), length 84) 10.244.29.4 > 10.244.243.197: ICMP echo request, id 40823, seq 7, length 64 10:57:54.845063 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 21434, offset 0, flags [none], proto ICMP (1), length 84) 10.244.243.197 > 10.244.29.4: ICMP echo reply, id 40823, seq 7, length 64 10:57:55.884993 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 45032, offset 0, flags [DF], proto ICMP (1), length 84) 10.244.29.4 > 10.244.243.197: ICMP echo request, id 40823, seq 8, length 64 10:57:55.885069 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 21618, offset 0, flags [none], proto ICMP (1), length 84) 10.244.243.197 > 10.244.29.4: ICMP echo reply, id 40823, seq 8, length 64 ^C 8 packets captured 8 packets received by filter 0 packets dropped by kernel root@ubuntu:~#
创建nodeport
root@ubuntu:~# cat web-ngx-svc.yml apiVersion: v1 kind: Service metadata: name: nodeport-svc spec: type: NodePort selector: app: web-nginx ports: - protocol: TCP port: 3000 targetPort: 80 nodePort: 30090
root@ubuntu:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.10.16.254 0.0.0.0 UG 0 0 0 enahisic2i0 10.10.16.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i0 10.244.2.0 10.10.16.47 255.255.255.0 UG 0 0 0 enahisic2i0 10.244.29.0 10.10.16.81 255.255.255.192 UG 0 0 0 enahisic2i0 10.244.41.0 10.10.16.47 255.255.255.192 UG 0 0 0 enahisic2i0 10.244.243.192 0.0.0.0 255.255.255.192 U 0 0 0 * 10.244.243.194 0.0.0.0 255.255.255.255 UH 0 0 0 cali0d27bc8b0f7 10.244.243.197 0.0.0.0 255.255.255.255 UH 0 0 0 cali090a0c3c4a2 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 root@ubuntu:~# iptables -nvL -t nat | grep 10.244.243.197 root@ubuntu:~# iptables -nvL -t nat | grep 10.244.243.194 root@ubuntu:~# kubectl apply -f web-ngx-svc.yml service/nodeport-svc created root@ubuntu:~# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d15h nodeport-svc NodePort 10.102.82.74 <none> 3000:30090/TCP 11s root@ubuntu:~# iptables -nvL -t nat | grep 10.244.243.197 0 0 KUBE-MARK-MASQ all -- * * 10.244.243.197 0.0.0.0/0 /* default/nodeport-svc: */ root@ubuntu:~# iptables -nvL -t nat | grep 10.244.243.194 root@ubuntu:~# iptables -nvL -t nat | grep 10.244.243.192 root@ubuntu:~#
访问 http://10.10.16.82:30090
源ip是ubuntu的IP
bogon tcpdump
[root@bogon ~]# tcpdump -i enahisic2i0 tcp and port 80 -ennvv tcpdump: listening on enahisic2i0, link-type EN10MB (Ethernet), capture size 262144 bytes 11:17:46.996627 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 56, id 57490, offset 0, flags [DF], proto TCP (6), length 60) 10.10.16.82.6086 > 10.244.29.4.80: Flags [S], cksum 0xa6c9 (correct), seq 4143434673, win 64240, options [mss 1460,nop,wscale 8,sackOK,TS val 7702716 ecr 0], length 0 11:17:46.996694 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 56, id 57491, offset 0, flags [DF], proto TCP (6), length 60) 10.10.16.82.38685 > 10.244.29.4.80: Flags [S], cksum 0xc0f6 (correct), seq 4263914238, win 64240, options [mss 1460,nop,wscale 8,sackOK,TS val 7702716 ecr 0], length 0 11:17:46.996793 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.244.29.4.80 > 10.10.16.82.6086: Flags [S.], cksum 0x4282 (incorrect -> 0xb7e8), seq 2234675113, ack 4143434674, win 27760, options [mss 1400,sackOK,TS val 2216559763 ecr 7702716,nop,wscale 7], length 0 11:17:46.996806 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.244.29.4.80 > 10.10.16.82.38685: Flags [S.], cksum 0x4282 (incorrect -> 0x7ef9), seq 4064105914, ack 4263914239, win 27760, options [mss 1400,sackOK,TS val 2216559763 ecr 7702716,nop,wscale 7], length 0 11:17:46.998825 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 56, id 57493, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.82.38685 > 10.244.29.4.80: Flags [.], cksum 0x17f5 (correct), seq 1, ack 1, win 515, options [nop,nop,TS val 7702718 ecr 2216559763], length 0 11:17:46.998841 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 56, id 57492, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.82.6086 > 10.244.29.4.80: Flags [.], cksum 0x50e4 (correct), seq 1, ack 1, win 515, options [nop,nop,TS val 7702718 ecr 2216559763], length 0 11:17:46.999988 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 501: (tos 0x0, ttl 56, id 57494, offset 0, flags [DF], proto TCP (6), length 487) 10.10.16.82.6086 > 10.244.29.4.80: Flags [P.], cksum 0x7a47 (correct), seq 1:436, ack 1, win 515, options [nop,nop,TS val 7702719 ecr 2216559763], length 435: HTTP, length: 435 GET / HTTP/1.1 Host: 10.10.16.82:30090 Connection: keep-alive Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 Accept-Encoding: gzip, deflate Accept-Language: zh-CN,zh;q=0.9 11:17:47.000030 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 14284, offset 0, flags [DF], proto TCP (6), length 52) 10.244.29.4.80 > 10.10.16.82.6086: Flags [.], cksum 0x427a (incorrect -> 0x504e), seq 1, ack 436, win 226, options [nop,nop,TS val 2216559766 ecr 7702719], length 0 11:17:47.000278 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 304: (tos 0x0, ttl 63, id 14285, offset 0, flags [DF], proto TCP (6), length 290) 10.244.29.4.80 > 10.10.16.82.6086: Flags [P.], cksum 0x4368 (incorrect -> 0x7a9d), seq 1:239, ack 436, win 226, options [nop,nop,TS val 2216559766 ecr 7702719], length 238: HTTP, length: 238 HTTP/1.1 200 OK Server: nginx/1.21.0 Date: Tue, 22 Jun 2021 03:17:47 GMT Content-Type: text/html Content-Length: 612 Last-Modified: Tue, 25 May 2021 12:28:56 GMT Connection: keep-alive ETag: "60aced88-264" Accept-Ranges: bytes 11:17:47.000393 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 678: (tos 0x0, ttl 63, id 14286, offset 0, flags [DF], proto TCP (6), length 664) 10.244.29.4.80 > 10.10.16.82.6086: Flags [P.], cksum 0x44de (incorrect -> 0x9128), seq 239:851, ack 436, win 226, options [nop,nop,TS val 2216559766 ecr 7702719], length 612: HTTP 11:17:47.001835 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 56, id 57495, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.82.6086 > 10.244.29.4.80: Flags [.], cksum 0x4bdc (correct), seq 436, ack 851, win 511, options [nop,nop,TS val 7702722 ecr 2216559766], length 0 11:17:47.107262 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 447: (tos 0x0, ttl 56, id 57496, offset 0, flags [DF], proto TCP (6), length 433) 10.10.16.82.6086 > 10.244.29.4.80: Flags [P.], cksum 0x5496 (correct), seq 436:817, ack 851, win 511, options [nop,nop,TS val 7702826 ecr 2216559766], length 381: HTTP, length: 381 GET /favicon.ico HTTP/1.1 Host: 10.10.16.82:30090 Connection: keep-alive User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36 Accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8 Referer: http://10.10.16.82:30090/ Accept-Encoding: gzip, deflate Accept-Language: zh-CN,zh;q=0.9 11:17:47.107469 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 776: (tos 0x0, ttl 63, id 14287, offset 0, flags [DF], proto TCP (6), length 762) 10.244.29.4.80 > 10.10.16.82.6086: Flags [P.], cksum 0x4540 (incorrect -> 0x1ba5), seq 851:1561, ack 817, win 234, options [nop,nop,TS val 2216559874 ecr 7702826], length 710: HTTP, length: 710 HTTP/1.1 404 Not Found Server: nginx/1.21.0 Date: Tue, 22 Jun 2021 03:17:47 GMT Content-Type: text/html Content-Length: 555 Connection: keep-alive <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx/1.21.0</center> </body> </html> <!-- a padding to disable MSIE and Chrome friendly error page --> <!-- a padding to disable MSIE and Chrome friendly error page --> <!-- a padding to disable MSIE and Chrome friendly error page --> <!-- a padding to disable MSIE and Chrome friendly error page --> <!-- a padding to disable MSIE and Chrome friendly error page --> <!-- a padding to disable MSIE and Chrome friendly error page --> 11:17:47.150111 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 56, id 57497, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.82.6086 > 10.244.29.4.80: Flags [.], cksum 0x4695 (correct), seq 817, ack 1561, win 515, options [nop,nop,TS val 7702870 ecr 2216559874], length 0 11:18:32.013058 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 56, id 57529, offset 0, flags [DF], proto TCP (6), length 41) 10.10.16.82.38685 > 10.244.29.4.80: Flags [.], cksum 0x62f1 (correct), seq 0:1, ack 1, win 515, length 1: HTTP 11:18:32.013151 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 63, id 3549, offset 0, flags [DF], proto TCP (6), length 64) 10.244.29.4.80 > 10.10.16.82.38685: Flags [.], cksum 0x4286 (incorrect -> 0xe0e3), seq 1, ack 1, win 217, options [nop,nop,TS val 2216604779 ecr 7702718,nop,nop,sack 1 {0:1}], length 0 11:18:32.124090 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 56, id 57530, offset 0, flags [DF], proto TCP (6), length 41) 10.10.16.82.6086 > 10.244.29.4.80: Flags [.], cksum 0x9298 (correct), seq 816:817, ack 1561, win 515, length 1: HTTP 11:18:32.124208 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 63, id 14288, offset 0, flags [DF], proto TCP (6), length 64) 10.244.29.4.80 > 10.10.16.82.6086: Flags [.], cksum 0x4286 (incorrect -> 0xd609), seq 1561, ack 817, win 234, options [nop,nop,TS val 2216604890 ecr 7702870,nop,nop,sack 1 {816:817}], length 0 11:18:47.052567 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 3550, offset 0, flags [DF], proto TCP (6), length 52) 10.244.29.4.80 > 10.10.16.82.38685: Flags [F.], cksum 0x427a (incorrect -> 0x2e86), seq 1, ack 1, win 217, options [nop,nop,TS val 2216619818 ecr 7702718], length 0 11:18:47.053849 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 56, id 57531, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.82.38685 > 10.244.29.4.80: Flags [.], cksum 0x42c4 (correct), seq 1, ack 2, win 515, options [nop,nop,TS val 7762773 ecr 2216619818], length 0 11:18:52.107778 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 14289, offset 0, flags [DF], proto TCP (6), length 52) 10.244.29.4.80 > 10.10.16.82.6086: Flags [F.], cksum 0x427a (incorrect -> 0x49c5), seq 1561, ack 817, win 234, options [nop,nop,TS val 2216624873 ecr 7702870], length 0 11:18:52.109056 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 56, id 57532, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.82.6086 > 10.244.29.4.80: Flags [.], cksum 0x4aed (correct), seq 817, ack 1562, win 515, options [nop,nop,TS val 7767828 ecr 2216624873], length 0 11:19:32.058301 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 56, id 57535, offset 0, flags [DF], proto TCP (6), length 41) 10.10.16.82.38685 > 10.244.29.4.80: Flags [.], cksum 0x62f0 (correct), seq 0:1, ack 2, win 515, length 1: HTTP 11:19:32.058399 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 1020, offset 0, flags [DF], proto TCP (6), length 52) 10.244.29.4.80 > 10.10.16.82.38685: Flags [.], cksum 0x941f (correct), seq 2, ack 1, win 217, options [nop,nop,TS val 2216664824 ecr 7762773], length 0 11:19:37.123367 48:57:02:64:e7:ab > 48:57:02:64:ea:1b, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 56, id 57536, offset 0, flags [DF], proto TCP (6), length 41) 10.10.16.82.6086 > 10.244.29.4.80: Flags [.], cksum 0x9297 (correct), seq 816:817, ack 1562, win 515, length 1: HTTP 11:19:37.123552 48:57:02:64:ea:1b > 48:57:02:64:e7:ab, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 1319, offset 0, flags [DF], proto TCP (6), length 52) 10.244.29.4.80 > 10.10.16.82.6086: Flags [.], cksum 0x9c2d (correct), seq 1562, ack 817, win 234, options [nop,nop,TS val 2216669889 ecr 7767828], length 0
[root@bogon ~]# conntrack -L -o ktimestamp | grep 6086 tcp 6 86362 ESTABLISHED src=10.10.16.82 dst=10.244.29.4 sport=6086 dport=80 src=10.244.29.4 dst=10.10.16.82 sport=80 dport=6086 [ASSURED] mark=0 use=1 tcp 6 86390 ESTABLISHED src=10.10.16.81 dst=10.10.16.81 sport=60860 dport=3306 src=10.10.16.81 dst=10.10.16.81 sport=3306 dport=60860 [ASSURED] mark=0 use=1 conntrack v1.4.4 (conntrack-tools): 511 flow entries have been shown. [root@bogon ~]#
bogon pod tcpdump
[root@bogon ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 4: eth0@if103: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether fa:34:76:a4:9d:f8 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.29.4/32 brd 10.244.29.4 scope global eth0 valid_lft forever preferred_lft forever You have mail in /var/spool/mail/root [root@bogon ~]# tcpdump -i eth0 tcp and port 80 -ennvv tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 11:17:46.996722 ee:ee:ee:ee:ee:ee > fa:34:76:a4:9d:f8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 55, id 57490, offset 0, flags [DF], proto TCP (6), length 60) 10.10.16.82.6086 > 10.244.29.4.80: Flags [S], cksum 0xa6c9 (correct), seq 4143434673, win 64240, options [mss 1460,nop,wscale 8,sackOK,TS val 7702716 ecr 0], length 0 11:17:46.996768 fa:34:76:a4:9d:f8 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.244.29.4.80 > 10.10.16.82.6086: Flags [S.], cksum 0x4282 (incorrect -> 0xb7e8), seq 2234675113, ack 4143434674, win 27760, options [mss 1400,sackOK,TS val 2216559763 ecr 7702716,nop,wscale 7], length 0 11:17:46.996771 ee:ee:ee:ee:ee:ee > fa:34:76:a4:9d:f8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 55, id 57491, offset 0, flags [DF], proto TCP (6), length 60) 10.10.16.82.38685 > 10.244.29.4.80: Flags [S], cksum 0xc0f6 (correct), seq 4263914238, win 64240, options [mss 1460,nop,wscale 8,sackOK,TS val 7702716 ecr 0], length 0 11:17:46.996778 fa:34:76:a4:9d:f8 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.244.29.4.80 > 10.10.16.82.38685: Flags [S.], cksum 0x4282 (incorrect -> 0x7ef9), seq 4064105914, ack 4263914239, win 27760, options [mss 1400,sackOK,TS val 2216559763 ecr 7702716,nop,wscale 7], length 0 11:17:46.998855 ee:ee:ee:ee:ee:ee > fa:34:76:a4:9d:f8, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 55, id 57493, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.82.38685 > 10.244.29.4.80: Flags [.], cksum 0x17f5 (correct), seq 1, ack 1, win 515, options [nop,nop,TS val 7702718 ecr 2216559763], length 0 11:17:46.998895 ee:ee:ee:ee:ee:ee > fa:34:76:a4:9d:f8, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 55, id 57492, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.82.6086 > 10.244.29.4.80: Flags [.], cksum 0x50e4 (correct), seq 1, ack 1, win 515, options [nop,nop,TS val 7702718 ecr 2216559763], length 0 11:17:47.000006 ee:ee:ee:ee:ee:ee > fa:34:76:a4:9d:f8, ethertype IPv4 (0x0800), length 501: (tos 0x0, ttl 55, id 57494, offset 0, flags [DF], proto TCP (6), length 487) 10.10.16.82.6086 > 10.244.29.4.80: Flags [P.], cksum 0x7a47 (correct), seq 1:436, ack 1, win 515, options [nop,nop,TS val 7702719 ecr 2216559763], length 435: HTTP, length: 435 GET / HTTP/1.1 Host: 10.10.16.82:30090 Connection: keep-alive Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 Accept-Encoding: gzip, deflate Accept-Language: zh-CN,zh;q=0.9 11:17:47.000020 fa:34:76:a4:9d:f8 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 14284, offset 0, flags [DF], proto TCP (6), length 52) 10.244.29.4.80 > 10.10.16.82.6086: Flags [.], cksum 0x427a (incorrect -> 0x504e), seq 1, ack 436, win 226, options [nop,nop,TS val 2216559766 ecr 7702719], length 0 11:17:47.000244 fa:34:76:a4:9d:f8 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 304: (tos 0x0, ttl 64, id 14285, offset 0, flags [DF], proto TCP (6), length 290) 10.244.29.4.80 > 10.10.16.82.6086: Flags [P.], cksum 0x4368 (incorrect -> 0x7a9d), seq 1:239, ack 436, win 226, options [nop,nop,TS val 2216559766 ecr 7702719], length 238: HTTP, length: 238 HTTP/1.1 200 OK Server: nginx/1.21.0 Date: Tue, 22 Jun 2021 03:17:47 GMT Content-Type: text/html Content-Length: 612 Last-Modified: Tue, 25 May 2021 12:28:56 GMT Connection: keep-alive ETag: "60aced88-264" Accept-Ranges: bytes 11:17:47.000375 fa:34:76:a4:9d:f8 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 678: (tos 0x0, ttl 64, id 14286, offset 0, flags [DF], proto TCP (6), length 664) 10.244.29.4.80 > 10.10.16.82.6086: Flags [P.], cksum 0x44de (incorrect -> 0x9128), seq 239:851, ack 436, win 226, options [nop,nop,TS val 2216559766 ecr 7702719], length 612: HTTP 11:17:47.001855 ee:ee:ee:ee:ee:ee > fa:34:76:a4:9d:f8, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 55, id 57495, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.82.6086 > 10.244.29.4.80: Flags [.], cksum 0x4bdc (correct), seq 436, ack 851, win 511, options [nop,nop,TS val 7702722 ecr 2216559766], length 0 11:17:47.107281 ee:ee:ee:ee:ee:ee > fa:34:76:a4:9d:f8, ethertype IPv4 (0x0800), length 447: (tos 0x0, ttl 55, id 57496, offset 0, flags [DF], proto TCP (6), length 433) 10.10.16.82.6086 > 10.244.29.4.80: Flags [P.], cksum 0x5496 (correct), seq 436:817, ack 851, win 511, options [nop,nop,TS val 7702826 ecr 2216559766], length 381: HTTP, length: 381 GET /favicon.ico HTTP/1.1 Host: 10.10.16.82:30090 Connection: keep-alive User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36 Accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8 Referer: http://10.10.16.82:30090/ Accept-Encoding: gzip, deflate Accept-Language: zh-CN,zh;q=0.9 11:17:47.107438 fa:34:76:a4:9d:f8 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 776: (tos 0x0, ttl 64, id 14287, offset 0, flags [DF], proto TCP (6), length 762) 10.244.29.4.80 > 10.10.16.82.6086: Flags [P.], cksum 0x4540 (incorrect -> 0x1ba5), seq 851:1561, ack 817, win 234, options [nop,nop,TS val 2216559874 ecr 7702826], length 710: HTTP, length: 710 HTTP/1.1 404 Not Found Server: nginx/1.21.0 Date: Tue, 22 Jun 2021 03:17:47 GMT Content-Type: text/html Content-Length: 555 Connection: keep-alive <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx/1.21.0</center> </body> </html> <!-- a padding to disable MSIE and Chrome friendly error page --> <!-- a padding to disable MSIE and Chrome friendly error page --> <!-- a padding to disable MSIE and Chrome friendly error page --> <!-- a padding to disable MSIE and Chrome friendly error page --> <!-- a padding to disable MSIE and Chrome friendly error page --> <!-- a padding to disable MSIE and Chrome friendly error page --> 11:17:47.150133 ee:ee:ee:ee:ee:ee > fa:34:76:a4:9d:f8, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 55, id 57497, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.82.6086 > 10.244.29.4.80: Flags [.], cksum 0x4695 (correct), seq 817, ack 1561, win 515, options [nop,nop,TS val 7702870 ecr 2216559874], length 0 11:18:32.013111 ee:ee:ee:ee:ee:ee > fa:34:76:a4:9d:f8, ethertype IPv4 (0x0800), length 55: (tos 0x0, ttl 55, id 57529, offset 0, flags [DF], proto TCP (6), length 41) 10.10.16.82.38685 > 10.244.29.4.80: Flags [.], cksum 0x62f1 (correct), seq 0:1, ack 1, win 515, length 1: HTTP 11:18:32.013134 fa:34:76:a4:9d:f8 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 64, id 3549, offset 0, flags [DF], proto TCP (6), length 64) 10.244.29.4.80 > 10.10.16.82.38685: Flags [.], cksum 0x4286 (incorrect -> 0xe0e3), seq 1, ack 1, win 217, options [nop,nop,TS val 2216604779 ecr 7702718,nop,nop,sack 1 {0:1}], length 0 11:18:32.124177 ee:ee:ee:ee:ee:ee > fa:34:76:a4:9d:f8, ethertype IPv4 (0x0800), length 55: (tos 0x0, ttl 55, id 57530, offset 0, flags [DF], proto TCP (6), length 41) 10.10.16.82.6086 > 10.244.29.4.80: Flags [.], cksum 0x9298 (correct), seq 816:817, ack 1561, win 515, length 1: HTTP 11:18:32.124193 fa:34:76:a4:9d:f8 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 64, id 14288, offset 0, flags [DF], proto TCP (6), length 64) 10.244.29.4.80 > 10.10.16.82.6086: Flags [.], cksum 0x4286 (incorrect -> 0xd609), seq 1561, ack 817, win 234, options [nop,nop,TS val 2216604890 ecr 7702870,nop,nop,sack 1 {816:817}], length 0 11:18:47.052530 fa:34:76:a4:9d:f8 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 3550, offset 0, flags [DF], proto TCP (6), length 52) 10.244.29.4.80 > 10.10.16.82.38685: Flags [F.], cksum 0x427a (incorrect -> 0x2e86), seq 1, ack 1, win 217, options [nop,nop,TS val 2216619818 ecr 7702718], length 0 11:18:47.053992 ee:ee:ee:ee:ee:ee > fa:34:76:a4:9d:f8, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 55, id 57531, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.82.38685 > 10.244.29.4.80: Flags [.], cksum 0x42c4 (correct), seq 1, ack 2, win 515, options [nop,nop,TS val 7762773 ecr 2216619818], length 0 11:18:52.107733 fa:34:76:a4:9d:f8 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 14289, offset 0, flags [DF], proto TCP (6), length 52) 10.244.29.4.80 > 10.10.16.82.6086: Flags [F.], cksum 0x427a (incorrect -> 0x49c5), seq 1561, ack 817, win 234, options [nop,nop,TS val 2216624873 ecr 7702870], length 0 11:18:52.109100 ee:ee:ee:ee:ee:ee > fa:34:76:a4:9d:f8, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 55, id 57532, offset 0, flags [DF], proto TCP (6), length 52) 10.10.16.82.6086 > 10.244.29.4.80: Flags [.], cksum 0x4aed (correct), seq 817, ack 1562, win 515, options [nop,nop,TS val 7767828 ecr 2216624873], length 0 11:19:32.058348 ee:ee:ee:ee:ee:ee > fa:34:76:a4:9d:f8, ethertype IPv4 (0x0800), length 55: (tos 0x0, ttl 55, id 57535, offset 0, flags [DF], proto TCP (6), length 41) 10.10.16.82.38685 > 10.244.29.4.80: Flags [.], cksum 0x62f0 (correct), seq 0:1, ack 2, win 515, length 1: HTTP 11:19:32.058380 fa:34:76:a4:9d:f8 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 1020, offset 0, flags [DF], proto TCP (6), length 52) 10.244.29.4.80 > 10.10.16.82.38685: Flags [.], cksum 0x941f (correct), seq 2, ack 1, win 217, options [nop,nop,TS val 2216664824 ecr 7762773], length 0 11:19:37.123509 ee:ee:ee:ee:ee:ee > fa:34:76:a4:9d:f8, ethertype IPv4 (0x0800), length 55: (tos 0x0, ttl 55, id 57536, offset 0, flags [DF], proto TCP (6), length 41) 10.10.16.82.6086 > 10.244.29.4.80: Flags [.], cksum 0x9297 (correct), seq 816:817, ack 1562, win 515, length 1: HTTP 11:19:37.123531 fa:34:76:a4:9d:f8 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 1319, offset 0, flags [DF], proto TCP (6), length 52) 10.244.29.4.80 > 10.10.16.82.6086: Flags [.], cksum 0x9c2d (correct), seq 1562, ack 817, win 234, options [nop,nop,TS val 2216669889 ecr 7767828], length 0 11:20:17.060789 ee:ee:ee:ee:ee:ee > fa:34:76:a4:9d:f8, ethertype IPv4 (0x0800), length 55: (tos 0x0, ttl 55, id 57538, offset 0, flags [DF], proto TCP (6), length 41) 10.10.16.82.38685 > 10.244.29.4.80: Flags [.], cksum 0x62f0 (correct), seq 0:1, ack 2, win 515, length 1: HTTP 11:20:17.060822 fa:34:76:a4:9d:f8 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 64, id 3695, offset 0, flags [DF], proto TCP (6), length 40) 10.244.29.4.80 > 10.10.16.82.38685: Flags [R], cksum 0x8e25 (correct), seq 4064105916, win 0, length 0 11:20:22.132633 ee:ee:ee:ee:ee:ee > fa:34:76:a4:9d:f8, ethertype IPv4 (0x0800), length 55: (tos 0x0, ttl 55, id 57540, offset 0, flags [DF], proto TCP (6), length 41) 10.10.16.82.6086 > 10.244.29.4.80: Flags [.], cksum 0x9297 (correct), seq 816:817, ack 1562, win 515, length 1: HTTP 11:20:22.132660 fa:34:76:a4:9d:f8 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 64, id 3840, offset 0, flags [DF], proto TCP (6), length 40) 10.244.29.4.80 > 10.10.16.82.6086: Flags [R], cksum 0x5a81 (correct), seq 2234676675, win 0, length 0
ubuntu
root@ubuntu:~# conntrack -L -o ktimestamp | grep 6086 conntrack v1.4.4 (conntrack-tools): 156 flow entries have been shown. tcp 6 86369 ESTABLISHED src=192.168.117.51 dst=10.10.16.82 sport=49623 dport=30090 src=10.244.29.4 dst=10.10.16.82 sport=80 dport=6086 [ASSURED] mark=0 use=1 root@ubuntu:~#
root@ubuntu:~# iptables -nvL -t nat | grep 10.16.82 0 0 KUBE-MARK-MASQ all -- * * 10.10.16.82 0.0.0.0/0 /* default/kubernetes:https */ root@ubuntu:~# iptables -nvL -t nat | grep 30090 2 120 KUBE-MARK-MASQ tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nodeport-svc: */ tcp dpt:30090 2 120 KUBE-SVC-GFPAJ7EGCNM4QF4H tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nodeport-svc: */ tcp dpt:30090 root@ubuntu:~# iptables -S -t nat | grep 30090 -A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport-svc:" -m tcp --dport 30090 -j KUBE-MARK-MASQ -A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport-svc:" -m tcp --dport 30090 -j KUBE-SVC-GFPAJ7EGCNM4QF4H root@ubuntu:~# iptables -S -t nat | grep KUBE-SVC-GFPAJ7EGCNM4QF4H -N KUBE-SVC-GFPAJ7EGCNM4QF4H -A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport-svc:" -m tcp --dport 30090 -j KUBE-SVC-GFPAJ7EGCNM4QF4H -A KUBE-SERVICES -d 10.102.82.74/32 -p tcp -m comment --comment "default/nodeport-svc: cluster IP" -m tcp --dport 3000 -j KUBE-SVC-GFPAJ7EGCNM4QF4H -A KUBE-SVC-GFPAJ7EGCNM4QF4H -m comment --comment "default/nodeport-svc:" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-I7IUICMFVGBZ4YHL -A KUBE-SVC-GFPAJ7EGCNM4QF4H -m comment --comment "default/nodeport-svc:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-5Y55J3IFI6JAIDAZ -A KUBE-SVC-GFPAJ7EGCNM4QF4H -m comment --comment "default/nodeport-svc:" -j KUBE-SEP-ABYWEQSNIXV3R64X
root@ubuntu:~# iptables -S -t nat | grep KUBE-SEP-ABYWEQSNIXV3R64X
-N KUBE-SEP-ABYWEQSNIXV3R64X
-A KUBE-SEP-ABYWEQSNIXV3R64X -s 10.244.41.3/32 -m comment --comment "default/nodeport-svc:" -j KUBE-MARK-MASQ
-A KUBE-SEP-ABYWEQSNIXV3R64X -p tcp -m comment --comment "default/nodeport-svc:" -m tcp -j DNAT [unsupported revision]
-A KUBE-SVC-GFPAJ7EGCNM4QF4H -m comment --comment "default/nodeport-svc:" -j KUBE-SEP-ABYWEQSNIXV3R64X
root@ubuntu:~# iptables -S -t nat | grep KUBE-SEP-I7IUICMFVGBZ4YHL -N KUBE-SEP-I7IUICMFVGBZ4YHL -A KUBE-SEP-I7IUICMFVGBZ4YHL -s 10.244.243.197/32 -m comment --comment "default/nodeport-svc:" -j KUBE-MARK-MASQ -A KUBE-SEP-I7IUICMFVGBZ4YHL -p tcp -m comment --comment "default/nodeport-svc:" -m tcp -j DNAT [unsupported revision] -A KUBE-SVC-GFPAJ7EGCNM4QF4H -m comment --comment "default/nodeport-svc:" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-I7IUICMFVGBZ4YHL root@ubuntu:~# iptables -S -t nat | grep KUBE-MARK-MASQ -N KUBE-MARK-MASQ -A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000 -A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport-svc:" -m tcp --dport 30090 -j KUBE-MARK-MASQ -A KUBE-SEP-4HSYGM5AX4RJ3SMI -s 10.10.16.82/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ -A KUBE-SEP-5Y55J3IFI6JAIDAZ -s 10.244.29.4/32 -m comment --comment "default/nodeport-svc:" -j KUBE-MARK-MASQ -A KUBE-SEP-75I6QPBZP6NPFMYC -s 10.244.29.1/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ -A KUBE-SEP-ABYWEQSNIXV3R64X -s 10.244.41.3/32 -m comment --comment "default/nodeport-svc:" -j KUBE-MARK-MASQ -A KUBE-SEP-I7IUICMFVGBZ4YHL -s 10.244.243.197/32 -m comment --comment "default/nodeport-svc:" -j KUBE-MARK-MASQ -A KUBE-SEP-IME5NDJ5NB2CIBBH -s 10.244.29.1/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ -A KUBE-SEP-PN4XDM7ZVTVWXEFK -s 10.244.29.1/32 -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-MARK-MASQ -A KUBE-SEP-XHDQHYDYIRFCR54C -s 10.244.41.1/32 -m comment --comment "kube-system/kube-dns:metrics" -j KUBE-MARK-MASQ -A KUBE-SEP-XPMB4DMGYR2YRIBM -s 10.244.41.1/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ -A KUBE-SEP-XW34Y4IZ5SRQMLRR -s 10.244.41.1/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ -A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ -A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-MARK-MASQ -A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ -A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.102.82.74/32 -p tcp -m comment --comment "default/nodeport-svc: cluster IP" -m tcp --dport 3000 -j KUBE-MARK-MASQ -A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ root@ubuntu:~# iptables -S -t nat | grep KUBE-MARK-MASQ | grep snat root@ubuntu:~# iptables -S -t nat | grep 4000 | grep snat root@ubuntu:~# iptables -S -t nat | grep MASQUERADE -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535 -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535 -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE -A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE -A cali-nat-outgoing -m comment --comment "cali:flqWnvo8yq4ULQLa" -m set --match-set cali40masq-ipam-pools src -m set ! --match-set cali40all-ipam-pools dst -j MASQUERADE root@ubuntu:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.10.16.254 0.0.0.0 UG 0 0 0 enahisic2i0 10.10.16.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i0 10.244.2.0 10.10.16.47 255.255.255.0 UG 0 0 0 enahisic2i0 10.244.29.0 10.10.16.81 255.255.255.192 UG 0 0 0 enahisic2i0 10.244.41.0 10.10.16.47 255.255.255.192 UG 0 0 0 enahisic2i0 10.244.243.192 0.0.0.0 255.255.255.192 U 0 0 0 * 10.244.243.194 0.0.0.0 255.255.255.255 UH 0 0 0 cali0d27bc8b0f7 10.244.243.197 0.0.0.0 255.255.255.255 UH 0 0 0 cali090a0c3c4a2 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 root@ubuntu:~#
BGP两种模式
-
全互联模式(node-to-node mesh)
全互联模式，每一个BGP Speaker都需要和其他BGP Speaker建立BGP连接，这样BGP连接总数就是N^2，如果数量过大会消耗大量连接。如果集群数量超过100台官方不建议使用此种模式。 -
路由反射模式Router Reflection（RR）
RR模式中会指定一个或多个BGP Speaker为RouterReflection，它与网络中其他Speaker建立连接，每个Speaker只要与Router Reflection建立BGP就可以获得全网的路由信息。在calico中可以通过Global Peer实现RR模式。
三个节点之间都建立了互连的关系，也就是一个节点要和其他节点建立TCP连接，这个就是BGP之间互联通信，要是节点好多的话连接就会好多
这个就是BGP client里面有个进程叫bird，这个bird就是负责BGP协议的通信完成路由表学习，随着连接的增多，下面这两个也会增多，使用的端口是179。
root@cloud:~# ./calicoctl node status Calico process is running. IPv4 BGP status +--------------+-------------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-------------------+-------+----------+-------------+ | 10.10.16.81 | node-to-node mesh | up | 02:49:55 | Established | | 10.10.16.82 | node-to-node mesh | up | 02:50:08 | Established | +--------------+-------------------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. root@cloud:~#
root@cloud:~# ss -antp | grep ESTAB | grep bird ESTAB 0 0 10.10.16.47:179 10.10.16.82:56069 users:(("bird",pid=570702,fd=9)) ESTAB 0 0 10.10.16.47:179 10.10.16.81:60407 users:(("bird",pid=570702,fd=8))
root@cloud:~# ./calicoctl get bgpconfig NAME LOGSEVERITY MESHENABLED ASNUMBER root@cloud:~#
root@cloud:~# ./calicoctl get node -o wide NAME ASN IPV4 IPV6 bogon (64512) 10.10.16.81/24 cloud (64512) 10.10.16.47/24 ubuntu (64512) 10.10.16.82/24 root@cloud:~#
1、关闭 node-to-node模式
关闭node to node模式就代表网络不通了，当完整切换了，网络就可以使用了
root@ubuntu:~# ./calicoctl apply -f bgpconfig.yaml Successfully applied 1 'BGPConfiguration' resource(s) root@ubuntu:~# ./calicoctl apply -f bgpconfig.yaml Successfully applied 1 'BGPConfiguration' resource(s) root@ubuntu:~# ./calicoctl get bgpconfig NAME LOGSEVERITY MESHENABLED ASNUMBER default Info false 64512 root@ubuntu:~# cat bgpconfig.yaml apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: logSeverityScreen: Info nodeToNodeMeshEnabled: false asNumber: 64512 root@ubuntu:~#
路由减少了
root@ubuntu:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.10.16.254 0.0.0.0 UG 0 0 0 enahisic2i0 10.10.16.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i0 10.244.243.192 0.0.0.0 255.255.255.192 U 0 0 0 * 10.244.243.194 0.0.0.0 255.255.255.255 UH 0 0 0 cali0d27bc8b0f7 10.244.243.197 0.0.0.0 255.255.255.255 UH 0 0 0 cali090a0c3c4a2 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 root@ubuntu:~#
无法ping 通了
[root@bogon ~]# ping 10.244.243.197 PING 10.244.243.197 (10.244.243.197) 56(84) bytes of data. ^C --- 10.244.243.197 ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 1058ms [root@bogon ~]#
root@ubuntu:~# ./calicoctl node status Calico process is running. IPv4 BGP status No IPv4 peers found. IPv6 BGP status No IPv6 peers found. root@ubuntu:~
2、配置指定节点充当路由反射器
从当前节点找两个节点充当路由反射器，作为路由反射器
为方便让BGPPeer轻松选择节点，通过标签选择器匹配。给路由器反射器节点打标签：
给选定的节点指定路由反射器ID
root@ubuntu:~# ./calicoctl get node ubuntu -o yaml > rr-node.yaml root@ubuntu:~# cat rr-node.yaml apiVersion: projectcalico.org/v3 kind: Node metadata: annotations: projectcalico.org/kube-labels: '{"beta.kubernetes.io/arch":"arm64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"arm64","kubernetes.io/hostname":"ubuntu","kubernetes.io/os":"linux","node-role.kubernetes.io/master":""}' creationTimestamp: "2021-06-18T11:12:32Z" labels: beta.kubernetes.io/arch: arm64 beta.kubernetes.io/os: linux kubernetes.io/arch: arm64 kubernetes.io/hostname: ubuntu kubernetes.io/os: linux node-role.kubernetes.io/master: "" name: ubuntu resourceVersion: "797539" uid: 369ae2bb-756d-446d-a044-9225d9849a13 spec: addresses: - address: 10.10.16.82/24 type: CalicoNodeIP - address: 10.10.16.82 type: InternalIP bgp: ipv4Address: 10.10.16.82/24 orchRefs: - nodeName: ubuntu orchestrator: k8s status: podCIDRs: - 10.244.0.0/24 root@ubuntu:~#
增加 routeReflectorClusterID: 244.0.0.1
root@ubuntu:~# ./calicoctl apply -f rr-node.yaml Successfully applied 1 'Node' resource(s) root@ubuntu:~# cat rr-node.yaml apiVersion: projectcalico.org/v3 kind: Node metadata: annotations: projectcalico.org/kube-labels: '{"beta.kubernetes.io/arch":"arm64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"arm64","kubernetes.io/hostname":"ubuntu","kubernetes.io/os":"linux","node-role.kubernetes.io/master":""}' creationTimestamp: "2021-06-18T11:12:32Z" labels: beta.kubernetes.io/arch: arm64 beta.kubernetes.io/os: linux kubernetes.io/arch: arm64 kubernetes.io/hostname: ubuntu kubernetes.io/os: linux node-role.kubernetes.io/master: "" name: ubuntu resourceVersion: "797539" uid: 369ae2bb-756d-446d-a044-9225d9849a13 spec: addresses: - address: 10.10.16.82/24 type: CalicoNodeIP - address: 10.10.16.82 type: InternalIP bgp: ipv4Address: 10.10.16.82/24 routeReflectorClusterID: 244.0.0.1 orchRefs: - nodeName: ubuntu orchestrator: k8s status: podCIDRs: - 10.244.0.0/24 root@ubuntu:~# ./calicoctl get bgpconfig NAME LOGSEVERITY MESHENABLED ASNUMBER default Info false 64512 root@ubuntu:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.10.16.254 0.0.0.0 UG 0 0 0 enahisic2i0 10.10.16.0 0.0.0.0 255.255.255.0 U 0 0 0 enahisic2i0 10.244.243.192 0.0.0.0 255.255.255.192 U 0 0 0 * 10.244.243.194 0.0.0.0 255.255.255.255 UH 0 0 0 cali0d27bc8b0f7 10.244.243.197 0.0.0.0 255.255.255.255 UH 0 0 0 cali090a0c3c4a2 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 root@ubuntu:~#
3、使用标签选择器将路由反射器节点与其他非路由反射器节点配置为对等
匹配所有节点带有这个标签的作为路由反射器，这就是非路由反射器和路由反射器建立关系
root@ubuntu:~# ./calicoctl apply -f bgppeer.yaml Successfully applied 1 'BGPPeer' resource(s) root@ubuntu:~# ./calicoctl node status Calico process is running. IPv4 BGP status No IPv4 peers found. IPv6 BGP status No IPv6 peers found. root@ubuntu:~# ./calicoctl node status Calico process is running. IPv4 BGP status No IPv4 peers found. IPv6 BGP status No IPv6 peers found. root@ubuntu:~# cat bgppeer.yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: peer-with-route-reflectors spec: nodeSelector: all() peerSelector: route-reflector == 'true'root@ubuntu:~# root@ubuntu:~# ./calicoctl node status Calico process is running. IPv4 BGP status No IPv4 peers found. IPv6 BGP status No IPv6 peers found. root@ubuntu:~#
root@ubuntu:~# ./calicoctl get bgppeer NAME PEERIP NODE ASN peer-with-route-reflectors all() 0 root@ubuntu:~#
root@ubuntu:~# vi bgppeer.yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: ubuntu spec: nodeSelector: all() peerSelector: route-reflector == 'true'
ot@ubuntu:~# ./calicoctl node status Calico process is running. IPv4 BGP status No IPv4 peers found. IPv6 BGP status No IPv6 peers found. root@ubuntu:~#
4、增加节点
apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: peer-with-route-reflectors spec: peerIP: 10.10.16.47 asNumber: 64512
root@ubuntu:~# ./calicoctl apply -f bgp-peer.yaml Successfully applied 1 'BGPPeer' resource(s) root@ubuntu:~# ./calicoctl node status Calico process is running. IPv4 BGP status +--------------+-----------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-----------+-------+----------+-------------+ | 10.10.16.47 | global | up | 06:55:18 | Established | +--------------+-----------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. root@ubuntu:~# cat bgp-peer.yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: peer-with-route-reflectors spec: peerIP: 10.10.16.47 asNumber: 64512root@ubuntu:~# root@ubuntu:~#
root@ubuntu:~# cat bgp-peer.yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: peer-with-route-reflectors spec: peerIP: 10.10.16.81 asNumber: 64512 root@ubuntu:~#
覆盖了
root@ubuntu:~# ./calicoctl apply -f bgp-peer.yaml Successfully applied 1 'BGPPeer' resource(s) root@ubuntu:~# ./calicoctl node status Calico process is running. IPv4 BGP status +--------------+-----------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-----------+-------+----------+-------------+ | 10.10.16.81 | global | up | 06:59:21 | Established | +--------------+-----------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. root@ubuntu:~# ./calicoctl get bgppeer NAME PEERIP NODE ASN peer-with-route-reflectors 10.10.16.81 (global) 64512 ubuntu all() 0 root@ubuntu:~#
root@ubuntu:~# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-nginx-7bdc6b976b-7454h 1/1 Running 0 4h41m 10.244.29.4 bogon <none> <none> web-nginx-7bdc6b976b-cml8v 1/1 Running 0 4h41m 10.244.41.3 cloud <none> <none> web-nginx-7bdc6b976b-p7m2b 1/1 Running 0 4h41m 10.244.243.197 ubuntu <none> <none> root@ubuntu:~#
bogon pod 可以访问ubuntu pod
[root@bogon ~]# ping 10.244.243.197 PING 10.244.243.197 (10.244.243.197) 56(84) bytes of data. 64 bytes from 10.244.243.197: icmp_seq=1 ttl=62 time=0.270 ms 64 bytes from 10.244.243.197: icmp_seq=2 ttl=62 time=0.174 ms 64 bytes from 10.244.243.197: icmp_seq=3 ttl=62 time=0.176 ms ^C --- 10.244.243.197 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2085ms rtt min/avg/max/mdev = 0.174/0.206/0.270/0.047 ms You have mail in /var/spool/mail/root [root@bogon ~]#
bogon pod 可以访问cloud pod
[root@bogon ~]# ping 10.244.41.3 PING 10.244.41.3 (10.244.41.3) 56(84) bytes of data. 64 bytes from 10.244.41.3: icmp_seq=1 ttl=62 time=0.307 ms 64 bytes from 10.244.41.3: icmp_seq=2 ttl=62 time=0.118 ms 64 bytes from 10.244.41.3: icmp_seq=3 ttl=62 time=0.224 ms 64 bytes from 10.244.41.3: icmp_seq=4 ttl=62 time=0.201 ms ^C --- 10.244.41.3 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3094ms rtt min/avg/max/mdev = 0.118/0.212/0.307/0.068 ms [root@bogon ~]#
cloud pod 不可以访问ubuntu pod
root@cloud:~# nsenter -n --target 533998 root@cloud:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 4: eth0@if44: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether f2:a1:42:ff:d4:86 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.41.3/32 brd 10.244.41.3 scope global eth0 valid_lft forever preferred_lft forever root@cloud:~# tcpdump -i eth0 tcp and port 80 -ennvv tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes ^C 0 packets captured 0 packets received by filter 0 packets dropped by kernel root@cloud:~# ping 10.244.243.197 PING 10.244.243.197 (10.244.243.197) 56(84) bytes of data. ^C --- 10.244.243.197 ping statistics --- 7 packets transmitted, 0 received, 100% packet loss, time 6136ms root@cloud:~#
root@ubuntu:~# ./calicoctl node status Calico process is running. IPv4 BGP status +--------------+-----------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-----------+-------+----------+-------------+ | 10.10.16.81 | global | up | 06:59:21 | Established | +--------------+-----------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. root@ubuntu:~#
root@cloud:~# ./calicoctl node status Calico process is running. IPv4 BGP status +--------------+-----------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-----------+-------+----------+-------------+ | 10.10.16.81 | global | up | 06:59:21 | Established | +--------------+-----------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. root@cloud:~#
[root@bogon ~]# ./calicoctl node status Calico process is running. IPv4 BGP status +--------------+---------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+---------------+-------+----------+-------------+ | 10.10.16.47 | node specific | up | 06:59:21 | Established | | 10.10.16.82 | node specific | up | 06:59:21 | Established | +--------------+---------------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. You have mail in /var/spool/mail/root [root@bogon ~]#
删掉
root@ubuntu:~# ./calicoctl delete -f bgppeer.yaml Successfully deleted 1 'BGPPeer' resource(s) root@ubuntu:~# ./calicoctl get bgppeer NAME PEERIP NODE ASN peer-with-route-reflectors 10.10.16.81 (global) 64512 root@ubuntu:~#
把81也删掉
root@ubuntu:~# ./calicoctl delete -f bgp-peer.yaml Successfully deleted 1 'BGPPeer' resource(s) root@ubuntu:~# ./calicoctl node status Calico process is running. IPv4 BGP status No IPv4 peers found. IPv6 BGP status No IPv6 peers found. root@ubuntu:~# ./calicoctl get bgppeer NAME PEERIP NODE ASN root@ubuntu:~# ./calicoctl node status Calico process is running. IPv4 BGP status No IPv4 peers found. IPv6 BGP status No IPv6 peers found.
root@ubuntu:~# ./calicoctl get bgppeer NAME PEERIP NODE ASN root@ubuntu:~#
global:
注意bgp对等体作用域范围，也可以理解为使用
node specific: 生效与特定节点global: 全局配置，使用该配置时不能配置node和nodeSelector字段，这个配置将集群中所有的节点都建立bgp关系node-to-node mesh：全互联模式
root@ubuntu:~# ./calicoctl apply -f bgp-peer-81.yaml Successfully applied 1 'BGPPeer' resource(s) root@ubuntu:~# ./calicoctl apply -f bgp-peer-82.yaml Successfully applied 1 'BGPPeer' resource(s) root@ubuntu:~# ./calicoctl get bgppeer NAME PEERIP NODE ASN peer2 10.10.16.82 (global) 64512 peer81 10.10.16.81 (global) 64512 root@ubuntu:~# ./calicoctl node status Calico process is running. IPv4 BGP status +--------------+---------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+---------------+-------+----------+-------------+ | 10.10.16.81 | global | up | 07:26:09 | Established | | 10.10.16.47 | node specific | up | 07:26:11 | Established | +--------------+---------------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. root@ubuntu:~#
root@ubuntu:~# cat bgp-peer-81.yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: peer81 spec: peerIP: 10.10.16.81 asNumber: 64512 root@ubuntu:~# cat bgp-peer-82.yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: peer2 spec: peerIP: 10.10.16.82 asNumber: 64512 root@ubuntu:~#
root@cloud:~# ./calicoctl node status Calico process is running. IPv4 BGP status +--------------+-----------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+-----------+-------+----------+-------------+ | 10.10.16.81 | global | up | 07:26:07 | Established | | 10.10.16.82 | global | up | 07:26:13 | Established | +--------------+-----------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. root@cloud:~#
[root@bogon ~]# ./calicoctl node status Calico process is running. IPv4 BGP status +--------------+---------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +--------------+---------------+-------+----------+-------------+ | 10.10.16.47 | node specific | up | 07:26:06 | Established | | 10.10.16.82 | global | up | 07:26:10 | Established | +--------------+---------------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. You have mail in /var/spool/mail/root [root@bogon ~]#
bogon pod ping
[root@bogon ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 4: eth0@if103: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether fa:34:76:a4:9d:f8 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.29.4/32 brd 10.244.29.4 scope global eth0 valid_lft forever preferred_lft forever [root@bogon ~]# ping 10.244.29.4 PING 10.244.29.4 (10.244.29.4) 56(84) bytes of data. 64 bytes from 10.244.29.4: icmp_seq=1 ttl=64 time=0.036 ms 64 bytes from 10.244.29.4: icmp_seq=2 ttl=64 time=0.022 ms 64 bytes from 10.244.29.4: icmp_seq=3 ttl=64 time=0.042 ms ^C --- 10.244.29.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2043ms rtt min/avg/max/mdev = 0.022/0.033/0.042/0.009 ms [root@bogon ~]# ping 10.244.243.197 PING 10.244.243.197 (10.244.243.197) 56(84) bytes of data. 64 bytes from 10.244.243.197: icmp_seq=1 ttl=62 time=0.298 ms 64 bytes from 10.244.243.197: icmp_seq=2 ttl=62 time=0.175 ms ^C --- 10.244.243.197 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1048ms rtt min/avg/max/mdev = 0.175/0.236/0.298/0.063 ms You have mail in /var/spool/mail/root [root@bogon ~]#
root@cloud:~# ping 10.244.243.197 PING 10.244.243.197 (10.244.243.197) 56(84) bytes of data. 64 bytes from 10.244.243.197: icmp_seq=1 ttl=62 time=0.259 ms 64 bytes from 10.244.243.197: icmp_seq=2 ttl=62 time=0.129 ms ^C --- 10.244.243.197 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 0.129/0.194/0.259/0.065 ms root@cloud:~# ping 10.244.29.4 PING 10.244.29.4 (10.244.29.4) 56(84) bytes of data. 64 bytes from 10.244.29.4: icmp_seq=1 ttl=62 time=0.266 ms ]64 bytes from 10.244.29.4: icmp_seq=2 ttl=62 time=0.206 ms ^C --- 10.244.29.4 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1018ms rtt min/avg/max/mdev = 0.206/0.236/0.266/0.030 ms root@cloud:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 4: eth0@if44: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default link/ether f2:a1:42:ff:d4:86 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.244.41.3/32 brd 10.244.41.3 scope global eth0 valid_lft forever preferred_lft forever root@cloud:~#
都可以相互访问了
Calico BGP跨网段(大型网络)
当节点位于不同的网络段时，我们需要在交换机或路由器上开启BGP协议，并配置BGPPeer将peerIP设置为路由器或交换机IP，我们需要做如下操作。