Calico是k8s诸多网络解决方案中比较优秀的一种，使用起来比较容易，且性能也很优秀，支持IPIP的隧道模式，也支持基于路由的全三层方案，特别是三层方案由于不需要额外的隧道开销，使其性能优秀，路由方案只支持BGP路由，这可能多少限制了一些其使用场景，特别是如果企业现有路由方案不是BGP的话，涉及到比较复杂的路由设计与分发考虑，但是由于k8s平台大部分情况下会是独立建设，因此对于数据中心运维人员来说，单纯的路由问题将会是比较熟悉的领域，因此建设难度并不会过于复杂。
F5是应用交付领域的领导者，凭借其优秀的3-7层控制性使得F5非常适合对应用进行发布和控制，典型的表现在应用路由控制、应用协议控制、灰度发布等。 F5通过在k8s环境下运行控制容器实现与k8s等容器方案的集成，可以替换或增强k8s现有的业务服务发布方法，例如替换Ingress Controller或将cluster方式的service发布到F5 BIGIP上，充分发挥F5服务接入的高级功能，如应用安全、复杂协议控制、应用级监控检查、限流、应用可视化等，对比Nginx有多达20-30种场景可以更好的让k8s适应企业复杂应用场景。
以下测试模拟Calico BGP路由协议与F5动态路由实现F5直接互通容器pod网络，实现F5对K8S服务的直接发布。
两台k8s主机，启用calico BGP路由方案。 k8s主机在同一个2层网络，F5也在同一个2层，模拟基于以太网模型的calico BGP方案。
1.Calico关闭ipip模式，改为使用BGP路由模式。
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 |
[root@k8s-master calico]# calicoctl get bgpPeer -o yaml apiVersion: projectcalico.org/v3 items: - apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: creationTimestamp: 2019-01-04T15:36:59Z name: bgppeer-global-f5-1 resourceVersion: "9530940" uid: 92ecf52a-1036-11e9-bf61-005056b34685 spec: asNumber: 64512 peerIP: 172.16.40.202 kind: BGPPeerList metadata: resourceVersion: "9531851" [root@k8s-master calico]# calicoctl get node -o yaml apiVersion: projectcalico.org/v3 items: - apiVersion: projectcalico.org/v3 kind: Node metadata: creationTimestamp: 2018-08-05T14:42:30Z name: k8s-master resourceVersion: "9531878" uid: c7b4efba-98bd-11e8-aeed-000c29850765 spec: bgp: ipv4Address: 172.16.40.199/24 ipv4IPIPTunnelAddr: 10.244.0.1 - apiVersion: projectcalico.org/v3 kind: Node metadata: creationTimestamp: 2018-08-05T17:10:45Z name: k8s-node1 resourceVersion: "9531882" uid: 7dff767b-98d2-11e8-aeed-000c29850765 spec: bgp: ipv4Address: 172.16.40.198/24 ipv4IPIPTunnelAddr: 10.244.1 [root@k8s-master calico]# calicoctl get ipPool -o yaml apiVersion: projectcalico.org/v3 items: - apiVersion: projectcalico.org/v3 kind: IPPool metadata: creationTimestamp: 2019-01-04T12:05:32Z name: default-ipv4-ippool resourceVersion: "9524594" uid: 08ffa4b6-1019-11e9-9cec-005056b34685 spec: blockSize: 26 cidr: 10.244.0.0/16 ipipMode: Never natOutgoing: true kind: IPPoolList metadata: resourceVersion: "9531814" |
2.配置calico 全局 bgp peer
|
1 2 3 4 5 6 7 8 9 10 |
[root@k8s-master calico]# cat global-bgp-peer.yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: bgppeer-global-f5-1 spec: peerIP: 172.16.40.202 asNumber: 64512 [root@k8s-master calico]# calicoctl apply -f global-bgp-peer.yaml |
3.F5上配置：
1). 在route domain下启动 BGP
2). 进入路由模块做最简配置：
|
1 2 3 4 5 6 7 8 |
[myf5@v13-common:Active:Not All Devices Synced] config # imish v13-common.lab.f5se.io[0]>en v13-common.lab.f5se.io[0]#configure terminal Enter configuration commands, one per line. End with CNTL/Z. v13-common.lab.f5se.io[0](config)#router bgp 64512 v13-common.lab.f5se.io[0](config-router)#neighbor 172.16.40.199 remote-as 64512 v13-common.lab.f5se.io[0](config-router)#neighbor 172.16.40.198 remote-as 64512 v13-common.lab.f5se.io[0](config-router)#redistribute kernel |
3.1 bgp邻居建立后
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 |
v13-common.lab.f5se.io[0]#show ip bgp neighbors BGP neighbor is 172.16.40.198, remote AS 64512, local AS 64512, internal link BGP version 4, remote router ID 172.16.40.198 BGP state = Established, up for 00:22:23 Last read 00:22:23, hold time is 90, keepalive interval is 30 seconds Neighbor capabilities: Route refresh: advertised and received (new) Address family IPv4 Unicast: advertised and received Received 78 messages, 2 notifications, 0 in queue Sent 78 messages, 0 notifications, 0 in queue Route refresh request: received 0, sent 0 Minimum time between advertisement runs is 5 seconds For address family: IPv4 Unicast BGP table version 11, neighbor version 11 Index 1, Offset 0, Mask 0x2 AF-dependant capabilities: Graceful restart: advertised, received Community attribute sent to this neighbor (both) 1 accepted prefixes 2 announced prefixes Connections established 3; dropped 2 Graceful-restart Status: Remote restart-time is 120 sec Local host: 172.16.40.202, Local port: 36730 Foreign host: 172.16.40.198, Foreign port: 179 Nexthop: 172.16.40.202 Nexthop global: fe80::250:56ff:feb3:cb7c Nexthop local: :: BGP connection: non shared network Last Reset: 00:23:43, due to BGP Notification received Notification Error Message: (Cease/Peer Unconfigured.) BGP neighbor is 172.16.40.199, remote AS 64512, local AS 64512, internal link BGP version 4, remote router ID 172.16.40.199 BGP state = Established, up for 00:22:15 Last read 00:22:15, hold time is 90, keepalive interval is 30 seconds Neighbor capabilities: Route refresh: advertised and received (new) Address family IPv4 Unicast: advertised and received Received 77 messages, 2 notifications, 0 in queue Sent 78 messages, 0 notifications, 0 in queue Route refresh request: received 0, sent 0 Minimum time between advertisement runs is 5 seconds For address family: IPv4 Unicast BGP table version 11, neighbor version 11 Index 2, Offset 0, Mask 0x4 AF-dependant capabilities: Graceful restart: advertised, received Community attribute sent to this neighbor (both) 1 accepted prefixes 2 announced prefixes Connections established 3; dropped 2 Graceful-restart Status: Remote restart-time is 120 sec Local host: 172.16.40.202, Local port: 35752 Foreign host: 172.16.40.199, Foreign port: 179 Nexthop: 172.16.40.202 Nexthop global: fe80::250:56ff:feb3:cb7c Nexthop local: :: BGP connection: non shared network Last Reset: 00:23:43, due to BGP Notification received Notification Error Message: (Cease/Peer Unconfigured.) |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
v13-common.lab.f5se.io[0]#sh ip route Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default C 10.244.0.0/16 is directly connected, flannel_vxlan B 10.244.0.0/24 [200/0] via 172.16.40.199, int_vlan, 00:22:53 B 10.244.1.0/24 [200/0] via 172.16.40.198, int_vlan, 00:23:01 C 127.0.0.1/32 is directly connected, lo C 127.1.1.254/32 is directly connected, tmm C 172.16.10.0/24 is directly connected, bigip_cc_vlan C 172.16.30.0/24 is directly connected, ext_vlan C 172.16.40.0/24 is directly connected, int_vlan K 172.16.40.81/32 [0/0] is directly connected, tmm K 172.16.99.99/32 [0/0] is directly connected, tmm |
可以看到F5上已经有到pod的BGP路由了
3.2 如果尝试将F5 VIP 注入到BGP的话，可以在k8s节点上看到VIP的路由
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
[root@k8s-master calico]# ip route default via 172.16.10.254 dev ens33 proto static metric 100 default via 172.16.40.254 dev ens160 proto dhcp metric 101 blackhole 10.244.0.0/24 proto bird 10.244.0.76 dev cali7adc62d51c0 scope link 10.244.0.77 dev cali0761ccbeace scope link 10.244.0.78 dev calicd88c4c95c8 scope link 10.244.0.79 dev cali3b3d3e2702c scope link 10.244.0.80 dev calif1c9cee6be2 scope link 10.244.0.81 dev calie89075bc69f scope link 10.244.1.0/24 via 172.16.40.198 dev ens160 proto bird 172.16.0.0/16 dev ens33 proto kernel scope link src 172.16.10.201 metric 100 172.16.40.0/24 dev ens160 proto kernel scope link src 172.16.40.199 metric 101 172.16.40.81 via 172.16.40.202 dev ens160 proto bird <<<<<<! 172.16.99.99 via 172.16.40.202 dev ens160 proto bird <<<<<<<! 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 |
4. 做一些ping测试进行验证：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
K8S节点ping VS IP [root@k8s-master calico]# ping 172.16.99.99 PING 172.16.99.99 (172.16.99.99) 56(84) bytes of data. 64 bytes from 172.16.99.99: icmp_seq=1 ttl=255 time=0.190 ms 64 bytes from 172.16.99.99: icmp_seq=2 ttl=255 time=0.224 ms 到一个容器里ping VSIP [root@k8s-master calico]# kubectl exec -it busybox2-54b7b7cd58-gn8pf sh / # ip route default via 169.254.1.1 dev eth0 169.254.1.1 dev eth0 scope link / # ping 172.16.99.99 PING 172.16.99.99 (172.16.99.99): 56 data bytes 64 bytes from 172.16.99.99: seq=0 ttl=254 time=1.543 ms 64 bytes from 172.16.99.99: seq=1 ttl=254 time=0.253 ms ^C F5上ping pod的IP [myf5@v13-common:Active:Not All Devices Synced] config # ping 10.244.1.37 PING 10.244.1.37 (10.244.1.37) 56(84) bytes of data. 64 bytes from 10.244.1.37: icmp_seq=1 ttl=63 time=0.797 ms 64 bytes from 10.244.1.37: icmp_seq=2 ttl=63 time=0.429 ms ^C --- 10.244.1.37 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1101ms rtt min/avg/max/mdev = 0.429/0.613/0.797/0.184 ms [myf5@v13-common:Active:Not All Devices Synced] config # ping 10.244.0.81 PING 10.244.0.81 (10.244.0.81) 56(84) bytes of data. 64 bytes from 10.244.0.81: icmp_seq=1 ttl=63 time=0.566 ms 64 bytes from 10.244.0.81: icmp_seq=2 ttl=63 time=0.523 ms |
5. 部署F5 bigip ctlr 容器，以及发布一个configmap 资源
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 |
[root@k8s-master f5-k8s]# cat bigip-ctlr-deployment-calico.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: k8s-bigip-ctlr-deployment namespace: kube-system spec: # DO NOT INCREASE REPLICA COUNT replicas: 1 template: metadata: name: k8s-bigip-ctlr-calico labels: app: k8s-bigip-ctlr-calico spec: # Name of the Service Account bound to a Cluster Role with the required # permissions serviceAccountName: bigip-ctlr containers: - name: k8s-bigip-ctlr-calico image: "myf5/k8s-bigip-ctlr:1.7.1" env: - name: BIGIP_USERNAME valueFrom: secretKeyRef: # Replace with the name of the Secret containing your login # credentials name: bigip-login key: username - name: BIGIP_PASSWORD valueFrom: secretKeyRef: # Replace with the name of the Secret containing your login # credentials name: bigip-login key: password - name: TZ value: Asia/Shanghai command: ["/app/bin/k8s-bigip-ctlr"] args: [ # See the k8s-bigip-ctlr documentation for information about # all config options # https://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/latest "--bigip-username=$(BIGIP_USERNAME)", "--bigip-password=$(BIGIP_PASSWORD)", "--bigip-url=172.16.20.202", "--bigip-partition=k8s", "--namespace=default", "--pool-member-type=cluster", ] imagePullSecrets: # Secret containing the BIG-IP system login credentials - name: bigip-login [root@k8s-master f5-k8s]# kubectl create -f bigip-ctlr-deployment-calico.yaml [root@k8s-master f5-k8s]# kubectl create -f f5-vs.yaml configmap "nginx-deploy-svc.vs" created [root@k8s-master f5-k8s]# cat f5-vs.yaml kind: ConfigMap apiVersion: v1 metadata: name: nginx-deploy-svc.vs labels: f5type: virtual-server data: # See the f5-schema table for schema-controller compatibility # https://clouddocs.f5.com/containers/latest/releases_and_versioning.html#f5-schema schema: "f5schemadb://bigip-virtual-server_v0.1.7.json" data: | { "virtualServer": { "backend": { "servicePort": 80, "serviceName": "nginx-deploy-svc", "healthMonitors": [{ "interval": 15, "protocol": "http", "send": "GET / HTTP/1.1\r\nConnection: close\r\nHost: 1.1.1.1\r\n\r\n", "recv": "cka", "timeout": 30 }] }, "frontend": { "virtualAddress": { "port": 80, "bindAddr": "172.16.40.80" }, "partition": "k8s", "balance": "least-connections-member", "mode": "http" } } } [root@k8s-master f5-k8s]# kubectl create -f f5-vs.yaml configmap "nginx-deploy-svc.vs" created |
6. 其它输出：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 |
[root@k8s-master f5-k8s]# calicoctl node status Calico process is running. IPv4 BGP status +---------------+-------------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +---------------+-------------------+-------+----------+-------------+ | 172.16.40.198 | node-to-node mesh | up | 14:02:04 | Established | | 172.16.40.202 | global | up | 15:37:55 | Established | +---------------+-------------------+-------+----------+-------------+ IPv6 BGP status No IPv6 peers found. Bird里的BGP配置 / # cat /etc/calico/confd/config/bird.cfg # Generated by confd include "bird_aggr.cfg"; include "bird_ipam.cfg"; router id 172.16.40.199; # Configure synchronization between routing tables and kernel. protocol kernel { learn; # Learn all alien routes from the kernel persist; # Don't remove routes on bird shutdown scan time 2; # Scan kernel routing table every 2 seconds import all; export filter calico_kernel_programming; # Default is export none graceful restart; # Turn on graceful restart to reduce potential flaps in # routes when reloading BIRD configuration. With a full # automatic mesh, there is no way to prevent BGP from # flapping since multiple nodes update their BGP # configuration at the same time, GR is not guaranteed to # work correctly in this scenario. } # Watch interface up/down events. protocol device { debug { states }; scan time 2; # Scan interfaces every 2 seconds } protocol direct { debug { states }; interface -"cali*", "*"; # Exclude cali* but include everything else. } # Template for all BGP clients template bgp bgp_template { debug { states }; description "Connection to BGP peer"; local as 64512; multihop; gateway recursive; # This should be the default, but just in case. import all; # Import all routes, since we don't know what the upstream # topology is and therefore have to trust the ToR/RR. export filter calico_export_to_bgp_peers; # Only want to export routes for workloads. source address 172.16.40.199; # The local address we use for the TCP connection add paths on; graceful restart; # See comment in kernel section about graceful restart. connect delay time 2; connect retry time 5; error wait time 5,30; } # ------------- Node-to-node mesh ------------- # For peer /host/k8s-master/ip_addr_v4 # Skipping ourselves (172.16.40.199) # For peer /host/k8s-node1/ip_addr_v4 protocol bgp Mesh_172_16_40_198 from bgp_template { neighbor 172.16.40.198 as 64512; } # ------------- Global peers ------------- # For peer /global/peer_v4/172.16.40.202 protocol bgp Global_172_16_40_202 from bgp_template { neighbor 172.16.40.202 as 64512; } # ------------- Node-specific peers ------------- # No node-specific peers configured. |
对于F5在k8s上的网络模型对接选型：
•如果一定要考虑隧道模型，那么选flannel vxlan （canal也可以），vxlan模型F5上需要注入太多静态arp条目，当容器实例很多的时候这是一个需要考虑的问题
•如果集群规模很小，flannel host-gw可选，F5上使用静态路由指向各个pod网络
•Calico的BGP路由模型适合较大的k8s集群，网络上对接更容易，F5路由模块可直接互通pod网络。 同时还可以继续保留Calico的对networ policy支持属性，可以实现较好的容器访问控制，加上F5的分区配置隔离，可以比较方便的实现租户隔离。 关于租户隔离方面文章请参考博客中其它文章。
•且双机时候不用考虑 flannel vxlan模型下备机tunnel不建立问题。
•Calico由于是动态路由，pod本身也可以访问F5注入的vip, 这样很适合让容器访问其它外部的资源。且calico在路由模式下性能很好，推荐优先考虑calico
文章评论