Vxlan模式
flannel的vxlan模式，数据包是通过kernal来转发的，而当使用udp模式时候是flannel自己转发。UDP模式下性能要差很多，性能差别可参考一些测试：
UDP模式：
60秒单线程测试
iperf3 -c x.x.x.x -t 60
结果：
[ 4] 0.00-60.00 sec 3.33 GBytes 477 Mbits/sec 15716 sender
[ 4] 0.00-60.00 sec 3.33 GBytes 476 Mbits/sec receiver开启30个线程传输
iperf3 -c x.x.x.x -t 60 -P 30
结果：
[SUM] 0.00-60.00 sec 3.08 GBytes 441 Mbits/sec 205692 sender
[SUM] 0.00-60.00 sec 3.06 GBytes 438 Mbits/sec receiver
多线程并没有带宽的提升反而由于多线程调度影响，降低带宽。在万兆网的环境下UDP的性能损耗还是挺大的。Vxlan模式：
60秒单线程测试
iperf3 -c x.x.x.x -t 60
结果：
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 10.1 GBytes 1.44 Gbits/sec 6957 sender
[ 4] 0.00-60.00 sec 10.1 GBytes 1.44 Gbits/sec receiver开启30个线程传输
iperf3 -c x.x.x.x -t 60 -P 30
结果：
[SUM] 0.00-60.00 sec 10.1 GBytes 1.45 Gbits/sec 90559 sender
[SUM] 0.00-60.00 sec 10.1 GBytes 1.44 Gbits/sec receiver
在vxlan下多线程也没有性能提升。从上述两组数据可以明显看出flannel在vxlan下性能要明显高于udp。
从UDP模式配置转换为VXLAN模式
flannel及安装参考：flannel-UDP一文 ,如果新装，方法一致，只是在etcd中配置config key时候配置的value不一样。如果是从udp转换到vxlan，参考本实验。
在所有节点上停止docker及flanneld服务
|
1 2 |
systemctl stop flanneld systemctl stop docker |
删除etcd中已经建立的子网配置和修改etcd中的config key
|
1 2 3 |
etcdctl rm --recursive /flannel/network/subnets etcdctl set /flannel/network/config '{"Network": "10.2.0.0/16", "Backend": {"Type": "vxlan"}}' |
重新启动所有节点的flanneld和docker，注意要先启flanneld
|
1 2 |
systemctl start flanneld systemctl start docker.service |
检查确认
|
1 2 3 4 |
[root@docker1 ~]# docker exec busybox-1 ping 10.2.101.2 PING 10.2.101.2 (10.2.101.2): 56 data bytes 64 bytes from 10.2.101.2: seq=0 ttl=62 time=0.701 ms 64 bytes from 10.2.101.2: seq=1 ttl=62 time=1.699 ms |
|
1 2 3 4 5 6 7 8 |
[root@docker1 ~]# arp -a ? (10.2.101.0) at da:12:3c:1e:af:8c [ether] on flannel.1 ? (172.16.199.37) at 00:0c:29:7d:f5:be [ether] on eno50332184 ? (192.168.188.254) at 00:50:56:f7:07:54 [ether] on eno33554960 ? (172.16.199.27) at 00:0c:29:ae:11:8d [ether] on eno50332184 ? (10.2.79.2) at 02:42:0a:02:4f:02 [ether] on docker0 ? (10.2.101.2) at da:12:3c:1e:af:8c [ether] on flannel.1 gateway (192.168.188.2) at 00:50:56:ee:c0:ec [ether] on eno33554960 |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 |
[root@docker1 ~]# ifconfig br-2aeddaf04b59: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.19.0.1 netmask 255.255.0.0 broadcast 0.0.0.0 inet6 fe80::42:39ff:fee4:447d prefixlen 64 scopeid 0x20<link> ether 02:42:39:e4:44:7d txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450 inet 10.2.79.1 netmask 255.255.255.0 broadcast 0.0.0.0 inet6 fe80::42:9cff:fe6c:62ef prefixlen 64 scopeid 0x20<link> ether 02:42:9c:6c:62:ef txqueuelen 0 (Ethernet) RX packets 5141 bytes 426300 (416.3 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 5152 bytes 499216 (487.5 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eno16777736: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.0.228 netmask 255.255.255.0 broadcast 192.168.0.255 inet6 fe80::20c:29ff:fe42:d98 prefixlen 64 scopeid 0x20<link> ether 00:0c:29:42:0d:98 txqueuelen 1000 (Ethernet) RX packets 1913 bytes 324392 (316.7 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1039 bytes 321158 (313.6 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eno33554960: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.188.131 netmask 255.255.255.0 broadcast 192.168.188.255 inet6 fe80::20c:29ff:fe42:da2 prefixlen 64 scopeid 0x20<link> ether 00:0c:29:42:0d:a2 txqueuelen 1000 (Ethernet) RX packets 167 bytes 15449 (15.0 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 194 bytes 16671 (16.2 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eno50332184: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.16.199.17 netmask 255.255.255.0 broadcast 172.16.199.255 inet6 fe80::20c:29ff:fe42:dac prefixlen 64 scopeid 0x20<link> ether 00:0c:29:42:0d:ac txqueuelen 1000 (Ethernet) RX packets 181893 bytes 18071701 (17.2 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 181427 bytes 17875322 (17.0 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450 inet 10.2.79.0 netmask 255.255.255.255 broadcast 0.0.0.0 ether ba:1e:bf:2f:cb:2f txqueuelen 0 (Ethernet) RX packets 2400 bytes 199584 (194.9 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 2508 bytes 202608 (197.8 KiB) TX errors 0 dropped 1 overruns 0 carrier 0 collisions 0 gre_sys: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 65490 inet6 fe80::28da:26ff:fe25:9f69 prefixlen 64 scopeid 0x20<link> ether 2a:da:26:25:9f:69 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 8 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1 (Local Loopback) RX packets 4129 bytes 238096 (232.5 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 4129 bytes 238096 (232.5 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 veth49e5bf6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::acae:b3ff:fe70:3b0b prefixlen 64 scopeid 0x20<link> ether ae:ae:b3:70:3b:0b txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 vethe91eedb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450 inet6 fe80::7cbf:1cff:fe16:7e3c prefixlen 64 scopeid 0x20<link> ether 7e:bf:1c:16:7e:3c txqueuelen 0 (Ethernet) RX packets 2414 bytes 233772 (228.2 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 2422 bytes 234420 (228.9 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 |
VXLAN封包：
|
1 2 3 4 5 6 7 8 9 10 11 12 |
Frame 1: 148 bytes on wire (1184 bits), 148 bytes captured (1184 bits) Ethernet II, Src: Vmware_42:0d:ac (00:0c:29:42:0d:ac), Dst: Vmware_7d:f5:be (00:0c:29:7d:f5:be) Internet Protocol Version 4, Src: 172.16.199.17, Dst: 172.16.199.37 User Datagram Protocol, Src Port: 47830, Dst Port: 8472 Virtual eXtensible Local Area Network Flags: 0x0800, VXLAN Network ID (VNI) Group Policy ID: 0 VXLAN Network Identifier (VNI): 1 Reserved: 0 Ethernet II, Src: ba:1e:bf:2f:cb:2f (ba:1e:bf:2f:cb:2f), Dst: da:12:3c:1e:af:8c (da:12:3c:1e:af:8c) Internet Protocol Version 4, Src: 10.2.79.0, Dst: 10.2.101.2 Internet Control Message Protocol |
文章评论