路由方式互联跨宿主机容器:
linux在容器docker0网路与host的网卡网咯之间是可以执行ip forwarding的,因此跨主机的容器网络可以通过路由的方式来实现彼此互通,如下结构:
只需要在host1 添加到达172.27.0.0/16网络的路由,在host2上添加172.17.0.0/16的路由即可。
需要注意的前提:
各个host上的容器docker0网络不能相同,也就是说各个host里的容器的ip地址不能存在网络重叠情况,而缺省情况下,docker在建立docker0的时候都默认使用172.17.0.0/16网络,因此有必要对docker这一默认行为进行修改,实验环境版本下的修改方法如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
[root@docker2 system]# pwd /usr/lib/systemd/system [root@docker2 system]# vi docker.service [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target firewalld.service Wants=network-online.target [Service] Type=notify # the default is not to use systemd for cgroups because the delegate issues still # exists and systemd currently does not support the cgroup feature set required # for containers run by docker ExecStart=/usr/bin/dockerd --bip 172.27.0.1/16 ExecReload=/bin/kill -s HUP $MAINPID # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity # Uncomment TasksMax if your systemd version supports it. # Only systemd 226 and above support this version. #TasksMax=infinity TimeoutStartSec=0 # set delegate yes so that systemd does not reset the cgroups of docker containers Delegate=yes # kill only the docker process, not all processes in the cgroup KillMode=process # restart the docker process if it exits prematurely Restart=on-failure StartLimitBurst=3 StartLimitInterval=60s [Install] WantedBy=multi-user.target |
修改缺省的 docker服务启动文件,在 execstart中增加 --bip参数来指定docker0网桥地址从而实现对应docker网络的修改。注意:修改此配置文件后,需要在ifconfig输出里没有docker0网桥或者docker0网桥的IP和你修改的配置文件里一致才可以正常启动服务。
修改完毕后,启动服务:
1 2 |
systemctl daemon-reload systemctl start docker |
检查docker0设置:
1 2 3 4 5 6 7 8 9 |
[root@docker2 system]# ifconfig docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.27.0.1 netmask 255.255.0.0 broadcast 0.0.0.0 inet6 fe80::42:24ff:fe2c:e4c7 prefixlen 64 scopeid 0x20<link> ether 02:42:24:2c:e4:c7 txqueuelen 0 (Ethernet) RX packets 551 bytes 45332 (44.2 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 27 bytes 1582 (1.5 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 |
host2增加路由;
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
[root@docker2 system]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.0.1 0.0.0.0 UG 100 0 0 eno16777736 172.16.199.0 0.0.0.0 255.255.255.0 U 100 0 0 eno50332184 172.27.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 192.168.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eno16777736 [root@docker2 system]# route add -net 172.17.0.0/16 gw 172.16.199.17 [root@docker2 system]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.0.1 0.0.0.0 UG 100 0 0 eno16777736 172.16.199.0 0.0.0.0 255.255.255.0 U 100 0 0 eno50332184 172.17.0.0 172.16.199.17 255.255.0.0 UG 0 0 0 eno50332184 172.27.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 192.168.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eno16777736 |
host1增加路由:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
[root@docker1 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.0.1 0.0.0.0 UG 100 0 0 eno16777736 172.16.199.0 0.0.0.0 255.255.255.0 U 100 0 0 eno50332184 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 172.19.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-2aeddaf04b59 192.168.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eno16777736 [root@docker1 ~]# route add -net 172.27.0.0/16 gw 172.16.199.27 [root@docker1 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.0.1 0.0.0.0 UG 100 0 0 eno16777736 172.16.199.0 0.0.0.0 255.255.255.0 U 100 0 0 eno50332184 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 172.19.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-2aeddaf04b59 172.27.0.0 172.16.199.27 255.255.0.0 UG 0 0 0 eno50332184 192.168.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eno16777736 |
测试:
host1上的容器里ping host2上的容器IP:
1 2 3 4 5 6 |
[root@docker1 ~]# docker exec busybox-11 ping 172.27.0.2 Error response from daemon: No such container: busybox-11 [root@docker1 ~]# docker exec busybox-1 ping 172.27.0.2 PING 172.27.0.2 (172.27.0.2): 56 data bytes 64 bytes from 172.27.0.2: seq=0 ttl=62 time=0.721 ms 64 bytes from 172.27.0.2: seq=1 ttl=62 time=0.948 ms |
可以正常ping 通。测试成功。
手工配置的问题及Quagga自动化方案
但是在一个生产环境中,会有很多主机,手工维护这样一个静态路由的添加修改必然是一件很恐怖的事情,因此有了这样一个工具可以帮助自动完成这些工作: http://www.quagga.net
需要在系统里运行这样一个软件路由器系统来实现彼此学习各个host里的docker0网络。为了简单化,可以将该软件容器化,然后以特权模式(--privileged)启动该容器,便可以自动彼此学习路由。网上灵雀云提供的那个容器image地址已经失效,可以手工从这里下载该镜像 https://pan.baidu.com/s/1sj26X8T
下载完毕后,将tar文件上传到所有host机器上,并执行命令:
1 2 3 |
[root@docker1 tmp]# ls abrt route.tar [root@docker1 tmp]# docker load -i route.tar |
tar文件将被load为image
1 2 3 4 5 6 |
[root@docker1 tmp]# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE nginx latest 46102226f2fd 8 weeks ago 109MB busybox latest 00f017a8c2a6 3 months ago 1.11MB hello-world latest 48b5124b2768 5 months ago 1.84kB index.alauda.cn/georce/router latest cc9f1eb39091 2 years ago 139MB |
在启动该容器前,检查一下当前路由表条目:
1 2 3 4 5 6 7 8 |
[root@docker1 tmp]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.0.1 0.0.0.0 UG 100 0 0 eno16777736 172.16.199.0 0.0.0.0 255.255.255.0 U 100 0 0 eno50332184 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 172.19.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-2aeddaf04b59 192.168.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eno16777736 |
所有host上关闭系统防火墙 systemctl stop firewalld.service
所有host上启动quagga容器:
1 |
docker run -itd --name router --privileged --net=host index.alauda.cn/georce/router |
稍等片刻,再次检查路由表,发现系统路由表已增加一条到达172.27.0.0/16的路由
1 2 3 4 5 6 7 8 9 |
[root@docker1 tmp]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.0.1 0.0.0.0 UG 100 0 0 eno16777736 172.16.199.0 0.0.0.0 255.255.255.0 U 100 0 0 eno50332184 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 172.19.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-2aeddaf04b59 172.27.0.0 192.168.0.183 255.255.0.0 UG 20 0 0 eno16777736 192.168.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eno16777736 |
再次执行ping,可以成功ping通:
1 2 3 4 |
[root@docker1 tmp]# docker exec busybox-1 ping 172.27.0.2 PING 172.27.0.2 (172.27.0.2): 56 data bytes 64 bytes from 172.27.0.2: seq=0 ttl=62 time=0.678 ms 64 bytes from 172.27.0.2: seq=1 ttl=62 time=0.907 ms |
如果此时连接到quagga, telnet localhost 2601: (密码zebra)可以看到当前路由表,docker容器启用了ospf和rip两个动态路由协议
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
Router# show ip route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - ISIS, B - BGP, > - selected route, * - FIB route K>* 0.0.0.0/0 via 192.168.0.1, eno16777736 C>* 127.0.0.0/8 is directly connected, lo O 172.16.199.0/24 [110/10] is directly connected, eno50332184, 00:25:21 C>* 172.16.199.0/24 is directly connected, eno50332184 O 172.17.0.0/16 [110/10] is directly connected, docker0, 00:25:21 K * 172.17.0.0/16 is directly connected, docker0 C>* 172.17.0.0/16 is directly connected, docker0 O 172.19.0.0/16 [110/10] is directly connected, br-2aeddaf04b59, 00:25:21 C>* 172.19.0.0/16 is directly connected, br-2aeddaf04b59 O>* 172.27.0.0/16 [110/20] via 192.168.0.183, eno16777736, 00:13:35 * via 172.16.199.27, eno50332184, 00:13:35 R 172.27.0.0/16 [120/2] via 192.168.0.183, eno16777736, 00:13:38 O 192.168.0.0/24 [110/10] is directly connected, eno16777736, 00:25:21 C>* 192.168.0.0/24 is directly connected, eno16777736 |
文章评论