Cloud Native应用交付
  • 首页
  • 关于本站
  • 个人介绍
  • Downloads
  • Repo
    • Github
    • Container
  • F5
    • F5 Python SDK
    • F5-container
    • F5-LBaaS
  • 社交
    • 联系我
    • 微信/微博
    • 公众号
    • 打赏赞助
行至水穷处 坐看云起时
☁️We are in new App Mesh era: imesh.club ☁️
  1. 首页
  2. 容器/k8s
  3. 正文

DOCKER容器跨宿主机通信方法-(3):Openvswitch

2017年06月26日 5198点热度 0人点赞 0条评论

Openvswitch overlay网络:

配置前网络状况

主机1:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
[root@docker1 ~]# ifconfig
br-2aeddaf04b59: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.19.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        ether 02:42:30:f2:2f:6b  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        ether 02:42:da:95:47:4e  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
eno16777736: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.228  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::20c:29ff:fe42:d98  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:42:0d:98  txqueuelen 1000  (Ethernet)
        RX packets 1802  bytes 252519 (246.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 768  bytes 148846 (145.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
eno33554960: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.188.130  netmask 255.255.255.0  broadcast 192.168.188.255
        inet6 fe80::20c:29ff:fe42:da2  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:42:0d:a2  txqueuelen 1000  (Ethernet)
        RX packets 12  bytes 1852 (1.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 17  bytes 2294 (2.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
eno50332184: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.199.17  netmask 255.255.255.0  broadcast 172.16.199.255
        inet6 fe80::20c:29ff:fe42:dac  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:42:0d:ac  txqueuelen 1000  (Ethernet)
        RX packets 3  bytes 218 (218.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 16  bytes 1172 (1.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 72  bytes 6184 (6.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 72  bytes 6184 (6.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

主机2:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
[root@docker2 ~]# ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.27.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        ether 02:42:e9:45:8c:a6  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
eno16777736: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.183  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::20c:29ff:feae:1179  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:ae:11:79  txqueuelen 1000  (Ethernet)
        RX packets 4424  bytes 817889 (798.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 413  bytes 48906 (47.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
eno33554960: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:0c:29:ae:11:83  txqueuelen 1000  (Ethernet)
        RX packets 18  bytes 1646 (1.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
eno50332184: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.199.27  netmask 255.255.255.0  broadcast 172.16.199.255
        inet6 fe80::20c:29ff:feae:118d  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:ae:11:8d  txqueuelen 1000  (Ethernet)
        RX packets 5  bytes 338 (338.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 13  bytes 962 (962.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 68  bytes 5844 (5.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 68  bytes 5844 (5.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

主机1设置:

添加ovs—br0网桥:

ovs-vsctl add-br ovs-br0

连接ovs-br0到docker0交换机中:

brctl addif docker0 ovs-br0

ovs-br0中增加gre接口并向对端设置gre隧道:

ovs-vsctl add-port ovs-br0 gre1 -- set interface gre1 type=gre option:remote_ip=172.16.199.27

相关输出

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[root@docker1 ~]# ovs-vsctl show
8f3683a9-75d5-4d64-8a0b-f6da2b132b35
    Bridge br-int
        Port br-int
            Interface br-int
                type: internal
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
    Bridge "ovs-br0"
        Port "gre1"
            Interface "gre1"
                type: gre
                options: {remote_ip="172.16.199.27"}
        Port "ovs-br0"
            Interface "ovs-br0"
                type: internal
    ovs_version: "2.5.0"
 
[root@docker1 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
br-2aeddaf04b59         8000.024230f22f6b       no
docker0         8000.0242da95474e       no              ovs-br0

增加到目的docker0网络的路由,使其通过本机docker0:

1
ip route add 172.27.0.0/16 dev docker0

增加前后路由对比

1
2
3
4
5
6
7
8
9
10
[root@docker1 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    100    0        0 eno16777736
0.0.0.0         192.168.188.2   0.0.0.0         UG    101    0        0 eno33554960
172.16.199.0    0.0.0.0         255.255.255.0   U     100    0        0 eno50332184
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.19.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-2aeddaf04b59
192.168.0.0     0.0.0.0         255.255.255.0   U     100    0        0 eno16777736
192.168.188.0   0.0.0.0         255.255.255.0   U     100    0        0 eno33554960

1
2
3
4
5
6
7
8
9
10
11
[root@docker1 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    100    0        0 eno16777736
0.0.0.0         192.168.188.2   0.0.0.0         UG    101    0        0 eno33554960
172.16.199.0    0.0.0.0         255.255.255.0   U     100    0        0 eno50332184
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.19.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-2aeddaf04b59
172.27.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.0.0     0.0.0.0         255.255.255.0   U     100    0        0 eno16777736
192.168.188.0   0.0.0.0         255.255.255.0   U     100    0        0 eno33554960

主机2设置:

添加ovs—br0网桥:

ovs-vsctl add-br ovs-br0

连接ovs-br0到docker0交换机中:

brctl addif docker0 ovs-br0

ovs-br0中增加gre接口并向对端设置gre隧道:

ovs-vsctl add-port ovs-br0 gre2 -- set interface gre2 type=gre option:remote_ip=172.16.199.17

增加到目的docker0网络的路由,使其通过本机docker0:

ip route add 172.17.0.0/16 dev docker0

测试输出:

主机1上的busybox-1容器ping主机2容器IP:

1
2
3
4
5
6
7
[root@docker1 ~]# docker exec busybox-1 ping 172.27.0.2
PING 172.27.0.2 (172.27.0.2): 56 data bytes
64 bytes from 172.27.0.2: seq=0 ttl=63 time=2.585 ms
64 bytes from 172.27.0.2: seq=1 ttl=63 time=0.839 ms
64 bytes from 172.27.0.2: seq=2 ttl=63 time=0.817 ms
64 bytes from 172.27.0.2: seq=3 ttl=63 time=0.827 ms
64 bytes from 172.27.0.2: seq=4 ttl=63 time=0.813 ms

主机2上busybox1容器ping主机1容器IP:

1
2
3
4
[root@docker2 ~]# docker exec busybox1 ping 172.17.0.2
PING 172.17.0.2 (172.17.0.2): 56 data bytes
64 bytes from 172.17.0.2: seq=0 ttl=63 time=0.775 ms
64 bytes from 172.17.0.2: seq=1 ttl=63 time=0.919 ms

主机1相关输出:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
[root@docker1 ~]# brctl show docker0
bridge name     bridge id               STP enabled     interfaces
docker0         8000.0242da95474e       no              ovs-br0
                                                        veth7af83ec
[root@docker1 ~]# docker exec busybox-1 ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: gre0@NONE: <NOARP> mtu 1476 qdisc noop qlen 1
    link/gre 0.0.0.0 brd 0.0.0.0
3: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
14: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 scope global eth0
       valid_lft forever preferred_lft forever
16: eth1@if17: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
    link/ether 02:42:ac:13:13:00 brd ff:ff:ff:ff:ff:ff
    inet 172.19.19.0/16 scope global eth1
       valid_lft forever preferred_lft forever
 
 
 
[root@docker1 ~]# ifconfig
br-2aeddaf04b59: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.19.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        inet6 fe80::42:30ff:fef2:2f6b  prefixlen 64  scopeid 0x20<link>
        ether 02:42:30:f2:2f:6b  txqueuelen 0  (Ethernet)
        RX packets 633  bytes 51892 (50.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 633  bytes 60754 (59.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        inet6 fe80::42:daff:fe95:474e  prefixlen 64  scopeid 0x20<link>
        ether 02:42:da:95:47:4e  txqueuelen 0  (Ethernet)
        RX packets 645  bytes 52564 (51.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 638  bytes 61408 (59.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
eno16777736: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.228  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::20c:29ff:fe42:d98  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:42:0d:98  txqueuelen 1000  (Ethernet)
        RX packets 6609  bytes 909652 (888.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2785  bytes 844864 (825.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
eno33554960: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.188.130  netmask 255.255.255.0  broadcast 192.168.188.255
        inet6 fe80::20c:29ff:fe42:da2  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:42:0d:a2  txqueuelen 1000  (Ethernet)
        RX packets 22  bytes 3862 (3.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 27  bytes 4304 (4.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
eno50332184: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.199.17  netmask 255.255.255.0  broadcast 172.16.199.255
        inet6 fe80::20c:29ff:fe42:dac  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:42:0d:ac  txqueuelen 1000  (Ethernet)
        RX packets 668  bytes 87994 (85.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 669  bytes 87180 (85.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
gre_sys: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 65490
        inet6 fe80::4091:8ff:fef8:7625  prefixlen 64  scopeid 0x20<link>
        ether 42:91:08:f8:76:25  txqueuelen 1000  (Ethernet)
        RX packets 633  bytes 60754 (59.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 633  bytes 51892 (50.6 KiB)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0
 
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 72  bytes 6184 (6.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 72  bytes 6184 (6.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
ovs-br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::8c6c:1aff:febc:f04f  prefixlen 64  scopeid 0x20<link>
        ether 8e:6c:1a:bc:f0:4f  txqueuelen 1000  (Ethernet)
        RX packets 633  bytes 51892 (50.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 633  bytes 60754 (59.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
veth7af83ec: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::8cb8:d7ff:fe35:b90f  prefixlen 64  scopeid 0x20<link>
        ether 8e:b8:d7:35:b9:0f  txqueuelen 0  (Ethernet)
        RX packets 618  bytes 59500 (58.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 637  bytes 61366 (59.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
vethc73f9a6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::e8a0:9dff:fe45:e8e3  prefixlen 64  scopeid 0x20<link>
        ether ea:a0:9d:45:e8:e3  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 16  bytes 1296 (1.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
 
 
[root@docker1 ~]# docker exec busybox-1 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: gre0@NONE: <NOARP> mtu 1476 qdisc noop qlen 1
    link/gre 0.0.0.0 brd 0.0.0.0
3: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
14: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
16: eth1@if17: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
    link/ether 02:42:ac:13:13:00 brd ff:ff:ff:ff:ff:ff

主机2相关输出:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
[root@docker2 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
docker0         8000.0242e9458ca6       no              ovs-br0
                                                        veth5c866ea
[root@docker2 ~]# docker exec busybox1 ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: gre0@NONE: <NOARP> mtu 1476 qdisc noop qlen 1
    link/gre 0.0.0.0 brd 0.0.0.0
3: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
13: eth0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
    link/ether 02:42:ac:1b:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.27.0.2/16 scope global eth0
       valid_lft forever preferred_lft forever
 
 
[root@docker2 ~]# docker exec busybox1 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: gre0@NONE: <NOARP> mtu 1476 qdisc noop qlen 1
    link/gre 0.0.0.0 brd 0.0.0.0
3: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
13: eth0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
    link/ether 02:42:ac:1b:00:02 brd ff:ff:ff:ff:ff:ff
 
 
 
[root@docker2 ~]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.27.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        inet6 fe80::42:e9ff:fe45:8ca6  prefixlen 64  scopeid 0x20<link>
        ether 02:42:e9:45:8c:a6  txqueuelen 0  (Ethernet)
        RX packets 2776  bytes 228144 (222.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2790  bytes 268580 (262.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
eno16777736: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.183  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::20c:29ff:feae:1179  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:ae:11:79  txqueuelen 1000  (Ethernet)
        RX packets 8271  bytes 1470666 (1.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1243  bytes 205416 (200.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
eno33554960: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:0c:29:ae:11:83  txqueuelen 1000  (Ethernet)
        RX packets 18  bytes 1646 (1.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
eno50332184: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.199.27  netmask 255.255.255.0  broadcast 172.16.199.255
        inet6 fe80::20c:29ff:feae:118d  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:ae:11:8d  txqueuelen 1000  (Ethernet)
        RX packets 2843  bytes 377038 (368.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2863  bytes 379430 (370.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
gre_sys: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 65490
        inet6 fe80::4848:3eff:fe38:c4c5  prefixlen 64  scopeid 0x20<link>
        ether 4a:48:3e:38:c4:c5  txqueuelen 1000  (Ethernet)
        RX packets 2753  bytes 266042 (259.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2769  bytes 228572 (223.2 KiB)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0
 
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 68  bytes 5844 (5.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 68  bytes 5844 (5.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
ovs-br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::5caa:1cff:fe1f:9842  prefixlen 64  scopeid 0x20<link>
        ether 5e:aa:1c:1f:98:42  txqueuelen 1000  (Ethernet)
        RX packets 2753  bytes 227500 (222.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2769  bytes 267338 (261.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 
veth5c866ea: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::589d:2fff:fe9a:eb7f  prefixlen 64  scopeid 0x20<link>
        ether 5a:9d:2f:9a:eb:7f  txqueuelen 0  (Ethernet)
        RX packets 2786  bytes 267428 (261.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2804  bytes 269168 (262.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

抓包分析:

可以看到,红色部分的外部IP包头,源mac和目的mac分别是两台主机的物理网卡mac

内层的以太网包头中,源mac是主机1的docker0 mac,目的mac是主机2上目标容器里mac

数据path拓扑:

可以想象为两台主机上的docker0被gre隧道虚拟直连起来。

此时在主机1中的docker容器内arp表:

1
2
3
[root@docker1 ~]# docker exec busybox-1 arp -a
? (172.17.0.1) at 02:42:da:95:47:4e [ether]  on eth0
? (172.27.0.1) at 02:42:e9:45:8c:a6 [ether]  on eth0

可以看到在容器内目的地址172.27.0.1的mac实际上是主机2内docker0桥的mac,这里比较特殊,注意到实际抓包中,GRE内层以太网的目的mac却是主机2内的容器真实mac, 此时如果直接查看主机1本身的arp表,可以看到172.27.0.2(目的容器IP)的mac是真实目的容器的mac(经过docker0接口):

1
2
3
4
5
6
7
8
[root@docker1 ~]# arp -a
gateway (192.168.188.2) at 00:50:56:ee:c0:ec [ether] on eno33554960
? (192.168.188.254) at 00:50:56:f4:68:91 [ether] on eno33554960
? (172.27.0.2) at 02:42:ac:1b:00:02 [ether] on docker0
? (172.16.199.27) at 00:0c:29:ae:11:8d [ether] on eno50332184
BEI-ML-JLIN.lan (192.168.0.157) at ac:bc:32:ae:0c:d5 [ether] on eno16777736
Hiwifi.lan (192.168.0.1) at d4:ee:07:44:e6:00 [ether] on eno16777736
? (172.17.0.2) at 02:42:ac:11:00:02 [ether] on docker0

  1. 所以最终的实际数据路径:
    主机1内的容器1向主机2的容器2发送数据包,在源容器内,目的mac是主机2中的docker0的mac
  2. 数据包根据目的IP查找主机1的系统路由表,将数据发往主机1的docker0桥,
  3. 数据包流经主机的docker0后,源mac被修改为docker0的mac,目的mac被修改为目标容器的真实mac
  4. 由于docker0向外直接对联了ovs-br0网桥,数据包被发送到ovs-br0并执行GRE封装
  5. 数据包通过GRE隧道(实际流经主机的物理网卡,经由外部网络)到达对端host2
  6. 对端host2首先解封装GRE,获得内层以太网数据包,并根据目的mac将数据包送往主机的docker0,并最终发送给目标容器

注意:

上述两个主机内的容器不在同一个网段,如果在同一个网段(但ip不重叠)也是可以的,配置中只是无需指定目标路由那个配置了。

对于这种手工的openvswitch配置方式,如果有很多host,需要在每台上配置gre,并彼此形成full mesh结构,手工配置起来还是很复杂。因此实际上,可以通过一些第三方的解决方案来自动化这些事情,比如flannel网络模式。

本作品采用 知识共享署名 4.0 国际许可协议 进行许可
标签: docker openvswitch ovs 容器网络
最后更新:2017年07月14日

纳米

http://linjing.io

打赏 点赞
< 上一篇
下一篇 >

文章评论

取消回复

纳米

http://linjing.io

☁️迈向Cloud Native ADC ☁️

认证获得:
Kubernetes: CKA #664
Microsoft: MCSE MCDBA
Cisco: CCNP
Juniper: JNCIS
F5:
F5 Certified Solution Expert, Security
F5 Certified Technology Specialist, LTM/GTM/APM/ASM
F5 Certified BIG-IP Administrator
  • 点击查看本博技术要素列表
  • 分类目录
    • Avi Networks (3)
    • Cisco ACI (1)
    • CISCO资源 (21)
    • F5 with ELK (8)
    • F5-Tech tips (38)
    • F5技术 (203)
    • Juniper (4)
    • Linux (7)
    • Nginx (18)
    • SDN (4)
    • ServiceMesh (19)
    • WEB编程 (8)
    • WINDOWS相关 (7)
    • 业界文章 (18)
    • 交换机技术 (20)
    • 化云为雨/Openstack (35)
    • 协议原理 (52)
    • 容器/k8s (64)
    • 我的工作 (19)
    • 我的生活 (70)
    • 网站技术 (19)
    • 路由器技术 (80)
    • 项目案例 (28)
    文章归档
    标签聚合
    F5 k8s openstack nginx istio DNS envoy gtm docker network flannel api irule bigip neutron cc kubernetes ELK vxlan BGP dhcp VPN IPSec lbaas ingress ingress controller nginx plus sidecar IPSec VPN NAT sql
    最新 热点 随机
    最新 热点 随机
    Say hello for 2021 二进制flannel部署,非cni网络模式下与k8s CIS结合方案 又是一年国庆 Service Account Token Volume Projection Istio ingressgateway 静态TLS证书加载与SDS发现方式配置区别 Istio里Gateway的port定义与实际ingressgateway的listener端口关系及规则 Helm 3 部署NGINX Ingress Controller 应用交付老兵眼中的Envoy, 云原生时代下的思考 Istio sidecar iptables以及流量控制分析 Istio 熔断策略及envoy配置
    Say hello for 2021
    TMOS_Order_of_Operations_v2 【原创】用CISCO VPN-Client4.01连接VPN-SERVER配置 国内各ISP及教育科研网段.F5必备! CISCO 网站导航 Openstack Tunnel 网络MTU的问题 k8s drain Strusts S2-045 CVE 2017-5638 BIGIP 防御措施 启用cloudflare免费服务 1924/2912交换机简单试验[转] 【转】openstack 的vlan配置方式
    链接表
    • Jimmy Song‘s Blog
    • SDNap
    • SDNlab
    • SDN论坛
    • Service Mesh社区
    • 三斗室
    • 个人profile

    COPYRIGHT © 2020 Cloud Native应用交付. ALL RIGHTS RESERVED.

    THEME KRATOS MADE BY VTROIS

    京ICP备14048088号-1

    京公网安备 11010502041506号

    [ Placeholder content for popup link ] WordPress Download Manager - Best Download Management Plugin