Cloud Native应用交付

  • 首页
  • 关于本站
  • 个人介绍
  • Downloads
  • Repo
    • Github
    • Container
  • F5
    • F5 Python SDK
    • F5-container
    • F5-LBaaS
  • 社交
    • 联系我
    • 微信/微博
    • 公众号
    • 打赏赞助
行至水穷处 坐看云起时
Cloud Native Application Services: cnadn.net
  1. 首页
  2. 化云为雨/Openstack
  3. 正文

openstack L3-GRE 网络结构分析记录 (Icehouse) 第一篇

2014年12月25日 11814点热度 0人点赞 1条评论

实验网络拓扑如下:

P6SU57JE1H7Y86U4U

 首先创建一个L3路由器,并给租户设置三个网络

网络节点情况:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
root@network:/home/mycisco# ovs-vsctl show
909c85d0-ff4e-446b-bf8a-9166f0fccd24
    Bridge br-int
        fail_mode: secure
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tap78dd7f1e-53"
            tag: 9
            Interface "tap78dd7f1e-53"
                type: internal
        Port "tap708ee330-0c"
            tag: 11
            Interface "tap708ee330-0c"
                type: internal
        Port "taped034aae-11"
            tag: 8
            Interface "taped034aae-11"
                type: internal
        Port "qr-d8fd27a6-d1"
            tag: 9
            Interface "qr-d8fd27a6-d1"
                type: internal
        Port "qr-09208a89-0b"
            tag: 8
            Interface "qr-09208a89-0b"
                type: internal
        Port "qr-4debe62b-e4"
            tag: 11
            Interface "qr-4debe62b-e4"
                type: internal
        Port br-int
            Interface br-int
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a0a648c"
            Interface "gre-0a0a648c"
                type: gre
                options: {in_key=flow, local_ip="10.10.100.139", out_key=flow, remote_ip="10.10.100.140"}
        Port br-tun
            Interface br-tun
                type: internal
    Bridge br-ex
        Port "eth2"
            Interface "eth2"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-6f726a6c-41"
            Interface "qg-6f726a6c-41"
                type: internal
    ovs_version: "2.0.2"

在br-int中,tap*为DHCP的接口, qr为租户的路由器接口,目前只有一个租户,可以看出该L3路由器上有三个网络,实际的网络拓扑结构如下:ERLGK$WSFHHU@SSGLRQB@$R

此时虚拟路由器上实际有4个接口,可以理解为br-int交换是租户私人网的内部交换机划分了三个独立vlan,每个vlan都有一个接口和虚拟路由器相连。

路由器左边的蓝色网络可以理解为provide网络,路由器通过接入br-ex使得路由器的蓝色网络与实际物理接口eth2桥接:

1
2
3
4
5
6
7
8
9
    Bridge br-ex
        Port "eth2"
            Interface "eth2"
        Port br-ex
            Interface br-ex
                type: internal
        Port "qg-6f726a6c-41"
            Interface "qg-6f726a6c-41"
                type: internal

事实上,这些网络的隔离是依靠namespace来进行的:

1
2
3
4
5
root@network:/home/mycisco# ip netns show
qdhcp-224f79e0-8068-4ef7-8c79-2326bc0cf5c4
qdhcp-70f7aa46-b66b-455e-896e-05f94a08fcb8
qrouter-040c1455-6096-4806-ba91-fec64cdaed81
qdhcp-4e3f621f-d6b9-4438-be58-aa51d3c2e061

每个网络都有一个自己的ns,路由器也有一个自己的ns,功能上这就类似于F5的route domain,存在一个上下文关系。

在网络节点内部dhcp接口以及namespace之间关系如下:

V7ZNZG~K6XAX89)E`@3U6`J

.计算节点情况:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
root@compute:/home/mycisco# ovs-vsctl show
b5553502-95e4-4ad2-90a6-a3da02d3819d
    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a0a648b"
            Interface "gre-0a0a648b"
                type: gre
                options: {in_key=flow, local_ip="10.10.100.140", out_key=flow, remote_ip="10.10.100.139"}
    Bridge br-int
        fail_mode: secure
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "2.0.2"

此时计算节点还没有启动instance,因此也就没有相关vm的interface, 此时计算节点的hypervisor上网络接口情况如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
root@compute:/home/mycisco# ifconfig
br-int    Link encap:Ethernet  HWaddr d2:2d:3d:fc:72:44  
          inet6 addr: fe80::b406:33ff:fea3:293c/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:41 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3258 (3.2 KB)  TX bytes:648 (648.0 B)
 
br-tun    Link encap:Ethernet  HWaddr 2a:fd:e3:83:b9:4f  
          inet6 addr: fe80::a8f6:49ff:fe0d:5dd5/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)
 
eth0      Link encap:Ethernet  HWaddr 00:0c:29:07:13:bf  
          inet addr:192.168.232.140  Bcast:192.168.232.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe07:13bf/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:270661 errors:0 dropped:0 overruns:0 frame:0
          TX packets:307615 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:67530181 (67.5 MB)  TX bytes:115865767 (115.8 MB)
 
eth1      Link encap:Ethernet  HWaddr 00:0c:29:07:13:c9  
          inet addr:10.10.100.140  Bcast:10.10.100.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe07:13c9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19723 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1859485 (1.8 MB)  TX bytes:2662 (2.6 KB)
          Interrupt:16 Base address:0x2000
 
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:3 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:333 (333.0 B)  TX bytes:333 (333.0 B)
 
virbr0    Link encap:Ethernet  HWaddr 6a:cc:bc:f5:91:be  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

 

下面,在网络private1网络中启动一个实例:

~Q[PIWW2G[G(4IE@R_2[B57)50LNXC6S61R9NMPNS}ML9E

实例自动获得10.10.10.7IP地址,此时网络拓扑如下:

CRQU2S{QXGBIYPCDQZ3{7_D

进入该实例查看:

G}XRA1SRC{H[8]JBD6[[BMR

可以看出该实例通过dhcp自动获取了ip地址以及缺省网关

FL$151LAH6N@[TAT]}`_0(1

在该实例中ping网关,以及ping路由器的外部网络接口都可以ping,说明路由器工作正常。

此时计算节点hypervisor上,网络变化为:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
root@compute:/home/mycisco# ifconfig
br-int    Link encap:Ethernet  HWaddr d2:2d:3d:fc:72:44  
          inet6 addr: fe80::b406:33ff:fea3:293c/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:64 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:5506 (5.5 KB)  TX bytes:648 (648.0 B)
 
br-tun    Link encap:Ethernet  HWaddr 2a:fd:e3:83:b9:4f  
          inet6 addr: fe80::a8f6:49ff:fe0d:5dd5/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)
 
eth0      Link encap:Ethernet  HWaddr 00:0c:29:07:13:bf  
          inet addr:192.168.232.140  Bcast:192.168.232.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe07:13bf/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:285451 errors:0 dropped:0 overruns:0 frame:0
          TX packets:320461 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:70009923 (70.0 MB)  TX bytes:118909346 (118.9 MB)
 
eth1      Link encap:Ethernet  HWaddr 00:0c:29:07:13:c9  
          inet addr:10.10.100.140  Bcast:10.10.100.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe07:13c9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:22244 errors:0 dropped:0 overruns:0 frame:0
          TX packets:181 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2101857 (2.1 MB)  TX bytes:23438 (23.4 KB)
          Interrupt:16 Base address:0x2000
 
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:3 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:333 (333.0 B)  TX bytes:333 (333.0 B)
 
qbr8cbe7bb6-18 Link encap:Ethernet  HWaddr 1a:8e:57:f2:7d:11  
          inet6 addr: fe80::f858:8cff:fe4e:dfcb/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1466 (1.4 KB)  TX bytes:648 (648.0 B)
 
qvb8cbe7bb6-18 Link encap:Ethernet  HWaddr 1a:8e:57:f2:7d:11  
          inet6 addr: fe80::188e:57ff:fef2:7d11/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:106 errors:0 dropped:0 overruns:0 frame:0
          TX packets:159 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:14872 (14.8 KB)  TX bytes:15102 (15.1 KB)
 
qvo8cbe7bb6-18 Link encap:Ethernet  HWaddr a6:98:52:ef:b1:0e  
          inet6 addr: fe80::a498:52ff:feef:b10e/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:159 errors:0 dropped:0 overruns:0 frame:0
          TX packets:106 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15102 (15.1 KB)  TX bytes:14872 (14.8 KB)
 
tap8cbe7bb6-18 Link encap:Ethernet  HWaddr fe:16:3e:32:28:af  
          inet6 addr: fe80::fc16:3eff:fe32:28af/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:144 errors:0 dropped:0 overruns:0 frame:0
          TX packets:113 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:13896 (13.8 KB)  TX bytes:15422 (15.4 KB)
 
virbr0    Link encap:Ethernet  HWaddr 6a:cc:bc:f5:91:be  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

可以看出,hypervisor的网络中多出了多个网络设备qbr***,qvb***,qvo***,tap****, 那么这些设备接口是如何将一个vm与物理网络(provide网络)对接起来的?

首先查看下多出来的虚拟网桥包含什么:

1
2
3
4
5
root@compute:/home/mycisco# brctl show
bridge name     bridge id               STP enabled     interfaces
<span style="color: #ff0000;">qbr8cbe7bb6-18          8000.1a8e57f27d11       no              qvb8cbe7bb6-18
                                                        tap8cbe7bb6-18</span>
virbr0          8000.000000000000       yes

qbr**虚拟网桥包含了两个接口,分别是 qvb**和tap***,tap***代表虚机实例的网卡,它和qvb***处于同一个二层网络,那么需要看看qvb***接口是连接到了哪里:

1
2
3
root@compute:/home/mycisco# ethtool -S qvb8cbe7bb6-18
NIC statistics:
     peer_ifindex: 19

该接口和一个编号索引为19的接口做了link。 来看看这个19接口是什么:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
root@compute:/home/mycisco# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:07:13:bf brd ff:ff:ff:ff:ff:ff
    inet 192.168.232.140/24 brd 192.168.232.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe07:13bf/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 00:0c:29:07:13:c9 brd ff:ff:ff:ff:ff:ff
    inet 10.10.100.140/24 brd 10.10.100.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe07:13c9/64 scope link
       valid_lft forever preferred_lft forever
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default
    link/ether e6:c2:4f:85:74:3b brd ff:ff:ff:ff:ff:ff
5: br-int: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether d2:2d:3d:fc:72:44 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b406:33ff:fea3:293c/64 scope link
       valid_lft forever preferred_lft forever
7: br-tun: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether 2a:fd:e3:83:b9:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a8f6:49ff:fe0d:5dd5/64 scope link
       valid_lft forever preferred_lft forever
8: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 6a:cc:bc:f5:91:be brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
18: qbr8cbe7bb6-18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 1a:8e:57:f2:7d:11 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f858:8cff:fe4e:dfcb/64 scope link
       valid_lft forever preferred_lft forever
19: qvo8cbe7bb6-18: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP group default qlen 1000
    link/ether a6:98:52:ef:b1:0e brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a498:52ff:feef:b10e/64 scope link
       valid_lft forever preferred_lft forever
20: qvb8cbe7bb6-18: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbr8cbe7bb6-18 state UP group default qlen 1000
    link/ether 1a:8e:57:f2:7d:11 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::188e:57ff:fef2:7d11/64 scope link
       valid_lft forever preferred_lft forever
21: tap8cbe7bb6-18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbr8cbe7bb6-18 state UNKNOWN group default qlen 500
    link/ether fe:16:3e:32:28:af brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe32:28af/64 scope link
       valid_lft forever preferred_lft forever

从输出中可以看到编号19是一个叫qvo8cbe7bb6-18的接口,该接口又位于哪?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
root@compute:/home/mycisco# ovs-vsctl show
b5553502-95e4-4ad2-90a6-a3da02d3819d
    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a0a648b"
            Interface "gre-0a0a648b"
                type: gre
                options: {in_key=flow, local_ip="10.10.100.140", out_key=flow, remote_ip="10.10.100.139"}
    Bridge br-int
        fail_mode: secure
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qvo8cbe7bb6-18"
            tag: 4
            Interface "qvo8cbe7bb6-18"
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "2.0.2

可以看到该qvo接口位于br-int网桥中。br-int网桥通过patch-tun和br-tun网桥连接,而br-tun网桥通过gre隧道接口gre-0a0a648b实现了通过底层物理接口(10.10.100.140)与外界的通信。

至此可以看到该vm通过如下路径实现了和外界的通信:

TC{LE5J5Q{XHJ7LKGLGDM$S

在本网络中,各个namespace的dhcp服务器都运行在network节点上,那么compute几点上的一个vm到底是如何通过2层网络与dhcp服务器通信的,network节点是如何知道该vm的接口是应该和哪个网络相通的,在horizon上,这是通过将某个网络关联到某个vm上,但是如果从底层来看,是如何在隧道中区分不同网络的。

首先查看网络节点中的openflow:

8
9
10
11
12
13
14
15
16
cookie=0x0, duration=128769.413s, table=2, n_packets=564, n_bytes=53473, idle_age=0, hard_age=65534, priority=1,tun_id=0x1 actions=mod_vlan_vid:9,resubmit(,10)
cookie=0x0, duration=120432.221s, table=2, n_packets=19, n_bytes=1398, idle_age=65534, hard_age=65534, priority=1,tun_id=0x4 actions=mod_vlan_vid:11,resubmit(,10)
cookie=0x0, duration=128779.008s, table=2, n_packets=197, n_bytes=18294, idle_age=65534, hard_age=65534, priority=1,tun_id=0x2 actions=mod_vlan_vid:8,resubmit(,10)
cookie=0x0, duration=143526.387s, table=2, n_packets=33, n_bytes=2526, idle_age=65534, hard_age=65534, priority=0 actions=drop
cookie=0x0, duration=143526.346s, table=3, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
cookie=0x0, duration=143526.302s, table=10, n_packets=1495, n_bytes=142220, idle_age=0, hard_age=65534, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1
cookie=0x0, duration=235.239s, table=20, n_packets=251, n_bytes=23702, hard_timeout=300, idle_age=0, hard_age=0, priority=1,vlan_tci=0x0009/0x0fff,dl_dst=fa:16:3e:32:28:af actions=load:0->NXM_OF_VLAN_TCI[],load:0x1->NXM_NX_TUN_ID[],output:2
cookie=0x0, duration=143526.245s, table=20, n_packets=1, n_bytes=98, idle_age=65534, hard_age=65534, priority=0 actions=resubmit(,21)
cookie=0x0, duration=120432.312s, table=21, n_packets=4, n_bytes=280, idle_age=65534, hard_age=65534, dl_vlan=11 actions=strip_vlan,set_tunnel:0x4,output:2

网络节点自动的为各个namespace的网络创建了对应的 tunnel (不同的tunnel id,这里分别是1,2,4)

再看计算节点的open flow:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
root@compute:/home/mycisco# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=76160.403s, table=0, n_packets=289, n_bytes=26654, idle_age=0, hard_age=65534, priority=1,in_port=1 actions=resubmit(,1)
cookie=0x0, duration=76159.597s, table=0, n_packets=186, n_bytes=22456, idle_age=0, hard_age=65534, priority=1,in_port=2 actions=resubmit(,2)
cookie=0x0, duration=76160.361s, table=0, n_packets=6, n_bytes=468, idle_age=65534, hard_age=65534, priority=0 actions=drop
cookie=0x0, duration=76160.294s, table=1, n_packets=221, n_bytes=20876, idle_age=0, hard_age=65534, priority=1,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
cookie=0x0, duration=76160.236s, table=1, n_packets=68, n_bytes=5778, idle_age=80, hard_age=65534, priority=1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,21)
cookie=0x0, duration=3669.216s, table=2, n_packets=186, n_bytes=22456, idle_age=0, priority=1,tun_id=0x1 actions=mod_vlan_vid:4,resubmit(,10)
cookie=0x0, duration=76160.193s, table=2, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
cookie=0x0, duration=76160.156s, table=3, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop
cookie=0x0, duration=76160.11s, table=10, n_packets=186, n_bytes=22456, idle_age=0, hard_age=65534, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1
cookie=0x0, duration=80.351s, table=20, n_packets=87, n_bytes=8190, hard_timeout=300, idle_age=0, hard_age=0, priority=1,vlan_tci=0x0004/0x0fff,dl_dst=fa:16:3e:2a:46:e0 actions=load:0->NXM_OF_VLAN_TCI[],load:0x1->NXM_NX_TUN_ID[],output:2
cookie=0x0, duration=76160.068s, table=20, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=resubmit(,21)
cookie=0x0, duration=3669.335s, table=21, n_packets=20, n_bytes=1962, idle_age=80, dl_vlan=4 actions=strip_vlan,set_tunnel:0x1,output:2
cookie=0x0, duration=76160.019s, table=21, n_packets=32, n_bytes=2616, idle_age=3669, hard_age=65534, priority=0 actions=drop

可以看出当计算节点的br-tun桥将某个tunnel ID与某个vlan ID进行了关联,当数据包从br-int收到tag=4的数据包会剔除相关vlan id,并将其送入对应的gre tunnel里,网络节点从tunnel中收到该数据包后,根据自己这边的tunnel id与vlan id的对应关系找到正确的vlan id,并将其封装并送回给br-int。在本例中 计算节点一侧虚机网卡位于vlan 4中:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
root@compute:/home/mycisco# ovs-vsctl show
b5553502-95e4-4ad2-90a6-a3da02d3819d
    Bridge br-tun
        Port br-tun
            Interface br-tun
                type: internal
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "gre-0a0a648b"
            Interface "gre-0a0a648b"
                type: gre
                options: {in_key=flow, local_ip="10.10.100.140", out_key=flow, remote_ip="10.10.100.139"}
    Bridge br-int
        fail_mode: secure
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "qvo8cbe7bb6-18"
            tag: 4
            Interface "qvo8cbe7bb6-18"
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "2.0.2"

而在网络节点中,相关网络位于vlan id 9, 因此这就像使用一个gre隧道将两个不同vlan直接互联起来一样,虽然位于不同vlan,但实际上是普通的2层,无tag:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
root@network:/home/mycisco# ovs-vsctl show
909c85d0-ff4e-446b-bf8a-9166f0fccd24
    Bridge br-int
        fail_mode: secure
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tap78dd7f1e-53"
            tag: 9
            Interface "tap78dd7f1e-53"
                type: internal
        Port "tap708ee330-0c"
            tag: 11
            Interface "tap708ee330-0c"
                type: internal
        Port "taped034aae-11"
            tag: 8
            Interface "taped034aae-11"
                type: internal
        Port "qr-d8fd27a6-d1"
            tag: 9
            Interface "qr-d8fd27a6-d1"
                type: internal
        Port "qr-09208a89-0b"
            tag: 8
            Interface "qr-09208a89-0b"
                type: internal
        Port "qr-4debe62b-e4"
            tag: 11
            Interface "qr-4debe62b-e4"
                type: internal
        Port br-int
            Interface br-int
                type: internal

上述tap设备对应下图中的dhcp端口

)ZCY~OM[)9XVFK52JWCZKU0

观察一下计算节点的open flow,在实例中ping网关,出现:

1
cookie=0x0, duration=151.529s, table=20, n_packets=18, n_bytes=1540, hard_timeout=300, idle_age=30, hard_age=29, priority=1,vlan_tci=0x0004/0x0fff,dl_dst=fa:16:3e:2a:46:e0 actions=load:0->NXM_OF_VLAN_TCI[],load:0x1->NXM_NX_TUN_ID[],output:2

可以看到上述条目中目标mac是对端网关的mac:

_U0FB72HI0B4GRH87NLTY@E

如果此时,你在同一个计算节点上开启同一网络下的另一个实例并ping相同的网关,open flow中并不会多出一条,因为此时是同一个目标mac。

若此时在同一个实例中,ping另一个ip(dhcp server)的话,会看到多出一条:

1
2
cookie=0x0, duration=151.529s, table=20, n_packets=18, n_bytes=1540, hard_timeout=300, idle_age=30, hard_age=29, priority=1,vlan_tci=0x0004/0x0fff,dl_dst=fa:16:3e:2a:46:e0 actions=load:0->NXM_OF_VLAN_TCI[],load:0x1->NXM_NX_TUN_ID[],output:2
cookie=0x0, duration=2.623s, table=20, n_packets=3, n_bytes=294, hard_timeout=300, idle_age=0, hard_age=0, priority=1,vlan_tci=0x0004/0x0fff,dl_dst=fa:16:3e:3c:54:50 actions=load:0->NXM_OF_VLAN_TCI[],load:0x1->NXM_NX_TUN_ID[],output:2

多出这一条的目的mac则对应于该网络中的dhcp服务器mac(10.10.10.2 IP对应的mac)

如果在同一个租户的同一个网络内部署两台虚机的话,则结构如下,计算节点的hypervisor对每个虚机实例都创建一个qbr**网桥:

20141225145502

上图中,两个实例位于同一网络,因此在br-int中,两个接口属于同一个vlan,同vlan之间通信直接通过br-int完成:

Bridge br-int
fail_mode: secure
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port "qvo8cbe7bb6-18"
          tag: 4
Interface "qvo8cbe7bb6-18"
Port br-int
Interface br-int
type: internal
Port "qvodd2c3906-f8"
tag: 4
Interface "qvodd2c3906-f8"

如果在同一个计算节点中再增加另一个网络的虚机实例的话,则彼此间数据流如下:

~XU_KGP9}3W$@DSEMN`GG32

instance1和instance3处于不同的网络中,他们之间通信需要依靠网络节点进行路由,通过网络节点的open flow可以看出这一点:

1
2
cookie=0x0, duration=5.791s, table=20, n_packets=8, n_bytes=672, hard_timeout=300, idle_age=0, hard_age=0, priority=1,vlan_tci=0x0008/0x0fff,dl_dst=fa:16:3e:cb:91:43 actions=load:0->NXM_OF_VLAN_TCI[],load:0x2->NXM_NX_TUN_ID[],output:2
cookie=0x0, duration=5.785s, table=20, n_packets=6, n_bytes=532, hard_timeout=300, idle_age=0, hard_age=0, priority=1,vlan_tci=0x0009/0x0fff,dl_dst=fa:16:3e:53:9b:3b actions=load:0->NXM_OF_VLAN_TCI[],load:0x1->NXM_NX_TUN_ID[],output:2

此时,计算节点上的open flow为:

1
2
cookie=0x0, duration=9.893s, table=20, n_packets=11, n_bytes=1022, hard_timeout=300, idle_age=0, hard_age=0, priority=1,vlan_tci=0x0004/0x0fff,dl_dst=fa:16:3e:2a:46:e0 actions=load:0->NXM_OF_VLAN_TCI[],load:0x1->NXM_NX_TUN_ID[],output:2
cookie=0x0, duration=9.895s, table=20, n_packets=11, n_bytes=1022, hard_timeout=300, idle_age=0, hard_age=0, priority=1,vlan_tci=0x0005/0x0fff,dl_dst=fa:16:3e:79:45:64 actions=load:0->NXM_OF_VLAN_TCI[],load:0x2->NXM_NX_TUN_ID[],output:2

可以利用上面的MAC,对应下表的IP结合起来看:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
root@network:/home/mycisco# neutron port-list
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                          |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+
| 09208a89-0b45-4b80-bf11-2aee68b30eb1 |      | fa:16:3e:79:45:64 | {"subnet_id": "f380d778-d576-4179-96df-cb3a2618dbd3", "ip_address": "10.10.20.1"}  |
| 4debe62b-e44c-47c7-a5e5-b6a11da8d31c |      | fa:16:3e:75:b5:d4 | {"subnet_id": "410a9d7c-6ae8-4059-954e-973529a933c5", "ip_address": "10.10.30.1"}  |
| 6f726a6c-412d-4d22-bcc7-eadedf407680 |      | fa:16:3e:fc:53:dc | {"subnet_id": "7b2cd216-1251-4eaa-9f05-b8cadc82bc1e", "ip_address": "192.168.0.2"} |
| 708ee330-0c8a-41bd-a2f5-271ca9662151 |      | fa:16:3e:75:82:cd | {"subnet_id": "410a9d7c-6ae8-4059-954e-973529a933c5", "ip_address": "10.10.30.2"}  |
| 78dd7f1e-53e6-4a2c-b166-5eef9add12b7 |      | fa:16:3e:3c:54:50 | {"subnet_id": "dcdd699a-c4b0-4360-9219-c62cb6ed0b3e", "ip_address": "10.10.10.2"}  |
| 8cbe7bb6-18d9-4455-abce-473ebe5a9591 |      | fa:16:3e:32:28:af | {"subnet_id": "dcdd699a-c4b0-4360-9219-c62cb6ed0b3e", "ip_address": "10.10.10.7"}  |
| a6f975bb-605b-4f36-ad3f-669ccdab5fe5 |      | fa:16:3e:cb:91:43 | {"subnet_id": "f380d778-d576-4179-96df-cb3a2618dbd3", "ip_address": "10.10.20.7"}  |
| d8fd27a6-d1cf-46f4-8e0e-4965001d547a |      | fa:16:3e:2a:46:e0 | {"subnet_id": "dcdd699a-c4b0-4360-9219-c62cb6ed0b3e", "ip_address": "10.10.10.1"}  |
| dd2c3906-f8cf-48cd-85bf-d952a11eeab4 |      | fa:16:3e:53:9b:3b | {"subnet_id": "dcdd699a-c4b0-4360-9219-c62cb6ed0b3e", "ip_address": "10.10.10.8"}  |
| ed034aae-1106-41b5-8b3f-2b8cb5776baf |      | fa:16:3e:fb:f3:33 | {"subnet_id": "f380d778-d576-4179-96df-cb3a2618dbd3", "ip_address": "10.10.20.2"}  |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+

 

 

最后,模拟从网络节点中的虚拟路由器上的external网络接口模拟ping 虚拟机:

root@network:/home/mycisco# ip netns exec qrouter-040c1455-6096-4806-ba91-fec64cdaed81  ping -I 192.168.0.2 10.10.10.7
PING 10.10.10.7 (10.10.10.7) from 192.168.0.2 : 56(84) bytes of data.
64 bytes from 10.10.10.7: icmp_seq=1 ttl=64 time=5.22 ms
64 bytes from 10.10.10.7: icmp_seq=2 ttl=64 time=6.31 ms
64 bytes from 10.10.10.7: icmp_seq=3 ttl=64 time=1.32 ms
64 bytes from 10.10.10.7: icmp_seq=4 ttl=64 time=1.33 ms
64 bytes from 10.10.10.7: icmp_seq=5 ttl=64 time=1.05 ms

 

相关文章

  • openstack L3-GRE 网络结构分析记录 (Icehouse) 第四篇(多计算节点)
  • openstack L3-GRE 网络结构分析记录 (Icehouse) 第三篇(多租户)
  • openstack L3-GRE 网络结构分析记录 (Icehouse) 第二篇
  • Neutron Networking: Neutron Routers and the L3 Agent
  • Provider network and tenant network in neutron
本作品采用 知识共享署名-非商业性使用 4.0 国际许可协议 进行许可
标签: gre l3agent network openstack
最后更新:2014年12月26日

纳米

linjing.io

打赏 点赞
< 上一篇
下一篇 >

文章评论

razz evil exclaim smile redface biggrin eek confused idea lol mad twisted rolleyes wink cool arrow neutral cry mrgreen drooling persevering
取消回复

这个站点使用 Akismet 来减少垃圾评论。了解你的评论数据如何被处理。

页面AI聊天助手
文章目录
  •  首先创建一个L3路由器,并给租户设置三个网络

纳米

linjing.io

☁️迈向Cloud Native ADC ☁️

认证获得:
TOGAF: ID 152743
Kubernetes: CKA #664
Microsoft: MCSE MCDBA
Cisco: CCNP
Juniper: JNCIS
F5:
F5 Certified Solution Expert, Security
F5 Certified Technology Specialist, LTM/GTM/APM/ASM
F5 Certified BIG-IP Administrator
  • 点击查看本博技术要素列表
  • 归档
    分类
    • AI
    • Automation
    • Avi Networks
    • Cisco ACI
    • CISCO资源
    • F5 with ELK
    • F5-Tech tips
    • F5技术
    • Juniper
    • Linux
    • NGINX
    • SDN
    • ServiceMesh
    • WEB编程
    • WINDOWS相关
    • 业界文章
    • 交换机技术
    • 化云为雨/Openstack
    • 协议原理
    • 容器/k8s
    • 我的工作
    • 我的生活
    • 网站技术
    • 路由器技术
    • 项目案例
    标签聚合
    DNS k8s network istio gtm F5 envoy irule nginx api docker bigip openstack neutron flannel
    最近评论
    汤姆 发布于 8 个月前(09月10日) 嗨,楼主,里面的json怎么下载啊,怎么收费啊?
    汤姆 发布于 8 个月前(09月09日) 大佬,kib的页面可以分享下吗?谢谢
    zhangsha 发布于 1 年前(05月12日) 资料发给我下,谢谢纳米同志!!!!lyx895@qq.com
    李成才 发布于 1 年前(01月02日) 麻烦了,谢谢大佬
    纳米 发布于 1 年前(01月02日) 你好。是的,因为以前下载系统插件在一次升级后将所有的下载生成信息全弄丢了。所以不少文件无法下载。DN...
    浏览次数
    • Downloads - 183,648 views
    • 联系我 - 118,966 views
    • 迄今为止最全最深入的BIGIP-DNS/GTM原理及培训资料 - 116,173 views
    • Github - 103,533 views
    • F5常见log日志解释 - 79,714 views
    • 从传统ADC迈向CLOUD NATIVE ADC - 下载 - 74,499 views
    • Sniffer Pro 4 70 530抓包软件 中文版+视频教程 - 74,320 views
    • 迄今为止最全最深入的BIGIP-DNS/GTM原理及培训资料 - 67,770 views
    • 关于本站 - 60,792 views
    • 这篇文档您是否感兴趣 - 55,459 views
    链接表
    • F5SE创新
    • Jimmy Song‘s Blog
    • SDNlab
    • Service Mesh社区
    • 三斗室
    • 个人profile
    • 云原生社区

    COPYRIGHT © 2023 Cloud Native 应用交付. ALL RIGHTS RESERVED.

    Theme Kratos Made By Seaton Jiang

    京ICP备14048088号-1

    京公网安备 11010502041506号