Warning: Array to string conversion in /home/wwwroot/myf5/wp-content/plugins/wpjam-basic/public/wpjam-utils.php on line 496
在上述两篇文章中描述的是单租户情况下同网段或不同网段间数据是如何走的，以及描述一个数据包是如何去往其他datacenter网络或互联网的。
openstack L3-GRE 网络结构分析记录 (Icehouse) 第一篇
openstack L3-GRE 网络结构分析记录 (Icehouse) 第二篇
本篇博文记录描述多租户情形下的网络及数据流情况。
首先，新建一个租户，并在该租户上下文下创建一个路由器和一个私网：
和另一个租户网络比较一下：
可以看出，两个租户共享了internet网络，这是因为在创建internet网络时，该网络是一个科共享的网络，即所有租户都可以使用。 两个租户的私有网络则完全不同，且两个网络拥有独立的虚拟路由器。
此时，在网络节点上，与前几篇文章描述的差异不大，neutron的l3agent会为每个租户创建独立的虚拟路由器，通过namespace隔离：
root@network:/home/mycisco# ip netns show qrouter-015840ab-38ed-4086-b78e-696664d59281 qdhcp-5286db3e-0025-4650-8a4d-3e3e1dd9f70c qrouter-040c1455-6096-4806-ba91-fec64cdaed81 qdhcp-224f79e0-8068-4ef7-8c79-2326bc0cf5c4 qdhcp-70f7aa46-b66b-455e-896e-05f94a08fcb8 qdhcp-4e3f621f-d6b9-4438-be58-aa51d3c2e061
上面 ***9281的虚拟路由器则是demo租户的。
而从openvswitch的角度来看，与之前的结构一样，只是多出一个port来连接demo租户的internal网络，对ovs来说没有租户的概念，租户和ovs中哪个网络连接是有neutron来负责的。
root@network:/home/mycisco# ovs-vsctl show
909c85d0-ff4e-446b-bf8a-9166f0fccd24
Bridge br-int
fail_mode: secure
Port "taped034aae-11"
tag: 1
Interface "taped034aae-11"
type: internal
Port "tap78dd7f1e-53"
tag: 2
Interface "tap78dd7f1e-53"
type: internal
Port "qr-219fc331-d3"
tag: 4
Interface "qr-219fc331-d3"
type: internal
Port "qr-09208a89-0b"
tag: 1
Interface "qr-09208a89-0b"
type: internal
Port "tap1b1b29de-97"
tag: 4
Interface "tap1b1b29de-97"
type: internal
Port "qr-d8fd27a6-d1"
tag: 2
Interface "qr-d8fd27a6-d1"
type: internal
Port "tap708ee330-0c"
tag: 3
Interface "tap708ee330-0c"
type: internal
Port br-int
Interface br-int
type: internal
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port "qr-4debe62b-e4"
tag: 3
Interface "qr-4debe62b-e4"
type: internal
Bridge br-ex
Port "qg-6f726a6c-41"
Interface "qg-6f726a6c-41"
type: internal
Port "eth2"
Interface "eth2"
Port br-ex
Interface br-ex
type: internal
Port "qg-814e9e39-99"
Interface "qg-814e9e39-99"
type: internal
Bridge br-tun
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "gre-0a0a648c"
Interface "gre-0a0a648c"
type: gre
options: {in_key=flow, local_ip="10.10.100.139", out_key=flow, remote_ip="10.10.100.140"}
Port br-tun
Interface br-tun
type: internal
ovs_version: "2.0.2"
计算节点情况：
root@compute:/home/mycisco# ovs-vsctl show
b5553502-95e4-4ad2-90a6-a3da02d3819d
Bridge br-tun
Port br-tun
Interface br-tun
type: internal
Port "gre-0a0a648b"
Interface "gre-0a0a648b"
type: gre
options: {in_key=flow, local_ip="10.10.100.140", out_key=flow, remote_ip="10.10.100.139"}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Bridge br-int
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port "qvo2e48b70e-e1"
tag: 3
Interface "qvo2e48b70e-e1"
Port "qvo8cbe7bb6-18"
tag: 2
Interface "qvo8cbe7bb6-18"
Port "qvoa6f975bb-60"
tag: 1
Interface "qvoa6f975bb-60"
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
ovs_version: "2.0.2"
root@network:/home/mycisco# neutron net-list +--------------------------------------+---------------+-----------------------------------------------------+ | id | name | subnets | +--------------------------------------+---------------+-----------------------------------------------------+ | 224f79e0-8068-4ef7-8c79-2326bc0cf5c4 | mgmt-net | 410a9d7c-6ae8-4059-954e-973529a933c5 10.10.30.0/24 | | 4e3f621f-d6b9-4438-be58-aa51d3c2e061 | private2 | f380d778-d576-4179-96df-cb3a2618dbd3 10.10.20.0/24 | | 5286db3e-0025-4650-8a4d-3e3e1dd9f70c | demo_internal | f43edf85-594e-4eb1-bd02-adc90c47ada1 10.90.10.0/24 | | 70f7aa46-b66b-455e-896e-05f94a08fcb8 | private1 | dcdd699a-c4b0-4360-9219-c62cb6ed0b3e 10.10.10.0/24 | | bbf38334-045a-4124-859d-419e7c7abbda | internet | 7b2cd216-1251-4eaa-9f05-b8cadc82bc1e 192.168.0.0/24 | +--------------------------------------+---------------+-----------------------------------------------------+ root@network:/home/mycisco# source creds_demo root@network:/home/mycisco# neutron net-list +--------------------------------------+---------------+-----------------------------------------------------+ | id | name | subnets | +--------------------------------------+---------------+-----------------------------------------------------+ | 5286db3e-0025-4650-8a4d-3e3e1dd9f70c | demo_internal | f43edf85-594e-4eb1-bd02-adc90c47ada1 10.90.10.0/24 | | bbf38334-045a-4124-859d-419e7c7abbda | internet | 7b2cd216-1251-4eaa-9f05-b8cadc82bc1e 192.168.0.0/24 | +--------------------------------------+---------------+-----------------------------------------------------+ root@network:/home/mycisco#
两个租户之间网络是完全不通和隔离的，且连个租户内的虚机实例所使用的安全组规则也是相互独立的，iptables利用为不同的qbr***设置各自独立的规则：
root@compute:/home/mycisco# brctl show
bridge name bridge id STP enabled interfaces
qbr2e48b70e-e1 8000.ee0883a9a73e no qvb2e48b70e-e1
tap2e48b70e-e1
qbr8cbe7bb6-18 8000.5e56d69f355c no qvb8cbe7bb6-18
tap8cbe7bb6-18
qbra6f975bb-60 8000.fe4f362748db no qvba6f975bb-60
virbr0 8000.000000000000 yes
root@compute:/home/mycisco#
root@compute:/home/mycisco#
root@compute:/home/mycisco# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
neutron-openvswi-INPUT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:53
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:67
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
Chain FORWARD (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- 0.0.0.0/0 0.0.0.0/0
neutron-openvswi-FORWARD all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED
ACCEPT all -- 192.168.122.0/24 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
neutron-filter-top all -- 0.0.0.0/0 0.0.0.0/0
neutron-openvswi-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:68
Chain neutron-filter-top (2 references)
target prot opt source destination
neutron-openvswi-local all -- 0.0.0.0/0 0.0.0.0/0
Chain neutron-openvswi-FORWARD (1 references)
target prot opt source destination
neutron-openvswi-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tapa6f975bb-60 --physdev-is-bridged
neutron-openvswi-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tapa6f975bb-60 --physdev-is-bridged
neutron-openvswi-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap2e48b70e-e1 --physdev-is-bridged
neutron-openvswi-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2e48b70e-e1 --physdev-is-bridged
neutron-openvswi-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap8cbe7bb6-18 --physdev-is-bridged
neutron-openvswi-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap8cbe7bb6-18 --physdev-is-bridged
Chain neutron-openvswi-INPUT (1 references)
target prot opt source destination
neutron-openvswi-oa6f975bb-6 all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tapa6f975bb-60 --physdev-is-bridged
neutron-openvswi-o2e48b70e-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2e48b70e-e1 --physdev-is-bridged
neutron-openvswi-o8cbe7bb6-1 all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap8cbe7bb6-18 --physdev-is-bridged
Chain neutron-openvswi-OUTPUT (1 references)
target prot opt source destination
Chain neutron-openvswi-i2e48b70e-e (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID
RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
RETURN icmp -- 0.0.0.0/0 0.0.0.0/0
RETURN udp -- 10.90.10.3 0.0.0.0/0 udp spt:67 dpt:68
neutron-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0
Chain neutron-openvswi-i8cbe7bb6-1 (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID
RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
RETURN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
RETURN icmp -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 10.10.20.1 0.0.0.0/0
RETURN all -- 10.10.30.1 0.0.0.0/0
RETURN all -- 10.10.20.7 0.0.0.0/0
RETURN all -- 10.10.10.1 0.0.0.0/0
RETURN udp -- 10.10.10.2 0.0.0.0/0 udp spt:67 dpt:68
neutron-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0
Chain neutron-openvswi-ia6f975bb-6 (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID
RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
RETURN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
RETURN icmp -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 10.10.20.1 0.0.0.0/0
RETURN all -- 10.10.30.1 0.0.0.0/0
RETURN all -- 10.10.10.7 0.0.0.0/0
RETURN all -- 10.10.10.1 0.0.0.0/0
RETURN udp -- 10.10.20.2 0.0.0.0/0 udp spt:67 dpt:68
neutron-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0
Chain neutron-openvswi-local (1 references)
target prot opt source destination
Chain neutron-openvswi-o2e48b70e-e (2 references)
target prot opt source destination
RETURN udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67
neutron-openvswi-s2e48b70e-e all -- 0.0.0.0/0 0.0.0.0/0
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID
RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
RETURN all -- 0.0.0.0/0 0.0.0.0/0
neutron-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0
Chain neutron-openvswi-o8cbe7bb6-1 (2 references)
target prot opt source destination
RETURN udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67
neutron-openvswi-s8cbe7bb6-1 all -- 0.0.0.0/0 0.0.0.0/0
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID
RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
RETURN icmp -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
neutron-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0
Chain neutron-openvswi-oa6f975bb-6 (2 references)
target prot opt source destination
RETURN udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67
neutron-openvswi-sa6f975bb-6 all -- 0.0.0.0/0 0.0.0.0/0
DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68
DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID
RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
RETURN icmp -- 0.0.0.0/0 0.0.0.0/0
RETURN all -- 0.0.0.0/0 0.0.0.0/0
neutron-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0
Chain neutron-openvswi-s2e48b70e-e (1 references)
target prot opt source destination
RETURN all -- 10.90.10.2 0.0.0.0/0 MAC FA:16:3E:99:88:5A
DROP all -- 0.0.0.0/0 0.0.0.0/0
Chain neutron-openvswi-s8cbe7bb6-1 (1 references)
target prot opt source destination
RETURN all -- 10.10.10.7 0.0.0.0/0 MAC FA:16:3E:32:28:AF
DROP all -- 0.0.0.0/0 0.0.0.0/0
Chain neutron-openvswi-sa6f975bb-6 (1 references)
target prot opt source destination
RETURN all -- 10.10.20.7 0.0.0.0/0 MAC FA:16:3E:CB:91:43
DROP all -- 0.0.0.0/0 0.0.0.0/0
Chain neutron-openvswi-sg-chain (6 references)
target prot opt source destination
neutron-openvswi-ia6f975bb-6 all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tapa6f975bb-60 --physdev-is-bridged
neutron-openvswi-oa6f975bb-6 all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tapa6f975bb-60 --physdev-is-bridged
neutron-openvswi-i2e48b70e-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap2e48b70e-e1 --physdev-is-bridged
neutron-openvswi-o2e48b70e-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2e48b70e-e1 --physdev-is-bridged
neutron-openvswi-i8cbe7bb6-1 all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap8cbe7bb6-18 --physdev-is-bridged
neutron-openvswi-o8cbe7bb6-1 all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap8cbe7bb6-18 --physdev-is-bridged
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain neutron-openvswi-sg-fallback (6 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0
root@compute:/home/mycisco#
所以网络的逻辑结构实际如下：
此时如果希望两个租户之间打通网络，可以通过外部网络路由将两个租户连接起来，或在两个虚拟路由器上设置路由指向彼此。
root@network:/home/mycisco# ip netns exec qrouter-040c1455-6096-4806-ba91-fec64cdaed81 netstat -nr Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 qg-6f726a6c-41 10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-d8fd27a6-d1 10.10.20.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-09208a89-0b 10.10.30.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-4debe62b-e4 10.90.10.0 192.168.0.4 255.255.255.0 UG 0 0 0 qg-6f726a6c-41 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-6f726a6c-41
root@network:/home/mycisco# ip netns exec qrouter-015840ab-38ed-4086-b78e-696664d59281 netstat -nr Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 qg-814e9e39-99 10.10.10.0 192.168.0.2 255.255.255.0 UG 0 0 0 qg-814e9e39-99 10.90.10.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-219fc331-d3 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-814e9e39-99
文章评论