在上述两篇文章中描述的是单租户情况下同网段或不同网段间数据是如何走的,以及描述一个数据包是如何去往其他datacenter网络或互联网的。
openstack L3-GRE 网络结构分析记录 (Icehouse) 第一篇
openstack L3-GRE 网络结构分析记录 (Icehouse) 第二篇
本篇博文记录描述多租户情形下的网络及数据流情况。
首先,新建一个租户,并在该租户上下文下创建一个路由器和一个私网:
可以看出,两个租户共享了internet网络,这是因为在创建internet网络时,该网络是一个科共享的网络,即所有租户都可以使用。 两个租户的私有网络则完全不同,且两个网络拥有独立的虚拟路由器。
此时,在网络节点上,与前几篇文章描述的差异不大,neutron的l3agent会为每个租户创建独立的虚拟路由器,通过namespace隔离:
1 2 3 4 5 6 7 |
root@network:/home/mycisco# ip netns show qrouter-015840ab-38ed-4086-b78e-696664d59281 qdhcp-5286db3e-0025-4650-8a4d-3e3e1dd9f70c qrouter-040c1455-6096-4806-ba91-fec64cdaed81 qdhcp-224f79e0-8068-4ef7-8c79-2326bc0cf5c4 qdhcp-70f7aa46-b66b-455e-896e-05f94a08fcb8 qdhcp-4e3f621f-d6b9-4438-be58-aa51d3c2e061 |
上面 ***9281的虚拟路由器则是demo租户的。
而从openvswitch的角度来看,与之前的结构一样,只是多出一个port来连接demo租户的internal网络,对ovs来说没有租户的概念,租户和ovs中哪个网络连接是有neutron来负责的。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 |
root@network:/home/mycisco# ovs-vsctl show 909c85d0-ff4e-446b-bf8a-9166f0fccd24 Bridge br-int fail_mode: secure Port "taped034aae-11" tag: 1 Interface "taped034aae-11" type: internal Port "tap78dd7f1e-53" tag: 2 Interface "tap78dd7f1e-53" type: internal Port "qr-219fc331-d3" tag: 4 Interface "qr-219fc331-d3" type: internal Port "qr-09208a89-0b" tag: 1 Interface "qr-09208a89-0b" type: internal Port "tap1b1b29de-97" tag: 4 Interface "tap1b1b29de-97" type: internal Port "qr-d8fd27a6-d1" tag: 2 Interface "qr-d8fd27a6-d1" type: internal Port "tap708ee330-0c" tag: 3 Interface "tap708ee330-0c" type: internal Port br-int Interface br-int type: internal Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port "qr-4debe62b-e4" tag: 3 Interface "qr-4debe62b-e4" type: internal Bridge br-ex Port "qg-6f726a6c-41" Interface "qg-6f726a6c-41" type: internal Port "eth2" Interface "eth2" Port br-ex Interface br-ex type: internal Port "qg-814e9e39-99" Interface "qg-814e9e39-99" type: internal Bridge br-tun Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Port "gre-0a0a648c" Interface "gre-0a0a648c" type: gre options: {in_key=flow, local_ip="10.10.100.139", out_key=flow, remote_ip="10.10.100.140"} Port br-tun Interface br-tun type: internal ovs_version: "2.0.2" |
计算节点情况:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
root@compute:/home/mycisco# ovs-vsctl show b5553502-95e4-4ad2-90a6-a3da02d3819d Bridge br-tun Port br-tun Interface br-tun type: internal Port "gre-0a0a648b" Interface "gre-0a0a648b" type: gre options: {in_key=flow, local_ip="10.10.100.140", out_key=flow, remote_ip="10.10.100.139"} Port patch-int Interface patch-int type: patch options: {peer=patch-tun} Bridge br-int fail_mode: secure Port br-int Interface br-int type: internal Port "qvo2e48b70e-e1" tag: 3 Interface "qvo2e48b70e-e1" Port "qvo8cbe7bb6-18" tag: 2 Interface "qvo8cbe7bb6-18" Port "qvoa6f975bb-60" tag: 1 Interface "qvoa6f975bb-60" Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} ovs_version: "2.0.2" |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
root@network:/home/mycisco# neutron net-list +--------------------------------------+---------------+-----------------------------------------------------+ | id | name | subnets | +--------------------------------------+---------------+-----------------------------------------------------+ | 224f79e0-8068-4ef7-8c79-2326bc0cf5c4 | mgmt-net | 410a9d7c-6ae8-4059-954e-973529a933c5 10.10.30.0/24 | | 4e3f621f-d6b9-4438-be58-aa51d3c2e061 | private2 | f380d778-d576-4179-96df-cb3a2618dbd3 10.10.20.0/24 | | 5286db3e-0025-4650-8a4d-3e3e1dd9f70c | demo_internal | f43edf85-594e-4eb1-bd02-adc90c47ada1 10.90.10.0/24 | | 70f7aa46-b66b-455e-896e-05f94a08fcb8 | private1 | dcdd699a-c4b0-4360-9219-c62cb6ed0b3e 10.10.10.0/24 | | bbf38334-045a-4124-859d-419e7c7abbda | internet | 7b2cd216-1251-4eaa-9f05-b8cadc82bc1e 192.168.0.0/24 | +--------------------------------------+---------------+-----------------------------------------------------+ root@network:/home/mycisco# source creds_demo root@network:/home/mycisco# neutron net-list +--------------------------------------+---------------+-----------------------------------------------------+ | id | name | subnets | +--------------------------------------+---------------+-----------------------------------------------------+ | 5286db3e-0025-4650-8a4d-3e3e1dd9f70c | demo_internal | f43edf85-594e-4eb1-bd02-adc90c47ada1 10.90.10.0/24 | | bbf38334-045a-4124-859d-419e7c7abbda | internet | 7b2cd216-1251-4eaa-9f05-b8cadc82bc1e 192.168.0.0/24 | +--------------------------------------+---------------+-----------------------------------------------------+ root@network:/home/mycisco# |
两个租户之间网络是完全不通和隔离的,且连个租户内的虚机实例所使用的安全组规则也是相互独立的,iptables利用为不同的qbr***设置各自独立的规则:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 |
root@compute:/home/mycisco# brctl show bridge name bridge id STP enabled interfaces qbr2e48b70e-e1 8000.ee0883a9a73e no qvb2e48b70e-e1 tap2e48b70e-e1 qbr8cbe7bb6-18 8000.5e56d69f355c no qvb8cbe7bb6-18 tap8cbe7bb6-18 qbra6f975bb-60 8000.fe4f362748db no qvba6f975bb-60 virbr0 8000.000000000000 yes root@compute:/home/mycisco# root@compute:/home/mycisco# root@compute:/home/mycisco# iptables -L -n Chain INPUT (policy ACCEPT) target prot opt source destination neutron-openvswi-INPUT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:53 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:67 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:67 Chain FORWARD (policy ACCEPT) target prot opt source destination neutron-filter-top all -- 0.0.0.0/0 0.0.0.0/0 neutron-openvswi-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination neutron-filter-top all -- 0.0.0.0/0 0.0.0.0/0 neutron-openvswi-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:68 Chain neutron-filter-top (2 references) target prot opt source destination neutron-openvswi-local all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-openvswi-FORWARD (1 references) target prot opt source destination neutron-openvswi-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tapa6f975bb-60 --physdev-is-bridged neutron-openvswi-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tapa6f975bb-60 --physdev-is-bridged neutron-openvswi-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap2e48b70e-e1 --physdev-is-bridged neutron-openvswi-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2e48b70e-e1 --physdev-is-bridged neutron-openvswi-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap8cbe7bb6-18 --physdev-is-bridged neutron-openvswi-sg-chain all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap8cbe7bb6-18 --physdev-is-bridged Chain neutron-openvswi-INPUT (1 references) target prot opt source destination neutron-openvswi-oa6f975bb-6 all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tapa6f975bb-60 --physdev-is-bridged neutron-openvswi-o2e48b70e-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2e48b70e-e1 --physdev-is-bridged neutron-openvswi-o8cbe7bb6-1 all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap8cbe7bb6-18 --physdev-is-bridged Chain neutron-openvswi-OUTPUT (1 references) target prot opt source destination Chain neutron-openvswi-i2e48b70e-e (1 references) target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED RETURN icmp -- 0.0.0.0/0 0.0.0.0/0 RETURN udp -- 10.90.10.3 0.0.0.0/0 udp spt:67 dpt:68 neutron-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-openvswi-i8cbe7bb6-1 (1 references) target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED RETURN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 RETURN icmp -- 0.0.0.0/0 0.0.0.0/0 RETURN all -- 10.10.20.1 0.0.0.0/0 RETURN all -- 10.10.30.1 0.0.0.0/0 RETURN all -- 10.10.20.7 0.0.0.0/0 RETURN all -- 10.10.10.1 0.0.0.0/0 RETURN udp -- 10.10.10.2 0.0.0.0/0 udp spt:67 dpt:68 neutron-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-openvswi-ia6f975bb-6 (1 references) target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED RETURN tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 RETURN icmp -- 0.0.0.0/0 0.0.0.0/0 RETURN all -- 10.10.20.1 0.0.0.0/0 RETURN all -- 10.10.30.1 0.0.0.0/0 RETURN all -- 10.10.10.7 0.0.0.0/0 RETURN all -- 10.10.10.1 0.0.0.0/0 RETURN udp -- 10.10.20.2 0.0.0.0/0 udp spt:67 dpt:68 neutron-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-openvswi-local (1 references) target prot opt source destination Chain neutron-openvswi-o2e48b70e-e (2 references) target prot opt source destination RETURN udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67 neutron-openvswi-s2e48b70e-e all -- 0.0.0.0/0 0.0.0.0/0 DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED RETURN all -- 0.0.0.0/0 0.0.0.0/0 neutron-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-openvswi-o8cbe7bb6-1 (2 references) target prot opt source destination RETURN udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67 neutron-openvswi-s8cbe7bb6-1 all -- 0.0.0.0/0 0.0.0.0/0 DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED RETURN icmp -- 0.0.0.0/0 0.0.0.0/0 RETURN all -- 0.0.0.0/0 0.0.0.0/0 neutron-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-openvswi-oa6f975bb-6 (2 references) target prot opt source destination RETURN udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:68 dpt:67 neutron-openvswi-sa6f975bb-6 all -- 0.0.0.0/0 0.0.0.0/0 DROP udp -- 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:68 DROP all -- 0.0.0.0/0 0.0.0.0/0 state INVALID RETURN all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED RETURN icmp -- 0.0.0.0/0 0.0.0.0/0 RETURN all -- 0.0.0.0/0 0.0.0.0/0 neutron-openvswi-sg-fallback all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-openvswi-s2e48b70e-e (1 references) target prot opt source destination RETURN all -- 10.90.10.2 0.0.0.0/0 MAC FA:16:3E:99:88:5A DROP all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-openvswi-s8cbe7bb6-1 (1 references) target prot opt source destination RETURN all -- 10.10.10.7 0.0.0.0/0 MAC FA:16:3E:32:28:AF DROP all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-openvswi-sa6f975bb-6 (1 references) target prot opt source destination RETURN all -- 10.10.20.7 0.0.0.0/0 MAC FA:16:3E:CB:91:43 DROP all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-openvswi-sg-chain (6 references) target prot opt source destination neutron-openvswi-ia6f975bb-6 all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tapa6f975bb-60 --physdev-is-bridged neutron-openvswi-oa6f975bb-6 all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tapa6f975bb-60 --physdev-is-bridged neutron-openvswi-i2e48b70e-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap2e48b70e-e1 --physdev-is-bridged neutron-openvswi-o2e48b70e-e all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap2e48b70e-e1 --physdev-is-bridged neutron-openvswi-i8cbe7bb6-1 all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap8cbe7bb6-18 --physdev-is-bridged neutron-openvswi-o8cbe7bb6-1 all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap8cbe7bb6-18 --physdev-is-bridged ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 Chain neutron-openvswi-sg-fallback (6 references) target prot opt source destination DROP all -- 0.0.0.0/0 0.0.0.0/0 root@compute:/home/mycisco# |
所以网络的逻辑结构实际如下:
此时如果希望两个租户之间打通网络,可以通过外部网络路由将两个租户连接起来,或在两个虚拟路由器上设置路由指向彼此。
1 2 3 4 5 6 7 8 9 |
root@network:/home/mycisco# ip netns exec qrouter-040c1455-6096-4806-ba91-fec64cdaed81 netstat -nr Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 qg-6f726a6c-41 10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-d8fd27a6-d1 10.10.20.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-09208a89-0b 10.10.30.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-4debe62b-e4 10.90.10.0 192.168.0.4 255.255.255.0 UG 0 0 0 qg-6f726a6c-41 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-6f726a6c-41 |
1 2 3 4 5 6 7 |
root@network:/home/mycisco# ip netns exec qrouter-015840ab-38ed-4086-b78e-696664d59281 netstat -nr Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 qg-814e9e39-99 10.10.10.0 192.168.0.2 255.255.255.0 UG 0 0 0 qg-814e9e39-99 10.90.10.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-219fc331-d3 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-814e9e39-99 |
文章评论