Neutron Networking: VLAN Provider Networks

In this multi-part blog series I intend to dive into the various components of the OpenStack Neutron project, and to also provide working examples of networking configurations for clouds built with Rackspace Private Cloud powered by OpenStackon Ubuntu 12.04 LTS.

In the previous installment, Neutron Networking: Simple Flat Network, I demonstrated an easy method of providing connectivity to instances using an untagged flat network. In this third installment, I’ll describe how to build multiple provider networks using 802.1q vlan tagging.<!-- more -->

Getting Started / VLAN vs Flat Design

One of the negative aspects of a flat network is that it’s one large broadcast domain. Virtual Local Area Networks, or VLANs, aim to solve this problem by creating smaller, more manageable broadcast domains. From a security standpoint, flat networks provide malicious users the potential to see the entire network from a single host.

VLAN segregation is often used in a web hosting environment where there’s one vlan for web servers (DMZ) and another for database servers (INSIDE). Neither network can communicate directly without a routing device to route between them. With proper security mechanisms in place, if a server becomes compromised in the DMZ it does not have the ability to determine or access the resources in the INSIDE vlan.

The diagrams below are examples of traditional flat and vlan-segregated networks:

VLAN Tagging / What is it and how does it work?

At a basic level on a Cisco switch there are two types of switchports: access ports and trunk ports. Switchports configured as access ports are placed into a single vlan and can communicate with other switchports in the same vlan. Switchports configured as trunks allow traffic from multiple vlans to traverse a single interface. The switch adds a tag to the Ethernet frame that contains the corresponding vlan ID as the frame enters the trunk. As the frame exits the trunk on the other side, the vlan tag is stripped and the traffic forwarded to its destination. Common uses of trunk ports include uplinks to other switches and more importantly in our case, hypervisors serving virtual machines from various networks.

VLAN Tagging / How does this apply to Neutron?

In the previous installment I discussed flat networks and their lack of vlan tagging. All hosts in the environment were connected to access ports in the same vlan, thereby allowing hosts and instances to communicate with one another on the same network. VLANs allow us to not only separate host and instance traffic, but to also create multiple networks for instances similar to the DMZ and INSIDE scenarios above.

Neutron allows users to create multiple provider or tenant networks using vlan IDs that correspond to real vlans in the data center. A single OVS bridge can be utilized by multiple provider and tenant networks using different vlan IDs, allowing instances to communicate with other instances across the environment, and also with dedicated servers, firewalls, load balancers and other networking gear on the same Layer 2 vlan.

Networking / Layout

For this installment, a Cisco ASA 5510 will once again serve as the lead gateway device. In fact, I’ll be building upon the configuration already in place from the flat networking demonstration in the previous installment. 10.240.0.0/24 will continue to serve as the management network for hosts and the flat provider network, and two new provider networks will be created:

• VLAN 100 - MGMT - 10.240.0.0/24 (Existing)
• VLAN 200 – DMZ – 192.168.100.0/24 (NEW)
• VLAN 300 – INSIDE – 172.16.0.0/24 (NEW)

A single interface on the servers will be used for both management and provider network connectivity.

Networking / Configuration of Network Devices

The Cisco ASA was chosen for this example since it’s readily available at my location. Interface e0/0 should be configured with your WAN IP. Interface e0/1 will be split into sub-interfaces in my example, but you can leverage separate interfaces if you’ve got them. e0/1.100 must be configured with an IP address that will serve as the gateway for hosts and instances using the flat provider network:

e0/1.200 and e0/1.300 will be utilized for the new INSIDE and DMZ vlans, respectively:

Based on the diagram above, the firewall’s switchport configuration may look something like this:

Server switchports will look similar, but vlan 100 will need to be the native vlan (untagged) due to its use as both a flat provider network and the management network:

Networking / Configuration of Servers

If you followed the previous installment and successfully configured a bridge in OVS and a flat provider network, the proper server network configuration should already be in place. If you haven’t, there’s a good chance your servers have their IP configured directly on eth0. To utilize Neutron, the hosts must have a network bridge configured. This can be accomplished one of two ways:

• Utilize a single interface for both management of the hosts and the network bridge
• Utilize one interface for management of the hosts, and another for the network bridge(s)

For this example, I’ll use a single interface and move the IP address from eth0 to the bridge.

Below is what a default eth0 configuration might look like:

In order to configure the bridge, eth0 must be modified and the bridge interface configured:

TIP: Do not set br-eth0 to auto. Because of the order that processes are started at boot, the interface must be brought up using rc.local. Instead, edit the /etc/rc.local file of each machine and add the following line before the 'exit 0' statement:

Networking / Open vSwitch Configuration

Creating the network bridge in Open vSwitch is a requirement for proper management of traffic. The previous installment covered this, so it may not be necessary to configure if it’s still in place.

The following will create a bridge called ‘br-eth0’ and place physical interface eth0 inside:

The creation and configuration of the bridge enables the instances to communicate on the network. At a minimum, bridges must be configured on the compute and network nodes. Since we're running the network services on the controller instead of using a dedicated network node, the compute and controller nodes should have the bridge configured.

Changes to Environment (RPC v4)

When using RPC v4, most configuration changes are handled via Chef. A few changes must be made to the environment file to utilize the bridge for Neutron networking. If using RPC v4.2 (Havana), all quantum references should be changed to neutron.

Look for the section 'quantum : ovs : provider_networks'

The bridge configuration should be modified to mirror the following example, if it doesn't already exist:

The “vlans” value above represents a range of vlans available for the automatic provisioning of vlan segmentation IDs to networks created in Neutron. A range of IDs isn’t necessary if the intent is to always pass an ID manually with the net-create command. If you’d prefer Neutron to handle the ID assignment automatically, which includes networks created through Horizon, the “vlans” value can be modified appropriately:

In the above example, only vlans 200 and 300 are available for automatic assignment. Non-contiguous vlan ranges can be specified and separated by commas. Neutron will assign the next available ID from the list when a segmentation ID is not passed in the net-create command.

The label ‘ph-eth0’ represents our provider interface, in this case the bridge ‘br-eth0’. It will be used during the creation of networks in Neutron.

The resulting file would look something like this:

Save the changes to the file and run chef-client on all the hosts to populate the changes.

Changes to Environment (Other)

When using something other than RPC v4, such as the vanilla OpenStack, configuration changes must be made to the appropriate configuration files and services restarted manually.

Within /etc/quantum/plugins/openvswitch/ovsquantumplugin.ini, update the following values:

The label ‘ph-eth0’ represents the provider bridge, in this case the bridge ‘br-eth0’. It will be used during the creation of networks in Neutron. It’s possible to have more than one provider bridge, especially when you have multiple switching infrastructures for various networks and services. Restart all Neutron and Open vSwitch services on all hosts for the changes to take effect.

Networking / OVS Confirmation

Remember the bridge (br-eth0) we created in OVS earlier? At a high level, it can be looked at as our bridge to the physical network infrastructure. Neutron requires an ‘Integration Bridge’ that serves as the bridge to our virtual instances. The integration bridge connects vNICs and Neutron DHCP and L3 agents with virtual networks. Overriding the default value of ‘br-int’ is not recommended, as the bridge must be named the same on each host (controller/network/compute). RPC v4 creates this bridge during the chef-client run.

For those installing from source, the name of the bridge is defined in /etc/quantum/plugins/openvswitch/ovsquantumplugin.ini:

The bridge will need to be created manually in OVS if you are not using Chef:

Networking / Building a vlan provider network in Neutron

With the network devices and OVS configured, it’s time to build a vlan provider network in Neutron. Creating a vlan provider network requires at least two values: the name of the network and the provider bridge label. However, I recommend specifying the vlan segmentation ID to ensure parity with the infrastructure that was previously configured.

Syntax:quantum net-create --provider:physical_network=<provider label> --provider:network_type=vlan –provider:segmentation_id=<vlan id> <network name>

With the two networks created, it’s time to create the subnets.

Syntax:quantum subnet-create <network_name> <subnet> --name <subnet_name> --no-gateway --host-route destination=<dest_network>,nexthop=<nexthop_ip> --allocation-pool start=<dhcp_start_ip>,end=<dhcp_end_ip> --dns-nameservers list=true <dns_1> <dns_2>

It’s a good idea to provide values for the following options:

When a gateway is defined, Neutron assumes all metadata routes are handled by the specified gateway and will not inject the metadata route via DHCP to instances. By specifying the --nogateway flag and defining a default route manually, we’re able to provide instances with a default route as well as an automatic metadata route to the namespace IP._

Networking / Testing Connectivity

Now that the networks have been built in Neutron, it’s time to test connectivity by spinning up some instances. Like the hosts, my instances are running Ubuntu 12.04 LTS.

Check the Nova console logs to verify the instances received their IPs, routes and metadata. Success here indicates the instances are able to communicate with the network namespace residing on the network or controller nodes.

The ability to connect to the instances from the controller confirms that proper vlan tagging is taking place:

Because the Cisco ASA is the gateway device, all NAT translations should be configured there to provide external connectivity to/from these instances. Neutron floating IPs cannot be used in this example as they require the use of a Neutron router.

Summary

With a limited amount of networking hardware one can create a functional private cloud based on Rackspace Private Cloudpowered by OpenStack. While the use of vlan provider networks allows for more flexibility over a ‘flat’ design, it still requires interaction with hardware routers and switches for every network and subnet that gets created. In the case where the use of the L3 agent isn’t possible or necessary, vlan-based configurations can still provide users with a secure and scalable cloud experience.