AN-003 Deploying aggregators on Microsoft Azure

Microsoft Azure is a popular cloud computing platform similar to Amazon Web Services that allows for quick deployment and management of Internet-connected servers. It is most popular for hosting web sites, but can be used to deploy aggregators as well, with some caveats.

The issues described here may also be applicable to other cloud providers that use private networking.

Installing the aggregator

Currently, aggregators are only supported using Debian 8 “Jessie”. When creating the instance, ensure the Debian 8 image is used. Once installed, follow the instructions here to complete the setup.

Networking considerations

The main issue with IPv4 networking in Azure is that there is no option for routing subnets to hosts. You can only allocate single IPs, and those IP addresses are randomly assigned, so it is not possible to even emulate subnet routing with IPv4. It can be emulated with IPv6 addresses, but each address in each range will need to be assigned to the aggregator interface, which is burdensome.

The other issue is that public IPv4 addresses are never assigned directly to instance interfaces. All public IPv4 addresses are assigned to a private address and translated using NAT. While aggregators are capable of handling tunnels in such a scenario, it makes bond routing complicated

However, even with the above limitations, there are some scenarios that work if set up in a particular way.

CPE NAT IP to individual bonds

This scenario involves providing individual per-bond access to the Internet, each with its own public IP address.

In a typical deployment, a subnet which contains the IPs available for use for CPE NAT IPs would be routed to the aggregator network, and the aggregator would announce each individual IP via OSPF or iBGP. If dynamic routing was not available, static routes could be made to the individual aggregators hosting the bonds, but that would require manual routing changes if a bond was moved.

In Azure, each public IP addresses is assigned to a private IP address, which would presumably be configured on an interface. By default, when a virtual instance is created, a network interface is created, a dynamically chosen private IP address is assigned to and configured on the interface, and a public IP address is associated with the private IP address.

For an aggregator, we can use this address for its primary communication, but we will need additional addresses for each CPE NAT IP. Azure public IP addresses can be used to provide these additional addresses.

Adding the Public IP Address

To add a new IP address in the Azure portal, perform the following steps:

  1. Navigate to the virtual machine running the aggregator the CPE NAT IP will be assigned to
  2. On the left menu, select Networking
  3. Click the link to the network interface near the top
  4. Select IP configurations on the left menu
  5. Click Add
  6. Set the Name
  7. Set Public IP address to Enabled
  8. Click Configure required settings under IP address
  9. Click Create new
  10. Set the name to anything. The same value as for the private IP address name given above is advised
  11. Set Assignment to Static
  12. Click OK
  13. Take note of the newly assigned private IP address and its associated public IP address

Note

You do not need to configure the private or public address in the aggregator’s network configuration

Configuring the CPE NAT IP

On a bond assigned to the aggregator, configure a private connected IP and add the CPE NAT IP using the private IP address we added to the interface. The Azure gateway will NAT the public IP address to the private CPE NAT IP address, and the bonder will NAT the private IP address to the connected IP.

Assuming the bonder is online, all of the routing will be in place.

Note

Due to the lack of dynamic routing for public IP addresses in Azure, moving bonds between aggregators will not normally work unless the static associations are updated via the Azure portal or API. As a consequence, aggregator failover will not work in this scenario.

Optional: Route CPE NAT IPs across a group of aggregators

Since the public IPs are associated with interfaces on specific instances, each bond must be assigned to a specific aggregator or routing will break if moved. However, as of SD-WAN 6.5, we can actually allow bonds to move to other aggregators without needing to update the association, at least not right away.

With SD-WAN 6.5 we can implement custom routing protocols to allow for an aggregator with associated IP to itself route to the correct aggregator.

In a typical data center deployment, CPE NAT IP routes are distributed across the main physical segment of the aggregators to a local router via a protocol such as BGP, OSPF, or Babel. Unfortunately Azure does not allow such peering and addresses must be routed statically via the Azure API or portal. We can, however, peer the aggregators with each other using BGP so that bonds can move between aggregators without breaking routing. Note that the Azure virtual networks do not allow multicast traffic, so OSPF and Babel will not work unless tunneled.

See here for more information on setting up BGP between aggregators. Note that CPE NAT IPs are always exposed in the main routing table, so the protocols should not have a space defined.

Private WAN

Note

The With private WAN routers mode of private WAN will not work with Azure since the GRE protocol used in that mode is blocked within Azure virtual networks. Use the “Managed mesh” or “Unmanaged” mode instead.

Private WAN can also be deployed in Azure, but due to the lack of layer-2 networking, the private WAN traffic will have to transit across VXLANs to get to other aggregators. This means using the Managed mesh or Unmanaged private WAN modes.

Also, note that Azure does not allow aggregators to connect to multiple virtual networks even with additional interfaces. To get around this we will need to set up VXLAN peering from the aggregator to a virtual machine hosted within the target resource group.

Due to these complications, it is highly recommended to simply place dedicated aggregators for each customer into their own Azure resource groups, disable private WAN, and set up peering between aggregators via BGP and static routing.

If you do need to extend a space into a network due to the need to integrate with existing aggregators, or in order to use larger sized aggregators for multiple spaces, follow the steps in this section.

Adding a new resource group for the space

If the customer does not already have it’s own Azure account or resource group, a new one will need to be added:

  1. Navigate to All services
  2. Under General select Resource groups
  3. Click Add
  4. Set Resource group to a descriptive name
  5. Set Region appropriately
  6. Click Review + create
  7. Click Create

Adding a VXLAN endpoint

We need to create a VXLAN endpoint on the target resource group. This machine can be any Linux distribution or router with VXLAN support. However, the easiest option is to simply add an aggregator to a private WAN in Managed mesh mode but without adding any bonds. The main interface will be used for the VXLAN traffic, while an additional interface will be used to carry the internal private WAN traffic.

Note

If you need to provide public Internet access for the private WAN, consider using a bare Linux distribution or a dedicated router distribution such as VyOS. They will be able to terminate the VXLAN, route the Azure networks, and NAT the private WAN networks. Setup of such a gateway is beyond the scope of this document.

Since this aggregator does not handle bonds, it does not have the same CPU and memory requirements of a normal aggregator. Any small instance should suffice.

Once the instance has been created, we will need to add a secondary interface:

  1. Navigate to the virtual machine for the endpoint
  2. At the top of the page, click Stop
  3. Wait for the machine to be stopped
  4. On the left menu, select Networking
  5. At the top of the page, click Attach network interface
  6. Click Create network interface
  7. Set Name
  8. Click Create
  9. Wait for the interface to deploy. This will take a minute or so and a notification will be displayed when it’s done
  10. Navigate to the newly created interface
  11. On the left menu, select IP configurations
  12. Set IP forwarding to Enabled
  13. Click Save
  14. Go back to the Attach network interface dialog for the virtual machine
  15. Select the newly created interface
  16. Click OK
  17. Wait for the interface to be added. A notification will be displayed
  18. Navigate back to Networking for the virtual machine
  19. Select the tab for the new interface
  20. Take note of the assigned private IP
  21. Navigate back to the virtual machine
  22. Click Start

Next, we need to set up the secondary interface on VXLAN peer. If using an aggregator, this can be configured in bondingadmin:

  1. Navigate to the aggregator defined for the virtual network
  2. Click the interfaces Add button
  3. Set Interface name to the second interface. It will likely be called eth1
  4. Set Space to the desired space
  5. Click Add IP address
  6. Set the private IP address assigned to the interface by Azure
  7. Click Add

Routing between Azure virtual networks and private WAN networks

Next, you will need to add routes on the private WAN as well as the Azure portal to ensure hosts can talk to each other.

When we added the interface to the VXLAN endpoint aggregator, it automatically injected the route for the subnet the interface’s IP is in. If all of the hosts the private WAN needs to access are within that subnet we don’t need to set up any additional routes in bondingadmin. However, if there are additional subnets in Azure that need to be accessed, we can set up a static protocol to inject those routes:

  1. Navigate to the VXLAN endpoint aggregator defined for the virtual network
  2. Click the protocols Add button
  3. Set Name
  4. Set Space to the desired space
  5. Set Protocol to Static
  6. For every additional subnet that needs to be routed:
    1. Click Add route
    2. Set Network to the subnet
    3. Set Destination to Gateway
    4. Set Address to the first address in the interface’s subnet

On the Azure side, routes for the bonds in the private WAN will need to be added to the Azure portal:

  1. Navigate to All services
  2. Under Networking select Route tables
  3. If no route table is associated with the resource group:
    1. Click Create route table
    2. Set Name
    3. Set Resource group to the resource group used by the space
    4. Click Create
    5. Wait for the route table to be created
  4. Navigate to the route table for the resource group
  5. In the left menu, select Routes
  6. For each route inside the private space that needs to access resources in the Azure resource group:
    1. Click Add
    2. Set Route name
    3. Set Address prefix to the subnet
    4. Set Next hop type to Virtual appliance
    5. Set Next hop address to the IP address configured on the secondary interface
    6. Click OK
  7. In the left menu, select Subnets
  8. Click Associate
  9. Set Virtual network to the virtual network associated with the space’s resource group
  10. Select all subnets that need to be routed to and from the private WAN networks
  11. Click OK

Optional: Add direct VXLAN peering

Note

This is optional for the Managed mesh mode, but required for the Unmanaged mode. Also, it is probably unnecessary unless you are routing large amounts of traffic over the private WAN.

Since we know what the private IP addresses of the aggregators are, we can provide a more direct path between the aggregators by adding custom VXLANs to the involved aggregators within the Azure virtual networks.

A unique VNI should be created for each space. Each VXLAN for the space should use that assigned VNI. Failure to do so will result in broken routing or unexpected cross routing.

Also, choose a unique subnet within the private WAN to assign to the VXLAN interfaces. Each aggregator will need a unique address in the subnet.

For the routing protocol, we will use Babel. OSPF will also work, but Babel is a more modern protocol that is better suited to this kind of application and is simpler to implement.

On each aggregator, for each space, do the following:

  1. Click the interfaces Add button:
    1. Set Type to VXLAN
    2. Set Interface name to something unique per space
    3. Set Space to the desired space
    4. Set the VNI to the unique VNI chosen for the space
    5. Click Add IP address and set a unique IP address n the subnet chosen for the space
    6. For all other aggregators involved in the space, click Add endpoint and set the aggregator’s private IP address and the VNI chosen for the space
  2. Click the protocols Add button:
    1. Set the Name
    2. Set Space to the desired space
    3. Set Protocol to Babel
    4. Click Add interface
    5. Click Add interface pattern
    6. Set Pattern 1 to the interface name chosen for the interface in the space