AN-001 Deploying aggregators on Amazon Web Services (AWS)

Amazon Web Services (AWS) is a popular cloud computing platform that allows for quick deployment and management of Internet-connected servers. It is most popular for hosting web sites, but can be used to deploy aggregators as well, with some caveats.

The issues described here may also be applicable to other cloud providers that use private networking.

Networking considerations

The main issue with IPv4 networking in AWS is that there is no option for routing subnets to AWS hosts. You can only allocate single IPs, and those IP addresses are randomly assigned, so it is not possible to even emulate subnet routing with IPv4. It can be emulated with IPv6 addresses, but each address in each range will need to be assigned to the aggregator interface, which is burdensome.

The other issue is that public IPv4 addresses are never assigned directly to instance interfaces. All public IPv4 addresses are assigned to a private address and translated using NAT. While aggregators are capable of handling tunnels in such a scenario, it makes bond routing complicated

However, even with the above limitations, there are some scenarios that work if set up in a particular way.

CPE NAT IP to individual bonds

This scenario involves providing individual per-bond access to the Internet, each with its own public IP address.

In a typical deployment, a subnet which contains the IPs available for use for CPE NAT IPs would be routed to the aggregator network, and the aggregator would announce each individual IP via OSPF or iBGP. If dynamic routing was not available, static routes could be made to the individual aggregators hosting the bonds, but that would require manual routing changes if a bond was moved.

In AWS, each public IP addresses is assigned to a private IP address, which would presumably be configured on an interface. By default, when an AWS EC2 instance is created, an Elastic Network Interface (ENI) is created, a dynamically chosen private IP address is assigned to and configured on the interface, and a public IP address is associated with the private IP address.

For an aggregator, we can use this address for it’s primary communication, but we will need additional addresses for each CPE NAT IP. The AWS Elastic IP (EIP) can be used to provide this, but it must be carefully configured to avoid losing the original assigned public IP address.

Adding the Elastic IP Address

To add a new IP address in the AWS console, perform the following steps:

  1. Under Network & Security, select Network Interfaces

  2. Right-click the primary interface for the aggregator interface and select Manage IP Addresses

  3. Click Assign new IP

  4. Click Yes, Update

  5. Take note of the newly assigned address

  6. Under Network & Security, select Elastic IPs

  7. Click Allocate Elastic IP address

  8. Select the pool and click Allocate

  9. Click the newly assigned public address

  10. Click Associate Elastic IP address

    1. Set Resource type to Network interface

    2. Select the interface for the instance

    3. Choose the newly assigned private IP address

    4. Click Associate

Note

You do not need to configure the address in the aggregator’s network configuration

Configuring the CPE NAT IP

On a bond assigned to the aggregator, configure a private connected IP and add the CPE NAT IP using the private IP address we added to the interface. The AWS gateway will NAT the public IP address to the private CPE NAT IP, and the bonder will NAT the private IP address to the connected IP.

Assuming the bonder is online, all of the routing will be in place.

Note

Due to the lack of dynamic routing in AWS VPC, moving bonds between aggregators will not normally work unless the static associations are updated via the AWS console or API. As a consequence, aggregator failover will not work in this scenario.

Optional: Route CPE NAT IPs across a group of aggregators

Since the public IPs are associated with interfaces on specific instances, each bond must be assigned to a specific aggregator or routing will break if moved. However, as of SD-WAN 6.5, we can actually allow bonds to move to other aggregators without needing to update the association, at least not right away.

With SD-WAN 6.5 we can implement custom routing protocols to allow for an aggregator with associated IP to itself route to the correct aggregator.

In a typical data center deployment, CPE NAT IP routes are distributed across the main physical segment of the aggregators to a local router via a protocol such as BGP, OSPF, or Babel. Unfortunately AWS does not allow such peering and addresses must be routed statically via the AWS API. We can, however, peer the aggregators with each other using BGP so that bonds can move between aggregators without breaking routing. Note that the AWS VPC does not allow multicast traffic, so OSPF and Babel will not work unless tunneled.

See here for more information on setting up BGP between aggregators. Note that CPE NAT IPs are always exposed in the main routing table, so the protocols should not have a space defined.

Private WAN

Note

The With private WAN routers mode of private WAN will not work with AWS due to the NAT involved and the way that the GRE tunnels are defined. Use the “Managed mesh” or “Unmanaged” mode instead.

Private WAN can also be deployed in AWS, but due to the lack of layer-2 networking, the private WAN traffic will have to transit across VXLANs to get to other aggregators. This means using the Managed mesh or Unmanaged private WAN modes.

Adding a new VPC

Since private WAN routing needs to be isolated, we will need to create a unique VPC for each space. To do this in the AWS console:

  1. At the top of the page, click Services then VPC

  2. Under Virtual Private Cloud click Your VPCs

  3. Click Create VPC

    1. Add an IPv4 and optionally an IPv6 subnet

    2. Click Create

Configuring the private WAN mesh

The Managed mesh mode is easiest to implement, since it automatically detects which aggregators have bonds in a space and sets up VXLANs between them. However, these VXLANs will transit between the aggregators via their public IP addresses, so the encapsulated traffic will have to hairpin at the AWS NAT gateway.

To configure this, just select the Managed mesh mode on the space’s Private WAN tab.

Optional: Add direct VXLAN peering

Note

This is optional for the Managed mesh mode, but required for the Unmanaged mode. Also, it is probably unnecessary unless you are routing large amounts of traffic over the private WAN.

Since we know what the private IP addresses of the aggregators are, we can provide a more direct path between the aggregators by adding custom VXLANs to the involved aggregators within the AWS VPC.

A unique VNI should be created for each space. Each VXLAN for the space should use that assigned VNI. Failure to do so will result in broken routing or unexpected cross routing.

Also, choose a unique subnet within the private WAN to assign to the VXLAN interfaces. Each aggregator will need a unique address in the subnet.

For the routing protocol, we will use Babel. OSPF will also work, but Babel is a more modern protocol that is better suited to this kind of application and is simpler to implement.

On each aggregator, for each space, do the following:

  1. Click the interfaces Add button:

    1. Set Type to VXLAN

    2. Set Interface name to something unique per space

    3. Set Space to the desired space

    4. Set the VNI to the unique VNI chosen for the space

    5. Click Add IP address and set a unique IP address n the subnet chosen for the space

    6. For all other aggregators involved in the space, click Add endpoint and set the aggregator’s private IP address and the VNI chosen for the space

  2. Click the protocols Add button:

    1. Set the Name

    2. Set Space to the desired space

    3. Set Protocol to Babel

    4. Click Add interface

    5. Click Add interface pattern

    6. Set Pattern 1 to the interface name chosen for the interface in the space

Adding a gateway

In a typical Managed mesh or Unmanaged private WAN deployment, gateways will be assigned to VLAN interfaces connected to individual or multi-tenant routers or firewalls. Since AWS does not support VLANs, we will need to create a separate Elastic Network Interface (ENI) and an Elastic IP (EIP) for each gateway. Each gateway will be assigned to a specific aggregator.

Note

AWS NAT gateways do not allow for individual port forwarding. This means that hosts in the private WAN will not be able to act as servers to provide services to public Internet hosts. If such functionality is desired, a separate NAT gateway appliance will need to be installed and configured. Such use is beyond the scope of this document.

To add the necessary resources on the AWS console:

  1. Under Network & Security, select Network Interfaces

  2. Click Create Network Interface

    1. Select the desired subnet. The same one as the aggregator’s main subnet is OK

    2. Click Create

  3. Click the new interface

  4. Take note of the Primary private IPv4 IP in the bottom pane. This will be used as the destination for private WAN

  5. Right-click the interface and select Attach, then select the aggregator instance

  6. Under Network & Security, select Elastic IPs

  7. Click Allocate Elastic IP address

  8. Select the pool and click Allocate

  9. Click the newly assigned public address

  10. Click Associate Elastic IP address

    1. Set Resource type to Network interface

    2. Select the interface for the instance

    3. Click Associate

  11. At the top of the page, click Services then VPC

  12. Under Virtual Private Cloud click Route Tables

  13. Right-click the route table for the space’s VPC and select Edit routes

    1. For each subnet defined in the private WAN add a route with the target of the interface defined for the gateway. It is advisable to use supernets here that encompass the multiple subnets defined on the bonds

    2. Click Save routes

Now, we can set up the gateway on the aggregator:

  1. SSH into the aggregator the interface was defined on and execute ip addr. You should see the new interface with no IP address assigned. Take note of its name

  2. Navigate to the aggregator in the management interface

  3. Click the interfaces Add button

    1. Set Type to Ethernet

    2. Set Interface name to the name we discovered via SSH

    3. Click Add IP address and set the private IP address that AWS associated with the interface

    4. Set Space to the desired space

    5. If an IPv6 subnet was defined, select Accept router advertisements

    6. Click Add

  4. Click the protocols Add button

    1. Set the Name

    2. Set Space to the desired space

    3. Set Protocol to Static

    4. Click Add route

    5. Set Network to 0.0.0.0/0

    6. Set Destination to Gateway

    7. Set Address to the first address of the main VPC subnet. For example, if the VPC subnet is 10.1.0.0/16, the address should be 10.1.0.1

At this point all of the routing should be in place. Bonds in the space should be able to access other bonds in the space as well as public Internet hosts.