=========================================================
AN-001 Deploying aggregators on Amazon Web Services (AWS)
=========================================================

Amazon Web Services (AWS) is a popular cloud computing platform that allows
for quick deployment and management of Internet-connected servers. It is most
popular for hosting web sites, but can be used to deploy aggregators as well,
with some caveats.

The issues described here may also be applicable to other cloud providers that
use private networking.

Networking considerations
=========================

The main issue with IPv4 networking in AWS is that there is no option for
routing subnets to AWS hosts. You can only allocate single IPs, and those IP
addresses are randomly assigned, so it is not possible to even emulate subnet
routing with IPv4. It can be emulated with IPv6 addresses, but each address in
each range will need to be assigned to the aggregator interface, which is
burdensome.

The other issue is that public IPv4 addresses are never assigned directly to
instance interfaces. All public IPv4 addresses are assigned to a private
address and translated using NAT. While aggregators are capable of handling
tunnels in such a scenario, it makes bond routing complicated

However, even with the above limitations, there are some scenarios that work
if set up in a particular way.

CPE NAT IP to individual bonds
==============================

This scenario involves providing individual per-bond access to the Internet,
each with its own public IP address.

In a typical deployment, a subnet which contains the IPs available for use for
CPE NAT IPs would be routed to the aggregator network, and the aggregator
would announce each individual IP via OSPF or iBGP. If dynamic routing was not
available, static routes could be made to the individual aggregators hosting
the bonds, but that would require manual routing changes if a bond was moved.

In AWS, each public IP addresses is assigned to a private IP address, which
would presumably be configured on an interface. By default, when an AWS EC2
instance is created, an Elastic Network Interface (ENI) is created, a
dynamically chosen private IP address is assigned to and configured on the
interface, and a public IP address is associated with the private IP address.

For an aggregator, we can use this address for it's primary communication, but
we will need additional addresses for each CPE NAT IP. The AWS Elastic IP
(EIP) can be used to provide this, but it must be carefully configured to
avoid losing the original assigned public IP address.

Adding the Elastic IP Address
-----------------------------

To add a new IP address in the AWS console, perform the following steps:

1. Under **Network & Security**, select **Network Interfaces**
#. Right-click the primary interface for the aggregator interface and select
   **Manage IP Addresses**
#. Click **Assign new IP**
#. Click **Yes, Update**
#. Take note of the newly assigned address
#. Under **Network & Security**, select **Elastic IPs**
#. Click **Allocate Elastic IP address**
#. Select the pool and click **Allocate**
#. Click the newly assigned public address
#. Click **Associate Elastic IP address**

   a) Set **Resource type** to **Network interface**
   #) Select the interface for the instance
   #) Choose the newly assigned private IP address
   #) Click **Associate**

.. note::
    You do **not** need to configure the address in the aggregator's network
    configuration

Configuring the CPE NAT IP
--------------------------

On a bond assigned to the aggregator, configure a private connected IP and add
the CPE NAT IP using the **private** IP address we added to the interface. The
AWS gateway will NAT the public IP address to the private CPE NAT IP, and the
bonder will NAT the private IP address to the connected IP.

Assuming the bonder is online, all of the routing will be in place.

.. note::
    Due to the lack of dynamic routing in AWS VPC, moving bonds between
    aggregators will not normally work unless the static associations are
    updated via the AWS console or API. As a consequence, aggregator failover
    will **not** work in this scenario.

Optional: Route CPE NAT IPs across a group of aggregators
---------------------------------------------------------

Since the public IPs are associated with interfaces on specific instances,
each bond must be assigned to a specific aggregator or routing will break if
moved. However, as of Bonded Internet 6.5, we can actually allow bonds to move
to other aggregators without needing to update the association, at least not
right away.

With Bonded Internet 6.5 we can implement custom routing protocols to allow
for an aggregator with associated IP to itself route to the correct
aggregator.

In a typical data center deployment, CPE NAT IP routes are distributed across
the main physical segment of the aggregators to a local router via a protocol
such as BGP, OSPF, or Babel. Unfortunately AWS does not allow such peering and
addresses must be routed statically via the AWS API. We can, however, peer the
aggregators with each other using BGP so that bonds can move between
aggregators without breaking routing. Note that the AWS VPC does not allow
multicast traffic, so OSPF and Babel will not work unless tunneled.

See `here <../dynamic-routing/configuring-dynamic-routing-in-bonding.html>`__ for
more information on setting up BGP between aggregators. Note that CPE NAT IPs
are always exposed in the main routing table, so the protocols should not have
a space defined.

Private WAN
===========

.. note::
    The **With private WAN routers** mode of private WAN will not work with
    AWS due to the NAT involved and the way that the GRE tunnels are defined.
    Use the "Managed mesh" or "Unmanaged" mode instead.

Private WAN can also be deployed in AWS, but due to the lack of layer-2
networking, the private WAN traffic will have to transit across VXLANs to get
to other aggregators. This means using the **Managed mesh** or **Unmanaged**
private WAN modes.

Adding a new VPC
----------------

Since private WAN routing needs to be isolated, we will need to create a
unique VPC for each space. To do this in the AWS console:

1. At the top of the page, click **Services** then **VPC**
#. Under **Virtual Private Cloud** click **Your VPCs**
#. Click **Create VPC**

   a) Add an IPv4 and optionally an IPv6 subnet
   #) Click **Create**

Configuring the private WAN mesh
--------------------------------

The **Managed mesh** mode is easiest to implement, since it automatically
detects which aggregators have bonds in a space and sets up VXLANs between
them. However, these VXLANs will transit between the aggregators via their
public IP addresses, so the encapsulated traffic will have to hairpin at the
AWS NAT gateway.

To configure this, just select the **Managed mesh** mode on the space's
**Private WAN** tab.

Optional: Add direct VXLAN peering
----------------------------------

.. note::
    This is optional for the **Managed mesh** mode, but required for the
    **Unmanaged** mode. Also, it is probably unnecessary unless you are
    routing large amounts of traffic over the private WAN.

Since we know what the private IP addresses of the aggregators are, we can
provide a more direct path between the aggregators by adding custom VXLANs to
the involved aggregators within the AWS VPC.

A unique VNI should be created for each space. Each VXLAN for the space should
use that assigned VNI. Failure to do so will result in broken routing or
unexpected cross routing.

Also, choose a unique subnet within the private WAN to assign to the VXLAN
interfaces. Each aggregator will need a unique address in the subnet.

For the routing protocol, we will use Babel. OSPF will also work, but Babel is
a more modern protocol that is better suited to this kind of application and
is simpler to implement.

On each aggregator, for each space, do the following:

1. Click the interfaces **Add** button:

   a) Set **Type** to **VXLAN**
   #) Set **Interface name** to something unique per space
   #) Set **Space** to the desired space
   #) Set the **VNI** to the unique VNI chosen for the space
   #) Click **Add IP address** and set a unique IP address n the subnet chosen
      for the space
   #) For all other aggregators involved in the space, click **Add endpoint**
      and set the aggregator's private IP address and the VNI chosen for the
      space

#. Click the protocols **Add** button:

   a) Set the **Name**
   #) Set **Space** to the desired space
   #) Set **Protocol** to **Babel**
   #) Click **Add interface**
   #) Click **Add interface pattern**
   #) Set **Pattern 1** to the interface name chosen for the interface in the
      space

Adding a gateway
----------------

In a typical **Managed mesh** or **Unmanaged** private WAN deployment, gateways
will be assigned to VLAN interfaces connected to individual or multi-tenant
routers or firewalls. Since AWS does not support VLANs, we will need to create
a separate Elastic Network Interface (ENI) and an Elastic IP (EIP) for each
gateway. Each gateway will be assigned to a specific aggregator.

.. note::
    AWS NAT gateways do not allow for individual port
    forwarding. This means that hosts in the private WAN will not be able to
    act as servers to provide services to public Internet hosts. If such
    functionality is desired, a separate NAT gateway appliance will need to be
    installed and configured. Such use is beyond the scope of this document.

To add the necessary resources on the AWS console:

1. Under **Network & Security**, select **Network Interfaces**
#. Click **Create Network Interface**

   a) Select the desired subnet. The same one as the aggregator's main subnet
      is OK
   #) Click **Create**

#. Click the new interface
#. Take note of the **Primary private IPv4 IP** in the bottom pane. This will
   be used as the destination for private WAN
#. Right-click the interface and select **Attach**, then select the aggregator
   instance
#. Under **Network & Security**, select **Elastic IPs**
#. Click **Allocate Elastic IP address**
#. Select the pool and click **Allocate**
#. Click the newly assigned public address
#. Click **Associate Elastic IP address**

   a) Set **Resource type** to **Network interface**
   #) Select the interface for the instance
   #) Click **Associate**

#. At the top of the page, click **Services** then **VPC**
#. Under **Virtual Private Cloud** click **Route Tables**
#. Right-click the route table for the space's VPC and select **Edit routes**

   a) For each subnet defined in the private WAN add a route with the target
      of the interface defined for the gateway. It is advisable to use
      supernets here that encompass the multiple subnets defined on the bonds
   #) Click **Save routes**

Now, we can set up the gateway on the aggregator:

1. SSH into the aggregator the interface was defined on and execute ``ip
   addr``. You should see the new interface with no IP address assigned. Take
   note of its name
#. Navigate to the aggregator in the management interface
#. Click the interfaces **Add** button

   a) Set **Type** to **Ethernet**
   #) Set **Interface name** to the name we discovered via SSH
   #) Click **Add IP address** and set the private IP address that AWS
      associated with the interface
   #) Set **Space** to the desired space
   #) If an IPv6 subnet was defined, select **Accept router advertisements**
   #) Click **Add**

#. Click the protocols **Add** button

   a) Set the **Name**
   #) Set **Space** to the desired space
   #) Set **Protocol** to **Static**
   #) Click **Add route**
   #) Set **Network** to ``0.0.0.0/0``
   #) Set **Destination** to **Gateway**
   #) Set **Address** to the first address of the main VPC subnet. For
      example, if the VPC subnet is ``10.1.0.0/16``, the address should be
      ``10.1.0.1``

At this point all of the routing should be in place. Bonds in the space should
be able to access other bonds in the space as well as public Internet hosts.
