Architecture

SD-WAN networking

All SD-WAN nodes run the same software and are capable of any SD-WAN function and can be used as a client, a concentrator, or both simultaneously.

Here is an example configuration that would be typical for a provider that has “east” and “west” data centers:

At each data center there are 2 nodes for redundancy and scalability. These are commonly called “core nodes” to reflect their position in the network.

At each site there is a node that can provide service to one or more local networks. These are commonly called “edge nodes” to, again, reflect their position in the network. These edge nodes can connect to multiple other nodes simultaneously. The node that actually handles traffic is determined by policy and availability.

Connections can be made across single or multiple network links. When multiple links are available, configurable policy is used to select which link or links are active for a connection. Customer connections that are made through these links maintain state even when the underlying links are changed to different providers thanks to packet-level redundancy.

In the above example, the Beta and Gamma nodes connect directly to the Alpha node. If the direct path between those nodes is considered available and policy allows, traffic will pass directly between them. If that link fails but the links to the core nodes are maintained, the traffic will pass though those core nodes instead.

Since these SD-WAN connections can carry traffic for multiple distinct VRFs, we do not need to establish redundant connections to nodes to handle these separate routing instances. The traffic is kept isolated within each VRF’s routing context so mixing between VRFs is not possible unless explicitly enabled through routing policies. You can even use the same private network addresses in different VRFs and they will not conflict with each other.

Management plane

The management plane is built on top of MQTT, a lightweight protocol that is popular in large IoT architectures.

On the manager there is a central broker that relays messages between the node manager and the nodes. MQTT is designed to handle many simultaneous connections and can also be distributed across multiple brokers if necessary.

On each node a local broker relays messages between the orchestration mesh and the services running on the node. The orchestration mesh relays messages between the local broker and the central one using any available local interface that allows connectivity to the central broker.

The use of the orchestration mesh here is to maintain reliability. As long as any single interface can connect to the central broker, the connection will be maintained. The actual path used at any point of time is invisible to local services.

Additionally, this architecture was specifically designed to enable custom software on the manager and nodes to utilize its infrastructure.

Manager

The manager is responsible for managing the networks, including the nodes and their configurations.

The manager consists of the following components:

Frontend

Provides a web-based interface to manage the system

API

Provides a programmable interface for managing the system

Database

Stores configuration, state, and historical telemetry data for nodes, groups, and other entities

Config Manager

Detects changes in the database and generates new node configurations when needed

Node Manager

Tracks the online state of nodes and sends configurations to them when necessary

Status Collector

Collects telemetry data from nodes and stores it in the database

MQTT Broker

Relays messages containing data such as configuration changes and telemetry between the manager and the nodes

Node

Nodes implement the SD-WAN routing in the system. All nodes run the same software and are capable of performing all SD-WAN functionality regardless of their position in the network. They can be organized into groups for simpler configuration management.

In general, nodes that reside in data centers that accept connections from other nodes are called “core nodes” while nodes that reside in customer premises that connect to other nodes are called “edge nodes”. However, since all nodes can perform all duties, even simultaneously, this distinction is merely conceptual and not tracked or enforced by the system.

The node consists of the following components:

MQTT Broker

Relays messages between local services, including custom software, on the node using the MQTT protocol

Orchestration Mesh

Relays messages between the local MQTT broker and the manager, using all available connections to maintain reliability

Agent

Implements the control plane by tracking and applying configurations from the manager, sending telemetry data, and making policy decisions based on network conditions

CLI

A local command line interface that allows for viewing the configuration and status, watching events, and manually updating configuration

Linux/XDP

Implements the data plane with Linux and XDP to handle packet processing, encapsulation/decapsulation, and forwarding