Leg bandwidth adaptation

SD-WAN is able to adjust leg download and upload speeds when it detects increases in leg latency or packet loss. This helps ensure that latency and loss remain as low as possible on a leg and maintains performance of both latency-sensitive and bulk applications.

Leg latency can increase for a number of reasons. Environments that bandwidth adaptation can handle include:

1. Leg download or upload speeds have been set higher than the bandwidth the ISP provides and the leg is approaching 100% load.
2. Leg speeds have been set properly for normal use, but equipment failure or environmental issues have reduced the bandwidth available from the ISP.

Bandwidth adaptation can help in the above situations because reducing the sending speed reduces the latency and packet loss rate of the leg.

Scenarios that bandwidth adaptation is not designed to handle include:

1. The leg is sharing bandwidth with one or more other devices. For example, if the bonder and another router are connected to the same cable modem via a switch, the bonder can’t fully control the bandwidth used on the Internet connection and the router’s traffic may induce latency.
2. The load on the ISP’s cable network, DSL central office, or wireless backhaul is greater than the network’s available bandwidth. At some sites this occurs regularly at the same time every day, such as during heavy residential use every evening.
3. The leg regularly experiences packet loss above the configured threshold, even though the throughput of the leg is low

In the above situations, latency or packet loss are caused by factors beyond the control of SD-WAN, and bandwidth adaptation will have no effect.

Leg bandwidth adaptation is available when both the bonder and aggregator run SD-WAN 2014.4 or later.

How it works

SD-WAN continually monitors latency on each leg. When the leg comes online, a test is performed to determine the leg’s idle latency and jitter. These values are called the “baseline” latency and jitter. When the leg is idle, the baseline values are updated using the regular heartbeat packets sent between the bonder and aggregator. This ensures both the bonder and aggregator always have an accurate understanding of the leg’s normal latency. If packet loss detection is enabled, the bonder and aggregator will also track a leg’s loss rate and use this to determine issues with the leg.

When the leg throughput is above 40% of its configured speed, the leg begins monitoring for deviations from its baseline latency. If it finds that the leg’s latency has increased above a threshold calculated from its baseline latency and jitter, the sending speed is reduced by 10% at approximately 30 second intervals. It is reduced repeatedly until the latency returns to the baseline threshold. If the latency increases while download traffic is 40% or more of the configured speed, the download speed is reduced. If it increases while upload traffic is 40% or more of the configured speed, the upload speed is reduced. If both download and upload traffic are greater than 40% of their configured speeds, both download and upload speeds are reduced.

In addition, when the leg throughput is above 40% of the configured speed and experiences packet loss above the configured bandwidth adaptation packet loss threshold, the leg speed will be reduced.

If the high latency or packet loss is due to sending faster than the leg’s available bandwidth, the latency or loss rate will return to the baseline after the speed is reduced to just below the available bandwidth. The speed will not be reduced below 50% (by default) of the leg’s configured speed. After finding the speed that results in normal latency or packet loss, the leg speed is gradually increased in order to resume the full sending speed. The speed increases slowly at first, but accelerates as long as the latency remains near the baseline. It increases by 10% in about 9 minutes and by 100% in about 14 minutes. If high latency or packet loss occurs after increasing the speed, the speed is reduced again and the the acceleration of the speed increases returns to the initial slow pace.

If the high latency or loss is due to traffic from other devices on the local or ISP network, the leg speed will be reduced but the latency will not return to baseline and packet loss will not change. In this case, the leg speed will eventually settle at 50% (by default) of the configured speed until the latency returns to the baseline or packet loss goes below the bandwidth adaptation threshold.

If leg traffic returns to idle, the speed slowly increases until it reaches the full configured speed.

Example

As an example, consider a leg with 10.0 Mbps download and 2.0 Mbps upload bandwidth, 40 milliseconds (ms) latency and 4 ms jitter at idle, configured with an adaptation limit of 60%. When the leg is idle, the latency and jitter are measured and a latency threshold of 52 ms is set. When the leg download throughput goes above 4.0 Mbps (which is 40% of the configured 10.0 Mbps speed), the latency is checked against the 52 ms threshold. If the latency increases above the threshold, the speed is reduced by 10%, or 1.0 Mbps, to 9.0 Mbps. If this doesn’t result in the latency returning to below the threshold, the speed is reduced to 8.1 Mbps. This process is repeated until the latency goes below the 52 ms threshold or until the speed is reduced to the 60% limit, or 6.0 Mbps. If the latency returns to below the threshold or the leg becomes idle, the speed will gradually be increased until the latency goes above the threshold or the speed reaches the full configured speed of 10.0 Mbps.

The chart below shows how latency and throughput can change due to changes in available bandwidth on a leg.

This setup simulated two legs, each initially with 10.0 Mbps download bandwidth. The leg shown with the orange series had about 35 ms idle latency and its bandwidth did not change. The leg shown with the blue series had about 40 ms idle latency and its bandwidth was changed as described below.

1. The blue leg was given 10.0 Mbps bandwidth and its speed was configured as 10.0 Mbps. A bulk download was started and the latency remained low.
2. The bandwidth available on the blue leg was changed to 5.0 Mbps, causing the latency to spike above 400 ms. Because traffic was balanced equally across both legs, the throughput on both legs slowed to 5.0 Mbps. As the speed of the blue leg was gradually reduced, more and more traffic was sent over the orange leg.
3. The speed of the blue leg reached its configured 60% limit, or 6.0 Mbps. Since this was still greater than the 5.0 Mbps available bandwidth, the latency remained high.
4. The bandwidth available on the blue leg was changed to 7.0 Mbps. Since it was still using the reduced speed of 6.0 Mbps, the latency immediately returned to 40 ms. In the following minutes, the speed was increased slowly at first, then gradually increased faster and faster.
5. The speed of the leg increased past 7.0 Mbps and latency spiked. The increased latency was noticed right away, the speed was reduced below 7.0 Mbps, and the latency returned to normal.
6. The speed again increased past 7.0 Mbps and latency spiked. After this spike, the bandwidth available on the blue leg was reset to 10.0 Mbps.
7. The speed of the leg eventually returned to the 10.0 maximum configured speed.

Configuration

Bandwidth adaptation is configured individually for each leg on the Optimization tab of the leg edit modal.

Bandwidth adaptation

Bandwidth adaptation adjusts leg download and upload speeds when it detects increases in leg latency or packet loss.

Enabled

Enables or disables bandwidth adaptation on the leg. Disabled by default.

Warning Threshold

The threshold, as a percentage of the configured speed, at which the warning icon is shown for the leg in the web interface. For example, if a leg is configured at 10.0 Mbps download and 2.0 Mbps upload and the warning threshold is 90%, the warning icon will be shown if the adapted download speed is 9.0 Mbps or less or the upload speed is 1.8 Mbps or less.

Limit Threshold

The lowest speed, as a percentage of the configured speed, that will be used for this leg. This prevents the speed of the leg from being reduced to 0 in some environments. For example, if a leg is configured at 10.0 Mbps download and 2.0 Mbps upload and the limit is 60%, the leg speed will never be reduced below 6.0 Mbps download or 1.2 Mbps upload.

Packet Loss Threshold

The amount of packet loss required on a leg to trigger bandwidth adaption.

Packet loss detection

Requires packet loss detection to be enabled on the bond.

Removal Threshold

The amount of packet loss required on a leg for it to be removed from the bond.

Monitoring

Information about bandwidth adaptation for a leg is provided in the management server. If a leg download or upload speed is reduced below the warn threshold (by default, 90% of the configured speed of the leg), its state is shown in the management server web interface as an orange circle with a down icon:

If the speed reaches the lower limit (by default, 50% of the configured speed), it is shown as an orange circle with an exclamation point:

Details about the adaptation state are shown in the Legs section of the bond details page. If the leg speed has been reduced, the background will change from green to yellow and the reduced speed will be shown. The configured speed can be displayed in a tooltip when hovering with the cursor.

In the example above, the download speed has been reduced from a maximum of 10 Mbps to a current value of 6.0 Mbps. The upload speed has not been reduced, so it is still green.