SD-WAN 2015.4 release notes

November 21, 2015

2015.4 is the most significant release of SD-WAN ever. It introduces a number of major features. Here are the highlights:

  • Bonds can be assigned to groups known as spaces, and users can be limited to view or manage bonds only in certain spaces. This allows partners to offer access to the management application to resellers and to representatives from end-user companies. Spaces are arranged in a hierarchy, allowing a variety of flexible arrangements.
  • Traffic can be routed privately between bonds in a space using the new private WAN functionality. This can be configured from the web application, and aggregator failover is supported. A new node type is introduced, the private WAN router, which performs the private routing between aggregators.
  • Users can be limited to perform only certain actions. Permission can be granted to view or manage bonds, legs, speed tests, spaces, users, etc. Partners can now give limited access to users who are trusted to view data but not change some or all data.
  • Branding options are available for customizing the web application appearance. Each space can have its own appearance.
  • IP management features allow subnets to be assigned to datacentre records and to be allocated to spaces so that space administrators can manage their own IP addresses.
  • Bonders and aggregators now detect link packet loss and can reduce a link’s speed or remove it from the bond based on its packet loss rate. Hosts can also detect a flapping link and remove it from the bond until the link becomes more stable.
  • Leg and bond tuning procedures are now automated. Appropriate leg speeds and bond settings can be automatically detected in order to maintain optimum performance with minimal effort.

A new service for managing nodes has been integrated with the management server. This service runs on TCP ports 4505 and 4506. If your management server is behind a firewall, please ensure you add inbound allow rules for those TCP ports.

As announced in 2015.3, this release removes support for the performance charting features of SD-WAN 2014.3 and earlier. Bonders and aggregators running 2014.3 and earlier will no longer show performance charts.

As always, all new functionality is available in the management server API.

Now, the details:

Bonding Node

Additions

  • Nodes detect link packet loss rates. The loss rate of a link is used to slow the link, if bandwidth adaptation is enabled, or to completely remove it from the bond, depending on the loss rate. The loss rate is also used to make MTU detection more reliable.
  • Nodes detect flapping links. If a link goes down and back up repeatedly, it is removed from the bond for longer and longer periods of time, until it becomes stable and remains up. An unstable link may be removed from the bond for up to 30 seconds.
  • Nodes can automatically set the length of time packets are kept in the reordering buffer that reduces or eliminates out-of-order packet delivery when bonding legs with different latencies.
  • PPTP, GRE, and certain types of FTP connections can now be used with CPE NAT IPs, because the appropriate Linux NAT modules are loaded on bonders and aggregators.
  • The files /root/.ssh/authorized_keys/, /etc/firewall.d/known_ips, and /etc/resolv.conf are no longer managed from a partner-specific Debian package. They are managed via the SaltStack configuration management service on the management server. resolv.conf is no longer managed on aggregators; it is only managed on bonders.

Removals

  • The CUSTOMER and DESCRIPTION fields have been removed from the bond hook environment.

Changes

  • Bandwidth adaptation is less sensitive to link jitter. This helps to avoid dropping speed on legs that normally increase in latency under load and that do not need to be slowed for minor increases in latency.
  • Legs, connected IPs, CPE NAT IPs, and routes start more reliably. Previously, if an object could not be started, it would remain down until the bond was restarted, but now these objects are restarted at increasing intervals.
  • Packet reordering is disabled when only a single link is available.

Fixes

  • When TCP proxy is enabled, bulk and interactive flows share bandwidth more evenly. Previously, interactive flows could hang under the load of bulk flows when TCP proxy was enabled.
  • A number of minor bugs have been fixed.

Patches

2015.4-17:Fix MTU detection on single leg bonds with encryption enabled. Improve packet loss reliability. Fix issue supervising DHCP on mobile broadband legs. Fix rare TCP proxy crash. Prevent starting multiple instances of cell service. Fix rare node service crash when stopping bonding.
2015.4-18:Fix issue preventing bonders behind bonders on same aggregator. Upgrade to GnuTLS 2 to avoid broken Debian update.
2015.4-19:Improve salt service logging. Fix rare crash in TCP proxy. Fix supervision for cell service. Handle “cannot find device” issue in node service.
2015.4-20:Restart leg when assigned port number changes. Fix minor issue related to detecting Ethernet modes. Improve reliability of stopping legs.
2015.4-21:Improve handling of TCP proxy buffers to ensure consistent performance.
2015.4-22:Send more detailed information about leg state changes; fix issue with Quagga routes persisting incorrectly after an aggregator reboot; fix issue with tunnel processes taking 100% CPU sometimes; no longer update /etc/salt/grains, so that partners can use it; improve Salt configuration with regards to default bonders and deconfigured bonders; fix a couple of rare crashes in the tunnel application.
2015.4-23:Update config service with changes to PWAN rule design.
2015.4-25:Fix memory leak in collectd statistics reporting service that occurs on aggregators when a bond is removed from the agg.
2015.4-26:Fix memory leak in collectd that occurs when the service accepting statistics on the management server is unavailable.
2015.4-28:Fix removal of all Quagga static routes when bonding starts—change it to only clear static routes that point to tunnel devices.
2015.4-29:Fix issue that allows management VPN tunnels to remain down for an increasing and unlimited amount of time if the management VPN server is unavailable. It now will try to start the tunnel at least once every 60 seconds.
2015.4-30:Fix a potential memory leak in the TCP proxy.
2015.4-31:Correct permissions on /var/log/bonding directory that prevented logrotate from rotating bonding log files.
2015.4-32:Messages sent from nodes to the management server are published from the config service. Messages can be seen using the bondevent application.
2015.4-38:Avoid crash at startup due to Debian changing file permissions on a Quagga configuration file: https://www.debian.org/security/2016/dsa-3654.en.html

Bonding Admin

Additions

  • Groups known as spaces have been introduced. Spaces create independent but related zones in the management server. They can have their own bonds, aggregators, QoS profiles, users, private WAN settings, and branding options. Spaces are organized in a hierarchy, with a root space, and each space able to have multiple child spaces. Users can view bonds and other objects within their assigned space and its descendant spaces. Spaces can be permitted to use aggregators and certain other objects from their parent space, allowing a number of deployment scenarios for organizations of different sizes.
  • The customer and description fields have been removed from the bond object and have been replaced with a single name field. During the 2015.4 upgrade, the name field of a bond is set to a value based on its customer and description fields in 2015.3. Bonds can now be uniquely identified by referring to the name of their space and the name of the bond.
  • Private WAN is available in the management application and offers many new features, including support for aggregator failover, per-space NAT and port forward rules, and automatic routing of traffic between aggregators in different datacentres. Private WAN requires deploying a new type of node in the datacentre, known as a “private WAN router,” which handles configuration of the specialized routing rules on the aggregators as well as routing private data between aggregators and between datacentres. Aggregators and private WAN routers are assigned to “routing groups,” which roughly correspond to datacentres. Outbound Internet access for bonds in a private WAN space is available using just a single IP routed to the private WAN router, and inbound access for servers hosted within the space is possible using 1:1 NAT or port forward rules.
  • Users can be granted a wide variety of granular permissions that restrict or allow them to do specific things in the management application. Permissions are assigned to groups, and users are given permissions by assigning them to those groups. A number of useful groups are included by default, and additional groups can be added to completely customize authorization in the management server. Available permissions include the ability to view, add, change, or delete bonds, legs, users, spaces, connected IPs, and more.
  • Users are assigned a role. Roles are selected from a few options such as “Systems Administrator”, “Technical support 2”, “Sales,” or “End-user.” Roles are used to suggest initial group memberships for the user.
  • The appearance of the web application can be customized for each space. Available branding options include the image, background colour, and text colour of the main navigation menu, and colours of text, buttons, and messages in the main web page body. Advanced users can provide custom CSS to override any other styling. Space administrators can supply technical support contact information, such as helpdesk email and phone numbers, that is shown to users in child spaces.
  • IP management features have been added. Administrators can allocate subnets to routing groups and then further delegate parts of those subnets to child spaces to allow administrators in child spaces to manage their own IP assignments. Reports are available that show the availability and usage of delegated IP subnets. IP allocations are applied to bond connected IPs, CPE NAT IPs, and routes, and private WAN SNAT, 1:1 NAT, and port forward rules. For example, if the subnet 203.0.113.0/28 is allocated to a certain routing group, and 203.0.113.0/29 is delegated to a space, then the subnet 203.0.113.0/30 can be used for a connected IP on a bond in that space. Subnets outside of 203.0.113.0/29 could not be assigned to that bond.
  • Leg and bond tuning procedures are now automated. A link on the leg actions menu starts the speed testing procedure for legs, and a link on the bond actions menu opens a dialog that offers a number of options for automatically tuning bonds. Bond tests can be run immediately, or at a scheduled time in the next 24 hours, or when the bonder is first connected. Since the tuning procedure can take a few minutes, the user that triggers the tuning is emailed when the procedure is complete. The recommended leg or bond settings can be applied by clicking a button on the tuning results page.
  • A new API version is available, version 3, at /api/v3/. Version 2 is deprecated but will continue to be available until at least December 2016. All new applications should be written against v3 and existing applications should be migrated to v3.
  • Bond options have been added for packet loss detection, flap detection, and automatic reorder max-hold. All options are enabled for existing bonds.
  • Leg options have been added for packet loss detection thresholds. This allows users to set packet loss rate thresholds for slowing a leg when bandwidth adaptation is enabled and for removing the leg from the bond.
  • The SaltStack configuration management service has been added. It requires TCP ports 4505 and 4506 on the management server to be available to the world.
  • User accounts can be managed from the API.
  • The first time a user visits the web application after the upgrade, a “What’s New in 2015.4” dialog is shown. This dialog describes the main features in 2015.4. The dialog is also shown to new users created after the 2015.4 upgrade.
  • Two text fields have been added to the bond object: “circuit ID” and “product.” These are free form fields that can be used to record relevant information from a partner’s own systems and sales processes.
  • Two text fields have been added to the bonder and aggregator objects: “serial number” and “asset tag.” Like the bond circuit ID and product fields, these are free-form fields that can be used to record relevant information about a bonder or aggregator.
  • A number of pages have been added to the Administration section:
    • System Charts—shows charts related to management server resources such as CPU, memory, and disk space. Previously this was available under the main Help menu.
    • Email—provides options for sending email from the management application
    • Support—provides options for technical support information shown to users in the root space
    • Versions—shows versions of software on the management server and web application. Previously this was available under the System Info page.
    • Other settings—shows values of settings coded in the main Python configuration file. Previously this was available under the System Info page.

Removals

  • The performance charting features of SD-WAN 2014.3 and earlier have been removed. Nodes running 2014.3 or earlier will no longer show performance charts.
  • Node passwords are no longer hidden on the HTML API view. This feature caused some browsers to hang on large documents.

Changes

  • New ISO files for provisioning nodes have been created. Please discard your existing provisioning disks and use the new ones.
  • The bond balancing algorithm field has been renamed distribution algorithm, in order to reduce confusion between bonding and load balancing technology.
  • Users created in 2015.4 and later must log in with their e-mail address, as users are no longer assigned usernames. Users created in 2015.3 and earlier can log in with their username or email address. Usernames should be considered deprecated and will be removed from a future version of SD-WAN.
  • Connected IP and route objects are described using CIDR notation, with no separate netmask field. For example, an IP of 203.1.113.0 and netmask 255.255.255.252 is now provided as 203.1.113.0/30.
  • Routing objects (connected IPs, CPE NAT IPs, and routes) can no longer be added at the same time as creating a new bond in the web interface. To create these objects, first create the bond, then edit it and add the routing records. This simplifies form validation methods in the application.
  • The default theme has been updated. Forms, dialogs, buttons, panels and typography have a modern and distinctive appearance.
  • The main navigation menu has been updated with new links for all the new management pages.
  • The main Help menu has been replaced with a menu showing documentation links, keyboard shortcuts, and technical support contact information.
  • Configuration updates to a node are attempted even if a node seems to not be connected to the management server.
  • Failing node configuration updates have ever-longer timeouts, up to five minutes between tries.
  • The mobile broadband provider profiles page has been move from the Administration page to the Policies menu.
  • Frameworks used by the management application have been upgraded—the Django application framework has been upgraded to 1.9, Bootstrap CSS to version 3, and Font Awesome icons to version 4.
  • Internal application structure has been reorganized in places to improve reliability and maintainability.
  • Performance of the nodeupdates and influxmux services, which collect and process a variety of information from nodes, has been improved under certain circumstances.

Fixes

  • Hung configuration updates are restarted automatically.
  • Numerous other bugs have been fixed.

Patches

2015.4-28:Improve leg form layout.
2015.4-29:Remove support for Munin charting. Fix various permission-related issues. Improve CSS generation and serving.
2015.4-30:Improve URL handling and filtering on bond list pages. Internal JavaScript improvements. Improve speed testing reliability.
2015.4-31:Handle unicode strings better.
2015.4-32:Add warnings in IP allocation/delegation section. Add space fields to various forms so that spaces can be specified when creating objects. Various UI improvements.
2015.4-33:Improve HTTP query performance. Remove ability to delete configuration updates. Provisioning ISOs no longer show graphical splash screen. Add permission to update permissions. Various UI improvements.
2015.4-34:Add ability to select columns shown on bond list tables. Improve service performance when handling updates from PWAN routers. Various UI improvements.
2015.4-35:Fix issues related to reporting of packet loss/flapping legs.
2015.4-36:Add before/after configuration to automatic tuning results. Clarify automatic tuning errors. Make automatic tuning more reliable. Various UI improvements.
2015.4-37:Add space note field. Various UI improvements.
2015.4-38:Update Salt configurations for PWAN routers. Update v3 API- move PPP, Ethernet, DHCP, and security fields to nested resources. Various UI improvements.
2015.4-39:Improve PWAN rule UI layout. Improve validation on IP allocations. Fix issues in column chooser and group/user chooser UIs. Fix issues on save buttons in administration section.
2015.4-40:Add permission for viewing/changing bonder, aggregator, and PWAN router username/password fields (6 permissions total). Change v2 API—now returns null instead of real information for bonder resource nested in bond resource when user doesn’t have appropriate permissions. Change v3 API—removed some fields from bonder resource nested in bond resource, move node username/password fields to nested object and only show when user has appropriate permissions. Remove username/password fields from bonder and aggregator edit forms and add button showing username/password dialog for users with appropriate permissions. Add priority and enabled fields to PWAN routers. Improve automated speed tuning reliability. Fix permission bug preventing some users from performing leg speed detection.
2015.4-41:Change node setup page from available to all users, to available per-space only to users with appropriate permissions. Make ISOs for each space. Add permission for viewing documentation. Don’t allow routing group to be changed for PWAN routers. Fix issues with validating uniqueness of certain PWAN router fields.
2015.4-42:Add failover leg counts to space statistics endpoint. Don’t send config updates to nodes that have never before downloaded their config. Add a tab to the user edit page that shows all permissions assigned to the user. Increase timeout for speed tuning. Prevent enabling private WAN on root space unless an internal setting is enabled.
2015.4-43:Fix issue with provisioning ISOs. UI refinements.
2015.4-44:Improve database schema of IP allocations and delegations. Update search and filter fields available for core resources. Fix an issue displaying certain documentation snippets. Include all routing groups and aggregators that the user has access to on aggregator and bonder forms. Fix issue that caused bonder password field to be set to blank after editing bond.
2015.4-45:Fix rendering issues in Microsoft Edge. Other UI improvements. Fix an issue in v1 aggregator API.
2015.4-46:Further improvements to database schema of IP allocations and delegations. Make prevented downtime calculations work in management server’s time zone. Nodeupdates service logs to its own file. Remove orphaned database records potentially added after 2015.4-40 upgrade. UI improvements.
2015.4-47:Use Salt connection to a central server to manage all management servers. Improve maintenance of Salt minion key database. Update Squeeze sources.list shown on Node Setup page.
2015.4-48:Improve handling of legs that have packet loss and flap status values inconsistent between bonder and aggregator.
2015.4-53:Improve validation of routing objects.
2015.4-59:Add update_leg_port management command.
2015.4-66:Roll up database schema migrations to accelerate installations and upgrades. Add update-squeeze-sources command. Update PWAN rule design, dropping distinction between local and global rules.
2015.4-75:Various IP routing validation fixes and UI fixes.
2015.4-76:Support running InfluxDB service on a dedicated host.
2015.4-77:Clear HTTP sessions when restoring database.
2015.4-79:Add capability for PWAN routers to have a dedicated interface for VLAN traffic.
2015.4-83:Improve UI performance on bond list page.
2015.4-92:Improve performance of nodeupdates service. Various other bugfixes.
2015.4-93:Improve API performance.
2015.4-96:Change space private WAN gateway rules to three different types of rules instead of a single type of rule.
2015.4-97:Fix bug in prevented downtime monitoring for bonds in non-root spaces.
2015.4-99:Make nodeupdates service multithreaded if option is enabled. Multithreaded mode is disabled by default.
2015.4-100:Improve bond details page UI performance.
2015.4-105:Fix bug that prevented deleting legs from bond edit page.
2015.4-107:Add port range capability to private WAN port forward rules.
2015.4-110:Fix issue that could cause config updates to be executed multiple times or in the wrong order.
2015.4-112:Add performance instrumentation to nodeupdates.