Upgrading a node

Before upgrading a node, please read our recommendations for managing a successful environment upgrade.

Note

Debian has removed the jessie-updates repository for Debian 8 (“Jessie”) and all of the Debian 7 (“Wheezy”) repositories from its mirror network as of March 24, 2019. Bonding version 6.4 introduced management of the apt repositories through salt, but if an error containing “404 Not Found” is encountered when running apt-get update, please refer to SB-7 2019-03-28 Debian repository removals.

Note

Bonding versions up to and including 6.5 supported 32-bit operating systems, but 6.6 and beyond will only support 64-bit. Attempting to upgrade a 32-bit node beyond 6.5 may leave the device in an inconsistent state if some supporting applications are upgraded beyond what the bonding software understands.

Automated bonder upgrades

Note

Aggregators must be upgraded using the individual method below, the automated method works only for bonders.

Bonders can be upgraded in bulk through the management server command line. This method runs upgrade and restart commands on each bonder in sequence. It is not intended as a completely unattended process—you may be prompted for input from time to time. It is strongly recommended that you configure SSH public keys; if you depend on password authentication, you will be prompted for the bonder root password for each bonder upgraded. If you have SSH public keys configured, ensure you enable “agent forwarding” in your SSH client to avoid the password prompts.

Only online bonders are upgraded. Bonders that are not online at the time the upgrade is executed will need to be upgraded manually later, or the bulk upgrade script will need to be run again.

To upgrade bonders in bulk, follow these steps:

  1. SSH into the management server.
  2. Run this command: upgrade-bonders
  3. Review the list of bonders to be upgraded. Confirm the upgrade by typing Y, then <enter>. Each bonder will be upgraded in turn. This process may take a long time. Bonders already at the latest software version are ignored.
  4. During the upgrade, each bonder downloads a new copy of its cached configuration file. If this fails, the bonder will offer the prompt “Go back and retry?”. We recommend you type 1 to not retry. The upgrade on that bonder will quit with a failure. You should log into that bonder, troubleshoot the network issue preventing the installer from downloading the config file, and manually upgrade the bonder.
  5. When the script finishes, it shows the name of any bonders whose upgrade failed. Investigate why these bonders failed to upgrade, then upgrade them manually or run the bulk upgrade script again.

Targeting particular bonders

The upgrade can be limited to certain bonders by providing arguments to the script.

Targeting by bond IDs

To upgrade individual bonders, provide their bond IDs on the command line. Multiple IDs can be set. For example,

upgrade-bonders 2 3 5

would upgrade only bonds 2, 3, and 5. No other bonders would be upgraded.

Warning

The argument refers to the bond ID, not the bonder or node ID.

Targeting by range of bond IDs

To upgrade a set of bonders whose bond ID falls into a certain range, provide a single argument on the command line in the form low-ID:high-ID. For example,

upgrade-bonders 2:19

would upgrade bonders whose bond ID was between 2 and 19, inclusive.

Note

Use the range method to speed up the upgrade process by updating multiple bonders in parallel. Suppose you have 240 bonders to upgrade. Running a single bulk upgrade process would take a very long time. However, you could run five bulk upgrades, each in its own SSH session with a different range of bond IDs. For example:

  • upgrade-bonders 1:49
  • upgrade-bonders 50:99
  • upgrade-bonders 100:149
  • upgrade-bonders 150:199
  • upgrade-bonders 200:250

The upgrade would complete about five times faster than if you only used a single upgrade process. Take care to ensure that the ranges don’t overlap.

Warning

Only bonders are upgraded, not aggregators. Because there are very few aggregators in a partner’s environment, and because aggregators are a critical part of the SD-WAN service, they should be upgraded and restarted manually.

Manual node upgrade

Debian 6 (“Squeeze”) reached end-of-life on February 29, 2016 and bonding discontinued support as of version 6.0. Please see SB-4 2016-03-31 Debian Squeeze end-of-life for details.

Nodes can be upgraded from any software version to the latest version. It is not expected (or even possible) to upgrade through each new version in turn until the most recent version is installed.

To upgrade Bonding on a single host, start a console session on the host.

If the node is an aggregator or private WAN router with dynamic routing set up, you should perform a Quagga configuration check before the upgrade. To view the running configuration, run:

vtysh -c "show running-config"

This configuration should be compared to the startup configuration. If using BGP, run:

cat /etc/quagga/bgpd.conf

If using OSPF, run:

cat /etc/quagga/ospfd.conf

If the configurations do not match and the changes to the running-config are required to be persistent, they should be saved:

vtysh -c write

If the running-config needs to be adjusted, perform the necessary changes using vtysh and verify that the peering is working and saved before continuing.

Initial upgrade to 6.1

Note

The private WAN packages pwan-aggregator and pwan-router had their functionality merged into the bonding package and will no longer need to be upgraded separately. Take note of the revised upgrade procedure below when upgrading aggregators integrated with private WAN and private WAN routers from 6.0 to 6.1. Bonders will continue to follow the standard upgrade procedure.

For aggregators integrated with a private WAN (PWAN) network, you must stop pwan-aggregator before upgrading to 6.1:

apt-get update
service pwan-aggregator stop
apt-get install bonding
apt-get upgrade
reboot

And for private WAN routers, you must also stop the pwan-router service before upgrading to 6.1:

apt-get update
service pwan-router stop
apt-get install bonding
apt-get upgrade
reboot

Upgrading 6.1 and newer nodes

Note

Future upgrades on bonders, aggregators, and private WAN routers can be simplified to just the following commands.

Note

If upgrading an aggregator to bonding 6.5, a reboot will be required to load the system into a supported kernel.

Note

If upgrading any node to bonding 6.6 or higher, a reboot will be required to load the system into a supported kernel as well as ensuring the correct kernel modules are loaded.

apt-get update
apt-get install bonding
apt-get upgrade
service bonding restart

While Bonding restarts, bonders will go offline and aggregators will drop traffic from their bonders. Bonding must be restarted immediately after upgrading—it is not supported to upgrade the software while allowing the current running version to continue running for an extended period of time.

If you see a message asking about overwriting a file called authorized_keys, known_ips, /etc/init.d/bonding, or other SD-WAN related files, we recommend installing the package maintainer’s version of the file unless you plan on managing the contents of the file manually. To do so, type “y” at the prompt and press <enter>.

Configuration file `/root/.ssh/authorized_keys'
 ==> File on system created by you or by a script.
 ==> File also in package provided by package maintainer.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** authorized_keys (Y/I/N/O/D/Z) [default=N] ? y
Installing new version of config file /root/.ssh/authorized_keys ...

If you see a similar message saying “Modified (by you or by a script) since installation. Package distributor has shipped an updated version.”, you should also type “y” to install the package maintainer’s version.

If you see a message like this:

W: A error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://bondingadmin.example.com wheezy Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY EC6F2DAFDD29650D

The cause is most likely a missing or out of date repository key. You can update the key on all nodes using the following command on the management server:

push-repository-key

You may also update the key on individual bonders using the instructions on the “Node setup” tab of the appropriate space on the server. See Node setup for more information.

If the node is an aggregator or private WAN router with dynamic routing set up, the peering status should be verified using vtysh after the upgrade.