=================
Upgrading a node
=================

Before upgrading a node, please read our recommendations for `managing a
successful environment upgrade <../../software-releases-and-upgrades/effective-upgrades.html>`__.

.. note::

    Debian has removed the jessie-updates repository for Debian 8 ("Jessie") and all of the
    Debian 7 ("Wheezy") repositories from its mirror network as of March 24, 2019.
    Bonding version 6.4 introduced management of the apt repositories through salt, but if
    an error containing "404 Not Found" is encountered when running `apt-get update`,
    please refer to :ref:`debian-repo-service-bulletin`.

.. note::

    Bonding versions up to and including 6.5 supported 32-bit operating systems,
    but 6.6 and beyond will only support 64-bit. Attempting to upgrade a 32-bit
    node beyond 6.5 may leave the device in an inconsistent state if some
    supporting applications are upgraded beyond what the bonding software
    understands.

Automated bonder upgrades
--------------------------

.. note::

    Aggregators must be upgraded using the individual method below, the
    automated method works only for bonders.

Bonders can be upgraded in bulk through the management server command
line. This method runs upgrade and restart commands on each bonder in
sequence. It is not intended as a completely unattended process—you may
be prompted for input from time to time. It is strongly recommended that
you configure SSH public keys; if you depend on password authentication,
you will be prompted for the bonder root password for each bonder
upgraded. If you have SSH public keys configured, ensure you enable
"agent forwarding" in your SSH client to avoid the password prompts.

Only online bonders are upgraded. Bonders that are not online at the
time the upgrade is executed will need to be upgraded manually later, or
the bulk upgrade script will need to be run again.

To upgrade bonders in bulk, follow these steps:

#. SSH into the management server.
#. Run this command:
   ``upgrade-bonders``
#. Review the list of bonders to be upgraded. Confirm the upgrade by
   typing Y, then <enter>. Each bonder will be upgraded in turn. This
   process may take a long time. Bonders already at the latest software
   version are ignored.
#. During the upgrade, each bonder downloads a new copy of its cached
   configuration file. If this fails, the bonder will offer the prompt
   "Go back and retry?". We recommend you type 1 to not retry. The
   upgrade on that bonder will quit with a failure. You should log into
   that bonder, troubleshoot the network issue preventing the installer
   from downloading the config file, and manually upgrade the bonder.
#. When the script finishes, it shows the name of any bonders whose
   upgrade failed. Investigate why these bonders failed to upgrade, then
   upgrade them manually or run the bulk upgrade script again.

Targeting particular bonders
+++++++++++++++++++++++++++++

The upgrade can be limited to certain bonders by providing arguments to
the script.

Targeting by bond IDs
^^^^^^^^^^^^^^^^^^^^^^

To upgrade individual bonders, provide their bond IDs on the command
line. Multiple IDs can be set. For example,

::

    upgrade-bonders 2 3 5

would upgrade only bonds 2, 3, and 5. No other bonders would be
upgraded.

.. warning::

    The argument refers to the bond ID, not the bonder or node ID.

Targeting by range of bond IDs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To upgrade a set of bonders whose bond ID falls into a certain range,
provide a single argument on the command line in the form
``low-ID:high-ID``. For example,

::

    upgrade-bonders 2:19

would upgrade bonders whose bond ID was between 2 and 19, inclusive.

.. note::
    Use the range method to speed up the upgrade process by updating
    multiple bonders in parallel. Suppose you have 240 bonders to upgrade.
    Running a single bulk upgrade process would take a very long time.
    However, you could run five bulk upgrades, each in its own SSH session
    with a different range of bond IDs. For example:

    - ``upgrade-bonders 1:49``
    - ``upgrade-bonders 50:99``
    - ``upgrade-bonders 100:149``
    - ``upgrade-bonders 150:199``
    - ``upgrade-bonders 200:250``

    The upgrade would complete about five times faster than if you only used
    a single upgrade process. Take care to ensure that the ranges don't
    overlap.

.. warning::

    Only bonders are upgraded, not aggregators. Because there are very few
    aggregators in a partner's environment, and because aggregators are a
    critical part of the SD-WAN service, they should be upgraded
    and restarted manually.

Manual node upgrade
--------------------


Debian 6 ("Squeeze") reached end-of-life on February 29, 2016 and
bonding discontinued support as of version 6.0. Please see `SB-4
2016-03-31 Debian Squeeze
end-of-life <../../service-bulletins/sb-4.html>`__
for details.

Nodes can be upgraded from any software version to the latest version.
It is not expected (or even possible) to upgrade through each new
version in turn until the most recent version is installed.

To upgrade Bonding on a single host, `start a console session on the
host <../accessing-a-node.html>`__.

If the node is an aggregator or private WAN router with dynamic routing
set up, you should perform a Quagga configuration check before the
upgrade. To view the running configuration, run:

::

    vtysh -c "show running-config"

This configuration should be compared to the startup configuration. If
using BGP, run:

::

    cat /etc/quagga/bgpd.conf

If using OSPF, run:

::

    cat /etc/quagga/ospfd.conf

If the configurations do not match and the changes to the running-config
are required to be persistent, they should be saved:

::

    vtysh -c write

If the running-config needs to be adjusted, perform the necessary
changes using vtysh and verify that the peering is working and saved
before continuing.

Initial upgrade to 6.1
-----------------------

.. note::

    The private WAN packages pwan-aggregator and pwan-router had their
    functionality merged into the bonding package and will no longer need
    to be upgraded separately.
    Take note of the revised upgrade procedure below when upgrading
    aggregators integrated with private WAN and private WAN routers from
    6.0 to 6.1. Bonders will continue to follow the standard upgrade
    procedure.

For aggregators integrated with a private WAN (PWAN) network, you must
stop pwan-aggregator before upgrading to 6.1:

::

    apt-get update
    service pwan-aggregator stop
    apt-get install bonding
    apt-get upgrade
    reboot

And for private WAN routers, you must also stop the pwan-router service
before upgrading to 6.1:

::

    apt-get update
    service pwan-router stop
    apt-get install bonding
    apt-get upgrade
    reboot

Upgrading 6.1 and newer nodes
-----------------------------

.. note::

    Future upgrades on bonders, aggregators, and private WAN routers can be
    simplified to just the following commands.


.. note::

    If upgrading an aggregator to bonding 6.5, a reboot will be required to load
    the system into a supported kernel.


.. note::

    If upgrading any node to bonding 6.6 or higher, a reboot will be required to
    load the system into a supported kernel as well as ensuring the correct kernel
    modules are loaded.


::

    apt-get update
    apt-get install bonding
    apt-get upgrade
    service bonding restart

While Bonding restarts, bonders will go offline and aggregators will
drop traffic from their bonders. Bonding must be restarted immediately
after upgrading—it is not supported to upgrade the software while
allowing the current running version to continue running for an extended
period of time.

If you see a message asking about overwriting a file called
``authorized_keys``, ``known_ips``, ``/etc/init.d/bonding``, or other
SD-WAN related files, we recommend installing the package
maintainer's version of the file unless you plan on managing the
contents of the file manually. To do so, type "y" at the prompt and
press <enter>.

::

    Configuration file `/root/.ssh/authorized_keys'
     ==> File on system created by you or by a script.
     ==> File also in package provided by package maintainer.
       What would you like to do about it ?  Your options are:
        Y or I  : install the package maintainer's version
        N or O  : keep your currently-installed version
          D     : show the differences between the versions
          Z     : start a shell to examine the situation
     The default action is to keep your current version.
    *** authorized_keys (Y/I/N/O/D/Z) [default=N] ? y
    Installing new version of config file /root/.ssh/authorized_keys ...

If you see a similar message saying "Modified (by you or by a script)
since installation. Package distributor has shipped an updated
version.", you should also type "y" to install the package maintainer's
version.

If you see a message like this:

::

    W: A error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://bondingadmin.example.com wheezy Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY EC6F2DAFDD29650D

The cause is most likely a missing or out of date repository key. You
can update the key on all nodes using the following command on the
management server:

::

    push-repository-key

You may also update the key on individual bonders using the instructions
on the "Node setup" tab of the appropriate space on the server. See
`Node setup <../../spaces/node-setup.html>`__ for more information.

If the node is an aggregator or private WAN router with dynamic routing
set up, the peering status should be verified using vtysh after the
upgrade.
