===================
Effective upgrades
===================

Updates are regularly issued to SD-WAN. Updates add new
features, improve performance, and fix issues, and we recommend you keep
your aggregators and bonders up to date. However, that doesn't mean you
should upgrade your whole environment at once. To reduce the risk of
encountering incompatible changes to updated features,
previously-unknown bugs, or issues related to your specific environment,
you should plan your upgrades carefully and follow these
recommendations.

For an example upgrade programs, refer to the last section of this page.

.. contents:: :depth: 2


Upgrade considerations
-----------------------

Document your upgrade process
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Your SD-WAN environment is unique, and your upgrade process
will be unique. You should develop an upgrade process that works well in
your environment, and document it. This will ensure the upgrade plans
are clear and complete, repeatable, and available to all members of your
team. The upgrade plan can be used with few changes for each upgrade,
saving you time and effort.

Upgrade plans should include:

-  A goal for scheduling upgrades—for example, document that you intend
   to upgrade staging nodes within two weeks of a new release, upgrade
   the first stage of production nodes within three weeks of the
   release, and finish the upgrades within five weeks of a release
-  Identification of which aggregators and bonders will be upgraded in
   which stages
-  The planned date of each upgrade stage
-  The actual date of each upgrade stage and solutions to any issues
   that were encountered

Read the release notes
^^^^^^^^^^^^^^^^^^^^^^^

`Release notes <../release-notes/index.html>`__ describe improvements and
changes in each version of SD-WAN. We know they are boring.
Please read them anyway. Release notes may announce a new feature that
improves your workflow or the resolution of a bug that affects you.

Aggregators must be upgraded before bonders
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Aggregators must be upgraded before bonders. If a bonder uses a later
major version of SD-WAN software than its aggregator (for
example, the bonder runs 6.8 and the aggregator runs 6.7), some
features may not work properly and the end-users' service may be
interrupted.

Limit the size of the default bonder inventory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You may keep an inventory of imaged `default
bonders <../nodes/provisioning-nodes/configuring-default-bonder-via-web-service.html>`__. We
strongly recommend you avoid shipping default bonders that are more than
one version out of date—for example, if the most recent release is
6.8, you might ship out default bonders installed with 6.7 but not
with 6.6. This is easier if you limit the size of your default bonder
inventory to not more than two to four weeks of deployments. If you have
an inventory of default bonders installed with an older software
version, we recommend you re-image them with the new software version
when a new version is released. This will reduce the number of remote
upgrades you will need to perform on newly-installed bonders.

Use a staging environment
^^^^^^^^^^^^^^^^^^^^^^^^^^

We recommend you keep at least one aggregator and one CPE dedicated to
testing upgrades. This is known as a staging environment. When we
release an update, you should upgrade the staging aggregator and CPE
first and verify that the upgraded software works as expected. When you
are confident that the update works, begin upgrading your production
environment.

To be effective, the staging environment should match the production
environment as closely as possible. For example, the staging aggregator
would be integrated into your data center OSPF routing, and the CPE
would have three different types of legs—static, DHCP, and PPPoE, and a
public /30 connected IP or CPE NAT IP. If CPE hardware cannot be
dedicated to the staging environment, the bonder could run as a guest in
the free `VirtualBox <https://www.virtualbox.org/>`__ virtualization
software on a Windows workstation.

The staging bond should be set as `proof of
concept <../bonds/proof-of-concept-bonds.html>`__ so that you are not
charged for its legs. Contact `technical
support <../spaces/technical-support.html>`__ to cancel the expiration
date for the proof of concept bond.

Upgrade production environment in stages
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We recommend you upgrade your aggregators and bonders in stages. If you
upgrade them all at once, you increase the risk of a new bug affecting
your whole environment. In large environments where all bonders cannot
be upgraded to each new version of SD-WAN, consider upgrading
in a "leap frog" pattern, where bonders are split into two groups, and
each group is upgraded to every other new version. With this pattern, no
bonder is ever out of date by more than one version for very long.

See the Examples section below for some specific ideas on staged upgrade
plans. These stages are only a suggestion—you may want to upgrade more
or less aggressively, depending on your own requirements.

Setting the version to be installed
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Before performing an upgrade, visit the `Software
repository <../administration/software-repository.html>`__ administration page to
set the version of SD-WAN available for upgrade. If you wish to
continue to deploy the older version for new nodes during the validation
process, you can set the repository to the previous version after the
initial nodes have been upgraded.

Rolling back to an earlier version
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

While most upgrades go as planned, it's not impossible for a software
upgrade to introduce major unforeseen problems in certain environments.
If an upgrade appears to be causing problems, please contact `Technical
support <../spaces/technical-support.html>`__. We will help you roll back
to an earlier software version if the incident has all these
characteristics:

#. The priority is Urgent or High, according to our incident severity
   matrix
#. It cannot be solved or worked around in a reasonable period of time,
   depending on the issue severity—approximately 1.5 to 3 hours from
   our first notification of the problem
#. It is likely to have been caused by the new software version

Rolling back to an earlier version is very uncommon and can only be done
with assistance `technical support <../spaces/technical-support.html>`__.

Example upgrade programs
-------------------------

Small environment
^^^^^^^^^^^^^^^^^^

In a small environment, each bonder can be upgraded to each new version.
The following method could be used:

-  Day 1: Upgrade staging aggregator and bonder and verify
   functionality.
-  Day 2: Upgrade one aggregator and two bonders on that aggregator.
   Monitor them for 48 hours.
-  Day 4: Upgrade all remaining bonders on the upgraded aggregator.
   Again, monitor them for 48 hours.
-  Day 6: Upgrade all remaining aggregators and two bonders on each of
   those aggregators. Again, monitor them for 48 hours.
-  Day 8: Upgrade all remaining CPEs.

Large environment
^^^^^^^^^^^^^^^^^^

In a large environment, it might not be possible to upgrade each bonder
to each new version of SD-WAN. Consider splitting up the
aggregators and bonders into groups, and upgrading the groups as
described below.

Groups:

-  Pilot group—a small set of production aggregators and bonders that
   are always upgraded first—for example, 10% of the aggregator fleet
   could be in the pilot group, and 30% of the bonders on the pilot
   aggregators could be in the pilot group

   -  Pilot group A—half the pilot aggregators and half of the pilot
      bonders on those aggregators
   -  Pilot group B—all the pilot nodes not in group A

-  Production group

   -  All aggregators not in the pilot group. In general, all
      aggregators must be upgraded before bonders because a bond should
      not be assigned to either a primary or secondary aggregator that
      runs a SD-WAN version earlier than the version on the
      bonder. Since partners use a variety of aggregator failover
      assignment patterns, it's very complicated to ensure that bonds
      are not assigned to either a primary or secondary aggregator with
      a lower software version unless all aggregators are upgraded
      first.
   -  Production group A—half the production bonders
   -  Production group B—all the production bonders not in group A

Because the pilot group nodes are actual production sites, operating a
pilot group does not eliminate the need for staging nodes as described
above.

Example upgrade plan:

-  Week 1: 6.8 released, upgrade staging aggregator and bonder and
   verify functionality.
-  Week 2: Upgrade pilot group A nodes to 6.8
-  Week 3: Upgrade pilot group B nodes to 6.8
-  Week 4: Upgrade all production aggregators to 6.8
-  Week 5: Upgrade production group A bonders to 6.8—if necessary,
   this could be done in parts through week 6 or 7
