Effective upgrades¶
Updates are regularly issued to SD-WAN. Updates add new features, improve performance, and fix issues, and we recommend you keep your aggregators and bonders up to date. However, that doesn’t mean you should upgrade your whole environment at once. To reduce the risk of encountering incompatible changes to updated features, previously-unknown bugs, or issues related to your specific environment, you should plan your upgrades carefully and follow these recommendations.
For an example upgrade programs, refer to the last section of this page.
Upgrade considerations¶
Document your upgrade process¶
Your SD-WAN environment is unique, and your upgrade process will be unique. You should develop an upgrade process that works well in your environment, and document it. This will ensure the upgrade plans are clear and complete, repeatable, and available to all members of your team. The upgrade plan can be used with few changes for each upgrade, saving you time and effort.
Upgrade plans should include:
A goal for scheduling upgrades—for example, document that you intend to upgrade staging nodes within two weeks of a new release, upgrade the first stage of production nodes within three weeks of the release, and finish the upgrades within five weeks of a release
Identification of which aggregators and bonders will be upgraded in which stages
The planned date of each upgrade stage
The actual date of each upgrade stage and solutions to any issues that were encountered
Read the release notes¶
Release notes describe improvements and changes in each version of SD-WAN. We know they are boring. Please read them anyway. Release notes may announce a new feature that improves your workflow or the resolution of a bug that affects you.
Aggregators must be upgraded before bonders¶
Aggregators must be upgraded before bonders. If a bonder uses a later major version of SD-WAN software than its aggregator (for example, the bonder runs 2015.2 and the aggregator runs 2015.1), some features may not work properly and the end-users’ service may be interrupted.
Limit the size of the default bonder inventory¶
You may keep an inventory of imaged default bonders. We strongly recommend you avoid shipping default bonders that are more than one version out of date—for example, if the most recent release is 2015.3, you might ship out default bonders installed with 2015.2 but not with 2015.1. This is easier if you limit the size of your default bonder inventory to not more than two to four weeks of deployments. If you have an inventory of default bonders installed with an older software version, we recommend you re-image them with the new software version when a new version is released. This will reduce the number of remote upgrades you will need to perform on newly-installed bonders.
Use a staging environment¶
We recommend you keep at least one aggregator and one CPE dedicated to testing upgrades. This is known as a staging environment. When we release an update, you should upgrade the staging aggregator and CPE first and verify that the upgraded software works as expected. When you are confident that the update works, begin upgrading your production environment.
To be effective, the staging environment should match the production environment as closely as possible. For example, the staging aggregator would be integrated into your data center OSPF routing, and the CPE would have three different types of legs—static, DHCP, and PPPoE, and a public /30 connected IP or CPE NAT IP. If CPE hardware cannot be dedicated to the staging environment, the bonder could run as a guest in the free VirtualBox virtualization software on a Windows workstation.
The staging bond should be set as proof of concept so that you are not charged for its legs. Contact technical support to cancel the expiration date for the proof of concept bond.
Upgrade production environment in stages¶
We recommend you upgrade your aggregators and bonders in stages. If you upgrade them all at once, you increase the risk of a new bug affecting your whole environment. In large environments where all bonders cannot be upgraded to each new version of SD-WAN, consider upgrading in a “leap frog” pattern, where bonders are split into two groups, and each group is upgraded to every other new version. With this pattern, no bonder is ever out of date by more than one version for very long.
See the Examples section below for some specific ideas on staged upgrade plans. These stages are only a suggestion—you may want to upgrade more or less aggressively, depending on your own requirements.
Setting the version to be installed¶
Before performing an upgrade, visit the Software repository administration page to set the version of SD-WAN available for upgrade. If you wish to continue to deploy the older version for new nodes during the validation process, you can set the repository to the previous version after the initial nodes have been upgraded.
Rolling back to an earlier version¶
While most upgrades go as planned, it’s not impossible for a software upgrade to introduce major unforeseen problems in certain environments. If an upgrade appears to be causing problems, please contact Technical support. We will help you roll back to an earlier software version if the incident has all these characteristics:
The priority is Urgent or High, according to our incident severity matrix
It cannot be solved or worked around in a reasonable period of time, depending on the issue severity—approximately 1.5 to 3 hours from our first notification of the problem
It is likely to have been caused by the new software version
Rolling back to an earlier version is very uncommon and can only be done with assistance technical support.
Example upgrade programs¶
Small environment¶
In a small environment, each bonder can be upgraded to each new version. The following method could be used:
Day 1: Upgrade staging aggregator and bonder and verify functionality.
Day 2: Upgrade one aggregator and two bonders on that aggregator. Monitor them for 48 hours.
Day 4: Upgrade all remaining bonders on the upgraded aggregator. Again, monitor them for 48 hours.
Day 6: Upgrade all remaining aggregators and two bonders on each of those aggregators. Again, monitor them for 48 hours.
Day 8: Upgrade all remaining CPEs.
Large environment¶
In a large environment, it might not be possible to upgrade each bonder to each new version of SD-WAN. Consider splitting up the aggregators and bonders into groups, and upgrading the groups as described below.
Groups:
Pilot group—a small set of production aggregators and bonders that are always upgraded first—for example, 10% of the aggregator fleet could be in the pilot group, and 30% of the bonders on the pilot aggregators could be in the pilot group
Pilot group A—half the pilot aggregators and half of the pilot bonders on those aggregators
Pilot group B—all the pilot nodes not in group A
Production group
All aggregators not in the pilot group. In general, all aggregators must be upgraded before bonders because a bond should not be assigned to either a primary or secondary aggregator that runs a SD-WAN version earlier than the version on the bonder. Since partners use a variety of aggregator failover assignment patterns, it’s very complicated to ensure that bonds are not assigned to either a primary or secondary aggregator with a lower software version unless all aggregators are upgraded first.
Production group A—half the production bonders
Production group B—all the production bonders not in group A
Because the pilot group nodes are actual production sites, operating a pilot group does not eliminate the need for staging nodes as described above.
Example upgrade plan:
Week 1: 2015.1 released, upgrade staging aggregator and bonder and verify functionality.
Week 2: Upgrade pilot group A nodes to 2015.1
Week 3: Upgrade pilot group B nodes to 2015.1
Week 4: Upgrade all production aggregators to 2015.1
Week 5: Upgrade production group A bonders to 2015.1—if necessary, this could be done in parts through week 6 or 7
Week 11: 2015.2 released, upgrade staging aggregator and bonder and verify functionality.
Week 12: Upgrade pilot group A nodes to 2015.2
Week 13: Upgrade pilot group B nodes to 2015.2
Week 14: Upgrade all production aggregators to 2015.2
Week 15: Upgrade production group B bonders to 2015.2—if necessary, this could be done in parts through week 16 or 17
Week 21: 2015.3 released, upgrade staging aggregator and bonder and verify functionality. And so on…