=====================================
Bonded Internet 2015.4 release notes
=====================================

November 21, 2015

2015.4 is the most significant release of Bonded Internet ever. It
introduces a number of major features. Here are the highlights:

-  Bonds can be assigned to groups known as spaces, and users can be
   limited to view or manage bonds only in certain spaces. This allows
   partners to offer access to the management application to resellers
   and to representatives from end-user companies. Spaces are arranged
   in a hierarchy, allowing a variety of flexible arrangements.
-  Traffic can be routed privately between bonds in a space using the
   new private WAN functionality. This can be configured from the web
   application, and aggregator failover is supported. A new node type is
   introduced, the private WAN router, which performs the private
   routing between aggregators.
-  Users can be limited to perform only certain actions. Permission can
   be granted to view or manage bonds, legs, speed tests, spaces, users,
   etc. Partners can now give limited access to users who are trusted to
   view data but not change some or all data.
-  Branding options are available for customizing the web application
   appearance. Each space can have its own appearance.
-  IP management features allow subnets to be assigned to datacentre
   records and to be allocated to spaces so that space administrators
   can manage their own IP addresses.
-  Bonders and aggregators now detect link packet loss and can reduce a
   link's speed or remove it from the bond based on its packet loss
   rate. Hosts can also detect a flapping link and remove it from the
   bond until the link becomes more stable.
-  Leg and bond tuning procedures are now automated. Appropriate leg
   speeds and bond settings can be automatically detected in order to
   maintain optimum performance with minimal effort.

A new service for managing nodes has been integrated with the management
server. This service runs on TCP ports 4505 and 4506. If your management
server is behind a firewall, please ensure you add inbound allow rules
for those TCP ports.

As announced in 2015.3, this release removes support for the performance
charting features of Bonded Internet 2014.3 and earlier. Bonders and
aggregators running 2014.3 and earlier will no longer show performance
charts.

As always, all new functionality is available in the management server
API.

Now, the details:

Bonding Node
-------------

Additions
^^^^^^^^^^

-  Nodes detect link packet loss rates. The loss rate of a link is used
   to slow the link, if bandwidth adaptation is enabled, or to
   completely remove it from the bond, depending on the loss rate. The
   loss rate is also used to make MTU detection more reliable.
-  Nodes detect flapping links. If a link goes down and back up
   repeatedly, it is removed from the bond for longer and longer periods
   of time, until it becomes stable and remains up. An unstable link may
   be removed from the bond for up to 30 seconds.
-  Nodes can automatically set the length of time packets are kept in
   the reordering buffer that reduces or eliminates out-of-order packet
   delivery when bonding legs with different latencies.
-  PPTP, GRE, and certain types of FTP connections can now be used with
   CPE NAT IPs, because the appropriate Linux NAT modules are loaded on
   bonders and aggregators.
-  The files ``/root/.ssh/authorized_keys/``,
   ``/etc/firewall.d/known_ips``, and ``/etc/resolv.conf`` are no longer
   managed from a partner-specific Debian package. They are managed via
   the SaltStack configuration management service on the management
   server. ``resolv.conf`` is no longer managed on aggregators; it is
   only managed on bonders.

Removals
^^^^^^^^^

-  The CUSTOMER and DESCRIPTION fields have been removed from the bond
   hook environment.

Changes
^^^^^^^^

-  Bandwidth adaptation is less sensitive to link jitter. This helps to
   avoid dropping speed on legs that normally increase in latency under
   load and that do not need to be slowed for minor increases in
   latency.
-  Legs, connected IPs, CPE NAT IPs, and routes start more reliably.
   Previously, if an object could not be started, it would remain down
   until the bond was restarted, but now these objects are restarted at
   increasing intervals.
-  Packet reordering is disabled when only a single link is available.

Fixes
^^^^^^

-  When TCP proxy is enabled, bulk and interactive flows share bandwidth
   more evenly. Previously, interactive flows could hang under the load
   of bulk flows when TCP proxy was enabled.
-  A number of minor bugs have been fixed.

Patches
^^^^^^^^

:2015.4-17: Fix MTU detection on single leg bonds with encryption
  enabled. Improve packet loss reliability. Fix issue supervising DHCP
  on mobile broadband legs. Fix rare TCP proxy crash. Prevent starting
  multiple instances of cell service. Fix rare node service crash when
  stopping bonding.
:2015.4-18: Fix issue preventing bonders behind bonders on same
  aggregator. Upgrade to GnuTLS 2 to avoid broken Debian update.
:2015.4-19: Improve salt service logging. Fix rare crash in TCP proxy.
  Fix supervision for cell service. Handle "cannot find device" issue in
  node service.
:2015.4-20: Restart leg when assigned port number changes. Fix minor
  issue related to detecting Ethernet modes. Improve reliability of
  stopping legs.
:2015.4-21: Improve handling of TCP proxy buffers to ensure consistent
  performance.
:2015.4-22: Send more detailed information about leg state changes; fix
  issue with Quagga routes persisting incorrectly after an aggregator
  reboot; fix issue with tunnel processes taking 100% CPU sometimes; no
  longer update ``/etc/salt/grains``, so that partners can use it;
  improve Salt configuration with regards to default bonders and
  deconfigured bonders; fix a couple of rare crashes in the tunnel
  application.
:2015.4-23: Update config service with changes to PWAN rule design.
:2015.4-25: Fix memory leak in collectd statistics reporting service
  that occurs on aggregators when a bond is removed from the agg.
:2015.4-26: Fix memory leak in collectd that occurs when the service
  accepting statistics on the management server is unavailable.
:2015.4-28: Fix removal of all Quagga static routes when bonding
  starts—change it to only clear static routes that point to tunnel
  devices.
:2015.4-29: Fix issue that allows management VPN tunnels to remain down
  for an increasing and unlimited amount of time if the management VPN
  server is unavailable. It now will try to start the tunnel at least
  once every 60 seconds.
:2015.4-30: Fix a potential memory leak in the TCP proxy.
:2015.4-31: Correct permissions on /var/log/bonding directory that
  prevented logrotate from rotating bonding log files.
:2015.4-32: Messages sent from nodes to the management server are
  published from the config service. Messages can be seen using the
  ``bondevent`` application.
:2015.4-38: Avoid crash at startup due to Debian changing file
  permissions on a Quagga configuration file:
  https://www.debian.org/security/2016/dsa-3654.en.html

Bonding Admin
--------------

Additions
^^^^^^^^^^

-  Groups known as spaces have been introduced. Spaces create
   independent but related zones in the management server. They can have
   their own bonds, aggregators, QoS profiles, users, private WAN
   settings, and branding options. Spaces are organized in a hierarchy,
   with a root space, and each space able to have multiple child spaces.
   Users can view bonds and other objects within their assigned space
   and its descendant spaces. Spaces can be permitted to use aggregators
   and certain other objects from their parent space, allowing a number
   of deployment scenarios for organizations of different sizes.
-  The customer and description fields have been removed from the bond
   object and have been replaced with a single name field. During the
   2015.4 upgrade, the name field of a bond is set to a value based on
   its customer and description fields in 2015.3. Bonds can now be
   uniquely identified by referring to the name of their space and the
   name of the bond.
-  Private WAN is available in the management application and offers
   many new features, including support for aggregator failover,
   per-space NAT and port forward rules, and automatic routing of
   traffic between aggregators in different datacentres. Private WAN
   requires deploying a new type of node in the datacentre, known as a
   "private WAN router," which handles configuration of the specialized
   routing rules on the aggregators as well as routing private data
   between aggregators and between datacentres. Aggregators and private
   WAN routers are assigned to "routing groups," which roughly
   correspond to datacentres. Outbound Internet access for bonds in a
   private WAN space is available using just a single IP routed to the
   private WAN router, and inbound access for servers hosted within the
   space is possible using 1:1 NAT or port forward rules.
-  Users can be granted a wide variety of granular permissions that
   restrict or allow them to do specific things in the management
   application. Permissions are assigned to groups, and users are given
   permissions by assigning them to those groups. A number of useful
   groups are included by default, and additional groups can be added to
   completely customize authorization in the management server.
   Available permissions include the ability to view, add, change, or
   delete bonds, legs, users, spaces, connected IPs, and more.
-  Users are assigned a role. Roles are selected from a few options such
   as "Systems Administrator", "Technical support 2", "Sales," or
   "End-user." Roles are used to suggest initial group memberships for
   the user.
-  The appearance of the web application can be customized for each
   space. Available branding options include the image, background
   colour, and text colour of the main navigation menu, and colours of
   text, buttons, and messages in the main web page body. Advanced users
   can provide custom CSS to override any other styling. Space
   administrators can supply technical support contact information, such
   as helpdesk email and phone numbers, that is shown to users in child
   spaces.
-  IP management features have been added. Administrators can allocate
   subnets to routing groups and then further delegate parts of those
   subnets to child spaces to allow administrators in child spaces to
   manage their own IP assignments. Reports are available that show the
   availability and usage of delegated IP subnets. IP allocations are
   applied to bond connected IPs, CPE NAT IPs, and routes, and private
   WAN SNAT, 1:1 NAT, and port forward rules. For example, if the subnet
   203.0.113.0/28 is allocated to a certain routing group, and
   203.0.113.0/29 is delegated to a space, then the subnet
   203.0.113.0/30 can be used for a connected IP on a bond in that
   space. Subnets outside of 203.0.113.0/29 could not be assigned to
   that bond.
-  Leg and bond tuning procedures are now automated. A link on the leg
   actions menu starts the speed testing procedure for legs, and a link
   on the bond actions menu opens a dialog that offers a number of
   options for automatically tuning bonds. Bond tests can be run
   immediately, or at a scheduled time in the next 24 hours, or when the
   bonder is first connected. Since the tuning procedure can take a few
   minutes, the user that triggers the tuning is emailed when the
   procedure is complete. The recommended leg or bond settings can be
   applied by clicking a button on the tuning results page.
-  A new API version is available, version 3, at ``/api/v3/``. Version 2
   is deprecated but will continue to be available until at least
   December 2016. All new applications should be written against v3 and
   existing applications should be migrated to v3.
-  Bond options have been added for packet loss detection, flap
   detection, and automatic reorder max-hold. All options are enabled
   for existing bonds.
-  Leg options have been added for packet loss detection thresholds.
   This allows users to set packet loss rate thresholds for slowing a
   leg when bandwidth adaptation is enabled and for removing the leg
   from the bond.
-  The SaltStack configuration management service has been added. It
   requires TCP ports 4505 and 4506 on the management server to be
   available to the world.
-  User accounts can be managed from the API.
-  The first time a user visits the web application after the upgrade, a
   "What's New in 2015.4" dialog is shown. This dialog describes the
   main features in 2015.4. The dialog is also shown to new users
   created after the 2015.4 upgrade.
-  Two text fields have been added to the bond object: "circuit ID" and
   "product." These are free form fields that can be used to record
   relevant information from a partner's own systems and sales
   processes.
-  Two text fields have been added to the bonder and aggregator objects:
   "serial number" and "asset tag." Like the bond circuit ID and product
   fields, these are free-form fields that can be used to record
   relevant information about a bonder or aggregator.
-  A number of pages have been added to the Administration section:

   -  System Charts—shows charts related to management server resources
      such as CPU, memory, and disk space. Previously this was available
      under the main Help menu.
   -  Email—provides options for sending email from the management
      application
   -  Support—provides options for technical support information shown
      to users in the root space
   -  Versions—shows versions of software on the management server and
      web application. Previously this was available under the System
      Info page.
   -  Other settings—shows values of settings coded in the main Python
      configuration file. Previously this was available under the System
      Info page.

Removals
^^^^^^^^^

-  The performance charting features of Bonded Internet 2014.3 and
   earlier have been removed. Nodes running 2014.3 or earlier will no
   longer show performance charts.
-  Node passwords are no longer hidden on the HTML API view. This
   feature caused some browsers to hang on large documents.

Changes
^^^^^^^^

-  New ISO files for provisioning nodes have been created. Please
   discard your existing provisioning disks and use the new ones.
-  The bond balancing algorithm field has been renamed distribution
   algorithm, in order to reduce confusion between bonding and load
   balancing technology.
-  Users created in 2015.4 and later must log in with their e-mail
   address, as users are no longer assigned usernames. Users created in
   2015.3 and earlier can log in with their username or email address.
   Usernames should be considered deprecated and will be removed from a
   future version of Bonded Internet.
-  Connected IP and route objects are described using CIDR notation,
   with no separate netmask field. For example, an IP of 203.1.113.0 and
   netmask 255.255.255.252 is now provided as 203.1.113.0/30.
-  Routing objects (connected IPs, CPE NAT IPs, and routes) can no
   longer be added at the same time as creating a new bond in the web
   interface. To create these objects, first create the bond, then edit
   it and add the routing records. This simplifies form validation
   methods in the application.
-  The default theme has been updated. Forms, dialogs, buttons, panels
   and typography have a modern and distinctive appearance.
-  The main navigation menu has been updated with new links for all the
   new management pages.
-  The main Help menu has been replaced with a menu showing
   documentation links, keyboard shortcuts, and technical support
   contact information.
-  Configuration updates to a node are attempted even if a node seems to
   not be connected to the management server.
-  Failing node configuration updates have ever-longer timeouts, up to
   five minutes between tries.
-  The mobile broadband provider profiles page has been move from the
   Administration page to the Policies menu.
-  Frameworks used by the management application have been upgraded—the
   Django application framework has been upgraded to 1.9, Bootstrap CSS
   to version 3, and Font Awesome icons to version 4.
-  Internal application structure has been reorganized in places to
   improve reliability and maintainability.
-  Performance of the nodeupdates and influxmux services, which collect
   and process a variety of information from nodes, has been improved
   under certain circumstances.

Fixes
^^^^^^

-  Hung configuration updates are restarted automatically.
-  Numerous other bugs have been fixed.

Patches
^^^^^^^^

:2015.4-28: Improve leg form layout.
:2015.4-29: Remove support for Munin charting. Fix various
  permission-related issues. Improve CSS generation and serving.
:2015.4-30: Improve URL handling and filtering on bond list pages.
  Internal JavaScript improvements. Improve speed testing reliability.
:2015.4-31: Handle unicode strings better.
:2015.4-32: Add warnings in IP allocation/delegation section. Add space
  fields to various forms so that spaces can be specified when creating
  objects. Various UI improvements.
:2015.4-33: Improve HTTP query performance. Remove ability to delete
  configuration updates. Provisioning ISOs no longer show graphical
  splash screen. Add permission to update permissions. Various UI
  improvements.
:2015.4-34: Add ability to select columns shown on bond list tables.
  Improve service performance when handling updates from PWAN routers.
  Various UI improvements.
:2015.4-35: Fix issues related to reporting of packet loss/flapping
  legs.
:2015.4-36: Add before/after configuration to automatic tuning results.
  Clarify automatic tuning errors. Make automatic tuning more reliable.
  Various UI improvements.
:2015.4-37: Add space note field. Various UI improvements.
:2015.4-38: Update Salt configurations for PWAN routers. Update v3 API-
  move PPP, Ethernet, DHCP, and security fields to nested resources.
  Various UI improvements.
:2015.4-39: Improve PWAN rule UI layout. Improve validation on IP
  allocations. Fix issues in column chooser and group/user chooser UIs.
  Fix issues on save buttons in administration section.
:2015.4-40: Add permission for viewing/changing bonder, aggregator, and
  PWAN router username/password fields (6 permissions total). Change v2
  API—now returns null instead of real information for bonder resource
  nested in bond resource when user doesn't have appropriate
  permissions. Change v3 API—removed some fields from bonder resource
  nested in bond resource, move node username/password fields to nested
  object and only show when user has appropriate permissions. Remove
  username/password fields from bonder and aggregator edit forms and add
  button showing username/password dialog for users with appropriate
  permissions. Add priority and enabled fields to PWAN routers. Improve
  automated speed tuning reliability. Fix permission bug preventing some
  users from performing leg speed detection.
:2015.4-41: Change node setup page from available to all users, to
  available per-space only to users with appropriate permissions. Make
  ISOs for each space. Add permission for viewing documentation. Don't
  allow routing group to be changed for PWAN routers. Fix issues with
  validating uniqueness of certain PWAN router fields.
:2015.4-42: Add failover leg counts to space statistics endpoint. Don't
  send config updates to nodes that have never before downloaded their
  config. Add a tab to the user edit page that shows all permissions
  assigned to the user. Increase timeout for speed tuning. Prevent
  enabling private WAN on root space unless an internal setting is
  enabled.
:2015.4-43: Fix issue with provisioning ISOs. UI refinements.
:2015.4-44: Improve database schema of IP allocations and delegations.
  Update search and filter fields available for core resources. Fix an
  issue displaying certain documentation snippets. Include all routing
  groups and aggregators that the user has access to on aggregator and
  bonder forms. Fix issue that caused bonder password field to be set to
  blank after editing bond.
:2015.4-45: Fix rendering issues in Microsoft Edge. Other UI
  improvements. Fix an issue in v1 aggregator API.
:2015.4-46: Further improvements to database schema of IP allocations
  and delegations. Make prevented downtime calculations work in
  management server's time zone. Nodeupdates service logs to its own
  file. Remove orphaned database records potentially added after
  2015.4-40 upgrade. UI improvements.
:2015.4-47: Use Salt connection to a central server to manage all
  management servers. Improve maintenance of Salt minion key database.
  Update Squeeze sources.list shown on Node Setup page.
:2015.4-48: Improve handling of legs that have packet loss and flap
  status values inconsistent between bonder and aggregator.
:2015.4-53: Improve validation of routing objects.
:2015.4-59: Add update_leg_port management command.
:2015.4-66: Roll up database schema migrations to accelerate
  installations and upgrades. Add ``update-squeeze-sources`` command.
  Update PWAN rule design, dropping distinction between local and global
  rules.
:2015.4-75: Various IP routing validation fixes and UI fixes.
:2015.4-76: Support running InfluxDB service on a dedicated host.
:2015.4-77: Clear HTTP sessions when restoring database.
:2015.4-79: Add capability for PWAN routers to have a dedicated
  interface for VLAN traffic.
:2015.4-83: Improve UI performance on bond list page.
:2015.4-92: Improve performance of nodeupdates service. Various other
  bugfixes.
:2015.4-93: Improve API performance.
:2015.4-96: Change space private WAN gateway rules to three different
  types of rules instead of a single type of rule.
:2015.4-97: Fix bug in prevented downtime monitoring for bonds in
  non-root spaces.
:2015.4-99: Make nodeupdates service multithreaded if option is
  enabled. Multithreaded mode is disabled by default.
:2015.4-100: Improve bond details page UI performance.
:2015.4-105: Fix bug that prevented deleting legs from bond edit page.
:2015.4-107: Add port range capability to private WAN port forward
  rules.
:2015.4-110: Fix issue that could cause config updates to be executed
  multiple times or in the wrong order.
:2015.4-112: Add performance instrumentation to nodeupdates.
