===============
Flow collectors
===============

Flow collectors are external hosts that can collect flow data for tracking and
reporting purposes. Bonders can export the flow data to these flow collectors
using industry-standard protocols.

.. note::
    Flow collection is only available on bonders running 6.2 or later.

.. contents:: :depth: 2

.. note::
    Flow collectors can use nDPI to enable some extra protocol identification
    but this requires some extra configuration so ask your support contact if
    you want this enabled

Protocols
=========

The following flow collection protocols are available.

IPFIX
-----

IPFIX (IP Flow Information Export) is a standard protocol for flow export
defined in `RFC7011 <https://tools.ietf.org/html/rfc7011>`__. It is supported
by many modern devices and collectors, and is the recommended protocol for
most situations.

Many collectors will listen for IPFIX data on port 2055 by default.

Netflow (v5 and v9)
-------------------

Netflow is a protocol for flow export designed by Cisco. It is available on
most Cisco products and some products from other vendors. It has been
superceded by the formally standardized IPFIX protocol, which is heavily based
on Netflow v9, but Netflow is still in common use.

Many collectors will listen for Netflow data on port 2055 by default.

sFlow
-----

sFlow is an alternative to the IPFIX/Netflow protocols that operates by random
sampling of traffic instead of tracking individual flows. As such, it is less
accurate, but scales to higher speed networks and provides a more real-time
view of traffic.

Collectors will listen for sFlow data on its official port, 6363.


Managing flow collectors
========================

Each flow collector may be used on one or more bonders. When a collector's
settings are changed, those settings are automatically applied to all bonds
configured to use the collector.

To see the list of flow collectors, select the *Flow Collectors* menu item in
the *Policies* menu at the top of the administration interface. The collectors
for the current space, as well as child spaces, are listed in the *Flow
Collectors* tab, while collectors for parent spaces that are available to the
current space, are listed in the *Inherited Flow Collectors* tab. Inherited
collectors are read-only and may not be edited.

Adding a flow collector
-----------------------

To add a flow collector, click *Add Flow Collector*.

 |add-flow-collector|

See the *Flow collector parameters* section below for a description of the flow collector
parameters.

Editing a flow collector
------------------------

To edit a flow collector, click the flow collector in the list.

See the *Flow collector parameters* section below for a description of the flow collector
parameters.

Deleting a flow collector
-------------------------

To delete a flow collector, click the flow collector in the list and click the *Delete*
button near the top of the page.

 |delete-flow-collector|

Flow collector parameters
-------------------------

.. glossary::

    Name
        A descriptive name for the flow collector.

    IP
        The IP address to send flow data to.

    Port
        The port to send flow data to. This is typically 2055 for IPFIX and
        Netflow, or 6363 for sFlow.

    Type
        The protocol type. See the *Protocols* section above for descriptions of
        the individual protocols.

    Source IP policy
        The policy to determine the IP address in packets sent to the collector.
        For IPFIX and Netflow, this determines the source address of the packets
        sent. For sFlow, this determines the ``agentIP`` field in the messages. If
        *Connected IP* is selected, the first Connected IP on the bond will be
        used. If *Management VPN IP* is selected, the management VPN IP will be
        used. If *Tunnel* is selected, the tunnel IP address will be used, which
        will be NAT'ed to the aggregator's IP address for IPFIX and Netflow
        packets. If *Connected IP* is selected and there are no connected IPs
        defined, the tunnel IP address is used.

    Space
        The space for the collector. The collector will be available to bonds in
        this space, as well as child spaces, if the child spaces have the *Use
        flow collectors from parent space* parameter enabled.

    IP precedence
        If set, packets sent to the collectors will contain the given IP
        precedence value.

    Direction
        Defines traffic direction. Can be ``in`` or ``out``.

    Timeouts
        Allows to tune a set of timeouts to be applied over collected packets. The
        value is expected in the following form: ``name=value:name2=value2:...``.
        The set of supported timeouts, in seconds, and their default values are
        listed below:

        * ``tcp``	(generic tcp flow life)	3600
        * ``tcp.rst`` (TCP RST flow life)	120
        * ``tcp.fin``	(TCP FIN flow life)	300
        * ``udp``	(UDP flow life)		300
        * ``icmp``	(ICMP flow life)	300
        * ``general``	(generic flow life)	3600
        * ``maxlife``	(maximum flow life)	604800
        * ``expint``	(expiry interval)	60

        ``expint`` is a passive timeout, ie. by default if a flow does not make
        any traffic in 60 secs it gets evicted from the cache. ``tcp``,
        ``tcp.rst``, ``tcp.fin``, ``udp``, ``icmp`` and ``general`` are active
        timeouts, ie. by default a tcp flow is evicted after 3600 secs even if
        still active, that is, it's still making traffic.

        This parameter only applies to IPFIX and NetFlow collectors.

    Hop limit
        Value of TTL for the newly generated NetFlow datagrams. This parameter
        only applies to IPFIX and NetFlow collectors.

    Maximum flows
        Maximum number of flows that can be tracked simultaneously. This parameter
        only applies to IPFIX and NetFlow collectors.

    Engine
        Allows to define ``Engine ID`` and ``Engine Type`` fields. It applies only
        to NetFlow v5/v9 and IPFIX. In NetFlow v9/IPFIX, the supplied value fills
        last two bytes of ``SourceID`` field. Expects two non-negative numbers, up
        to 255 each and separated by the ``:`` symbol. It also allows a collector
        to distinguish between distinct probe instances running on the same
        device, which is also important for letting NetFlow v9/IPFIX templates
        work correctly: template IDs will only get automatically selected inside
        single instances.

    Set AgentSubId
        If enabled, sets the value of ``agentSubId`` field inside the sFlow
        datagram header to the bond ID. This parameter only applies to sFlow
        collectors.

    Interface speed
        Statically associates an interface speed (in bits/second) for an interface
        whose traffic is exported to an sFlow collector.


Considerations for NAT
======================

If a bond is using private addresses via NAT, it is important to note that
some collectors may be unable to properly distinguish flow data in certain
cases.

Duplicate address ranges
------------------------

If multiple bonds are using the same address ranges, some collectors will
aggregate the data for duplicate IP addresses, making it impossible to
distinguish which bond the traffic passed through. Some of those collectors
have ways to distinguish this traffic using, for example, the source IP
address of the flow collection traffic, or via multiple collectors defined on
separate ports.

Source IP of flow collection traffic
------------------------------------

Some collectors use the address the flow collection traffic arrived from to
determine which device the traffic passed though. If this traffic passed
through NAT, the address seen by the collector will match the NAT'ed IP, not
the private IP.

The *Source IP policy* parameter on the collector setup defines what IP the
bonder will use when it sends the traffic data to the collector in order to
help the collector distinguish between bonds. For IPFIX and NetFlow, the
source IP of the packets will match the IP for the policy.

If the bond is using a CPE NAT IP, the collector will see the CPE NAT IP as
the source address. If the bond is in a private WAN space with a NAT-via-PWAN
gateway, the collector will see the gateway IP as the source address.

Counteracting NAT effects
-------------------------

In the case where a CPE NAT IP is used, the collector should simply be able to
differentiate the traffic by tracking the CPE NAT IP.

Private WAN is not as simple, however, since multiple bonds share a single
external address. One way to counteract the effect of NAT would be to pass
traffic at the gateway without NAT, using an external gateway. That gateway
can be set up to perform NAT when communicating with external networks, but
not when sending traffic to the flow collector, preserving the source IP
address.

Another option is to use the management VPN address. This involves placing a
central collector in the management VPN network, terminated at the management
server. This will require some planning and manual setup. Please contact
support if you wish to pursue this option.


.. |add-flow-collector| image:: ./add-flow-collector.png
.. |delete-flow-collector| image:: ./delete-flow-collector.png
