This document is published under the Creative Commons license Attribution 3.0 Netherlands. More information about the license can be found at http://creativecommons.org/licenses/by/3.0/nl/ Ethernet OAM in Icinga and Cacti Erik Ruiter, SURFsara December 2013 Project : Research on Networks (RoN) Projectyear : 2013 Projectmanager : Ronald van der Pol Authors (s) : Erik Ruiter Releasedate : 31-12-2013 Version : 1 Summary Network operators can monitor network connections on the Ethernet layer using the Ethernet Operations, Administration and Monitoring (Ethernet OAM) standards, similar to monitoring on the IP layer using ping and traceroute. This document describes how to integrate Ethernet OAM monitoring in existing Icinga and Cacti monitoring tools. The latest version of this document is also available from https://github.com/sara-nl/eth- oam/wiki/.
Transcript
This document is published under the Creative Commons license Attribution 3.0 Netherlands.
More information about the license can be found at http://creativecommons.org/licenses/by/3.0/nl/
Ethernet OAM in Icinga and Cacti Erik Ruiter, SURFsara
December 2013
Project : Research on Networks (RoN) Projectyear : 2013 Projectmanager : Ronald van der Pol Authors (s) : Erik Ruiter Releasedate : 31-12-2013 Version : 1
Summary Network operators can monitor network connections on the Ethernet layer using the Ethernet Operations, Administration and Monitoring (Ethernet OAM) standards, similar to monitoring on the IP layer using ping and traceroute. This document describes how to integrate Ethernet OAM monitoring in existing Icinga and Cacti monitoring tools.
The latest version of this document is also available from https://github.com/sara-nl/eth-oam/wiki/.
2
Colophon Programme line : Gigaport3 Part : NI hybrid network Activity : Network Virtualisation in the Cloud Deliverable : EDS-13-12R Access rights : public External party : SURFsara
This project was made possible by the support of SURF, the collaborative organisation for higher education institutes and research institutes aimed at breakthrough innovations in ICT. More information
on SURF is available on the website www.surf.nl.
3
Fact Sheet Six matters one should know about Ethernet OAM in Icinga and Cacti.
Scenario Network monitoring between Ethernet switches, where the monitoring results are collected by Icinga or Cacti.
What is it? A how to describing how to deploy the given scenario, including scripts for use in Icinga, Nagios and Cacti.
Whom is it for? Network operators that like to deploy Ethernet OAM in their network.
How does it work? Ethernet switches are configured to enable Ethernet OAM monitoring. A central monitoring tool like Icinga or Cacti runs one of the provided scripts which collect the monitoring results from these switches.
What can one do with it?
Monitor network connection between Ethernet switches, similar to how network connection are monitored between IP routers.
More information This document was prepared by Erik Ruiter with help of Freek Dijkstra. For more information, contact [email protected].
4
Table of Contents 1 . Introduction ..................................................................................................................................... 5 2. Prerequisites and dependencies ....................................................................................... 6
4. Configuring Icinga to use the OAM plugin scripts ............................................. 14 4.1. eth-oam.cfg Example configuration file .......................................................................... 14
1. Introduction Ethernet OAM support is becoming more mature on various switching platforms, however Ethernet OAM is not yet widely supported on monitoring applications. Icinga and Cacti are two widely used monitoring applications that normally focus on layer three devices. Without too much effort SURFsara wrote some extensions that can be used in these applications. This proves that it is not necessarily to have a separate application for Ethernet OAM. This saves the time and resources when deploying Ethernet OAM.
One benefit of Ethernet OAM is the possibility to monitor Ethernet switches which have no IP connectivity. The 802.1ag CFM standard provides a standardized SNMP MIB, which allows monitoring of Maintenance End Points (MEP). This makes it easy to implement it on commonly used monitoring applications like Icinga and Cacti. These applications are open source and widely used. SURFsara developed a plugin for Icinga and Cacti to be able to use this MIB. This was tested on an Ethernet OAM testbed between TNO, SURFnet and SURFsara.
The plugins were developed as part of the Research On Networks (RON) project in 2013, which is funded and organised by SURFnet.
This document describes a HOWTO, which focuses at implementing the plugins for several brands of Ethernet switching hardware.
6
2. Prerequisites and dependencies
2.1. Icinga Icinga is an enterprise grade open source monitoring system which keeps watch over a network and any conceivable network resource, notifies the user of errors and recoveries, and generates performance data for reporting. Scalable and extensible, Icinga can monitor complex, large environments across dispersed locations. Icinga is licensed under GPL v2 and is free to use, distribute and modify.
Icinga is a fork of the Nagios platform. It was created by people from the Nagios community who were displeased with the direction that Nagios was developing. Mainly the lack of development on the core application was reason for people to initiate the development of Icinga. A quickstart guide, including installation instructions can be found on: http://docs.icinga.org/latest/en/quickstart-icinga.html
2.2. Cacti Cacti is a complete frontend to RRDTool, it stores all of the necessary information to create graphs and populate them with data in a MySQL database. The frontend is completely PHP driven. Along with being able to maintain Graphs, Data Sources, and Round Robin Archives in a database, cacti handles the data gathering. There is also SNMP support for those used to creating traffic graphs with MRTG. A complete manual can be found at: http://www.cacti.net/downloads/docs/html/index.html
2.3. net-snmp + OAM mibs To install net-snmp, use the following command:
These can be found at: http://www.ieee802.org/1/files/public/MIBs/
For the Ciena version of the plugin, the following MIB is required:
● WWP-LEOS-CFM-MIB.my
Unfortunately this MIB is not available publicly.
Copy the necessary mibs to /usr/share/mibs/site or any other directory which is accessible by Icinga and net-snmp.
7
2.4. Python net-snmp bindings The Icinga plugin is written in Python and makes use of the Python net-snmp bindings. These can be installed by using the following command:
2.5. ncclient: Python NETconf library There are two scripts as part of the plugins that require the ncclient NETconf library. This mainly used to support Ethernet OAM for Juniper based devices. There are serveral forks of ncclient. Please see https://github.com/Juniper/ncclient for installation instructions and dependencies.
2.6. dot1ag utils The dot1ag-utils software package is an Open Source (new BSD License) implementation of the IEEE 802.1ag protocol and is supported on Linux, FreeBSD and MacOSX servers. It provides several debugging tools that interact with IEEE 802.1ag enabled routers and switches. The package implements the MEP Down functionality with a loopback message initiator (ethping), a link trace message initiator (ethtrace) and a daemon (dot1agd) that responds to loopback messages and link trace messages. Sending and receiving of continuity check messages is done by dot1ag_ccd.
Dot1ag-utils is maintained by SURFnet. See for more information: https://svn.surfnet.nl/trac/dot1ag-utils/wiki
8
3. Icinga plugin documentation The Ethernet OAM Icinga plugin consists of a number of scripts, which can be used to check several OAM features of supported devices.
Overview:
Name Functionality Supported platform
Method
check_cfm_state_8021ag.py check CCM state Any 8021.ag supported device
SNMP standardized MIB
check_cfm_state_ciena.py check CCM state Ciena SNMP proprietary MIB
check_cfm_state_juniper.py check CCM state Juniper Netconf
check_ethping check LBR/LBM Any 8021.ag supported device
dot1ag utils
check_ethtrace check LTR/LTM Any 8021.ag supported device
dot1ag utils
3.1. Loopback message check script The check_ethping script can be used to send loopback requests (LBR) to remote hosts and monitor the results (LBM).
It is actually a wrapper script for the dot1ag-utils ethping utility.
The ethping and ethtrace utility require sudo permissions to execute.
To avoid a password prompt when executing the script, it is possible to add the ethping command to the sudoers file. This can be accomplished by executing the visudo tool.
where <user> is the username that will be used to execute the command (eg. icinga).
3.1.1. Usage check_ethping.py To see all possible flags, issue the `--help` option:
$ ./check_ethping.py --help Usage: check_ethping.py [options] destination_MAC Options: -h, --help show this help message and exit -i INTERFACE, --interface=INTERFACE
9
interface to use -v VLAN, --vlan=VLAN vlan to query -l MDLEVEL, --mdlevel=MDLEVEL OAM Maintenance Level -c COUNT, --count=COUNT number of ethpings to send -w WARN_ON_PACKETLOSS, --warn_on_packetloss=WARN_ON_PACKETLOSS Return warning on packetloss 1=yes 0=no (default=1)
-h / --help This option shows the above listed help message.
-i / --interface This option specifies the interface to use when sending loopback requests.
-v / --vlan This option specifies the vlan in which the destination address is present.
-l / --mdlevel This option specifies management domain level of the destination address.
-c / --count This option specifies the amount of loop back requests to be sent.
-w / --count This option specifies if a warning should be returned, in case of packet-loss of loopback messages. Normally the script only generates a warning if all packets are lost.
3.1.2. Output of loopback message check script Normal result:
$ check_ethping.py -i em1.1235 -l 7 -v 1235 aa:aa:bb:bb:cc:cc PING aa:aa:bb:bb:cc:cc OK - Packet loss = 0%, RTA = 7.8806 ms
3.2. Linktrace check script The check_ethtrace script can be used to execute a linktrace request (LTR) to a MAC address.
It is actually a wrapper script for the dot1ag-utils ethping utility.
It will return the number of hops that it took to reach the MAC address. The script can be optionally be configured to expect a specific number of hops, or a specific path that needs to be taken to reach a MAC address. The ethping and ethtrace utility require sudo permissions to execute.
To avoid a password prompt when executing the script, it is possible to add the ethping command to the sudoers file. This can be accomplished by executing the visudo tool.
where <user> is the username that will be used to execute the command (eg. icinga).
10
3.2.1. 3.2.1 Usage check_trace.py To see all possible flags, issue the `--help` option:
$ sudo ./check_ethtrace.py --help Usage: check_ethtrace.py [options] destination_MAC Options: -h, --help show this help message and exit -i INTERFACE, --interface=INTERFACE interface to use -v VLAN, --vlan=VLAN vlan to query -l MDLEVEL, --mdlevel=MDLEVEL OAM Maintentance Level --hops=HOPS Allowed number of hops (number or range eg. 2:3) --mac_path=MACPATH Specified trace path (use comma separated mac addresses)
-h / --help This option shows the above listed help message.
-i / --interface This option specifies the interface to use when sending loopback requests.
-v / --vlan This option specifies the vlan in which the destination address is present.
-l / --mdlevel This option specifies management domain level of the destination address.
--hops This option specifies a number of expected hops that the linktrace will take. If the result doesn't match the configured hop count, it will return a WARNING.
The hop count can also be configured as a range of minimum and maximum hops. This can be done with the following syntax: --hops 2:4. This means minimum allowed hops is 2, maximum allowed hops is 4.
--mac_path With this option you can specify the path of mac addresses, which is to be expected as the result of the linktrace request. If the path doesn't match, it will return a warning.
The format is --mac_path 11:22:33:44:55:66,aa:bb:cc:dd:ee:ff,99:88:77:66:55:44
3.2.2. Output of link trace check script When the script finishes it will return an errorstate code which is used by Icinga to detect the service state. The script will return 0 ('OK') if there are no issues. The script wil return 1 ('WARNING') if there are problems detected.
3.3. Continuity check script The check_cfm_state scripts can be used to monitor the state of CCM messages, related to specific MEPs. It will detect if there are any of the alarm flags set, and if the port and interface TLVs are 'Up'. The scripts do not provide continuous checking of the CCM state, since the nature of Icinga is to poll only once in a while. This means if there is an alarm occurring just between two polls, this is not detected.
There are currently three implementations of the script, which are functionally almost the same, but are used for different devices:
check_cfm_state_8021ag.py
This implementation supports any device which makes use of the standardized 802.1ag SNMP MIB. It has been tested on a number of devices from different vendors.
check_cfm_state_ciena.py
This implementation makes use of a more or less proprietary Ciena MIB. This MIB is not freely available and should be obtained from Ciena. It has been tested on a Ciena 3960.
check_cfm_state_juniper.py
This implementation supports the Juniper Ex4200 switch. It makes use of the NETconf protocol to retrieve OAM related information from the device. Juniper currently has no 802.1ag SNMP support. This is expected in Q2 2014.
3.3.1. Usage check_cfm_state_8021ag.py and check_cfm_state_ciena.py To see all possible flags, issue the `--help` option:
$ ./check_cfm_state_8021ag.py --help Usage: check_cfm_state_8021ag.py [options] hostname Options: -h, --help show this help message and exit -v SNMP_VERSION, --version=SNMP_VERSION Use specific SNMP version default = 1 -p PORT, --port=PORT SNMP port default = 161 -c COMMUNITY, --community=COMMUNITY SNMP community -m LIST, --mep=LIST comma separated list to specify remote MEPs to monitor, (all = all available MEPs)
-h / --help This option shows the above listed help message.
-v / --version This option specifies the SNMP version, when omitted it uses version 1 as default. Valid options are '1' or '2'.
-p / --port This option allows you to override the default UDP 161 port for SNMP.
12
-c / --community This option specifies the SNMP community of the device you want to query. This option is mandatory.
-m / --mep This option allows you to specify a single or a list of remote MEPs which should be monitored by the plugin. A list can be created by using commas between the individual meps: Single MEP: --mep 800 List of MEPs: --mep 800,810,830 All MEPs: --mep all
You can also query all detected remote MEPs using the --mep all option. This will detect the status of all discovered remote meps. However if the plugin cannot retrieve data for one of the meps, it will not generate any 'NO DATA' message, since the MEP was not specifically mentioned. This option is mandatory.
3.3.2. 3.3.2 Usage check_cfm_state_juniper.py: To see all possible flags, issue the `--help` option:
$ ./check_cfm_state_juniper.py --help Usage: check_cfm_state_juniper.py [options] hostname Options: -h, --help show this help message and exit -P PORT, --port=PORT NETconf port default = 22 -u USERNAME, --username=USERNAME ssh username -p password, --password=password ssh password (can be ommited when using remote ssh key) -m LIST, --mep=LIST comma separated list to specify remote MEPs to monitor, (all = all available MEPs)
-h / --help This option shows the above listed help message.
-P / --port This option allows you to override the default port for SSH (TCP port 22).
-u / --username This option is used to specify the SSH username for Netconf. This option is mandatory.
-p / --password This option is used to specify the SSH password for Netconf. Alternatively you can use a remotely stored SSH key. In this case the password can be omitted.
-m / --mep This option allows you to specify a single or a list of remote MEPs which should be monitored by the plugin. A list can be created by using commas between the individual meps: Single MEP: --mep 800 List of MEPs: --mep 800,810,830 All MEPs: --mep all
You can also query all detected remote MEPs using the --mep all option. This will detect the status of all discovered remote meps. However if the plugin cannot retrieve data for one of the meps, it will not generate any 'NO DATA' message, since the MEP was not specifically mentioned. This option is mandatory.
13
3.3.3. 3.3.3 Output of continuity check scripts When the script finishes it will return an error state code which is used by Icinga to detect the service state. The script will return 0 ('OK') if there are no issues. The script will return 1 ('WARNING') if there are problems detected with one or more MEPs.
This shows that the remote MEP 2022 has a 'Remote MEP State Error'.
14
4. Configuring Icinga to use the OAM plugin scripts Icinga works with configuration files that contain the information needed to define monitoring services.
Configuration files can be added to the configuration directory of Icinga. The location differs per implementation.
Possible locations are /usr/local/icinga/etc/ or /etc/icinga.
You can use the whereis icinga command to find out where Icinga is located. Any additional files can be added to the configuration by specifying them in the icinga.cfg global configuration file. For example you can add our file with the following statement:
Note: normally there is a separate 'objects' folder where configuration files related to objects go.
After updating the file, Icinga has to be reloaded. This can be done with sudo /etc/init.d/icinga reload.
This example configuration file has four elements:
Host definition Specifies information about the host that is being monitored
Host_group definition groups all hosts in a single group.
Service definition Specifies information about what service checks are to be initiated by the host.
Check command definition
Specifies the command-line call to the plugin, including arguments, which is used by the service.
4.1. eth-oam.cfg Example configuration file # HOST DEFINITIONS
define host{ use generic-switch ; Inherit default values from a template host_name nge-2.lighthouse.sara.nl ; The name we're giving to this switch alias SURFsara CIENA 3960 NGE-1 ; longer name associated with the switch address 10.0.102.11 ; IP address of the switch hostgroups TNC2013_DEMO ; Host groups associated with this switch } define host{ use generic-switch host_name oam-nid1.lighthouse.sara.nl alias SURFsara Overture ISG24 bridge address 10.0.102.9 hostgroups TNC2013_DEMO } # HOST GROUP DEFINITIONS define hostgroup{ hostgroup_name TNC2013_DEMO ; The name of the hostgroup
15
alias TNC2013 DEMO ; Long name of the group } # SERVICE DEFINITIONS define service{ use generic-service ; Inherit values from a template host_name oam-nid1.lighthouse.sara.nl ; The name of the host associated with the service service_description check_cfm_state_8021ag ; The service description check_command check_cfm_state_8021ag!TNCDEMO!872 ; The command used to monitor the service normal_check_interval 5 ; Check the service every 5 minutes under normal ; conditions retry_check_interval 1 ; Re-check the service every minute until its ; final/hard state is determined } define service{ use generic-service host_name nge-2.lighthouse.sara.nl service_description check_cfm_state_ciena check_command check_cfm_state_ciena!TNCDEMO!871 normal_check_interval 5 retry_check_interval 1 } # CHECKCOMMAND DEFINITIONS define command { command_name check_cfm_state_ciena command_line $USER1$/plugins-sara/check_cfm_state_ciena.py \ -v 2 -c $ARG1$ -m $ARG2$ $HOSTADDRESS$ } define command { command_name check_cfm_state_8021ag command_line $USER1$/plugins-sara/check_cfm_state_8021ag.py \ -c $ARG1$ -m $ARG2$ $HOSTADDRESS$ }
16
5. Cacti template documentation There are two Cacti templates available, which can monitor and graph Delay Measurement DMM information.
The provided templates consists of a number of XML files, which contain a complete template of graphs, queries and data templates, and can be imported to Cacti using the ‘Import Templates’ Function from the console menu. There are also some supporting scripts provided, information about these files is listed below.
5.1. Ciena Template This template provides a SNMP data query to monitor the Ciena 3960. It provides graphs for monitoring L2 Delay and jitter between 2 endpoints.
Ciena Cacti template locations:
File Required location
cacti_host_template_ciena.xml Import into Cacti using the immport template function.
This file is an XML template which needs to be imported to Cacti. It provides a Ciena host template, with a number of data queries, data templates and graph templates. You can use this to create new devices and enable the Ciena DDM statistics data query.
ciena_cfm.xml
This file contains the data template of the SNMP data query.
5.2. Juniper Template This template provides a Script data query to monitor the Juniper Ex4200. It provides graphs for monitoring L2 Delay and jitter between 2 endpoints. It makes use of Netconf instead of SNMP.
Juniper Cacti template locations:
File Required location
cacti_host_template_juniper.xml
Import into Cacti using the import template function.
This file is an XML template that needs to be imported to Cacti. It provides a Juniper host template, with a number of data queries, data templates and graph templates. You can us this to create new devices and enable the Juniper DDM statistics data query.
juniper_dmm.py:
This file is a python script which handles the data query request from cacti. The Juniper DMM template makes use of NETconf, and therefor requires an external script.
The script references to the netconf_auth file location. Adjust this to your needs.
The script does not do a SSH hostkey verification. Cacti is most often run by a system user, which does not have a home directory with a known_host file. If you would like to enable this check, it can be enabled by adjusting the script. Look for the variable hostkey_verify=False, and adjust it to True.
juniper_dmm.xml:
This file contains the data template of the data query, and references to the juniper_dmm.py script. Adjust your Cacti directory and Python binary location within the file when necessary.
netconf_auth:
This file handles NETconf authentication per host. For each host configured in cacti, you can add a line, and add the username, password, and TCP port which is used by Netconf.
Cacti does not natively support NETconf, since it is an SNMP monitoring tool, therefor this file is required.
The format is: <host>:<username>:<password>:<port>
18
6. Some issues
6.1. Overture 24 The initial version of the Overture firmware that we used had a bug, which caused that no CCM messages were processed by the CPU. This was initially fixed by reverting to an older Firmware. Later on Overture released an update, which fixed the problem.
Overture firmware version
CCM issue
4.9.1 No issue
4.9.2.1 YES
4.9.3.1 No issue
6.2. MRV The MRV device that we use currently has an issue, which does not correctly display the MA name correctly when querying it in SNMP. Instead of displaying the name, it displays the oid leaf index of the MA. This still needs to be further investigated.
6.3. Ciena MIB problem During implementation it was discovered that the WWP-LEOS-CFM-MIB.my did not response to SNMP queries. It turned out that our MEP IDs were higher than 100. This was in conflict with the wwpLeosCfmRemoteMEPID range value defined in the MIB. This was set to 'Unsigned32 (1..100)' at line 2959. When it was adjusted to 'Unsigned32 (1..8192)'. SNMP response messages were received normally.
6.4. Cacti template import issue If you get the error in subject importing a template, simply edit /var/www/cacti/include/global_array.php, at line 558 add a comma at the end "0.8.7i" => "0023", and after it add a new line "0.8.8" => "0024" This should resolve the problem.
19
7. Configuration examples
7.1. OAM configuration for Ciena 3960 This section shows an overview of commands and examples which are used to configure Ethernet OAM related settings for the Ciena 3960 Ethernet switch.
Creating a MEP:
NGE-1> cfm enable NGE-1> cfm service create vlan 2003 name vpls2003 md md2 ma-name-string vpls2003 NGE-1> cfm service enable service vpls2003 NGE-1> cfm mep create service vpls2003 port 2 type down mepid 851 NGE-1> cfm mip create vlan 2003 port 2 level 2
NGE-1> cfm show +------------ CFM GLOBAL CONFIGURATION -------------+ | Parameter | Value | +-------------------------------+-------------------+ | Admin State | Enabled | | L2 Loop Detection State | Disabled | | Ethertype | 0x8902 | | Y1731 Ethertype | 0x8902 | | Remote MEP Hold Time (ms) | 10000 | | 802.1ad Strict Mode | Off | | MIP Level Enforcement | On | | MIP CCM Database Learning | Enabled | | VS Automatic MEP Creation | On | | Frames/sec Avail | 1496 | | Loopback Default Count | 1 | | Loopback Default Interval (ms)| 100 | | Loopback Default Timeout (ms) | 5000 | | Source MAC For PBT CFM Frames | 00:03:18:89:A3:20 | | Frame Classification Mode | Per Snet | | Total Rx Frames | 237262639 | | Total Tx Frames | 280820416 | +-------------------------------+-------------------+
+--------------- CFM GLOBAL RESOURCES ---------------+ | Resource | Current | Maximum | +--------------------------------+---------+---------+ | CFM Frames per Second | 504 | 2000 |
7.2. OAM configuration example for Overture 24 The Overture ISG 24 can be configured through CLI or webGUI. The webGUI however is different per firmware version.
7.3. Netconf / OAM configuration example for Juniper EX4200 This example creates a MEP with number 888 in the maintenance domain md7 with association TNCDEMO. It sens out CCM frames in an interval of 100 milliseconds. It also transmits the port-status and interface-status TLVs. In the performance-monitoring section a DMM sla-iterator-profile is configured. This is assigned to a MEP and provides DMM delay and measurement statistics.
IEEE Std 802.1agTM-2007 IEEE Standard for Local and metropolitan area networks; Virtual Bridged Local Area Networks Amendment 5: Connectivity Fault Management: http://standards.ieee.org/getieee802/download/802.1ag-2007.pdf
Juniper Day One: Navigating The Junos XML Hierarchy: http://www.juniper.net/us/en/community/junos/training-certification/day-one/automation-series/junos-xml-hierarchy/
Juniper ITU-T Y.1731 Ethernet Service OAM Feature Guide for Routing Devices: http://www.juniper.net/techpubs/en_US/junos13.2/information-products/pathway-pages/config-guide-network-interfaces/ethernet-itu-t-y-1731-ethernet-service-oam.pdf
Acknowledgements The plugins were developed in 2013 as part of the Research on Networks (RoN) track with the Gigaport project, which is funded by SURF and organised by SURFnet.
