BAROMETER - OPNFV

BAROMETERRelease Latest

Open Platform for NFV

Jul 01, 2021

CONTENTS

1 OPNFV Barometer configuration Guide 31.1 Barometer post installation procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.1.1 Automated post installation activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.1.2 Barometer post configuration procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Anuket Barometer User Guide 52.1 OPNFV Barometer User Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.1 Barometer collectd plugins description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.1.2 Collectd capabilities and usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2 VES Application User Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252.2.1 Install Kafka Broker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262.2.2 Install collectd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272.2.3 Setup VES application (guest mode) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282.2.4 Setup VES application (hypervisor mode) . . . . . . . . . . . . . . . . . . . . . . . . . . . 302.2.5 Setup VES application (host mode) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312.2.6 Setup VES Test Collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332.2.7 VES application configuration description . . . . . . . . . . . . . . . . . . . . . . . . . . . 342.2.8 VES YAML configuration format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342.2.9 VES message types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352.2.10 Collectd metrics in VES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362.2.11 VES YAML references . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362.2.12 Collectd metric mapping YAML tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382.2.13 Collectd event mapping YAML tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402.2.14 Helper YAML tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422.2.15 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

2.3 Anuket Barometer Docker Install Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442.3.1 Barometer Docker Images Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462.3.2 Installing Docker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472.3.3 Build and Run Collectd Docker Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492.3.4 Build and Run InfluxDB and Grafana docker images . . . . . . . . . . . . . . . . . . . . . 532.3.5 Run the Influxdb and Grafana Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552.3.6 Build and Run VES and Kafka Docker Images . . . . . . . . . . . . . . . . . . . . . . . . 572.3.7 Build and Run DMA and Redis Docker Images . . . . . . . . . . . . . . . . . . . . . . . . 59

2.4 OPNFV Barometer One Click Install Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 612.4.1 One Click Install with Ansible . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

3 Anuket Barometer Release Notes 693.1 Barometer Kali Release Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

3.1.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 693.1.2 Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

i

3.1.3 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 693.2 Barometer Release Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

3.2.1 Important notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 703.2.2 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 703.2.3 Release Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 703.2.4 Deliverables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 713.2.5 Known Limitations, Issues and Workarounds . . . . . . . . . . . . . . . . . . . . . . . . . 723.2.6 Test Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 733.2.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

4 OPNFV Barometer Requirements 754.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 754.2 Barometer updated scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

4.2.1 Scope of SFQM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 764.2.2 Consumption Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

4.3 collectd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 774.4 collectd statistics and Notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 774.5 DPDK Enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

4.5.1 Measuring Telco Traffic and Performance KPIs . . . . . . . . . . . . . . . . . . . . . . . . 784.5.2 Monitoring DPDK interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 794.5.3 DPDK Keep Alive description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

5 Indices 83

ii

BAROMETER, Release Latest

Project Barometer

Authors Maryam Tahhan <[email protected]>

History

Date Description16.12.2014 Project creation

Barometer is the project that renames Software Fastpath service Quality Metrics (SFQM) and updates its scope whichwas networking centric.

The goal of SFQM was to develop the utilities and libraries in DPDK to support:

• Measuring Telco Traffic and Performance KPIs. Including:

– Packet Delay Variation (by enabling TX and RX time stamping).

– Packet loss (by exposing extended NIC stats).

• Performance Monitoring of the DPDK interfaces (by exposing extended NIC stats + collectd Plugin).

• Detecting and reporting violations that can be consumed by VNFs and higher level management systems(through DPDK Keep Alive).

With Barometer the scope is extended to monitoring the NFVI. The ability to monitor the Network Function Vir-tualization Infrastructure (NFVI) where VNFs are in operation will be a key part of Service Assurance within anNFV environment, in order to enforce SLAs or to detect violations, faults or degradation in the performance of NFVIresources so that events and relevant metrics are reported to higher level fault management systems. If physical appli-ances are going to be replaced by virtualized appliances the service levels, manageability and service assurance needsto remain consistent or improve on what is available today. As such, the NFVI needs to support the ability to monitor:

• Traffic monitoring and performance monitoring of the components that provide networking functionality tothe VNF, including: physical interfaces, virtual switch interfaces and flows, as well as the virtual interfacesthemselves and their status, etc.

• Platform monitoring including: CPU, memory, load, cache, themals, fan speeds, voltages and machine checkexceptions, etc.

All of the statistics and events gathered must be collected in-service and must be capable of being reported by standardTelco mechanisms (e.g. SNMP), for potential enforcement or correction actions. In addition, this information couldbe fed to analytics systems to enable failure prediction, and can also be used for intelligent workload placement.

All developed features will be upstreamed to Open Source projects relevant to telemetry such as collectd, and relevantOpenstack projects.

CONTENTS 1

https://wiki.anuket.io/display/HOME/Barometer

mailto:[email protected]

https://collectd.org/


2 CONTENTS

CHAPTER

ONE

OPNFV BAROMETER CONFIGURATION GUIDE

1.1 Barometer post installation procedures

This document describes briefly the methods of validating the Barometer installation.

1.1.1 Automated post installation activities

The Barometer test-suite in Functest is called barometercollectd and is part of the Features tier. Runningthese tests is done automatically by the OPNFV deployment pipeline on the supported scenarios. The testing consistsof basic verifications that each plugin is functional per their default configurations. Inside the Functest container, thedetailed results can be found in the /home/opnfv/functest/results/barometercollectd.log.

1.1.2 Barometer post configuration procedures

The functionality for each plugin (such as enabling/disabling and configuring its capabilities) is controlled as describedin the User Guide through their individual .conf file located in the /etc/collectd/collectd.conf.d/folder on the compute node(s). In order for any changes to take effect, the collectd service must be stopped and thenstarted again.

3


4 Chapter 1. OPNFV Barometer configuration Guide

CHAPTER

TWO

ANUKET BAROMETER USER GUIDE

2.1 OPNFV Barometer User Guide

2.1.1 Barometer collectd plugins description

Collectd is a daemon which collects system performance statistics periodically and provides a variety of mechanismsto publish the collected metrics. It supports more than 90 different input and output plugins. Input plugins retrievemetrics and publish them to the collectd deamon, while output plugins publish the data they receive to an end point.collectd also has infrastructure to support thresholding and notification.

Barometer has enabled the following collectd plugins:

• dpdkstat plugin: A read plugin that retrieves stats from the DPDK extended NIC stats API.

• dpdkevents plugin: A read plugin that retrieves DPDK link status and DPDK forwarding cores liveliness status(DPDK Keep Alive).

• dpdk_telemetry plugin: A read plugin to collect dpdk interface stats and application or global stats fromdpdk telemetry library. Both ‘dpdkstat’ and ‘dpdk_telemetry’ plugins provides dpdk NIC Stats, but only‘dpdk_telemetry’ provides the DPDK Application stats. So in other words, ‘dpdk_telemetry’ is an advancedversion of dpdkstat. This plugin don’t deal with dpdk events. So not in related with ‘dpdkevents’ plugin. Themimimum dpdk version required to use this plugin is 19.08.

Note: dpdpkstat and dpdk_telemetry should not be used together. Use dpdk_telemetry if your version of dpdksupports it (i.e. DPDK >= 19.08) and use dpdkstat otherwise. dpdkstat, dpdkevents and dpdk_telemetry pluginsshould only be used if your dpdk application doesn’t already have more relevant metrics available(e.g.ovs_stats).

• gnocchi plugin: A write plugin that pushes the retrieved stats to Gnocchi. It’s capable of pushing any stats readthrough collectd to Gnocchi, not just the DPDK stats.

• aodh plugin: A notification plugin that pushes events to Aodh, and creates/updates alarms appropriately.

• hugepages plugin: A read plugin that retrieves the number of available and free hugepages on a platform as wellas what is available in terms of hugepages per socket.

• Open vSwitch events Plugin: A read plugin that retrieves events from OVS.

• Open vSwitch stats Plugin: A read plugin that retrieves flow and interface stats from OVS.

• mcelog plugin: A read plugin that uses mcelog client protocol to check for memory Machine Check Exceptionsand sends the stats for reported exceptions.

• PMU plugin: A read plugin that provides performance counters data on Intel CPUs using Linux perf interface.

• RDT plugin: A read plugin that provides the last level cache utilization and memory bandwidth utilization.

5

https://github.com/openstack/collectd-openstack-plugins/tree/stable/ocata/

https://github.com/openstack/collectd-openstack-plugins/tree/stable/ocata/


• virt: A read plugin that uses virtualization API libvirt to gather statistics about virtualized guests on a systemdirectly from the hypervisor, without a need to install collectd instance on the guest.

• SNMP Agent: A write plugin that will act as a AgentX subagent that receives and handles queries from SNMPmaster agent and returns the data collected by read plugins. The SNMP Agent plugin handles requests only forOIDs specified in configuration file. To handle SNMP queries the plugin gets data from collectd and translatesrequested values from collectd’s internal format to SNMP format. Supports SNMP: get, getnext and walkrequests.

All the plugins above are available on the collectd main branch, except for the Gnocchi and Aodh plugins as they arePython-based plugins and only C plugins are accepted by the collectd community. The Gnocchi and Aodh plugins livein the OpenStack repositories.

Other plugins existing as a pull request into collectd main:

• Legacy/IPMI: A read plugin that reports platform thermals, voltages, fanspeed, current, flow, power etc. Also,the plugin monitors Intelligent Platform Management Interface (IPMI) System Event Log (SEL) and sends theappropriate notifications based on monitored SEL events.

• PCIe AER: A read plugin that monitors PCIe standard and advanced errors and sends notifications about thoseerrors.

Third party application in Barometer repository:

• Open vSwitch PMD stats: An aplication that retrieves PMD stats from OVS. It is run through exec plugin.

Plugins and application included in the Euphrates release:

Write Plugins: aodh plugin, SNMP agent plugin, gnocchi plugin.

Read Plugins/application: Intel RDT plugin, virt plugin, Open vSwitch stats plugin, Open vSwitch PMD stats appli-cation.

2.1.2 Collectd capabilities and usage

Note: Plugins included in the OPNFV E release will be built-in for Apex integration and can be configured as shownin the examples below.

The collectd plugins in OPNFV are configured with reasonable defaults, but can be overridden.

Building all Barometer upstreamed plugins from scratch

The plugins that have been merged to the collectd main branch can all be built and configured through the barometerrepository.

Note:

• sudo permissions are required to install collectd.

• These instructions are for Centos 7.

To build all the upstream plugins, clone the barometer repo:

$ git clone https://gerrit.opnfv.org/gerrit/barometer

To install collectd as a service and install all it’s dependencies:

6 Chapter 2. Anuket Barometer User Guide


$ cd barometer/systems && ./build_base_machine.sh

This will install collectd as a service and the base install directory will be /opt/collectd.

Sample configuration files can be found in ‘/opt/collectd/etc/collectd.conf.d’

Note: If you don’t want to use one of the Barometer plugins, simply remove the sample config file from‘/opt/collectd/etc/collectd.conf.d’

Note: If you plan on using the Exec plugin (for OVS_PMD_STATS or for executing scripts on notification gener-ation), the plugin requires a non-root user to execute scripts. By default, collectd_exec user is used in the exec.confprovided in the sample configurations directory under src/collectd in the Barometer repo. These scripts DO NOT cre-ate this user. You need to create this user or modify the configuration in the sample configurations directory undersrc/collectd to use another existing non root user before running build_base_machine.sh.

Note: If you are using any Open vSwitch plugins you need to run:

$ sudo ovs-vsctl set-manager ptcp:6640

After this, you should be able to start collectd as a service

$ sudo systemctl status collectd

If you want to use granfana to display the metrics you collect, please see: grafana guide

For more information on configuring and installing OpenStack plugins for collectd, check out the collectd-openstack-plugins GSG.

Below is the per plugin installation and configuration guide, if you only want to install some/particular plugins.

DPDK plugins

Repo: https://github.com/collectd/collectd

Branch: main

Dependencies: DPDK (https://dpdk.org/)

Note: DPDK statistics plugin requires DPDK version 16.04 or later.

To build and install DPDK to /usr please see: https://github.com/collectd/collectd/blob/main/docs/BUILD.dpdkstat.md

Building and installing collectd:

$ git clone https://github.com/collectd/collectd.git$ cd collectd$ ./build.sh$ ./configure --enable-syslog --enable-logfile --enable-debug$ make$ sudo make install

2.1. OPNFV Barometer User Guide 7

https://wiki.anuket.io/display/HOME/Installing+and+configuring+InfluxDB+and+Grafana+to+display+metrics+with+collectd

https://opendev.org/x/collectd-openstack-plugins/src/branch/master/doc/source/GSG.rst


https://github.com/collectd/collectd

https://dpdk.org/

https://github.com/collectd/collectd/blob/main/docs/BUILD.dpdkstat.md


Note: If DPDK was installed in a non standard location you will need to specify paths to the header files and librariesusing LIBDPDK_CPPFLAGS and LIBDPDK_LDFLAGS. You will also need to add the DPDK library symbols to theshared library path using ldconfig. Note that this update to the shared library path is not persistant (i.e. it will notsurvive a reboot).

Example of specifying custom paths to DPDK headers and libraries:

$ ./configure LIBDPDK_CPPFLAGS="path to DPDK header files" LIBDPDK_LDFLAGS="path to→˓DPDK libraries"

This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf)can be found at /opt/collectd/etc. To configure the dpdkstats plugin you need to modify the configuration fileto include:

LoadPlugin dpdkstat<Plugin dpdkstat>

Coremask "0xf"ProcessType "secondary"FilePrefix "rte"EnabledPortMask 0xffffPortName "interface1"PortName "interface2"

</Plugin>

To configure the dpdkevents plugin you need to modify the configuration file to include:

<LoadPlugin dpdkevents>Interval 1

</LoadPlugin>

<Plugin "dpdkevents"><EAL>Coremask "0x1"MemoryChannels "4"FilePrefix "rte"

</EAL><Event "link_status">SendEventsOnUpdate falseEnabledPortMask 0xffffSendNotification true

</Event><Event "keep_alive">SendEventsOnUpdate falseLCoreMask "0xf"KeepAliveShmName "/dpdk_keepalive_shm_name"SendNotification true

</Event></Plugin>

Note: Currently, the DPDK library doesn’t support API to de-initialize the DPDK resources allocated on the ini-tialization. It means, the collectd plugin will not be able to release the allocated DPDK resources (locks/memory/pcibindings etc.) correctly on collectd shutdown or reinitialize the DPDK library if primary DPDK process is restarted.The only way to release those resources is to terminate the process itself. For this reason, the plugin forks off a sep-arate collectd process. This child process becomes a secondary DPDK process which can be run on specific CPUcores configured by user through collectd configuration file (“Coremask” EAL configuration option, the hexadecimal



bitmask of the cores to run on).

For more information on the plugin parameters, please see: https://github.com/collectd/collectd/blob/main/src/collectd.conf.pod

Note: dpdkstat plugin initialization time depends on read interval. It requires 5 read cycles to set up internal buffersand states, during that time no statistics are submitted. Also, if plugin is running and the number of DPDK ports isincreased, internal buffers are resized. That requires 3 read cycles and no port statistics are submitted during that time.

The Address-Space Layout Randomization (ASLR) security feature in Linux should be disabled, in order for the samehugepage memory mappings to be present in all DPDK multi-process applications.

To disable ASLR:

$ sudo echo 0 > /proc/sys/kernel/randomize_va_space

To fully enable ASLR:

$ sudo echo 2 > /proc/sys/kernel/randomize_va_space

Warning: Disabling Address-Space Layout Randomization (ASLR) may have security implications. It is recom-mended to be disabled only when absolutely necessary, and only when all implications of this change have beenunderstood.

For more information on multi-process support, please see: https://doc.dpdk.org/guides/prog_guide/multi_proc_support.html

DPDK stats plugin limitations:

1. The DPDK primary process application should use the same version of DPDK that collectd DPDK plugin isusing;

2. L2 statistics are only supported;

3. The plugin has been tested on Intel NIC’s only.

DPDK stats known issues:

• DPDK port visibility

When network port controlled by Linux is bound to DPDK driver, the port will not be available in the OS. Itaffects the SNMP write plugin as those ports will not be present in standard IF-MIB. Thus, additional work isrequired to be done to support DPDK ports and statistics.

DPDK telemetry plugin

Please refer to https://wiki.anuket.io/display/HOME/DPDK+Telemetry+Plugin


https://github.com/collectd/collectd/blob/main/src/collectd.conf.pod


https://doc.dpdk.org/guides/prog_guide/multi_proc_support.html

https://doc.dpdk.org/guides/prog_guide/multi_proc_support.html

https://wiki.anuket.io/display/HOME/DPDK+Telemetry+Plugin


Hugepages Plugin


Branch: main

Dependencies: None, but assumes hugepages are configured.

To configure some hugepages:

$ sudo mkdir -p /mnt/huge$ sudo mount -t hugetlbfs nodev /mnt/huge$ sudo bash -c "echo 14336 > /sys/devices/system/node/node0/hugepages/hugepages-→˓2048kB/nr_hugepages"


$ git clone https://github.com/collectd/collectd.git$ cd collectd$ ./build.sh$ ./configure --enable-syslog --enable-logfile --enable-hugepages --enable-debug$ make$ sudo make install

This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf)can be found at /opt/collectd/etc. To configure the hugepages plugin you need to modify the configurationfile to include:

LoadPlugin hugepages<Plugin hugepages>

ReportPerNodeHP trueReportRootHP trueValuesPages trueValuesBytes falseValuesPercentage false

</Plugin>


Intel PMU Plugin


Branch: main

Dependencies:

• PMU tools (jevents library) https://github.com/andikleen/pmu-tools

To be suitable for use in collectd plugin shared library libjevents should be compiled as position-independent code. Todo this add the following line to pmu-tools/jevents/Makefile:

CFLAGS += -fPIC

Building and installing jevents library:






https://github.com/andikleen/pmu-tools


$ git clone https://github.com/andikleen/pmu-tools.git$ cd pmu-tools/jevents/$ make$ sudo make install

Download the Hardware Events that are relevant to your CPU, download the appropriate CPU event list json file:

$ wget https://raw.githubusercontent.com/andikleen/pmu-tools/main/event_download.py$ python event_download.py

This will download the json files to the location: $HOME/.cache/pmu-events/. If you don’t want to download thesefiles to the aforementioned location, set the environment variable XDG_CACHE_HOME to the location you want thefiles downloaded to.


$ git clone https://github.com/collectd/collectd.git$ cd collectd$ ./build.sh$ ./configure --enable-syslog --enable-logfile --with-libjevents=/usr/local --enable-→˓debug$ make$ sudo make install

This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf)can be found at /opt/collectd/etc. To configure the PMU plugin you need to modify the configuration file toinclude:

<LoadPlugin intel_pmu>Interval 1

</LoadPlugin><Plugin "intel_pmu">

ReportHardwareCacheEvents trueReportKernelPMUEvents trueReportSoftwareEvents trueCores ""

</Plugin>

If you want to monitor Intel CPU specific CPU events, make sure to enable the additional two options shown below:

<Plugin intel_pmu>ReportHardwareCacheEvents trueReportKernelPMUEvents trueReportSoftwareEvents trueEventList "$HOME/.cache/pmu-events/GenuineIntel-6-2D-core.json"HardwareEvents "L2_RQSTS.CODE_RD_HIT,L2_RQSTS.CODE_RD_MISS" "L2_RQSTS.ALL_CODE_RD"Cores ""

</Plugin>

Note: If you set XDG_CACHE_HOME to anything other than the variable above - you will need to modify the pathfor the EventList configuration.

Use “Cores” option to monitor metrics only for configured cores. If an empty string is provided as value for this fielddefault cores configuration is applied - that is all available cores are monitored separately. To limit monitoring to cores0-7 set the option as shown below:



Cores "[0-7]"


Note: The plugin opens file descriptors whose quantity depends on number of monitored CPUs and number ofmonitored counters. Depending on configuration, it might be required to increase the limit on the number of open filedescriptors allowed. This can be done using ‘ulimit -n’ command. If collectd is executed as a service ‘LimitNOFILE=’directive should be defined in [Service] section of collectd.service file.

Intel RDT Plugin


Branch: main

Dependencies:

• PQoS/Intel RDT library https://github.com/intel/intel-cmt-cat

• msr kernel module

Building and installing PQoS/Intel RDT library:

$ git clone https://github.com/intel/intel-cmt-cat$ cd intel-cmt-cat$ make$ make install PREFIX=/usr

You will need to insert the msr kernel module:

$ modprobe msr


$ git clone https://github.com/collectd/collectd.git$ cd collectd$ ./build.sh$ ./configure --enable-syslog --enable-logfile --with-libpqos=/usr/ --enable-debug$ make$ sudo make install

This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf)can be found at /opt/collectd/etc. To configure the RDT plugin you need to modify the configuration file toinclude:

<LoadPlugin intel_rdt>Interval 1

</LoadPlugin><Plugin "intel_rdt">

Cores ""</Plugin>






https://github.com/intel/intel-cmt-cat




IPMI Plugin


Branch: feat_ipmi_events, feat_ipmi_analog

Dependencies: OpenIPMI library (http://openipmi.sourceforge.net/)

The IPMI plugin is already implemented in the latest collectd and sensors like temperature, voltage, fanspeed, currentare already supported there. The list of supported IPMI sensors has been extended and sensors like flow, power aresupported now. Also, a System Event Log (SEL) notification feature has been introduced.

• The feat_ipmi_events branch includes new SEL feature support in collectd IPMI plugin. If this feature is en-abled, the collectd IPMI plugin will dispatch notifications about new events in System Event Log.

• The feat_ipmi_analog branch includes the support of extended IPMI sensors in collectd IPMI plugin.

Install dependencies

On Centos, install OpenIPMI library:

$ sudo yum install OpenIPMI ipmitool

Anyway, it’s recommended to use the latest version of the OpenIPMI library as it includes fixes of known issueswhich aren’t included in standard OpenIPMI library package. The latest version of the library can be found at https://sourceforge.net/p/openipmi/code/ci/master/tree/. Steps to install the library from sources are described below.

Remove old version of OpenIPMI library:

$ sudo yum remove OpenIPMI ipmitool

Build and install OpenIPMI library:

$ git clone https://git.code.sf.net/p/openipmi/code openipmi-code$ cd openipmi-code$ autoreconf --install$ ./configure --prefix=/usr$ make$ sudo make install

Add the directory containing OpenIPMI*.pc files to the PKG_CONFIG_PATH environment variable:

export PKG_CONFIG_PATH=/usr/lib/pkgconfig

Enable IPMI support in the kernel:

$ sudo modprobe ipmi_devintf$ sudo modprobe ipmi_si

Note: If HW supports IPMI, the /dev/ipmi0 character device will be created.

Clone and install the collectd IPMI plugin:

$ git clone https://github.com/collectd/collectd$ cd collectd$ ./build.sh$ ./configure --enable-syslog --enable-logfile --enable-debug$ make$ sudo make install



http://openipmi.sourceforge.net/

https://sourceforge.net/p/openipmi/code/ci/master/tree/

https://sourceforge.net/p/openipmi/code/ci/master/tree/


This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf)can be found at /opt/collectd/etc. To configure the IPMI plugin you need to modify the file to include:

LoadPlugin ipmi<Plugin ipmi>

<Instance "local">SELEnabled true # only feat_ipmi_events branch supports this

</Instance></Plugin>

Note: By default, IPMI plugin will read all available analog sensor values, dispatch the values to collectd and sendSEL notifications.

For more information on the IPMI plugin parameters and SEL feature configuration, please see: https://github.com/collectd/collectd/blob/main/src/collectd.conf.pod

Extended analog sensors support doesn’t require additional configuration. The usual collectd IPMI documentation canbe used:

• https://collectd.org/wiki/index.php/Plugin:IPMI

• https://collectd.org/documentation/manpages/collectd.conf.5.shtml#plugin_ipmi

IPMI documentation:

• https://www.kernel.org/doc/Documentation/IPMI.txt

• https://www.intel.com/content/www/us/en/products/docs/servers/ipmi/ipmi-second-gen-interface-spec-v2-rev1-1.html

Mcelog Plugin


Branch: main

Dependencies: mcelog

Start by installing mcelog.

Note: The kernel has to have CONFIG_X86_MCE enabled. For 32bit kernels you need atleast a 2.6,30 kernel.

On Centos:

$ sudo yum install mcelog

Or build from source

$ git clone https://git.kernel.org/pub/scm/utils/cpu/mce/mcelog.git$ cd mcelog$ make... become root ...$ make install$ cp mcelog.service /etc/systemd/system/$ systemctl enable mcelog.service$ systemctl start mcelog.service




https://collectd.org/wiki/index.php/Plugin:IPMI

https://collectd.org/documentation/manpages/collectd.conf.5.shtml#plugin_ipmi

https://www.kernel.org/doc/Documentation/IPMI.txt

https://www.intel.com/content/www/us/en/products/docs/servers/ipmi/ipmi-second-gen-interface-spec-v2-rev1-1.html

https://www.intel.com/content/www/us/en/products/docs/servers/ipmi/ipmi-second-gen-interface-spec-v2-rev1-1.html



Verify you got a /dev/mcelog. You can verify the daemon is running completely by running:

$ mcelog --client

This should query the information in the running daemon. If it prints nothing that is fine (no errors logged yet). Moreinfo @ http://www.mcelog.org/installation.html

Modify the mcelog configuration file “/etc/mcelog/mcelog.conf” to include or enable:

socket-path = /var/run/mcelog-client[dimm]dimm-tracking-enabled = yesdmi-prepopulate = yesuc-error-threshold = 1 / 24hce-error-threshold = 10 / 24h

[socket]socket-tracking-enabled = yesmem-uc-error-threshold = 100 / 24hmem-ce-error-threshold = 100 / 24hmem-ce-error-log = yes

[page]memory-ce-threshold = 10 / 24hmemory-ce-log = yesmemory-ce-action = soft

[trigger]children-max = 2directory = /etc/mcelog

Clone and install the collectd mcelog plugin:

$ git clone https://github.com/collectd/collectd$ cd collectd$ ./build.sh$ ./configure --enable-syslog --enable-logfile --enable-debug$ make$ sudo make install

This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf)can be found at /opt/collectd/etc. To configure the mcelog plugin you need to modify the configuration fileto include:

<LoadPlugin mcelog>Interval 1

</LoadPlugin><Plugin mcelog>

<Memory>McelogClientSocket "/var/run/mcelog-client"PersistentNotification false

</Memory>#McelogLogfile "/var/log/mcelog"

</Plugin>


Simulating a Machine Check Exception can be done in one of 3 ways:


http://www.mcelog.org/installation.html




• Running $make test in the mcelog cloned directory - mcelog test suite

• using mce-inject

• using mce-test

mcelog test suite:

It is always a good idea to test an error handling mechanism before it is really needed. mcelog includes a test suite.The test suite relies on mce-inject which needs to be installed and in $PATH.

You also need the mce-inject kernel module configured (with CONFIG_X86_MCE_INJECT=y), compiled, installedand loaded:

$ modprobe mce-inject

Then you can run the mcelog test suite with

$ make test

This will inject different classes of errors and check that the mcelog triggers runs. There will be some kernel messagesabout page offlining attempts. The test will also lose a few pages of memory in your system (not significant).

Note: This test will kill any running mcelog, which needs to be restarted manually afterwards.

mce-inject:

A utility to inject corrected, uncorrected and fatal machine check exceptions

$ git clone https://git.kernel.org/pub/scm/utils/cpu/mce/mce-inject.git$ cd mce-inject$ make$ modprobe mce-inject

Modify the test/corrected script to include the following:

CPU 0 BANK 0STATUS 0xcc00008000010090ADDR 0x0010FFFFFFF

Inject the error: .. code:: bash

$ ./mce-inject < test/corrected

Note: The uncorrected and fatal scripts under test will cause a platform reset. Only the fatal script generates thememory errors**. In order to quickly emulate uncorrected memory errors and avoid host reboot following test errorsfrom mce-test suite can be injected:

$ mce-inject mce-test/cases/coverage/soft-inj/recoverable_ucr/data/srao_mem_scrub

mce-test:

In addition a more in-depth test of the Linux kernel machine check facilities can be done with the mce-test test suite.mce-test supports testing uncorrected error handling, real error injection, handling of different soft offlining cases, andother tests.

Corrected memory error injection:



To inject corrected memory errors:

• Remove sb_edac and edac_core kernel modules: rmmod sb_edac rmmod edac_core

• Insert einj module: modprobe einj param_extension=1

• Inject an error by specifying details (last command should be repeated at least two times):

$ APEI_IF=/sys/kernel/debug/apei/einj$ echo 0x8 > $APEI_IF/error_type$ echo 0x01f5591000 > $APEI_IF/param1$ echo 0xfffffffffffff000 > $APEI_IF/param2$ echo 1 > $APEI_IF/notrigger$ echo 1 > $APEI_IF/error_inject

• Check the MCE statistic: mcelog –client. Check the mcelog log for injected error details: less /var/log/mcelog.

Open vSwitch Plugins

OvS Plugins Repo: https://github.com/collectd/collectd

OvS Plugins Branch: main

OvS Events MIBs: The SNMP OVS interface link status is provided by standard IF-MIB (http://www.net-snmp.org/docs/mibs/IF-MIB.txt)

Dependencies: Open vSwitch, Yet Another JSON Library (https://github.com/lloyd/yajl)

On Centos, install the dependencies and Open vSwitch:

$ sudo yum install yajl-devel

Steps to install Open vSwtich can be found at https://docs.openvswitch.org/en/latest/intro/install/fedora/

Start the Open vSwitch service:

$ sudo service openvswitch-switch start

Configure the ovsdb-server manager:

$ sudo ovs-vsctl set-manager ptcp:6640

Clone and install the collectd ovs plugin:

$ git clone $REPO$ cd collectd$ git checkout main$ ./build.sh$ ./configure --enable-syslog --enable-logfile --enable-debug$ make$ sudo make install

This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf)can be found at /opt/collectd/etc. To configure the OVS events plugin you need to modify the configurationfile to include:

<LoadPlugin ovs_events>Interval 1

</LoadPlugin>

(continues on next page)



http://www.net-snmp.org/docs/mibs/IF-MIB.txt

http://www.net-snmp.org/docs/mibs/IF-MIB.txt

https://github.com/lloyd/yajl

https://docs.openvswitch.org/en/latest/intro/install/fedora/


(continued from previous page)

<Plugin ovs_events>Port "6640"Address "127.0.0.1"Socket "/var/run/openvswitch/db.sock"Interfaces "br0" "veth0"SendNotification true

</Plugin>

To configure the OVS stats plugin you need to modify the configuration file to include:

<LoadPlugin ovs_stats>Interval 1

</LoadPlugin><Plugin ovs_stats>

Port "6640"Address "127.0.0.1"Socket "/var/run/openvswitch/db.sock"Bridges "br0"

</Plugin>


OVS PMD stats

Repo: https://gerrit.opnfv.org/gerrit/barometer

Prequistes: 1. Open vSwitch dependencies are installed. 2. Open vSwitch service is running. 3. Ovsdb-servermanager is configured. You can refer Open vSwitch Plugins section above for each one of them.

OVS PMD stats application is run through the exec plugin.

To configure the OVS PMD stats application you need to modify the exec plugin configuration to include:

<LoadPlugin exec>Interval 1

</LoadPlugin<Plugin exec>

Exec "user:group" "<path to ovs_pmd_stat.sh>"</Plugin>

Note: Exec plugin configuration has to be changed to use appropriate user before starting collectd service.

ovs_pmd_stat.sh calls the script for OVS PMD stats application with its argument:

sudo python /usr/local/src/ovs_pmd_stats.py" "--socket-pid-file""/var/run/openvswitch/ovs-vswitchd.pid"




https://gerrit.opnfv.org/gerrit/barometer


SNMP Agent Plugin


Branch: main

Dependencies: NET-SNMP library

Start by installing net-snmp and dependencies.

On Centos 7:

$ sudo yum install net-snmp net-snmp-libs net-snmp-utils net-snmp-devel$ sudo systemctl start snmpd.service

go to the snmp configuration steps.

From source:

Clone and build net-snmp:

$ git clone https://github.com/haad/net-snmp.git$ cd net-snmp$ ./configure --with-persistent-directory="/var/net-snmp" --with-systemd --enable-→˓shared --prefix=/usr$ make

Become root

$ make install

Copy default configuration to persistent folder:

$ cp EXAMPLE.conf /usr/share/snmp/snmpd.conf

Set library path and default MIB configuration:

$ cd ~/$ echo export LD_LIBRARY_PATH=/usr/lib >> .bashrc$ net-snmp-config --default-mibdirs$ net-snmp-config --snmpconfpath

Configure snmpd as a service:

$ cd net-snmp$ cp ./dist/snmpd.service /etc/systemd/system/$ systemctl enable snmpd.service$ systemctl start snmpd.service

Add the following line to snmpd.conf configuration file /etc/snmp/snmpd.conf to make all OID tree visible forSNMP clients:

view systemview included .1

To verify that SNMP is working you can get IF-MIB table using SNMP client to view the list of Linux interfaces:

$ snmpwalk -v 2c -c public localhost IF-MIB::interfaces

Get the default MIB location:




$ net-snmp-config --default-mibdirs/opt/stack/.snmp/mibs:/usr/share/snmp/mibs

Install Intel specific MIBs (if needed) into location received by net-snmp-config command (e.g. /usr/share/snmp/mibs).

$ git clone https://gerrit.opnfv.org/gerrit/barometer.git$ sudo cp -f barometer/mibs/*.txt /usr/share/snmp/mibs/$ sudo systemctl restart snmpd.service

Clone and install the collectd snmp_agent plugin:

$ cd ~$ git clone https://github.com/collectd/collectd$ cd collectd$ ./build.sh$ ./configure --enable-syslog --enable-logfile --enable-debug --enable-snmp --with-→˓libnetsnmp$ make$ sudo make install

This will install collectd to default folder /opt/collectd. The collectd configuration file (collectd.conf)can be found at /opt/collectd/etc.

SNMP Agent plugin is a generic plugin and cannot work without configuration. To configure the snmp_agentplugin you need to modify the configuration file to include OIDs mapped to collectd types. The following examplemaps scalar memAvailReal OID to value represented as free memory type of memory plugin:

LoadPlugin snmp_agent<Plugin "snmp_agent">

<Data "memAvailReal">Plugin "memory"Type "memory"TypeInstance "free"OIDs "1.3.6.1.4.1.2021.4.6.0"

</Data></Plugin>

The snmpwalk command can be used to validate the collectd configuration:

$ snmpwalk -v 2c -c public localhost 1.3.6.1.4.1.2021.4.6.0UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 135237632 kB

Limitations

• Object instance with Counter64 type is not supported in SNMPv1. When GetNext request is received, Counter64type objects will be skipped. When Get request is received for Counter64 type object, the error will be returned.

• Interfaces that are not visible to Linux like DPDK interfaces cannot be retreived using standard IF-MIB tables.


For more details on AgentX subagent, please see: http://www.net-snmp.org/tutorial/tutorial-5/toolkit/demon/




http://www.net-snmp.org/tutorial/tutorial-5/toolkit/demon/


virt plugin


Branch: main

Dependencies: libvirt (https://libvirt.org/), libxml2

On Centos, install the dependencies:

$ sudo yum install libxml2-devel libpciaccess-devel yajl-devel device-mapper-devel

Install libvirt:

Note: libvirt version in package manager might be quite old and offer only limited functionality. Hence, building andinstalling libvirt from sources is recommended. Detailed instructions can bet found at: https://libvirt.org/compiling.html

$ sudo yum install libvirt-devel

Certain metrics provided by the plugin have a requirement on a minimal version of the libvirt API. File system infor-mation statistics require a Guest Agent (GA) to be installed and configured in a VM. User must make sure that installedGA version supports retrieving file system information. Number of Performance monitoring events metrics dependson running libvirt daemon version.

Note: Please keep in mind that RDT metrics (part of Performance monitoring events) have to be supported byhardware. For more details on hardware support, please see: https://github.com/intel/intel-cmt-cat

Additionally perf metrics cannot be collected if Intel RDT plugin is enabled.

libvirt version can be checked with following commands:

$ virsh --version$ libvirtd --version

Table 1: Extended statistics requirementsStatistic Min. libvirt API version Requires GADomain reason 0.9.2 NoDisk errors 0.9.10 NoJob statistics 1.2.9 NoFile system information 1.2.11 YesPerformance monitoring events 1.3.3 No

Start libvirt daemon:

$ systemctl start libvirtd

Create domain (VM) XML configuration file. For more information on domain XML format and examples, pleasesee: https://libvirt.org/formatdomain.html

Note: Installing additional hypervisor dependencies might be required before deploying virtual machine.



https://libvirt.org/

https://libvirt.org/compiling.html

https://libvirt.org/compiling.html

https://github.com/intel/intel-cmt-cat

https://libvirt.org/formatdomain.html


Create domain, based on created XML file:

$ virsh define DOMAIN_CFG_FILE.xml

Start domain:

$ virsh start DOMAIN_NAME

Check if domain is running:

$ virsh list

Check list of available Performance monitoring events and their settings:

$ virsh perf DOMAIN_NAME

Enable or disable Performance monitoring events for domain:

$ virsh perf DOMAIN_NAME [--enable | --disable] EVENT_NAME --live

Clone and install the collectd virt plugin:

$ git clone $REPO$ cd collectd$ ./build.sh$ ./configure --enable-syslog --enable-logfile --enable-debug$ make$ sudo make install

Where $REPO is equal to information provided above.

This will install collectd to /opt/collectd. The collectd configuration file collectd.conf can be found at/opt/collectd/etc. To load the virt plugin user needs to modify the configuration file to include:

LoadPlugin virt

Additionally, user can specify plugin configuration parameters in this file, such as connection URL, domain name andmuch more. By default extended virt plugin statistics are disabled. They can be enabled with ExtraStats option.

<Plugin virt>RefreshInterval 60ExtraStats "cpu_util disk disk_err domain_state fs_info job_stats_background pcpu

→˓perf vcpupin"</Plugin>






Installing collectd as a service

NOTE: In an OPNFV installation, collectd is installed and configured as a service.

Collectd service scripts are available in the collectd/contrib directory. To install collectd as a service:

$ sudo cp contrib/systemd.collectd.service /etc/systemd/system/$ cd /etc/systemd/system/$ sudo mv systemd.collectd.service collectd.service$ sudo chmod +x collectd.service

Modify collectd.service

[Service]ExecStart=/opt/collectd/sbin/collectdEnvironmentFile=-/opt/collectd/etc/EnvironmentFile=-/opt/collectd/etc/CapabilityBoundingSet=CAP_SETUID CAP_SETGID

Reload

$ sudo systemctl daemon-reload$ sudo systemctl start collectd.service$ sudo systemctl status collectd.service should show success

Additional useful plugins

Exec Plugin : Can be used to show you when notifications are being generated by calling a bash script that dumpsnotifications to file. (handy for debug). Modify /opt/collectd/etc/collectd.conf:

LoadPlugin exec<Plugin exec># Exec "user:group" "/path/to/exec"

NotificationExec "user" "<path to barometer>/barometer/src/collectd/collectd_→˓sample_configs/write_notification.sh"</Plugin>

write_notification.sh (just writes the notification passed from exec through STDIN to a file (/tmp/notifications)):

#!/bin/bashrm -f /tmp/notificationswhile read x ydo

echo $x$y >> /tmp/notificationsdone

output to /tmp/notifications should look like:

Severity:WARNINGTime:1479991318.806Host:localhostPlugin:ovs_eventsPluginInstance:br-exType:gaugeTypeInstance:link_statusuuid:f2aafeec-fa98-4e76-aec5-18ae9fc74589





linkstate of "br-ex" interface has been changed to "DOWN"

• logfile plugin: Can be used to log collectd activity. Modify /opt/collectd/etc/collectd.conf to include:

LoadPlugin logfile<Plugin logfile>

LogLevel infoFile "/var/log/collectd.log"Timestamp truePrintSeverity false

</Plugin>

Monitoring Interfaces and Openstack Support

Fig. 1: Monitoring Interfaces and Openstack Support

The figure above shows the DPDK L2 forwarding application running on a compute node, sending and receiving traffic.Collectd is also running on this compute node retrieving the stats periodically from DPDK through the dpdkstat pluginand publishing the retrieved stats to OpenStack through the collectd-openstack-plugins.

To see this demo in action please checkout: Barometer OPNFV Summit demo

For more information on configuring and installing OpenStack plugins for collectd, check out the collectd-openstack-plugins GSG.


https://prezi.com/kjv6o8ixs6se/software-fastpath-service-quality-metrics-demo/




Security

• AAA – on top of collectd there secure agents like SNMP V3, Openstack agents etc. with their own AAAmethods.

• Collectd runs as a daemon with root permissions.

• The Exec plugin allows the execution of external programs but counters the security concerns by:

– Ensuring that only one instance of the program is executed by collectd at any time

– Forcing the plugin to check that custom programs are never executed with superuser

privileges.

• Protection of Data in flight:

– It’s recommend to use a minimum version of 4.7 of the Network plugin which provides the possibility tocryptographically sign or encrypt the network traffic.

– Write Redis plugin or the Write MongoDB plugin are recommended to store the data.

– For more information, please see: https://collectd.org/wiki/index.php?title=Networking_introduction

• Known vulnerabilities include:

– https://www.cvedetails.com/vulnerability-list/vendor_id-11242/Collectd.html

* CVE-2017-7401 fixed https://github.com/collectd/collectd/issues/2174 in Version 5.7.2.

* CVE-2016-6254 fixed https://mailman.verplant.org/pipermail/collectd/2016-July/006838.htmlin Version 5.4.3.

* CVE-2010-4336 fixed https://mailman.verplant.org/pipermail/collectd/2010-November/004277.htmlin Version 4.10.2.

– https://www.cvedetails.com/product/20310/Collectd-Collectd.html?vendor_id=11242

• It’s recommended to only use collectd plugins from signed packages.

References

2.2 VES Application User Guide

The Barometer repository contains a python based application for VES (VNF Event Stream) which receives the col-lectd specific metrics via Kafka bus, normalizes the metric data into the VES message format and sends it into theVES collector.

The application currently supports pushing platform relevant metrics through the additional measurements field forVES.

Collectd has a write_kafka plugin that sends collectd metrics and values to a Kafka Broker. The VES messageformatting application, ves_app.py, receives metrics from the Kafka broker, normalises the data to VES messageformat for forwarding to VES collector. The VES message formatting application will be simply referred to as the“VES application” within this userguide

The VES application can be run in host mode (baremetal), hypervisor mode (on a host with a hypervisor and VMsrunning) or guest mode(within a VM). The main software blocks that are required to run the VES application demoare:

1. Kafka

2. Collectd

2.2. VES Application User Guide 25

https://collectd.org/wiki/index.php/Plugin:Exec

https://collectd.org/wiki/index.php?title=Networking_introduction

https://www.cvedetails.com/vulnerability-list/vendor_id-11242/Collectd.html

https://www.cvedetails.com/cve/CVE-2017-7401/

https://github.com/collectd/collectd/issues/2174


https://mailman.verplant.org/pipermail/collectd/2016-July/006838.html


https://mailman.verplant.org/pipermail/collectd/2010-November/004277.html

https://www.cvedetails.com/product/20310/Collectd-Collectd.html?vendor_id=11242



https://kafka.apache.org/


3. VES Application

4. VES Collector

2.2.1 Install Kafka Broker

1. Dependencies: install JAVA & Zookeeper.

Ubuntu 16.04:

$ sudo apt-get install default-jre$ sudo apt-get install zookeeperd$ sudo apt-get install python-pip

CentOS:

$ sudo yum update -y$ sudo yum install java-1.8.0-openjdk$ sudo yum install epel-release$ sudo yum install python-pip$ sudo yum install zookeeper$ sudo yum install telnet$ sudo yum install wget

Note: You may need to add the repository that contains zookeeper. To do so, follow the step belowand try to install zookeeper again using steps above. Otherwise, skip next step.

$ sudo yum installhttps://archive.cloudera.com/cdh5/one-click-install/redhat/7/x86_64/→˓cloudera-cdh-5-0.x86_64.rpm

Start zookeeper:

$ sudo zookeeper-server start

if you get the error message like ZooKeeper data directory is missing at /var/lib/zookeeper during the start of zookeeper, initialize zookeeper data directory using the com-mand below and start zookeeper again, otherwise skip the next step.

$ sudo /usr/lib/zookeeper/bin/zkServer-initialize.shNo myid provided, be sure to specify it in /var/lib/zookeeper/myid if→˓using non-standalone

2. Test if Zookeeper is running as a daemon.

$ telnet localhost 2181

Type ‘ruok’ & hit enter. Expected response is ‘imok’ which means that Zookeeper is up running.

3. Install Kafka

Note: VES doesn’t work with version 0.9.4 of kafka-python. The recommended/tested version is1.3.3.



$ sudo pip install kafka-python$ wget "https://archive.apache.org/dist/kafka/1.0.0/kafka_2.11-1.0.0.tgz"$ tar -xvzf kafka_2.11-1.0.0.tgz$ sed -i -- 's/#delete.topic.enable=true/delete.topic.enable=true/' kafka_→˓2.11-1.0.0/config/server.properties$ sudo nohup kafka_2.11-1.0.0/bin/kafka-server-start.sh \kafka_2.11-1.0.0/config/server.properties > kafka_2.11-1.0.0/kafka.log

→˓2>&1 &

Note: If Kafka server fails to start, please check if the platform IP address is associated with thehostname in the static host lookup table. If it doesn’t exist, use the following command to add it.

$ echo "$(ip route get 8.8.8.8 | awk '{print $NF; exit}') $HOSTNAME" |→˓sudo tee -a /etc/hosts

4. Test the Kafka Installation

To test if the installation worked correctly there are two scripts, producer and consumer scripts. Thesewill allow you to see messages pushed to broker and receive from broker.

Producer (Publish “Hello World”):

$ echo "Hello, World" | kafka_2.11-1.0.0/bin/kafka-console-producer.sh \--broker-list localhost:9092 --topic TopicTest > /dev/null

Consumer (Receive “Hello World”):

$ kafka_2.11-1.0.0/bin/kafka-console-consumer.sh --zookeeper \localhost:2181 --topic TopicTest --from-beginning --max-messages 1 --

→˓timeout-ms 3000

2.2.2 Install collectd

Install development tools:

Ubuntu 16.04:

$ sudo apt-get install build-essential bison autotools-dev autoconf$ sudo apt-get install pkg-config flex libtool

CentOS:

$ sudo yum group install 'Development Tools'

Install Apache Kafka C/C++ client library:

$ git clone https://github.com/edenhill/librdkafka.git ~/librdkafka$ cd ~/librdkafka$ git checkout -b v0.9.5 v0.9.5$ ./configure --prefix=/usr$ make$ sudo make install

Build collectd with Kafka support:



$ git clone https://github.com/collectd/collectd.git ~/collectd$ cd ~/collectd$ ./build.sh$ ./configure --with-librdkafka=/usr --without-perl-bindings --enable-perl=no$ make && sudo make install

Note: If installing from git repository collectd.conf configuration file will be located in directory /opt/collectd/etc/. If installing from via a package manager collectd.conf configuration file will be located indirectory /etc/collectd/

Configure and start collectd. Modify Collectd configuration file collectd.conf as following:

• Within a VM: Setup VES application (guest mode)

• On Host with VMs: Setup VES application (hypervisor mode)

• No Virtualization: Setup VES application (host mode)

Start collectd process as a service as described in Installing collectd as a service.

2.2.3 Setup VES application (guest mode)

In this mode Collectd runs from within a VM and sends metrics to the VES collector.

Fig. 2: VES guest mode setup

Install dependencies:



$ sudo pip install pyyaml python-kafka

Clone Barometer repo and start the VES application:

$ git clone https://gerrit.opnfv.org/gerrit/barometer$ cd barometer/3rd_party/collectd-ves-app/ves_app$ nohup python ves_app.py --events-schema=yaml/guest.yaml --config=config/ves_app_→˓config.conf > ves_app.stdout.log &

Modify Collectd configuration file collectd.conf as following:

LoadPlugin logfileLoadPlugin interfaceLoadPlugin memoryLoadPlugin loadLoadPlugin diskLoadPlugin uuidLoadPlugin write_kafka

<Plugin logfile>LogLevel infoFile "/opt/collectd/var/log/collectd.log"Timestamp truePrintSeverity false

</Plugin>

<Plugin cpu>ReportByCpu trueReportByState trueValuesPercentage true

</Plugin>

<Plugin write_kafka>Property "metadata.broker.list" "localhost:9092"<Topic "collectd">

Format JSON</Topic>

</Plugin>


Note: The above configuration is used for a localhost. The VES application can be configured to use remote VEScollector and remote Kafka server. To do so, the IP addresses/host names needs to be changed in collector.confand ves_app_config.conf files accordingly.



2.2.4 Setup VES application (hypervisor mode)

This mode is used to collect hypervisor statistics about guest VMs and to send those metrics into the VES collector.Also, this mode collects host statistics and send them as part of the guest VES message.

Fig. 3: VES hypervisor mode setup

Running the VES in hypervisor mode looks like steps described in Setup VES application (guest mode) but with thefollowing exceptions:

• The hypervisor.yaml configuration file should be used instead of guest.yaml file when VES applica-tion is running.

• Collectd should be running on hypervisor machine only.

• Addition libvirtd dependencies needs to be installed on where collectd daemon is running. To install thosedependencies, see virt plugin section of Barometer user guide.

• The next (minimum) configuration needs to be provided to collectd to be able to generate the VES message toVES collector.

Note: At least one VM instance should be up and running by hypervisor on the host.

LoadPlugin logfileLoadPlugin cpuLoadPlugin virtLoadPlugin write_kafka





<Plugin logfile>LogLevel infoFile "/opt/collectd/var/log/collectd.log"Timestamp truePrintSeverity false

</Plugin>

<Plugin virt>Connection "qemu:///system"RefreshInterval 60HostnameFormat uuidPluginInstanceFormat nameExtraStats "cpu_util"

</Plugin>

<Plugin write_kafka>Property "metadata.broker.list" "localhost:9092"<Topic "collectd">Format JSON

</Topic></Plugin>



Note: The list of the plugins can be extented depends on your needs.

2.2.5 Setup VES application (host mode)

This mode is used to collect platform wide metrics and to send those metrics into the VES collector. It is most suitablefor running within a baremetal platform.


$ sudo pip install pyyaml

Clone Barometer repo and start the VES application:

$ git clone https://gerrit.opnfv.org/gerrit/barometer$ cd barometer/3rd_party/collectd-ves-app/ves_app$ nohup python ves_app.py --events-schema=yaml/host.yaml --config=config/ves_app_→˓config.conf > ves_app.stdout.log &

Modify collectd configuration file collectd.conf as following:

LoadPlugin interfaceLoadPlugin memoryLoadPlugin disk




Fig. 4: VES Native mode setup




LoadPlugin cpu<Plugin cpu>

ReportByCpu trueReportByState trueValuesPercentage true

</Plugin>

LoadPlugin write_kafka<Plugin write_kafka>

Property "metadata.broker.list" "localhost:9092"<Topic "collectd">Format JSON

</Topic></Plugin>



Note: The list of the plugins can be extented depends on your needs.

2.2.6 Setup VES Test Collector

Note: Test Collector setup is required only for VES application testing purposes.


$ sudo pip install jsonschema

Clone VES Test Collector:

$ git clone https://github.com/att/evel-test-collector.git ~/evel-test-collector

Modify VES Test Collector config file to point to existing log directory and schema file:

$ sed -i.back 's/^$log_file[ ]*=[ ]*$.*/\1collector.log/' ~/evel-test-collector/→˓config/collector.conf$ sed -i.back 's/^$schema_file[ ]*=.*$event_format_updated.json$/\→˓1CommonEventFormat.json/' ~/evel-test-collector/config/collector.conf

Start VES Test Collector:

$ cd ~/evel-test-collector/code/collector$ nohup python ./collector.py --config ../../config/collector.conf > collector.stdout.→˓log &



2.2.7 VES application configuration description

Details of the Vendor Event Listener REST service

REST resources are defined with respect to a ServerRoot:

ServerRoot = https://{Domain}:{Port}/{optionalRoutingPath}

REST resources are of the form:

{ServerRoot}/eventListener/v{apiVersion}`{ServerRoot}/eventListener/v{apiVersion}/{topicName}`{ServerRoot}/eventListener/v{apiVersion}/eventBatch`

Within the VES directory (3rd_party/collectd-ves-app/ves_app/config) there is a configuration filecalled ves_app_conf.conf. The description of the configuration options are described below:

Domain “host” VES domain name. It can be IP address or hostname of VES collector (default: 127.0.0.1)

Port port VES port (default: 30000)

Path “path” Used as the “optionalRoutingPath” element in the REST path (default: vendor_event_listener)

Topic “path” Used as the “topicName” element in the REST path (default: example_vnf)

UseHttps true|false Allow application to use HTTPS instead of HTTP (default: false)

Username “username” VES collector user name (default: empty)

Password “passwd” VES collector password (default: empty)

SendEventInterval interval This configuration option controls how often (sec) collectd data is sent to Vendor EventListener (default: 20)

ApiVersion version Used as the “apiVersion” element in the REST path (default: 3)

KafkaPort port Kafka Port (Default 9092)

KafkaBroker host Kafka Broker domain name. It can be an IP address or hostname of local or remote server (default:localhost)

2.2.8 VES YAML configuration format

The format of the VES message is generated by the VES application based on the YAML schema configuration fileprovided by user via --events-schema command-line option of the application.

Note: Use --help option of VES application to see the description of all available options

Note: The detailed installation guide of the VES application is described in the VES Application User Guide.

The message defined by YAML schema should correspond to format defined in VES shema definition.

Warning: This section doesn’t explain the YAML language itself, so the knowledge of the YAML language isrequired before continue reading it!


https://gerrit.onap.org/r/gitweb?p=demo.git;a=tree;f=vnfs/VES5.0/evel/evel-test-collector/docs/att_interface_definition;hb=refs/heads/master


Since, the VES message output is a JSON format, it’s recommended to understand how YAML document is convertedto JSON before starting to create a new YAML definition for the VES. E.g.:

The following YAML document:

---additionalMeasurements:

plugin: somenameinstance: someinstance

will be converted to JSON like this:

{"additionalMeasurements": {"instance": "someinstance","plugin": "somename"

}}

Note: The YAML syntax section of PyYAML documentation can be used as a reference for this.

2.2.9 VES message types

The VES agent can generate two type of messages which are sent to the VES collector. Each message type must bespecified in the YAML configuration file using a specific YAML tag.

Measurements This type is used to send a message defined in YAML configuration to the VES collector with aspecified interval (default is 20 sec and it’s configurable via command line option of the application). This typecan be specified in the configuration using !Measurements tag. For instance:

---# My commentsMy Host Measurements: !Measurements... # message definition

Events This type is used to send a message defined in YAML configuration to the VES collector when collectdnotification is received from Kafka bus (collectd write_kafka plugin). This type can be specified in theconfiguration using !Events tag. For instance:

---# My commentsMy Events: !Events... # event definition


https://pyyaml.org/wiki/PyYAMLDocumentation


2.2.10 Collectd metrics in VES

The VES application caches collectd metrics received via Kafka bus. The data is stored in a table format. It’s importantto know it before mapping collectd metric values to message defined in YAML configuration file.

VES collectd metric cache example:

host plugin plu-gin_instance

type type_instace time value ds_name inter-val

local-host

cpu per-cent

user 1509535512.30567 16 value 10

local-host

mem-ory

mem-ory

free 1509535573.448014798456 value 10

local-host

inter-face

eth0 if_packets 1509544183.956496253 rx 10

7ec333e7 virt Ubuntu-12.04.5-LTS

per-cent

virt_cpu_total 1509544402.3780350.2 value 10


mem-ory

rss 1509544638.55119 123405 value 10


if_octets vnet1 1509544646.27108 67 tx 10

cc659a52 virt Ubuntu-16.04 per-cent

virt_cpu_total 1509544745.6172070.3 value 10

cc659a52 virt Ubuntu-16.04 mem-ory

rss 1509544754.3294114567 value 10

cc659a52 virt Ubuntu-16.04 if_octets vnet0 1509544760.7203810 rx 10

It’s possible from YAML configuration file to refer to any field of any row of the table via special YAML tags likeValueItem or ArrayItem. See the Collectd metric reference reference for more details.

Note: The collectd data types file contains map of type to ds_name field. It can be used to get possible value fords_name field. See the collectd data types description for more details on collectd data types.

Aging of collectd metric

If the metric will not be updated by the collectd during the double metric interval time, it will be removed (aged) fromVES internal cache.

2.2.11 VES YAML references

There are four type of references supported by the YAML format: System, Config, Collectd metric and Collectdnotification. The format of the reference is the following:

"{<ref type>.<attribute name>}"

Note: Possible values for <ref type> and <attribute name> are described in farther sections.


https://github.com/collectd/collectd/blob/main/src/types.db

https://github.com/collectd/collectd/blob/main/src/types.db.pod


System reference

This reference is used to get system statistics like time, date etc. The system reference (<ref type> = system)can be used in any place of the YAML configuration file. This type of reference provides the following attributes:

hostname The name of the host where VES application is running.

id Unique ID during VES application runtime.

time Current time in seconds since the Epoch. For example 1509641411.631951.

date Current date in ISO 8601 format, YYYY-MM-DD. For instance 2017-11-02.

For example:

Date: "{system.date}"

Config reference

This reference is used to get VES configuration described in VES application configuration description sectoin. Thereference (<ref type> = config) can be used in any place of the YAML configuration file. This type of referenceprovides the following attributes:

interval Measurements dispatch interval. It referenses to SendEventInterval configuration of the VESapplication.

For example:

Interval: "{config.interval}"

Collectd metric reference

This reference is used to get the attribute value of collectd metric from the VES cache. The reference (<ref type>= vl) can be used ONLY inside Measurements, ValueItem and ArrayItem tags. Using the reference inside ahelper tag is also allowed if the helper tag is located inside the tag where the reference is allowed (e.g.: ArrayItem).The <attribute name> name corresponds to the table field name described in Collectd metrics in VES section.For example:

name: "{vl.type}-{vl.type_instance}"

Collectd notification reference

This reference is used to get the attribute value of received collectd notification. The reference (<ref type> = n)can be used ONLY inside Events tag. Using the reference inside a helper tag is also allowed if the helper tag islocated inside the Events tag. This type of reference provides the following attributes:

host The hostname of received collectd notification.

plugin The plugin name of received collectd notification.

plugin_instance The plugin instance of the received collectd notification.

type The type name of the received collectd notification.

type_instance The type instance of the received collectd notification.

severity The severity of the received collectd notification.



message The message of the received collectd notification.

Note: The exact value for each attribute depends on the collectd plugin which may generate the notification. Pleaserefer to the collectd plugin description document to get more details on the specific collectd plugin.

YAML config example:

sourceId: "{n.plugin_instance}"

2.2.12 Collectd metric mapping YAML tags

This section describes the YAML tags used to map collectd metric values in the YAML configuration file.

Measurements tag

This tag is a YAML map which is used to define the VES measurement message. It’s allowed to be used multipletimes in the document (e.g.: you can define multiple VES messages in one YAML document). This tag works in thesame way as ArrayItem tag does and all keys have the same description/rules.

ValueItem tag

This tag is used to select a collectd metric and get its attribute value using Collectd metric reference. The type of thistag is a YAML array of maps with the possible keys described below.

SELECT (required) Is a YAML map which describes the select metric criteria. Each key name of the map mustcorrespond to the table field name described in Collectd metrics in VES section.

VALUE (optional) Describes the value to be assigned. If not provided, the default !Number "{vl.value}"expression is used.

DEFAULT (optional) Describes the default value which will be assigned if no metric is selected by SELECT criteria.

ValueItem tag description example:

memoryFree: !ValueItem- SELECT:

plugin: memorytype: memorytype_instance: rss

- VALUE: !Bytes2Kibibytes "{vl.value}"- DEFAULT: 0

The tag process workflow is described on the figure below.




Fig. 5: YAML ValueItem tag process workflow



ArrayItem tag

This tag is used to select a list of collectd metrics and generate a YAML array of YAML items described byITEM-DESC key. If no collectd metrics are selected by the given criteria, the empty array will be returned.

SELECT (optional) Is a YAML map which describes the select metrics criteria. Each key name of the map mustcorrespond to the table field name described in Collectd metrics in VES section. The value of the key maybe regular expression. To enable regular expression in the value, the YAML string containing / char at thebeginning and at the end should be used. For example:

plugin: "/^(?!virt).*$/" # selected all metrics except ``virt`` plugin

The VES application uses the python RE library to work with regular expression specified in the YAML con-figuration. Please refer to python regular expression syntax documentation for more details on a syntax used bythe VES.

Multiple SELECT keys are allowed by the tag. If nor SELECT or INDEX-KEY key is specified, the VES erroris generated.

INDEX-KEY (optional) Is a YAML array which describes the unique fields to be selected by the tag. Each elementof array is a YAML string which should be one of the table field name described in Collectd metrics in VESsection. Please note, if this key is used, only fields specified by the key can be used as a collectd reference in theITEM-DESC key.

ITEM-DESC (required) Is a YAML map which describes each element of the YAML array generated by the tag.Using ArrayItem tags and other Helper YAML tags are also allowed in the definition of the key.

In the example below, the ArrayItem tag is used to generate an array of ITEM-DESC items for each collectd metricsexcept virt plugin with unique plugin, plugin_instance attribute values.

Measurements: !ArrayItem- SELECT:

plugin: "/^(?!virt).*$/"- INDEX-KEY:

- plugin- plugin_instance

- ITEM-DESC:name: !StripExtraDash "{vl.plugin}-{vl.plugin_instance}"

The tag process workflow is described on the figure below.

2.2.13 Collectd event mapping YAML tags

This section describes the YAML tags used to map the collectd notification to the VES event message in the YAMLconfiguration file.

Events tag

This tag is a YAML map which is used to define the VES event message. It’s allowed to be used multiple times in thedocument (e.g.: you can map multiple collectd notification into VES message in one YAML document). The possiblekeys of the tag are described below.

CONDITION (optional) Is a YAML map which describes the select metric criteria. Each key name of the map mustcorrespond to the name of attribute provided by Collectd notification reference. If no such key provided, anycollectd notification will map the defined YAML message.


https://docs.python.org/2/library/re.html#regular-expression-syntax


Fig. 6: YAML ArrayItem tag process workflow



ITEM-DESC (required) Is a YAML map which describes the message generated by this tag. Only Helper YAMLtags are allowed in the definition of the key.

The example of the VES event message which will be generate by the VES application when collectd notification ofthe virt plugin is triggered is described below.

---Virt Event: !Events- ITEM-DESC:

event:commonEventHeader:domain: faulteventType: NotificationsourceId: &event_sourceId "{n.plugin_instance}"sourceName: *event_sourceIdlastEpochMicrosec: !Number "{n.time}"startEpochMicrosec: !Number "{n.time}"

faultFields:alarmInterfaceA: !StripExtraDash "{n.plugin}-{n.plugin_instance}"alarmCondition: "{n.severity}"faultFieldsVersion: 1.1

- CONDITION:plugin: virt

2.2.14 Helper YAML tags

This section describes the YAML tags used as utilities for formatting the output message. The YAML configurationprocess workflow is described on the figure below.

Fig. 7: YAML configuration process workflow

Convert string to number tag

The !Number tag is used in YAML configuration file to convert string value into the number type. For instance:

lastEpochMicrosec: !Number "3456"

The output of the tag will be the JSON number.

{lastEpochMicrosec: 3456

}



Convert bytes to Kibibytes tag

The !Bytes2Kibibytes tag is used in YAML configuration file to convert bytes into kibibytes (1 kibibyte = 1024bytes). For instance:

memoryConfigured: !Bytes2Kibibytes 4098memoryConfigured: !Bytes2Kibibytes "1024"

The output of the tag will be the JSON number.

{memoryConfigured: 4memoryConfigured: 1

}

Convert one value to another tag

The !MapValue tag is used in YAML configuration file to map one value into another value defined in the configu-ration. For instance:

Severity: !MapValueVALUE: FailureTO:Failure: CriticalError: Warnign

The output of the tag will be the mapped value.

{Severity: Critical

}

Strip extra dash tag

The !StripExtraDash tag is used in YAML configuration file to strip extra dashes in the string (dashes at thebeginning, at the end and double dashes). For example:

name: !StripExtraDash string-with--extra-dashes-

The output of the tag will be the JSON string with extra dashes removed.

{name: string-with-extra-dashes

}



2.2.15 Limitations

1. Only one document can be defined in the same YAML configuration file.

2. The collectd notification is not supported by Kafka collectd plugin. Due to this limitation, the collectd notifica-tions cannot be received by the VES application and the defined YAML event will not be generated and sent tothe VES collector. Please note, that VES YAML format already supports the events definition and the format isdescibed in the document.

2.3 Anuket Barometer Docker Install Guide

• Barometer Docker Images Description

– Barometer Collectd Image

– InfluxDB + Grafana Docker Images

– VES + Kafka Docker Images

• Installing Docker

– On Ubuntu

– On CentOS

– Manual proxy configuration for docker

– Test docker installation

• Build and Run Collectd Docker Image

– Collectd-barometer flavors

– Download the collectd docker image

– Build stable collectd container

– Build barometer-collectd-latest container

– Build barometer-collectd-experimental container

– Build collectd-6

– Run the collectd stable docker image

– Run the barometer-collectd-latest docker image

– Run the barometer-collectd-experimental docker image

• Build and Run InfluxDB and Grafana docker images

– Overview

– Download the InfluxDB and Grafana docker images

– Build InfluxDB docker image

– Build Grafana docker image

• Run the Influxdb and Grafana Images

– Run the InfluxDB docker image


https://collectd.org/wiki/index.php/Plugin:Write_Kafka


– Modify collectd to support InfluxDB on another host

– Run the Grafana docker image

– Cleanup of influxdb/grafana configuration

• Build and Run VES and Kafka Docker Images

– Download VES and Kafka docker images

– Build Kafka docker image

– Build VES docker image

– Run Kafka docker image

– Run VES Application docker image

– Run VES Test Collector application

• Build and Run DMA and Redis Docker Images

– Download DMA docker images

– Build DMA docker image

– Run Redis docker image

– Run DMA docker image

– References

The intention of this user guide is to outline how to install and test the Barometer project’s docker images. The OPNFVdocker hub contains 5 docker images from the Barometer project:

1. Collectd docker image

2. Influxdb docker image

3. Grafana docker image

4. Kafka docker image

5. VES application docker image

For description of images please see section Barometer Docker Images Description

For steps to build and run Collectd image please see section Build and Run Collectd Docker Image

For steps to build and run InfluxDB and Grafana images please see section Build and Run InfluxDB and GrafanaDocker Images

For steps to build and run VES and Kafka images please see section Build and Run VES and Kafka Docker Images

For overview of running VES application with Kafka please see the VES Application User Guide

For an alternative installation method using ansible, please see the Barometer One Click Install Guide.

2.3. Anuket Barometer Docker Install Guide 45

https://hub.docker.com/u/opnfv/?page=1

https://hub.docker.com/u/opnfv/?page=1

https://hub.docker.com/r/opnfv/barometer-collectd/

https://hub.docker.com/r/opnfv/barometer-influxdb/

https://hub.docker.com/r/opnfv/barometer-grafana/

https://hub.docker.com/r/opnfv/barometer-kafka/

https://hub.docker.com/r/opnfv/barometer-ves/


2.3.1 Barometer Docker Images Description

Barometer Collectd Image

The barometer collectd docker image gives you a collectd installation that includes all the barometer plugins.

Note: The Dockerfile is available in the docker/barometer-collectd directory in the barometer repo. The Dockerfilebuilds a CentOS 8 docker image. The container MUST be run as a privileged container.

Collectd is a daemon which collects system performance statistics periodically and provides a variety of mechanismsto publish the collected metrics. It supports more than 90 different input and output plugins. Input plugins retrievemetrics and publish them to the collectd deamon, while output plugins publish the data they receive to an end point.Collectd also has infrastructure to support thresholding and notification.

Collectd docker image has enabled the following collectd plugins (in addition to the standard collectd plugins):

• hugepages plugin

• Open vSwitch events Plugin

• Open vSwitch stats Plugin

• mcelog plugin

• PMU plugin

• RDT plugin

• virt

• SNMP Agent

• Kafka_write plugin

Plugins and third party applications in Barometer repository that will be available in the docker image:

• Open vSwitch PMD stats

• ONAP VES application

• gnocchi plugin

• aodh plugin

• Legacy/IPMI

InfluxDB + Grafana Docker Images

The Barometer project’s InfluxDB and Grafana docker images are 2 docker images that database and graph statisticsreported by the Barometer collectd docker. InfluxDB is an open-source time series database tool which stores the datafrom collectd for future analysis via Grafana, which is a open-source metrics anlytics and visualisation suite whichcan be accessed through any browser.



VES + Kafka Docker Images

The Barometer project’s VES application and Kafka docker images are based on a CentOS 7 image. Kafka dockerimage has a dependancy on Zookeeper. Kafka must be able to connect and register with an instance of Zookeeper thatis either running on local or remote host. Kafka recieves and stores metrics recieved from Collectd. VES applicationpulls latest metrics from Kafka which it normalizes into VES format for sending to a VES collector. Please see detailsin VES Application User Guide

2.3.2 Installing Docker

Note: The below sections provide steps for manual installation and configuration of docker images. They are notneccessary if docker images were installed with use of Ansible-Playbook.

On Ubuntu

Note:

• sudo permissions are required to install docker.

• These instructions are for Ubuntu 16.10

To install docker:

$ sudo apt-get install curl$ sudo curl -fsSL https://get.docker.com/ | sh$ sudo usermod -aG docker <username>$ sudo systemctl status docker

Replace <username> above with an appropriate user name.

On CentOS

Note:

• sudo permissions are required to install docker.

• These instructions are for CentOS 7

To install docker:

$ sudo yum remove docker docker-common docker-selinux docker-engine$ sudo yum install -y yum-utils device-mapper-persistent-data lvm2$ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/→˓docker-ce.repo$ sudo yum-config-manager --enable docker-ce-edge$ sudo yum-config-manager --enable docker-ce-test$ sudo yum install docker-ce$ sudo usermod -aG docker <username>$ sudo systemctl status docker


https://zookeeper.apache.org/


Replace <username> above with an appropriate user name.

Note: If this is the first time you are installing a package from a recently added repository, you will be prompted toaccept the GPG key, and the key’s fingerprint will be shown. Verify that the fingerprint is correct, and if so, accept thekey. The fingerprint should match060A 61C5 1B55 8A7F 742B 77AA C52F EB6B 621E 9F35.

Retrieving key from https://download.docker.com/linux/centos/gpg Importing GPG key 0x621E9F35:

Manual proxy configuration for docker

Note: This applies for both CentOS and Ubuntu.

If you are behind an HTTP or HTTPS proxy server, you will need to add this configuration in the Docker systemdservice file.

1. Create a systemd drop-in directory for the docker service:

$ sudo mkdir -p /etc/systemd/system/docker.service.d

2. Create a file called /etc/systemd/system/docker.service.d/http-proxy.conf that adds the HTTP_PROXY environmentvariable:

[Service]Environment="HTTP_PROXY=http://proxy.example.com:80/"

Or, if you are behind an HTTPS proxy server, create a file called /etc/systemd/system/docker.service.d/https-proxy.conf that adds the HTTPS_PROXY environment variable:

[Service]Environment="HTTPS_PROXY=https://proxy.example.com:443/"

Or create a single file with all the proxy configurations: /etc/systemd/system/docker.service.d/proxy.conf

[Service]Environment="HTTP_PROXY=http://proxy.example.com:80/"Environment="HTTPS_PROXY=https://proxy.example.com:443/"Environment="FTP_PROXY=ftp://proxy.example.com:443/"Environment="NO_PROXY=localhost"

3. Flush changes:

$ sudo systemctl daemon-reload

4. Restart Docker:

$ sudo systemctl restart docker

5. Check docker environment variables:

sudo systemctl show --property=Environment docker


https://download.docker.com/linux/centos/gpg


Test docker installation

Note: This applies for both CentOS and Ubuntu.

$ sudo docker run hello-world

The output should be something like:

Trying to pull docker.io/library/hello-world...Getting image source signaturesCopying blob 0e03bdcc26d7 doneCopying config bf756fb1ae doneWriting manifest to image destinationStoring signatures

Hello from Docker!This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:1. The Docker client contacted the Docker daemon.2. The Docker daemon pulled the "hello-world" image from the Docker Hub.3. The Docker daemon created a new container from that image which runs the

executable that produces the output you are currently reading.4. The Docker daemon streamed that output to the Docker client, which sent it

to your terminal.

To try something more ambitious, you can run an Ubuntu container with:$ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:https://hub.docker.com/

For more examples and ideas, visit:https://docs.docker.com/get-started/

2.3.3 Build and Run Collectd Docker Image

Collectd-barometer flavors

Before starting to build and run the Collectd container, understand the available flavors of Collectd containers:

• barometer-collectd - stable release, based on collectd 5.11

• barometer-collectd-latest - release based on collectd ‘main’ branch

• barometer-collectd-experimental - release based on collectd ‘main’ branch that can also include a set of experi-mental (not yet merged into upstream) pull requests

Note: Experimental container is not tested across various OS’es and the stability of the container can change. Usageof experimental flavor is at users risk.

Stable barometer-collectd container is intended for work in production environment as it is based on latest collectdofficial release. barometer-collectd-latest and barometer-collectd-experimental containers can be used in order totry new collectd features. All flavors are located in barometer git repository - respective Dockerfiles are stored insubdirectories of docker/ directory



$ git clone https://gerrit.opnfv.org/gerrit/barometer$ ls barometer/docker|grep collectdbarometer-collectdbarometer-collectd-latestbarometer-collectd-experimental

Note: Main directory of barometer source code (directory that contains ‘docker’, ‘docs’, ‘src’ and systems sub-directories) will be referred as <BAROMETER_REPO_DIR>

Download the collectd docker image

If you wish to use a pre-built barometer image, you can pull the barometer image from https://hub.docker.com/r/opnfv/barometer-collectd/

$ docker pull opnfv/barometer-collectd

Build stable collectd container

$ cd <BAROMETER_REPO_DIR>/docker/barometer-collectd$ sudo docker build -t opnfv/barometer-collectd --build-arg http_proxy=ècho $http_→˓proxy` \--build-arg https_proxy=ècho $https_proxy` --network=host -f Dockerfile .

Note: In the above mentioned docker build command, http_proxy & https_proxy arguments needs to be passedonly if system is behind an HTTP or HTTPS proxy server.

Check the docker images:

$ sudo docker images

Output should contain a barometer-collectd image:

REPOSITORY TAG IMAGE ID CREATED→˓ SIZEopnfv/barometer-collectd latest 05f2a3edd96b 3 hours ago→˓ 1.2GBcentos 7 196e0ce0c9fb 4 weeks ago→˓ 197MBcentos latest 196e0ce0c9fb 4 weeks ago→˓ 197MBhello-world latest 05a3bd381fc2 4 weeks ago→˓ 1.84kB

Note: If you do not plan to use barometer-collectd-latest and barometer-collectd-experimental containers, then youcan proceed directly to section Run the collectd stable docker image





Build barometer-collectd-latest container

$ cd <BAROMETER_REPO_DIR>$ sudo docker build -t opnfv/barometer-collectd-latest \--build-arg http_proxy=ècho $http_proxy` \--build-arg https_proxy=ècho $https_proxy` --network=host -f \docker/barometer-collectd-latest/Dockerfile .

Note: For barometer-collectd-latest and barometer-collectd-experimental containers proxy parameters should bepassed only if system is behind an HTTP or HTTPS proxy server (same as for stable collectd container)

Build barometer-collectd-experimental container

The barometer-collectd-experimental container use the main branch of collectd, but allows the user to apply a numberof pull requests, which are passed via the COLLECTD_PULL_REQUESTS build arg, which is passed to docker asshown in the example below. COLLECTD_PULL_REQUESTS should be a comma-delimited string of pull requestIDs.

$ cd <BAROMETER_REPO_DIR>$ sudo docker build -t opnfv/barometer-collectd-experimental \--build-arg http_proxy=ècho $http_proxy` \--build-arg https_proxy=ècho $https_proxy` \--build-arg COLLECTD_PULL_REQUESTS=1234,5678 \--network=host -f docker/barometer-collectd-experimental/Dockerfile .

Note: For barometer-collectd-latest and barometer-collectd-experimental containers proxy parameters should bepassed only if system is behind an HTTP or HTTPS proxy server (same as for stable collectd container)

Build collectd-6

The barometer-collectd-experimental Dockerfile can be used to build collectd-6.0, which is currently under devel-opment. In order to do this, the COLLECTD_FLAVOR build arg can be passed to the docker build command. Theoptional COLLECTD_PULL_REQUESTS arg can be passed as well, to test proposed patches to collectd.

$ cd <BAROMETER_REPO_DIR>$ sudo docker build -t opnfv/barometer-collectd-6 \

--build-arg COLLECTD_FLAVOR=collectd-6 \--build-arg COLLECTD_PULL_REQUESTS=1234,5678 \--network=host -f docker/barometer-collectd-experimental/Dockerfile .

The instructions for running the collectd-6 container are the same as for the collectd-experimental container.

There are a few useful build args that can be used to further customise the collectd-6 build:

* COLLECTD_CONFIG_CMD_ARGS

For testing with new plugins for collectd-6, as un-ported plugins are disabled by default. This new optionlets the ./configure command be run with extra args, e.g. –enable-cpu –enable-<my-newly-ported-plugin>,which means that plugin can be enabled for the PR that is being tested.



• COLLECTD_TAG This overrides the default tag selected by the flavors, and allows checking out out an ar-bitrary branch (e.g. PR branch instead of using the COLLECTD_PULL_REQUESTS arg, which rebases eachPR on top of the nominal branch. To check out a PR, use the following args with the docker build command:--build-arg COLLECTD_TAG=pull/<PR_ID>/head

Run the collectd stable docker image

$ cd <BAROMETER_REPO_DIR>$ sudo docker run -ti --net=host -v \`pwd`/src/collectd/collectd_sample_configs:/opt/collectd/etc/collectd.conf.d \-v /var/run:/var/run -v /tmp:/tmp -v /sys/fs/resctrl:/sys/fs/resctrl \--privileged opnfv/barometer-collectd

Note: The docker collectd image contains configuration for all the collectd plugins. In the command above weare overriding /opt/collectd/etc/collectd.conf.d by mounting a host directory src/collectd/collectd_sample_configs thatcontains only the sample configurations we are interested in running.

If some dependencies for plugins listed in configuration directory aren’t met, then collectd startup may fail(collectdtries to initialize plugins configurations for all given config files that can be found in shared configs directory and mayfail if some dependency is missing).

If DPDK or RDT can’t be installed on host, then corresponding config files should be removed from shared config-uration directory (<BAROMETER_REPO_DIR>/src/collectd/collectd_sample_configs/ ) prior to starting barometer-collectd container. By example: in case of missing DPDK functionality on the host, dpdkstat.conf and dpdkevents.confshould be removed.

Sample configurations can be found at: https://github.com/opnfv/barometer/tree/master/src/collectd/collectd_sample_configs

List of barometer-collectd dependencies on host for various plugins can be found at: https://wiki.anuket.io/display/HOME/Barometer-collectd+host+dependencies

The Resource Control file system (/sys/fs/resctrl) can be bound from host to container only if this directory exists onthe host system. Otherwise omit the ‘-v /sys/fs/resctrl:/sys/fs/resctrl’ part in docker run command. More informationabout resctrl can be found at: https://github.com/intel/intel-cmt-cat/wiki/resctrl

Check your docker image is running

sudo docker ps

To make some changes when the container is running run:

sudo docker exec -ti <CONTAINER ID> /bin/bash

Run the barometer-collectd-latest docker image

Run command for barometer-collectd-latest container is very similar to command used for stable container- the only differences are name of the image and location of the sample configuration files (as different version ofcollectd plugins requiring different configuration files)

$ cd <BAROMETER_REPO_DIR>$ sudo docker run -ti --net=host -v \`pwd`/src/collectd/collectd_sample_configs-latest:/opt/collectd/etc/collectd.conf.d \



https://github.com/opnfv/barometer/tree/master/src/collectd/collectd_sample_configs

https://github.com/opnfv/barometer/tree/master/src/collectd/collectd_sample_configs

https://wiki.anuket.io/display/HOME/Barometer-collectd+host+dependencies

https://wiki.anuket.io/display/HOME/Barometer-collectd+host+dependencies



-v /var/run:/var/run -v /tmp:/tmp -v /sys/fs/resctrl:/sys/fs/resctrl \--privileged opnfv/barometer-collectd-latest

Note: Barometer collectd docker images are sharing some directories with host (e.g. /tmp) therefore only oneof collectd barometer flavors can be run at a time. In other words, if you want to try barometer-collectd-latest orbarometer-collectd-experimental image, please stop instance of barometer-collectd(stable) image first.

The Resource Control file system (/sys/fs/resctrl) can be bound from host to container only if this directory exists onthe host system. Otherwise omit the ‘-v /sys/fs/resctrl:/sys/fs/resctrl’ part in docker run command. More informationabout resctrl can be found at: https://github.com/intel/intel-cmt-cat/wiki/resctrl

Run the barometer-collectd-experimental docker image

Barometer-collectd-experimental container shares default configuration files with ‘barometer-collectd-latest’ equiv-alent but some of experimental pull requests may require modified configuration. Additional configura-tion files that are required specifically by experimental container can be found in docker/barometer-collectd-experimental/experimental-configs/ directory. Content of this directory (all *.conf files) should be copied to src/collectd/collectd_sample_configs-latest directory before first run of experimental container.

$ cd <BAROMETER_REPO_DIR>$ cp docker/barometer-collectd-experimental/experimental-configs/*.conf \src/collectd/collectd_sample_configs-latest

When configuration files are up to date for experimental container, it can be launched using following command(almost identical to run-command for latest collectd container)

$ cd <BAROMETER_REPO_DIR>$ sudo docker run -ti --net=host -v \`pwd`/src/collectd/collectd_sample_configs-latest:/opt/collectd/etc/collectd.conf.d \-v /var/run:/var/run -v /tmp:/tmp -v /sys/fs/resctrl:/sys/fs/resctrl --privileged \opnfv/barometer-collectd-experimental

Note: The Resource Control file system (/sys/fs/resctrl) can be bound from host to container only if this directoryexists on the host system. Otherwise omit the ‘-v /sys/fs/resctrl:/sys/fs/resctrl’ part in docker run command. Moreinformation about resctrl can be found at: https://github.com/intel/intel-cmt-cat/wiki/resctrl

2.3.4 Build and Run InfluxDB and Grafana docker images

Overview

The barometer-influxdb image is based on the influxdb:1.3.7 image from the influxdb dockerhub. To view detils onthe base image please visit https://hub.docker.com/_/influxdb/ Page includes details of exposed ports and configurableenviromental variables of the base image.

The barometer-grafana image is based on grafana:4.6.3 image from the grafana dockerhub. To view details on the baseimage please visit https://hub.docker.com/r/grafana/grafana/ Page includes details on exposed ports and configurableenviromental variables of the base image.

The barometer-grafana image includes pre-configured source and dashboards to display statistics exposed by thebarometer-collectd image. The default datasource is an influxdb database running on localhost but the address of


https://hub.docker.com/_/influxdb/

https://hub.docker.com/r/grafana/grafana/


the influxdb server can be modified when launching the image by setting the environmental variables influxdb_host toIP or hostname of host on which influxdb server is running.

Additional dashboards can be added to barometer-grafana by mapping a volume to /opt/grafana/dashboards. Incasewhere a folder is mounted to this volume only files included in this folder will be visible inside barometer-grafana. Toensure all default files are also loaded please ensure they are included in volume folder been mounted. Appropriateexample are given in section Run the Grafana docker image

Download the InfluxDB and Grafana docker images

If you wish to use pre-built barometer project’s influxdb and grafana images, you can pull the images from https://hub.docker.com/r/opnfv/barometer-influxdb/ and https://hub.docker.com/r/opnfv/barometer-grafana/

Note: If your preference is to build images locally please see sections Build InfluxDB Docker Image and BuildGrafana Docker Image

$ docker pull opnfv/barometer-influxdb$ docker pull opnfv/barometer-grafana

Note: If you have pulled the pre-built barometer-influxdb and barometer-grafana images there is no requirementto complete steps outlined in sections Build InfluxDB Docker Image and Build Grafana Docker Image and you canproceed directly to section Run the Influxdb and Grafana Images

Build InfluxDB docker image

Build influxdb image from Dockerfile

$ cd barometer/docker/barometer-influxdb$ sudo docker build -t opnfv/barometer-influxdb --build-arg http_proxy=ècho $http_→˓proxy` \--build-arg https_proxy=ècho $https_proxy` --network=host -f Dockerfile .




Output should contain an influxdb image:

REPOSITORY TAG IMAGE ID CREATED→˓ SIZEopnfv/barometer-influxdb latest 1e4623a59fe5 3 days ago→˓ 191MB




https://hub.docker.com/r/opnfv/barometer-grafana/


Build Grafana docker image

Build Grafana image from Dockerfile

$ cd barometer/docker/barometer-grafana$ sudo docker build -t opnfv/barometer-grafana --build-arg http_proxy=ècho $http_→˓proxy` \--build-arg https_proxy=ècho $https_proxy` -f Dockerfile .




Output should contain an influxdb image:

REPOSITORY TAG IMAGE ID CREATED→˓ SIZEopnfv/barometer-grafana latest 05f2a3edd96b 3 hours ago→˓ 1.2GB

2.3.5 Run the Influxdb and Grafana Images

Run the InfluxDB docker image

$ sudo docker run -tid -v /var/lib/influxdb:/var/lib/influxdb --net=host\--name bar-influxdb opnfv/barometer-influxdb


sudo docker ps



When both collectd and InfluxDB containers are located on the same host, then no additional configuration have to beadded and you can proceed directly to Run the Grafana docker image section.

Modify collectd to support InfluxDB on another host

If InfluxDB and collectd containers are located on separate hosts, then additional configuration have to be done incollectd container - it normally sends data using network plugin to ‘localhost/127.0.0.1’ therefore changing outputlocation is required:

1. Stop and remove running bar-collectd container (if it is running)

$ sudo docker ps #to get collectd container name$ sudo docker rm -f <COLLECTD_CONTAINER_NAME>

2. Go to location where shared collectd config files are stored



$ cd <BAROMETER_REPO_DIR>$ cd src/collectd/collectd_sample_configs

3. Edit content of network.conf file. By default this file looks like that:

LoadPlugin network<Plugin network>Server "127.0.0.1" "25826"</Plugin>

127.0.0.1 string has to be replaced with the IP address of host where InfluxDB container is running (e.g.192.168.121.111). Edit this using your favorite text editor.

4. Start again collectd container like it is described in Run the collectd stable docker image chapter

$ cd <BAROMETER_REPO_DIR>$ sudo docker run -ti --name bar-collectd --net=host -v \`pwd`/src/collectd/collectd_sample_configs:/opt/collectd/etc/collectd.conf.d \-v /var/run:/var/run -v /tmp:/tmp --privileged opnfv/barometer-collectd

Now collectd container will be sending data to InfluxDB container located on remote Host pointed by IP configuredin step 3.

Run the Grafana docker image

Connecting to an influxdb instance running on local system and adding own custom dashboards

$ cd <BAROMETER_REPO_DIR>$ sudo docker run -tid -v /var/lib/grafana:/var/lib/grafana \

-v ${PWD}/docker/barometer-grafana/dashboards:/opt/grafana/dashboards \--name bar-grafana --net=host opnfv/barometer-grafana

Connecting to an influxdb instance running on remote system with hostname of someserver and IP address of192.168.121.111

$ sudo docker run -tid -v /var/lib/grafana:/var/lib/grafana --net=host -e \influxdb_host=someserver --add-host someserver:192.168.121.111 --name \bar-grafana opnfv/barometer-grafana


sudo docker ps



Connect to <host_ip>:3000 with a browser and log into grafana: admin/admin



Cleanup of influxdb/grafana configuration

When user wants to remove current grafana and influxdb configuration, folowing actions have to be performed

1. Stop and remove running influxdb and grafana containers

sudo docker rm -f bar-grafana bar-influxdb

2. Remove shared influxdb and grafana folders from the Host

sudo rm -rf /var/lib/grafanasudo rm -rf /var/lib/influxdb

Note: Shared folders are storing configuration of grafana and influxdb containers. In case of changing influxdb orgrafana configuration (e.g. moving influxdb to another host) it is good to perform cleanup on shared folders to notaffect new setup with an old configuration.

2.3.6 Build and Run VES and Kafka Docker Images

Download VES and Kafka docker images

If you wish to use pre-built barometer project’s VES and kafka images, you can pull the images from https://hub.docker.com/r/opnfv/barometer-ves/ and https://hub.docker.com/r/opnfv/barometer-kafka/

Note: If your preference is to build images locally please see sections `Build the Kafka Image`_ and `Build VESImage`_

$ docker pull opnfv/barometer-kafka$ docker pull opnfv/barometer-ves

Note: If you have pulled the pre-built images there is no requirement to complete steps outlined in sections BuildKafka Docker Image and Build VES Docker Image and you can proceed directly to section Run Kafka Docker Image

Build Kafka docker image

Build Kafka docker image:

$ cd barometer/docker/barometer-kafka$ sudo docker build -t opnfv/barometer-kafka --build-arg http_proxy=ècho $http_→˓proxy` \--build-arg https_proxy=ècho $https_proxy` -f Dockerfile .






https://hub.docker.com/r/opnfv/barometer-kafka/



Output should contain a barometer image:

REPOSITORY TAG IMAGE ID CREATED→˓SIZEopnfv/barometer-kafka latest 05f2a3edd96b 3 hours ago→˓1.2GB

Build VES docker image

Build VES application docker image:

$ cd barometer/docker/barometer-ves$ sudo docker build -t opnfv/barometer-ves --build-arg http_proxy=ècho $http_proxy` \

--build-arg https_proxy=ècho $https_proxy` -f Dockerfile .





REPOSITORY TAG IMAGE ID CREATED→˓SIZEopnfv/barometer-ves latest 05f2a3edd96b 3 hours ago→˓1.2GB

Run Kafka docker image

Note: Before running Kafka an instance of Zookeeper must be running for the Kafka broker to register with.Zookeeper can be running locally or on a remote platform. Kafka’s broker_id and address of its zookeeper instancecan be configured by setting values for environmental variables ‘broker_id’ and ‘zookeeper_node’. In instance where‘broker_id’ and/or ‘zookeeper_node’ is not set the default setting of broker_id=0 and zookeeper_node=localhost isused. In intance where Zookeeper is running on same node as Kafka and there is a one to one relationship betweenZookeeper and Kafka, default setting can be used. The docker argument add-host adds hostname and IP address to/etc/hosts file in container

Run zookeeper docker image:

$ sudo docker run -tid --net=host -p 2181:2181 zookeeper:3.4.11

Run kafka docker image which connects with a zookeeper instance running on same node with a 1:1 relationship

$ sudo docker run -tid --net=host -p 9092:9092 opnfv/barometer-kafka

Run kafka docker image which connects with a zookeeper instance running on a node with IP address of192.168.121.111 using broker ID of 1



$ sudo docker run -tid --net=host -p 9092:9092 --env broker_id=1 --env zookeeper_→˓node=zookeeper --add-host \zookeeper:192.168.121.111 opnfv/barometer-kafka

Run VES Application docker image

Note: VES application uses configuration file ves_app_config.conf from directory barometer/3rd_party/collectd-ves-app/ves_app/config/ and host.yaml file from barometer/3rd_party/collectd-ves-app/ves_app/yaml/ by default. If youwish to use a custom config file it should be mounted to mount point /opt/ves/config/ves_app_config.conf. To use analternative yaml file from folder barometer/3rd_party/collectd-ves-app/ves_app/yaml the name of the yaml file to useshould be passed as an additional command. If you wish to use a custom file the file should be mounted to mount point/opt/ves/yaml/ Please see examples below

Run VES docker image with default configuration

$ sudo docker run -tid --net=host opnfv/barometer-ves

Run VES docker image with guest.yaml files from barometer/3rd_party/collectd-ves-app/ves_app/yaml/

$ sudo docker run -tid --net=host opnfv/barometer-ves guest.yaml

Run VES docker image with using custom config and yaml files. In example below yaml/ folder cotains file namedcustom.yaml

$ sudo docker run -tid --net=host -v ${PWD}/custom.config:/opt/ves/config/ves_app_→˓config.conf \-v ${PWD}/yaml/:/opt/ves/yaml/ opnfv/barometer-ves custom.yaml

Run VES Test Collector application

VES Test Collector application can be used for displaying platform wide metrics that are collected by barometer-vescontainer. Setup instructions are located in: Setup VES Test Collector

2.3.7 Build and Run DMA and Redis Docker Images

Download DMA docker images

If you wish to use pre-built barometer project’s DMA images, you can pull the images from https://hub.docker.com/r/opnfv/barometer-dma/

Note: If your preference is to build images locally please see sections Build DMA Docker Image

$ docker pull opnfv/barometer-dma

Note: If you have pulled the pre-built images there is no requirement to complete steps outlined in sections BuildDMA Docker Image and you can proceed directly to section Run DMA Docker Image


https://hub.docker.com/r/opnfv/barometer-dma/

https://hub.docker.com/r/opnfv/barometer-dma/


Build DMA docker image

Build DMA docker image:

$ cd barometer/docker/barometer-dma$ sudo docker build -t opnfv/barometer-dma --build-arg http_proxy=ècho $http_proxy` \

--build-arg https_proxy=ècho $https_proxy` -f Dockerfile .





REPOSITORY TAG IMAGE ID CREATED→˓ SIZEopnfv/barometer-dma latest 2f14fbdbd498 3 hours ago→˓ 941 MB

Run Redis docker image

Note: Before running DMA, Redis must be running.

Run Redis docker image:

$ sudo docker run -tid -p 6379:6379 --name barometer-redis redis


sudo docker ps

Run DMA docker image

Run DMA docker image with default configuration

$ cd barometer/docker/barometer-dma$ sudo mkdir /etc/barometer-dma$ sudo cp ../../src/dma/examples/config.toml /etc/barometer-dma/$ sudo vi /etc/barometer-dma/config.toml(edit amqp_password and os_password:OpenStack admin password)

$ sudo su -(When there is no key for SSH access authentication)# ssh-keygen(Press Enter until done)(Backup if necessary)# cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys_org# cat ~/.ssh/authorized_keys_org ~/.ssh/id_rsa.pub \





> ~/.ssh/authorized_keys# exit

$ sudo docker run -tid --net=host --name server \-v /etc/barometer-dma:/etc/barometer-dma \-v /root/.ssh/id_rsa:/root/.ssh/id_rsa \-v /etc/collectd/collectd.conf.d:/etc/collectd/collectd.conf.d \opnfv/barometer-dma /server

$ sudo docker run -tid --net=host --name infofetch \-v /etc/barometer-dma:/etc/barometer-dma \-v /var/run/libvirt:/var/run/libvirt \opnfv/barometer-dma /infofetch

(Execute when installing the threshold evaluation binary)$ sudo docker cp infofetch:/threshold ./$ sudo ln -s ${PWD}/threshold /usr/local/bin/

References

2.4 OPNFV Barometer One Click Install Guide

• One Click Install with Ansible

– Proxy for package manager on host

– Proxy environment variables (for docker and pip)

– Install Ansible

– Clone barometer repo

– Edit inventory file

– Additional plugin dependencies

– Configure ssh keys

– Download and run Collectd+Influxdb+Grafana containers

– Download and run collectd+kafka+ves containers

– List of default plugins for collectd container

– List and description of tags used in ansible scripts

The intention of this user guide is to outline how to use the ansible playbooks for a one click installation of Barometer.A more in-depth installation guide is available with the Docker user guide.

2.4. OPNFV Barometer One Click Install Guide 61


2.4.1 One Click Install with Ansible

Proxy for package manager on host

Note: This step has to be performed only if host is behind HTTP/HTTPS proxy

Proxy URL have to be set in dedicated config file

1. CentOS - /etc/yum.conf

proxy=http://your.proxy.domain:1234

2. Ubuntu - /etc/apt/apt.conf

Acquire::http::Proxy "http://your.proxy.domain:1234"

After update of config file, apt mirrors have to be updaited via apt-get update

$ sudo apt-get update

Proxy environment variables (for docker and pip)

Note: This step has to be performed only if host is behind HTTP/HTTPS proxy

Configuring proxy for packaging system is not enough, also some proxy environment variables have to be set in thesystem before ansible scripts can be started. Barometer configures docker proxy automatically via ansible task as apart of one click install process - user only has to provide proxy URL using common shell environment variables andansible will automatically configure proxies for docker(to be able to fetch barometer images). Another componentused by ansible (e.g. pip is used for downloading python dependencies) will also benefit from setting proxy variablesproperly in the system.

Proxy variables used by ansible One Click Install:

• http_proxy

• https_proxy

• ftp_proxy

• no_proxy

Variables mentioned above have to be visible for superuser (because most actions involving ansible-barometerinstallation require root privileges). Proxy variables are commonly defined in /etc/environment file (but anyother place is good as long as variables can be seen by commands using su).

Sample proxy configuration in /etc/environment:

http_proxy=http://your.proxy.domain:1234https_proxy=http://your.proxy.domain:1234ftp_proxy=http://your.proxy.domain:1234no_proxy=localhost



Install Ansible

Note:

• sudo permissions or root access are required to install ansible.

• ansible version needs to be 2.4+, because usage of import/include statements

The following steps have been verified with Ansible 2.6.3 on Ubuntu 16.04 and 18.04. To install Ansible 2.6.3 onUbuntu:

$ sudo apt-get install python$ sudo apt-get install python-pip$ sudo -H pip install 'ansible==2.6.3'$ sudo apt-get install git

The following steps have been verified with Ansible 2.6.3 on Centos 7.5. To install Ansible 2.6.3 on Centos:

$ sudo yum install python$ sudo yum install epel-release$ sudo yum install python-pip$ sudo -H pip install 'ansible==2.6.3'$ sudo yum install git

Note: When using multi-node-setup, please make sure that python package is installed on all of the target nodes(ansible during ‘Gathering facts’ phase is using python2 and it may not be installed by default on some distributions- e.g. on Ubuntu 16.04 it has to be installed manually)

Clone barometer repo

$ git clone https://gerrit.opnfv.org/gerrit/barometer$ cd barometer/docker/ansible

Edit inventory file

Edit inventory file and add hosts: $barometer_dir/docker/ansible/default.inv

[collectd_hosts]localhost

[collectd_hosts:vars]install_mcelog=trueinsert_ipmi_modules=true#to use master or experimental container set the collectd flavor below#possible values: stable|master|experimentalflavor=stable

[influxdb_hosts]#hostname or ip must be used.#using localhost will cause issues with collectd network plugin.#hostname





[grafana_hosts]#NOTE: As per current support, Grafana and Influxdb should be same host.#hostname

[prometheus_hosts]#localhost

[zookeeper_hosts]#NOTE: currently one zookeeper host is supported#hostname

[kafka_hosts]#hostname

[ves_hosts]#hostname

Change localhost to different hosts where neccessary. Hosts for influxdb and grafana are required only forcollectd_service.yml. Hosts for zookeeper, kafka and ves are required only for collectd_ves.yml.

Note: Zookeeper, Kafka and VES need to be on the same host, there is no support for multi node setup.

To change host for kafka edit kafka_ip_addr in ./roles/config_files/vars/main.yml.

Additional plugin dependencies

By default ansible will try to fulfill dependencies for mcelog and ipmi plugin. For mcelog plugin it installs mcelogdaemon. For ipmi it tries to insert ipmi_devintf and ipmi_si kernel modules. This can be changed in inventoryfile with use of variables install_mcelog and insert_ipmi_modules, both variables are independent:

[collectd_hosts:vars]install_mcelog=falseinsert_ipmi_modules=false

Note: On Ubuntu 18.04 the deb package for mcelog daemon is not available in official Ubuntu repository. In thatcase ansible scripts will try to download, make and install the daemon from mcelog git repository.

Configure ssh keys

Generate ssh keys if not present, otherwise move onto next step. ssh keys are required for Ansible to connect the hostyou use for Barometer Installation.

$ sudo ssh-keygen

Copy ssh key to all target hosts. It requires to provide root password. The example is for localhost.

$ sudo -i$ ssh-copy-id root@localhost

Verify that key is added and password is not required to connect.



$ sudo ssh root@localhost

Note: Keys should be added to every target host and [localhost] is only used as an example. For multinode installationkeys need to be copied for each node: [collectd_hostname], [influxdb_hostname] etc.

Download and run Collectd+Influxdb+Grafana containers

The One Click installation features easy and scalable deployment of Collectd, Influxdb and Grafana containers usingAnsible playbook. The following steps goes through more details.

$ sudo -H ansible-playbook -i default.inv collectd_service.yml

Check the three containers are running, the output of docker ps should be similar to:

$ sudo docker psCONTAINER ID IMAGE COMMAND CREATED→˓ STATUS PORTS NAMESa033aeea180d opnfv/barometer-grafana "/run.sh" 9 days ago→˓ Up 7 minutes bar-grafana1bca2e4562ab opnfv/barometer-influxdb "/entrypoint.sh in..." 9 days ago→˓ Up 7 minutes bar-influxdbdaeeb68ad1d5 opnfv/barometer-collectd "/run_collectd.sh ..." 9 days ago→˓ Up 7 minutes bar-collectd

To make some changes when a container is running run:

$ sudo docker exec -ti <CONTAINER ID> /bin/bash

Connect to <host_ip>:3000 with a browser and log into Grafana: admin/admin. For short introduction please seethe: Grafana guide.

The collectd configuration files can be accessed directly on target system in /opt/collectd/etc/collectd.conf.d. It can be used for manual changes or enable/disable plugins. If configuration has been modified it is requiredto restart collectd:

$ sudo docker restart bar-collectd

Download and run collectd+kafka+ves containers

$ sudo ansible-playbook -i default.inv collectd_ves.yml

Check the containers are running, the output of docker ps should be similar to:

$ sudo docker psCONTAINER ID IMAGE COMMAND CREATED→˓ STATUS PORTS NAMES29035be2dab5 zookeeper:3.4.11 "/docker-entrypoint._" 7 minutes ago→˓ Up 7 minutes bar-zookeepereb8bba3c0b76 opnfv/barometer-ves "./start_ves_app.s..." 6 minutes ago→˓ Up 6 minutes bar-ves



https://grafana.com/docs/grafana/latest/guides/getting_started/



86702a96a68c opnfv/barometer-kafka "/src/start_kafka.sh" 6 minutes ago→˓ Up 6 minutes bar-kafkadaeeb68ad1d5 opnfv/barometer-collectd "/run_collectd.sh ..." 6 minutes ago→˓ Up 6 minutes bar-collectd

To make some changes when a container is running run:

$ sudo docker exec -ti <CONTAINER ID> /bin/bash

List of default plugins for collectd container

Note: From Jerma release, the supported dpdk version is 19.11

If you would like to use v18.11, Do the following changes: 1.Update the dpdk version to v18.11 in<barometer>/src/package-list.mk 2.Replace all ‘common_linux’ string with ‘common_linuxapp’ in <barome-ter>/src/dpdk/Makefile

If you would like to downgrade to a version lower than v18.11, Do the following changes: 1.Update the dpdkversion to a version lower than v18.11(Eg:- v16.11) in <barometer>/src/package-list.mk 2.Replace all‘common_linux’ string with ‘common_linuxapp’ in <barometer>/src/dpdk/Makefile 3.Change the Makefilepath from ‘(WORKDIR)/kernel/linux/kni/Makefile’ to (WORKDIR)/lib/librte_eal/linuxapp/kni/Makefile in‘(WORK_DIR)/src/dpdk/Makefile’.

By default the collectd is started with default configuration which includes the following plugins:

• csv, contextswitch, cpu, cpufreq, df, disk, ethstat, ipc, irq, load, memory,numa, processes, swap, turbostat, uuid, uptime, exec, hugepages, intel_pmu, ipmi,write_kafka, logfile, mcelog, network, intel_rdt, rrdtool, snmp_agent, syslog, virt,ovs_stats, ovs_events, dpdk_telemetry

Note: Some of the plugins are loaded depending on specific system requirements and can be omitted if dependencyis not met, this is the case for: * hugepages, ipmi, mcelog, intel_rdt, virt, ovs_stats, ovs_events

Note: The dpdkstat and dpdkevents plugins are disabled by default (in favour of the dpdk_telemetryplugin) and need to be explicitly enabled in order to use them:

$ sudo ansible-playbook -i default.inv collectd_service.yml –tags “all,dpdkstats,dpdkevents”

List and description of tags used in ansible scripts

Tags can be used to run a specific part of the configuration without running the whole playbook. To run a specific partsonly:

$ sudo ansible-playbook -i default.inv collectd_service.yml --tags "syslog,cpu,uuid"

To disable some parts or plugins:

$ sudo ansible-playbook -i default.inv collectd_service.yml --skip-tags "en_default_→˓all,syslog,cpu,uuid"



List of available tags:

install_docker Install docker and required dependencies with package manager.

add_docker_proxy Configure proxy file for docker service if proxy is set on host environment.

rm_config_dir Remove collectd config files.

copy_additional_configs Copy additional configuration files to target system. Path to additional configu-ration is stored in $barometer_dir/docker/ansible/roles/config_files/vars/main.ymlas additional_configs_path.

en_default_all Set of default read plugins: contextswitch, cpu, cpufreq, df, disk, ethstat, ipc,irq, load, memory, numa, processes, swap, turbostat, uptime.

plugins tags The following tags can be used to enable/disable plugins: csv, contextswitch, cpu,cpufreq, df, disk, ethstat, ipc, irq, load, memory, numa, processes, swap, turbostat,uptime, exec, hugepages, ipmi, kafka, logfile, mcelogs, n`ètwork,`` pmu, rdt,rrdtool, snmp, syslog, virt, ovs_stats, ovs_events, uuid, dpdkevents, dpdkstat,dpdk_telemetry.




CHAPTER

THREE

ANUKET BAROMETER RELEASE NOTES

3.1 Barometer Kali Release Notes

This document provides the release notes for Kali release of Barometer.

3.1.1 Summary

The Kali release is the first one since becoming part of Anuket, and focussed on changes that will make testing andintegrating easier.

3.1.2 Details

Testing and build tools were developed and updated to do the following:

• A new reference container was added for the collectd-6.0 version, which is under development and represents abig API change that is not backwards compatible. This reference build should facilitate porting the plugins thatwere previously developed by the Barometer project. https://jira.anuket.io/browse/BAROMETER-184

• Updated to the stable version of collectd to collectd 5.12.

• Removed duplication in the three existing containers (stable, latest and experimental). https://jira.anuket.io/browse/BAROMETER-179

Some work was started but not completed in the Kali release:

• Updating of the ansible playbooks for generating configs so that they will be easier to maintain and extend inthe future.

• Additional testing tools for verifying plugin functionality

3.1.3 References

• Barometer Kali release plan

• Kali Release on Jira

69

https://jira.anuket.io/browse/BAROMETER-184



https://wiki.anuket.io/display/HOME/Barometer+Kali+Release+Planning

https://jira.anuket.io/projects/BAROMETER/versions/10224


3.2 Barometer Release Notes

This document provides the release notes for Euphrates release of Barometer.

3.2.1 Important notes

None to date.

3.2.2 Summary

The Barometer@OPNFV project adds a platform telemetry agent to compute nodes that is capable of retrieving plat-form statistics and events, and relay them to Openstack Gnocchi and Aodh. The telemetry agent currently supportedby barometer is collectd. Some additional collectd plugins and application were developed to add the following func-tionality:

Write/publishing Plugins:

• aodh plugin: A notification plugin that pushes events to Aodh, and creates/updates alarms appropriately.

• SNMP agent plugin: A write plugin that will act as an AgentX subagent that receives and handles queries fromSNMP master agent and returns the data collected by read plugins. The SNMP Agent plugin handles requestsonly for OIDs specified in configuration file. To handle SNMP queries the plugin gets data from collectd andtranslates requested values from collectd’s internal format to SNMP format. Supports SNMP: get, getnext andwalk requests.

• gnocchi plugin: A write plugin that pushes the retrieved stats to Gnocchi. It’s capable of pushing any stats readthrough collectd to Gnocchi, not just the DPDK stats.

Read Plugins:

• Intel RDT plugin: A read plugin that provides the last level cache utilization and memory bandwidth utilization.

• virt plugin: A read plugin that uses virtualization API libvirt to gather statistics and events about virtualizedguests on a system directly from the hypervisor, without a need to install collectd instance on the guest.

• Open vSwitch stats plugin: A read plugin that retrieves interface stats from OVS.

In addition to the previous plugins from the Danube Release described below.

3.2.3 Release Data

Project Euphrates/barometer/barometer@opnfvRepo/commit-ID barometer/opnfv-5.1.0Release designation Euphrates 5.1.0Release date December 15th, 2017Purpose of the delivery Official OPNFV release

70 Chapter 3. Anuket Barometer Release Notes

mailto:Barometer@OPNFV

mailto:Euphrates/barometer/barometer@opnfv


Version change

Module version changes

• There have been no version changes.

Document version changes

Reason for version

Feature additions

JIRA BACK-LOG:

JIRA REFERENCE SLOGANBAROMETER-51 RDT Cache FeatureBAROMETER-53 RAS Metrics and Events/ MCELOG Memory ErrorsBAROMETER-55 Libvirt Metrics and EventsBAROMETER-56 Openvswitch Mrtics and EventsBAROMETER-59 AODH pluginBAROMETER-60 Gnocchi PluginBAROMETER-58 SNMP Agent

Bugs

JIRA TICKETS:

JIRA REFER-ENCE

SLOGAN

BAROMETER-80

SNMP Agent testing with Intel RDT, MCELOG, Hugepages, and OVS Stats not functional inthe Apex image of OPNFV Release E

3.2.4 Deliverables

Software deliverables

Features to Date

Release B

The features implemented for OPNFV release B (as part of SFQM) in DPDK include:

• Callback API to enable TX/RX timestamping to measure latency through DPDK.

• Extended NIC statistics API for 1GB, 10GB and 40GB NICs to expose detailed statistics for DPDK interfacesin addition to the overall aggregate statistics.

• DPDK Keep Alive.

3.2. Barometer Release Notes 71


Release C

The features implemented for OPNFV release C (as part of SFQM) include:

• DPDK extended NIC stats API improvement; migrate from key value pairs to using id value pairs.

• DPDK Keep Alive improvement, so that core status is exposed through a posix shared memory object.

• collectd dpdkstat plugin that can retrieve DPDK interface statistics.

• collectd ceilometer plugin that can publish any statistics collected by collectd to ceilometer.

• Fuel plugin support for the collectd ceilometer plugin for OPNFV.

Release D

The features implemented for OPNFV release D include:

• collectd hugepages plugin that can retrieves the number of available and free hugepages on a platform as wellas what is available in terms of hugepages per socket.

• collectd Open vSwitch Events plugin that can retrieves events from OVS.

• collectd mcelog plugin that can use mcelog client protocol to check for memory Machine Check Exceptions andsends the stats for reported exceptions.

• collectd ceilometer plugin that can publish any statistics collected by collectd to ceilometer.

Documentation deliverables

• Configuration guide

• User guide

• Release notes

• Scenario documentation.

3.2.5 Known Limitations, Issues and Workarounds

System Limitations

For Intel RDT plugin, compute node needs to support Intel RDT.

Known issues

No known issues to date.

JIRA TICKETS:

JIRA REFERENCE SLOGAN



Workarounds

• None to date.

3.2.6 Test Result

Barometer@OPNFV Euphrates has undergone QA test runs with the following results:

TEST-SUITE Results:barometercollectd

3.2.7 References

3.2. Barometer Release Notes 73

mailto:Barometer@OPNFV



CHAPTER

FOUR

OPNFV BAROMETER REQUIREMENTS

4.1 Problem Statement

Providing carrier grade Service Assurance is critical in the network transformation to a software defined and virtualizednetwork (NFV). Medium-/large-scale cloud environments account for between hundreds and hundreds of thousandsof infrastructure systems. It is vital to monitor systems for malfunctions that could lead to users application servicedisruption and promptly react to these fault events to facilitate improving overall system performance. As the size ofinfrastructure and virtual resources grow, so does the effort of monitoring back-ends. SFQM aims to expose as muchuseful information as possible off the platform so that faults and errors in the NFVI can be detected promptly andreported to the appropriate fault management entity.

The Anuket platform (NFVI) requires functionality to:

• Create a low latency, high performance packet processing path (fast path) through the NFVI that VNFs can takeadvantage of;

• Measure Telco Traffic and Performance KPIs through that fast path;

• Detect and report violations that can be consumed by VNFs and higher level EMS/OSS systems

Examples of local measurable QoS factors for Traffic Monitoring which impact both Quality of Experience and five9’s availability would be (using Metro Ethernet Forum Guidelines as reference):

• Packet loss

• Packet Delay Variation

• Uni-directional frame delay

Other KPIs such as Call drops, Call Setup Success Rate, Call Setup time etc. are measured by the VNF.

In addition to Traffic Monitoring, the NFVI must also support Performance Monitoring of the physical interfacesthemselves (e.g. NICs), i.e. an ability to monitor and trace errors on the physical interfaces and report them.

All these traffic statistics for Traffic and Performance Monitoring must be measured in-service and must be capable ofbeing reported by standard Telco mechanisms (e.g. SNMP traps), for potential enforcement actions.

75


4.2 Barometer updated scope

The scope of the project is to provide interfaces to support monitoring of the NFVI. The project will develop pluginsfor telemetry frameworks to enable the collection of platform stats and events and relay gathered information to faultmanagement applications or the VIM. The scope is limited to collecting/gathering the events and stats and relayingthem to a relevant endpoint. The project will not enforce or take any actions based on the gathered information.

4.2.1 Scope of SFQM

NOTE: The SFQM project has been replaced by Barometer. The output of the project will provide interfaces andfunctions to support monitoring of Packet Latency and Network Interfaces while the VNF is in service.

The DPDK interface/API will be updated to support:

• Exposure of NIC MAC/PHY Level Counters

• Interface for Time stamp on RX

• Interface for Time stamp on TX

• Exposure of DPDK events

collectd will be updated to support the exposure of DPDK metrics and events.

Specific testing and integration will be carried out to cover:

• Unit/Integration Test plans: A sample application provided to demonstrate packet latency monitoring and inter-face monitoring

The following list of features and functionality will be developed:

• DPDK APIs and functions for latency and interface monitoring

• A sample application to demonstrate usage

• collectd plugins

The scope of the project involves developing the relavant DPDK APIs, OVS APIs, sample applications, as well as theutilities in collectd to export all the relavent information to a telemetry and events consumer.

VNF specific processing, Traffic Monitoring, Performance Monitoring and Management Agent are out of scope.

The Proposed Interface counters include:

• Packet RX

• Packet TX

• Packet loss

• Interface errors + other stats

The Proposed Packet Latency Monitor include:

• Cycle accurate stamping on ingress

• Supports latency measurements on egress

Support for failover of DPDK enabled cores is also out of scope of the current proposal. However, this is an importantrequirement and must-have functionality for any DPDK enabled framework in the NFVI. To that end, a second phaseof this project will be to implement DPDK Keep Alive functionality that would address this and would report to aVNF-level Failover and High Availability mechanism that would then determine what actions, including failover, maybe triggered.

76 Chapter 4. OPNFV Barometer Requirements


4.2.2 Consumption Models

In reality many VNFs will have an existing performance or traffic monitoring utility used to monitor VNF behaviorand report statistics, counters, etc.

The consumption of performance and traffic related information/events provided by this project should be a logicalextension of any existing VNF/NFVI monitoring framework. It should not require a new framework to be developed.We do not see the Barometer gathered metrics and evetns as major additional effort for monitoring frameworks toconsume; this project would be sympathetic to existing monitoring frameworks. The intention is that this projectrepresents an interface for NFVI monitoring to be used by higher level fault management entities (see below).

Allowing the Barometer metrics and events to be handled within existing telemetry frameoworks makes it simpler foroverall interfacing with higher level management components in the VIM, MANO and OSS/BSS. The Barometer pro-posal would be complementary to the Doctor project, which addresses NFVI Fault Management support in the VIM,and the VES project, which addresses the integration of VNF telemetry-related data into automated VNF managementsystems. To that end, the project committers and contributors for the Barometer project wish to collaborate with theDoctor and VES projects to facilitate this.

4.3 collectd

collectd is a daemon which collects system performance statistics periodically and provides a variety of mechanismsto publish the collected metrics. It supports more than 90 different input and output plugins. Input plugins retrievemetrics and publish them to the collectd deamon, while output plugins publish the data they receive to an end point.collectd also has infrastructure to support thresholding and notification.

4.4 collectd statistics and Notifications

Within collectd notifications and performance data are dispatched in the same way. There are producer plugins (pluginsthat create notifications/metrics), and consumer plugins (plugins that receive notifications/metrics and do somethingwith them).

Statistics in collectd consist of a value list. A value list includes:

• Values, can be one of:

– Derive: used for values where a change in the value since it’s last been read is of interest. Can be used tocalculate and store a rate.

– Counter: similar to derive values, but take the possibility of a counter wrap around into consideration.

– Gauge: used for values that are stored as is.

– Absolute: used for counters that are reset after reading.

• Value length: the number of values in the data set.

• Time: timestamp at which the value was collected.

• Interval: interval at which to expect a new value.

• Host: used to identify the host.

• Plugin: used to identify the plugin.

• Plugin instance (optional): used to group a set of values together. For e.g. values belonging to a DPDK interface.

• Type: unit used to measure a value. In other words used to refer to a data set.

• Type instance (optional): used to distinguish between values that have an identical type.

4.3. collectd 77


• meta data: an opaque data structure that enables the passing of additional information about a value list. “Metadata in the global cache can be used to store arbitrary information about an identifier” [7].

Host, plugin, plugin instance, type and type instance uniquely identify a collectd value.

Values lists are often accompanied by data sets that describe the values in more detail. Data sets consist of:

• A type: a name which uniquely identifies a data set.

• One or more data sources (entries in a data set) which include:

– The name of the data source. If there is only a single data source this is set to “value”.

– The type of the data source, one of: counter, gauge, absolute or derive.

– A min and a max value.

Types in collectd are defined in types.db. Examples of types in types.db:

bitrate value:GAUGE:0:4294967295counter value:COUNTER:U:Uif_octets rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295

In the example above if_octets has two data sources: tx and rx.

Notifications in collectd are generic messages containing:

• An associated severity, which can be one of OKAY, WARNING, and FAILURE.

• A time.

• A Message

• A host.

• A plugin.

• A plugin instance (optional).

• A type.

• A types instance (optional).

• Meta-data.

4.5 DPDK Enhancements

This section will discuss the Barometer features that were integrated with DPDK.

4.5.1 Measuring Telco Traffic and Performance KPIs

This section will discuss the Barometer features that enable Measuring Telco Traffic and Performance KPIs.

• The very first thing Barometer enabled was a call-back API in DPDK and an associated application that used theAPI to demonstrate how to timestamp packets and measure packet latency in DPDK (the sample app is calledrxtx_callbacks). This was upstreamed to DPDK 2.0 and is represented by the interfaces 1 and 2 in Figure 1.2.

• The second thing Barometer implemented in DPDK is the extended NIC statistics API, which exposes NIC statsincluding error stats to the DPDK user by reading the registers on the NIC. This is represented by interface 3 inFigure 1.2.



Fig. 1: Measuring Telco Traffic and Performance KPIs

– For DPDK 2.1 this API was only implemented for the ixgbe (10Gb) NIC driver, in association with asample application that runs as a DPDK secondary process and retrieves the extended NIC stats.

– For DPDK 2.2 the API was implemented for igb, i40e and all the Virtual Functions (VFs) for all drivers.

– For DPDK 16.07 the API migrated from using string value pairs to using id value pairs, improving theoverall performance of the API.

4.5.2 Monitoring DPDK interfaces

With the features Barometer enabled in DPDK to enable measuring Telco traffic and performance KPIs, we can nowretrieve NIC statistics including error stats and relay them to a DPDK user. The next step is to enable monitoring ofthe DPDK interfaces based on the stats that we are retrieving from the NICs, by relaying the information to a higherlevel Fault Management entity. To enable this Barometer has been enabling a number of plugins for collectd.

4.5.3 DPDK Keep Alive description

SFQM aims to enable fault detection within DPDK, the very first feature to meet this goal is the DPDK Keep AliveSample app that is part of DPDK 2.2.

DPDK Keep Alive or KA is a sample application that acts as a heartbeat/watchdog for DPDK packet processing cores,to detect application thread failure. The application supports the detection of ‘failed’ DPDK cores and notification toa HA/SA middleware. The purpose is to detect Packet Processing Core fails (e.g. infinite loop) and ensure the failureof the core does not result in a fault that is not detectable by a management entity.

Essentially the app demonstrates how to detect ‘silent outages’ on DPDK packet processing cores. The applicationcan be decomposed into two specific parts: detection and notification.

• The detection period is programmable/configurable but defaults to 5ms if no timeout is specified.

4.5. DPDK Enhancements 79


Fig. 2: DPDK Keep Alive Sample Application

• The Notification support is enabled by simply having a hook function that where this can be ‘call back support’for a fault management application with a compliant heartbeat mechanism.

DPDK Keep Alive Sample App Internals

This section provides some explanation of the The Keep-Alive/’Liveliness’ conceptual scheme as well as the DPDKKeep Alive App. The initialization and run-time paths are very similar to those of the L2 forwarding application (seeL2 Forwarding Sample Application (in Real and Virtualized Environments) for more information).

There are two types of cores: a Keep Alive Monitor Agent Core (master DPDK core) and Worker cores(Tx/Rx/Forwarding cores). The Keep Alive Monitor Agent Core will supervise worker cores and report any failure (2successive missed pings). The Keep-Alive/’Liveliness’ conceptual scheme is:

• DPDK worker cores mark their liveliness as they forward traffic.

• A Keep Alive Monitor Agent Core runs a function every N Milliseconds to inspect worker core liveliness.

• If keep-alive agent detects time-outs, it notifies the fault management entity through a call-back function.

Note: Only the worker cores state is monitored. There is no mechanism or agent to monitor the Keep Alive MonitorAgent Core.


http://doc.dpdk.org/guides/sample_app_ug/l2_forward_real_virtual.html


DPDK Keep Alive Sample App Code Internals

The following section provides some explanation of the code aspects that are specific to the Keep Alive sample appli-cation.

The heartbeat functionality is initialized with a struct rte_heartbeat and the callback function to invoke in the case of atimeout.

rte_global_keepalive_info = rte_keepalive_create(&dead_core, NULL);if (rte_global_hbeat_info == NULL)

rte_exit(EXIT_FAILURE, "keepalive_create() failed");

The function that issues the pings hbeat_dispatch_pings() is configured to run every check_period milliseconds.

if (rte_timer_reset(&hb_timer,(check_period * rte_get_timer_hz()) / 1000,PERIODICAL,rte_lcore_id(),&hbeat_dispatch_pings, rte_global_keepalive_info) != 0 )

rte_exit(EXIT_FAILURE, "Keepalive setup failure.\n");

The rest of the initialization and run-time path follows the same paths as the the L2 forwarding application. The onlyaddition to the main processing loop is the mark alive functionality and the example random failures.

rte_keepalive_mark_alive(&rte_global_hbeat_info);cur_tsc = rte_rdtsc();

/* Die randomly within 7 secs for demo purposes.. */if (cur_tsc - tsc_initial > tsc_lifetime)break;

The rte_keepalive_mark_alive() function simply sets the core state to alive.

static inline voidrte_keepalive_mark_alive(struct rte_heartbeat *keepcfg){

keepcfg->state_flags[rte_lcore_id()] = 1;}

Keep Alive Monitor Agent Core Monitoring Options The application can run on either a host or a guest. As such thereare a number of options for monitoring the Keep Alive Monitor Agent Core through a Local Agent on the computenode:

Application Location DPDK KA LOCAL AGENTHOST X HOST/GUESTGUEST X HOST/GUEST

For the first implementation of a Local Agent SFQM will enable:

Application Location DPDK KA LOCAL AGENTHOST X HOST

Through extending the dpdkstat plugin for collectd with KA functionality, and integrating the extended plugin withMonasca for high performing, resilient, and scalable fault detection.

4.5. DPDK Enhancements 81



CHAPTER

FIVE

INDICES

• search

83

Date post:	15-Oct-2021
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

BAROMETER - OPNFV

Documents