These materials are licensed under the Creative Commons Attribution-Noncommercial 3.0 Unported license (http://creativecommons.org/licenses/by-nc/3.0/)

Measuring Delay with

Network Management & Monitoring

l  Based on RRDTool (the same author) l  Measures ICMP delay and can measure

status of services such as HTTP, DNS, SMTP, SSH, LDAP, etc.

l  Define ranges on statistics and generate alarms.

l  Written in Perl for portability l  Easy to install - harder to configure.

Introduction

•  SmokePing keeps track of your network latency: •  Best of breed latency visualization. •  Interactive graph explorer. •  Wide range of latency measurement plugins. •  Master/Slave System for distributed measurement. •  Highly configurable alerting system. •  Live Latency Charts with the most 'interesting' graphs. •  Free and OpenSource Software written in Perl written by Tobi

Oetiker, the creator of MRTG and RRDtool

Introduction: “Marketing”

The “Smoke” and the “Pings”

How to Read Smokeping Graphs How to Read Smokeping Graphs

l  Smokeping sends multiples tests (pings), makes note of RTT, orders these and selects the median.

l  The different values of RTT are shown graphically as lighter and darker shades of grey (the “smoke”). This conveys the idea of variable round trip times or jitter.

l  The number of lost packets (if any) changes the color of the horizontal line across the graph.

An Example

The following packages are needed or recommended: -  rrdtool http://oss.oetiker.ch/rrdtool/ -  fping http://www.fping.com/ -  echoping http://echoping.sourceforge.net/ -  speedyCGI http://www.daemoninc.com/SpeedyCGI/

- Apache http://httpd.apache.org/ -  Perl http://www.perl.org/

What makes it tick!

Debian/Ubuntu: l  apt-get install smokeping

l  Configure /etc/smokeping/config.d/* l  Change Smokeping's appearance here:

-  /etc/smokeping/basepage.html l  Restart the service: -  service smokeping {start|stop|restart|reload}!

Smokeping: Installation

You will find Smokeping running here: http://pcN.ws.nsrc.org/cgi-bin/smokeping.cgi

Smokeping Installation

Smokeping configuration files in Ubuntu 12.04 include: /etc/smokeping/config.d/Alerts /etc/smokeping/config.d/Database /etc/smokeping/config.d/General /etc/smokeping/config.d/pathnames /etc/smokeping/config.d/Presentation /etc/smokeping/config.d/Probes /etc/smokeping/config.d/Slaves /etc/smokeping/config.d/Targets

Generally we spend most of our time in Alerts, General, Probes and Targets.

Configuration

*** General *** owner = NOC contact = sysadm@pcN.ws.nsrc.org mailhost = localhost # NOTE: do not put the Image Cache below cgi-bin # since all files under cgi-bin will be executed ... this is not # good for images. cgiurl = http://pcN.ws.nsrc.org/cgi-bin/smokeping.cgi # specify this to get syslog logging syslogfacility = local5 # each probe is now run in its own process # disable this to revert to the old behaviour # concurrentprobes = no @include /etc/smokeping/config.d/pathnames

Configuration: General To be updated:

•  owner è NOC •  contact è sysadm@pcN.ws.nsrc.org •  cgiurl è http://pcN.ws.nsrc.org/cgi-bin/smokeping.cgi •  mailhost è localhost •  syslogfacility è local5

*** Targets *** probe = FPing menu = Top title = Network Latency Grapher + UO menu = University of Oregon title = UO webserver host = www.uoregon.edu + NSRC menu = NSRC title = Network Startup Resource Center host = www.nsrc.org ++ HTTP menu = HTTP probe = EchoPingHttp +++ www menu = NSRC web host = www.nsrc.org ++ DNS menu = DNS probe = DNS +++ dns menu = NSRC DNS host = www.nsrc.org

•  Where we spend most of our time configuring Smokeping. •  Web menu hierarchy defined by “+”, “++”, etc. •  Each new probe statement resets the default probe in use. •  Probes have defaults set in the Probes config file. These can be overridden in Targets.

Configuration: Targets

+ UO menu = University of Oregon title = UO webserver host = www.uoregon.edu

Target entry

RRD filename on disk: UO.rrd Must not contain spaces!

Submenu depth (+ = top level, ++ = 2nd level, +++ = 3rd level...)

Label in left-side menu

Label at top of screen

The actual hostname (or IP address) to test

*** Targets *** probe = FPing menu = Top title = Network Latency Grapher remark = SmokePing Latency Monitoring \ Network Monitoring and Management Workshop + Local menu = Local title = Local Network ++ LocalMachine menu = Local Machine title = This host host = localhost ++ NSRC menu = Network Startup Resource Center title = Latency to Network Startup Resource Center host = nsrc.org

Configuration: Targets Example

Targets file below produces the following default SmokePing page:

Configuration: Targets Example

Clicking on “Local” in the previous slide gives us:

Configuration: Targets Example

Clicking “Network Startup Resource Center” in the previous slides gives us:

*** Targets *** probe = FPing menu = Top title = Network Latency Grapher remark = SmokePing Latency Monitor… \ Network Monitoring and Mana… + Local menu = Local title = Local Network ++ LocalMachine menu = Local Machine title = This host host = localhost ++ NSRC menu = Network Startup Resource Center title = Latency to Network Startup Re… host = nsrc.org

Configuration: Targets Example

Hierarchy of web interface to Targets file explained:

+ Local à /var/lib/smokeping/Local ++ LocalMachine à /var/lib/smokeping/Local/LocalMachine.rrd ++ NSRC à /var/lib/smokeping/Local/NSRC.rrd

1st level

2nd level

2nd level

Questions?

*** Alerts *** to = root@localhost from = smokeping-alert@localhost +someloss type = loss # in percent pattern = >0%,*12*,>0%,*12*,>0% comment = loss 3 times in a row over 12 samples

Configuration: Alerts •  Very flexible. Create your own type of alert. •  Send alerts to ticket queues (RT using rt-mailgate, for instance) •  Somewhat complex to understand. Read the Alerts section of the Smokeping on-line configuration documentation:

http://oss.oetiker.ch/smokeping/doc/smokeping_config.en.html

This could go to a ticketing queue instead.

++ LocalMachine menu = localhost

title = This host

host = localhost

alerts = startloss,someloss,bigloss,rttdetect

Target

*** Probes *** + FPing binary = /usr/sbin/fping + DNS binary = /usr/bin/dig lookup = nsrc.org pings = 5 step = 180 + EchoPingHttp binary = /usr/bin/echoping ignore_cache = yes pings = 5 url = / + EchoPingHttps binary = /usr/bin/echoping pings = 5 url = / + EchoPingSmtp binary = /usr/bin/echoping forks = 5

Configuration: Probes Smokeping is installed with a number of additional probes. They must, however, be specified here – including their default behaviors.

Use the DNS probe to verify that your services are available and responding as expected. We use ”nsrc.org” as a sample hostname to lookup, to verify that the DNS works. Note: Initial Probes file only has FPing defined.

l  Probing for delay and jitter (ping) l  Entry belongs in the Targets file

Network Latency probe = FPing ... ++ LocalMachine menu = localhost title = This host host = localhost

Default Probe: FPing

DNS Latency ++ DNS probe = DNS menu = External DNS Check title = DNS Latency +++ GoogleA menu = 8.8.8.8 Title = DNS Latency GoogleA host = google-public-dns-

a.google.com

Probe: DNS Check

In /etc/smokeping/config.d/Targets:

More information available here: http://oss.oetiker.ch/smokeping/probe/index.en.html

A few more probes... - DNS - CiscoRTTMonDNS - Radius - HTTP(S) - CiscoRTTMonTcpCon - IOS -  LDAP - Tacacs - FPing6 - Whois - WebProxyFilter - Etc. -  SMTP - WWW-Cache

More Types of Probes

sendmail = /usr/sbin/sendmail imgcache = /var/cache/smokeping/images imgurl = ../smokeping/images datadir = /var/lib/smokeping piddir = /var/run/smokeping smokemail = /etc/smokeping/smokemail tmail = /etc/smokeping/tmail

Normally you should not need to update this file:

Configuration: pathnames

*** Presentation *** template = /etc/smokeping/basepage.html charset = utf-8 + charts menu = Charts title = The most interesting destinations ++ stddev sorter = StdDev(entries=>4) title = Top Standard Deviation menu = Std Deviation format = Standard Deviation %f ++ max sorter = Max(entries=>5) title = Top Max Roundtrip Time menu = by Max format = Max Roundtrip Time %f seconds

Configuration: Presentation •  If you wish to customize Smokeping’s look and feel you can edit the file /etc/smokeping/basepage.html •  To change how Smokeping presents graphs you can edit this file.

File continues…

*** Database *** step = 300 pings = 20 # consfn mrhb steps total AVERAGE 0.5 1 1008 AVERAGE 0.5 12 4320 MIN 0.5 12 4320 MAX 0.5 12 4320 AVERAGE 0.5 144 720 MAX 0.5 144 720 MIN 0.5 144 720

Configuration: Database

•  Defines how RRDtool will save data over time in Round Robin Archives (RRAs) •  By default each step is 300 seconds (5 minutes). •  You cannot trivially change the step setting once data has been collected. •  Details on each column in the Database section of the Smokeping on-line •  configuration documentation:

http://oss.oetiker.ch/smokeping/doc/smokeping_config.en.html

consfn: Consolidation function mrhb: Percent of consolidated steps that must be known to warrant an entry. steps: How many steps to consolidate for

each entry in the RRA. total: Total number of rows to keep in the

RRA. Use rows and steps to determine time data will be saved.

12 steps = 12 x 300 sec = 1 hour 4320 rows = 4320 hours = 180 days

*** Slaves *** secrets=/etc/smokeping/smokeping_secrets #+boomer #display_name=boomer #color=0000ff #+slave2 #display_name=another #color=00ff00

Configuration: Slaves

Smokeping slave servers allow for multi-viewpoint monitoring and graphing of the same services, machines or links. Details here:

http://oss.oetiker.ch/smokeping/doc/smokeping_master_slave.en.html

That  is,  you  can  externally  monitor  your  network!  

MultiHost Graphing

Solve the issue of multiple hosts, one probe and missing differences in the Y axis (time): http://oss.oetiker.ch/smokeping/doc/smokeping_examples.en.html

Sample configuration +++MultihostRouters menu = MutihostRouters title = Combined Router Results host = /Local/Routers/gw /Local/Routers/rtr1 /Local/Routers/rtr2

Example Multihost Graph

Summary

•  Simple but powerful network monitoring •  Monitor machines, services and link health •  Distributed instances for external views –

often a paid-for service •  Easy to configure and customize, but very

extensible. •  Can be used with Ticketing Systems to

automate alerts •  Very small disk and CPU footprint

Smokeping website: http://oss.oetiker.ch/smokeping/

Smokeping Demo: http://oss.oetiker.ch/smokeping-demo/?target=Customers.OP

Good examples: http://oss.oetiker.ch/smokeping/doc/smokeping_examples.en.html

References