We’reHiring
• LinuxdevelopersC/C++orPython
• Anywhereintheworld,paidrelocaJontoTorontoorfullJmeremoteopportuniJes
• Funandrelaxedworkenvironment
2
SangomaTechnologies-©2016
Agenda
• MonitoringBasics
• MetricsandLogs
• AlerJng
• TradiJonalSoluJons• EmergingSoluJons
• A?erword(FreeSWITCHOxidized)
3
SangomaTechnologies-©2016
MonitoringBasics
• Monitoringisaboutknowingwhatishappeningonyoursystems.Thegoodandthebad
• MonitoringhelpsyouwithpredicJon/forecasJng.PlanyourfuturegrowthandanJcipateproblems
• Monitoringprovidesyouwiththedatayouneedtotroubleshootproblemsfaster
4
SangomaTechnologies-©2016
TypesofMonitoring
• ApplicaJonMetricMonitoring• NumbersaboutapplicaJonusage&performance• Youcanthenusethosenumberstodefinealerts
• ServiceStatusMonitoring• Simplebinarychecks.It’stheserviceupornot
5
SangomaTechnologies-©2016
MonitoringModels
• PushvsPullsystems• Push:Graphite,StatsD,CollectD,SNMP• Pull:Prometheus,SNMP
• DetermineswhoiniJatesthemetrictransfer
• Pushsystemsaremoredynamicandtendtorequirelessmaintenance
• Pullsystemsrequireasourceoftruthandnodediscovery
• PullinghoweverallowsyoutomoreeffecJvelydetectifanodeorapplicaJonisdown
6
SangomaTechnologies-©2016
PushMonitoring
7
SangomaTechnologies-©2016
MonitoringServer
Node1
Node2
Node3
MetricsandCheckResults
PullMonitoring
8
SangomaTechnologies-©2016
MonitoringServer
Node1
Node2
Node3
MetricandCheckRequests
Metrics
• Timeseriesdata.SequenceofnumericaldatapointslistedinJmeorder,usuallysampledatregularintervals.
• DifferentgranulariJesofdatastoredandaggregatedoverJmeonfixed-sizestorage
• Timeseriesanalysisoverthedataresultsinfancygraphs
• Exampledatabases:RRDTool,Graphite(Whisper),InfluxDB,Prometheus
9
SangomaTechnologies-©2016
FreeSWITCHMetrics
• AcJveCalls
• TotalCalls
• FailedCalls
• RegistraJons
• FailedRegistraJons
10
SangomaTechnologies-©2016
FreeSWITCHMetrics
• ASR(Answer/SeizureRaJo)
• NER(NetworkEffecJvenessRaJo)
• ACD(AverageCallDuraJon)
• PDD(PostDialDelay)
11
SangomaTechnologies-©2016
Logs
• Logscanbeasourceofmetricswhenmetricsandservicechecksarenotenough
• Youmaywanttosendalertsbasedonlogpagernsandrepeatedlogerrors
• ELKStack• ElasJcSearch• Logstash• Kabana
12
SangomaTechnologies-©2016
AlerMng
• Email,Chat(Slack/HipChat),SMS,etc
• Alertbasedonexpectedmetricsandcheckfailures
• ProcuretousepercenJlesinsteadofaverages
• SeveralalternaJves:Prometheus,Alerta,Flapjack
13
SangomaTechnologies-©2016
AlertFaMgue
14
SangomaTechnologies-©2016
• Thishappenswhenalertsaretriggeredtooo?enandcausesthereceivertobedesensiJzed
AlertFaMgue
• Youknowyouhaveitwhenyoustartignoringalerts
• Ignoringalertsleadstomissingrealproblemsortakingtoolongtorespondtothem
15
SangomaTechnologies-©2016
AlertFaMgue
• WarningsmightbewellintendedandinisolaJonworkwell,however,whentheyaddup,theymaycausemoreharmthangood
• Adjustyourthresholds.BecauJouswithwarningvscriJcalseverityandadjustyournoJficaJonsaccordingly
• Killorfixalertsthatarefiringtooo?en.YouneedawaytoeasilymutetheminthemeanJme
16
SangomaTechnologies-©2016
• They’reallaboutgejngthejobdone.Andtheydo:
• Butatsomepointyougogaaskyourselfifthere’sabegerway...
TradiMonalSoluMons
17
SangomaTechnologies-©2016
SNMP
18
SangomaTechnologies-©2016
SNMP
• MIBs/OIDsarecumbersometouse(tosaytheleast)• OIDsavailableinFreeSWITCHgiveyounumberofsessions,acJvecalls,sessionspersecondetc.
• Noteasytocollectcustommetrics
• VerylimitedinformaJonexposedbymod_snmpandoverallseemskindofabandoned
• Themonitoring/managementinterfaceinFreeSWITCHneedssomework
19
SangomaTechnologies-©2016
CACTI&RRDtool
• WebfrontendfortheRRDtoolJme-seriesdatabase
• CollectmetricsfrommulJplesources(e.gsnmp)
• Noonecaneverrememberhowtoaddaserver(convolutedprocess)
20
SangomaTechnologies-©2016
CACTI&RRDtool
21
SangomaTechnologies-©2016
• Youendupwithuglygraphslikethis:
It’sbegerthannothingthough….
MonitoringSucks
22
SangomaTechnologies-©2016
• Somuchthataround2011a“MonitoringSucks”communitywassettofixit:
• hgps://github.com/monitoringsucks
• hgps://github.com/monitoringsucks/blog-posts
• hgps://vimeo.com/monitorama
NewSoluMons
23
SangomaTechnologies-©2016
• Becausenewisalwaysbeger…
Sensu
• Monitoringthatdoesn’tsuck(theirmogo)
• Youcanreusenagios/zabbixchecks.SensuwasdesignedasareplacementforagingNagiosinstallaJons
• NoJme-seriesdatabaseincluded,youhaveseveralopJonsavailable(e.gGraphite,InfluxDB)
24
SangomaTechnologies-©2016
Sensu
• Pull/Pushsystemwithanagent
• Pub/Submodelwithpluggabletransports(RabbitMQ,Redis,AmazonSQS)
• Nodessubscribetogroupsofchecks.Thechecksarescheduledbytheserverorthenode.
• Bothserviceandmetriccheckssupported
• CheckresultscanbesentasynchronouslybyexternaljobsviarawTCP+JSON(supportinagithubPRforHTTP)
25
SangomaTechnologies-©2016
Sensu
• HTTPAPItoreadaccessnodes,checks,silencealarms,etc
• NiceandsimpledashboardUI
• Dynamicinfrastructure.Serversaddthemselvestothemonitoringsystem.
• ComposablejsonconfiguraJonforeasyautomaJon(e.gAnsible,SaltStack,Chefetc)
26
SangomaTechnologies-©2016
Sensu
27
SangomaTechnologies-©2016
Source:sensuapp.org
Graphite
• Time-seriesstoragesystem
• WhisperdatabaseformatdesignedasanimprovementoverRRDtooltomanageoutofJmedatapoints
• Carbon-cacheisadaemonlisteningformetricsonaTCPsocket(typicallyusedintandemwithStatsD)
• Metricsarestoredinfixed-sizefiles
• Comeswitha(quiteoutdated)UIformetricgraphs.Thisismosto?enusedasanhgpendpointbyothergraphingtoolslikeGrafana
28
SangomaTechnologies-©2016
Graphite
• EveryJmeseriesisidenJfiedbyahierarchyofdot-separatedidenJfiers(e.gstats.freeswitch.total_calls)
• Thetextformattosendstatstographiteisdeadsimple:<metricname><value><Jmestamp>• stats.freeswitch.total_calls501470843323
• Nofurtherdimensionscanbeencoded.Thetypicalwork-aroundistoencodetheminthemetricname:• stats.freeswitch.profile.internal.total_calls20
1470843323
29
SangomaTechnologies-©2016
Graphite
30
SangomaTechnologies-©2016
Source:hgp://www.aosabook.org/en/graphite.html
Prometheus
• NewJme-seriesmonitoringsystemwithbuilt-inalerJng
• IncludesitsownJme-seriesdatabaseformat
• MulJ-dimensionaldatamodel(fancytermforkey/valuepairslabelsagachedtotheJme-seriesmetrics)
• HTTP-basedpullmodel(PrometheusscrapsnodesformetricsviaHTTP)
• PushgatewayssupportedforapplicaJonswithoutanaJveHTTPendpoint
31
SangomaTechnologies-©2016
Prometheus
• ClientlibrariesavailableformanylanguagessoyoucaninstrumentyourapplicaJons
• Flexiblequerylanguage(thinkSQLforforJme-series)
• Commandlinequeryingtool
• Built-inalerJng
• CanbefederatedforaggregaJonofdatainmulJpledatacenters
32
SangomaTechnologies-©2016
Prometheus
• GostandaloneapplicaJon(nodependencies)
• Allmetricsstored,nolossofgranularity
• Similarformattographite,withtheaddeddimensions:• freeswitch_total_calls{profile=internal}20
1470843323
33
SangomaTechnologies-©2016
Prometheus
34
SangomaTechnologies-©2016
GraphingwithGrafana
• PowerfuldashboardbuilderapplicaJon
• Supportformanybackenddatabases(Prometheus,Graphite)
• AutomatecreaJonofdashboardswiththeirHTTPAPIandJSONdashboarddefiniJonformat
• RichgraphingcustomizaJons,styling,accesscontrol,etc
35
SangomaTechnologies-©2016
GraphingwithGrafana
36
SangomaTechnologies-©2016
GraphingwithGrafana
37
SangomaTechnologies-©2016
mod_prometheus
• NaJvePrometheusFreeSWITCHintegraJonmodulewrigeninRust
• Earlydevelopment(startedjustafewdaysago)butalreadyprovidesusefulmetricsoutofthebox,readytobescrappedbyPrometheus
• MetricslikeCalls,Failedcalls,RegistraJons,CPS,ASR,etc
• YoucanthenusePrometheusAlertmanagertocreatealerttriggerrulesbasedonthosemetrics
• hgps://github.com/moises-silva/mod_prometheus
38
SangomaTechnologies-©2016
mod_prometheus
• InstallingrequiresRustandCargo(theRustpackage/buildtoolandFreeSWITCHmaster)
#gitclonehgps://github.com/moises-silva/mod_prometheus#cdmod_prometheus#cargobuild#cptarget/debug/libmod_prometheus.so/usr/local/freeswitch/mod(Donotagempttorenamethemoduletomod_prometheus,itwon’twork)
39
SangomaTechnologies-©2016
mod_prometheus
• Thenjustloadit
40
SangomaTechnologies-©2016
mod_prometheus
• Testwithcurl
41
SangomaTechnologies-©2016
mod_prometheus
• Usecustomcountersandgaugesfromthedialplan
42
SangomaTechnologies-©2016
mod_prometheus
• NextPrometheusscrapwillgetthosemetrics
43
SangomaTechnologies-©2016
FinalThoughts
• ApplicaJoninstrumentaJonmustbepartofapplicaJondevelopment
• Monitoringtoolsofallkindsareavailable.Noexcusesforlackofmonitoring
• Monitorallthethings!
44
SangomaTechnologies-©2016
AUerword
• BringingRusttoFreeSWITCH.Themod_prometheusmoduleisthefirstRustmodulecomingintoFreeSWITCH
• RustisanewsystemsprogramminglanguagewithinthesamespeedrangeasC/C++butwithmemorysafetyguaranteesusinganownershipsystemandmovesemanJcsbydefault
• RustbindingsareavailablebythefreeswitchrsprojectcreatedbyMichaelGiagnocavo:hgps://gitlab.com/wiresight/freeswitchrs/
• RustmemorysafetycouldmakeforaninteresJngaddiJontothestabilityofFreeSWITCH.Ifyou’rewriJngmodulespleaseconsiderusingRust
45
SangomaTechnologies-©2016
QUESTIONS
ContactUs
• SangomaTechnologies100RenfrewDrive,Suite100Markham,OntarioL3R9R6Canada
• Websitehgp://www.sangoma.com/
• Telephone+19054741990x2(forSales)
SangomaTechnologies-©2016
47
THANKYOU