Page 1
1
SplunkITSISandboxGuidebookSTARTHERE....................................................................................................................................................................................................................................21-FlyOvertheProduct..............................................................................................................................................................................................................32-PreparefortheJourney:CoreConcepts.......................................................................................................................................................................43-TourtheGlassTables.........................................................................................................................................................................................................134-TroubleshootingTourwithGlassTablesandDeepDives.................................................................................................................................165-DiveintoDeepDive...........................................................................................................................................................................................................257-TourMulti-KPIAlerts.........................................................................................................................................................................................................286-DiveintotheNotableEventsReview.........................................................................................................................................................................347-DiveintotheServiceAnalyzer......................................................................................................................................................................................368-SideTriptoOSHostDetails............................................................................................................................................................................................377b-AndBacktoServiceAnalyzer......................................................................................................................................................................................37
DocumentRevisionHistoryDate Notes2016Apr18 AddedMulti-KPIAlerts,tweakedvariouschapters,editedforconsistency..dmillis2016Apr08 Updatedfor2.2..lsnow2015Dec07 Completed"tour"chapters..jlebaugh,dmillis2015Dec03 Filledoutfirst4chapters..dmillis2015Nov29 Initialversion..dmillis
Page 2
2
STARTHEREWelcometotheITSISandboxPlaybook!Itisintendedasatravelguidetohelpyouexplorethefeatures,capabilitiesandpossibilitiesofITServiceIntelligence,usingyournewSplunkITSIOnlineSandbox.IfyoudonotalreadyhaveanITSISandbox,gototheITSIHomepage(http://www.splunk.com/itsi)andclickthegreen"FreeOnlineSandbox"button.Itonlytakesafewminutes!Theplaybookcontainsaseriesofchapters,orexercises,tofacilitatetheexplorationofITSIandillustratehowitcouldbeusefulinactual"realworld"environments.ThestudentshouldalreadyhaveabasicunderstandingofcoreSplunk,especiallyhowtocreatesearchesandreports.Thisplaybookshouldnotbeconsidered"realtraining";pleaseseeSplunkEducation(http://www.splunk.com/view/education/SP-CAAAAH9)forin-depthcoursesonITSIandothertopics."Fly-Over"and"Tour"chaptersshowfeaturesandcapabilities,inlessdetailandmoredetail,respectively."Divein"chaptersgointothemostdetailabouthowtosetupandconfigure.Otherchapterscoverhowtocreatenewcomponents,howtouseITSItotroubleshootproblemsquickly,andhowtomock-upvisualizationsforyourownhigh-valueservices.AlthoughtheITSISandboxisnotsetuptoallowoutsidemachine-datatobebroughtin,itdoescontainaneventgeneratortosimulatetheeventswhichmightbeseeninatypicalITenvironment,includingfailurescenarios.Italsocontainsanumberofpre-builtKPIs,services,GlassTablesandothergoodiestomakethejourneymoreinteresting.Generally,thechaptersarelaidoutwiththemorebasicconceptsandexercisesfirst,andmoreadvancedtopicslater.Studentscanskipchaptersandjumparoundastheycareto;eachchapterliststherecommendedpre-requisitechapters.Ultimately,thepurposeofthisplaybookisallowstudentstoworkwithandunderstandthefullcapabilitiesofITServiceIntelligence,andexplorehowITSIcouldhelpsolveactual,useful,high-valuechallengesintheirownITenvironments.
Page 3
3
1-FlyOvertheProductForthetravellerwhoisinahurry,whowantsthe30,000-footview,thisisthesectionforyou!Itisalsothebestplacetobegin,forthestudentwhoislargelyunfamiliarwithITServiceIntelligence.
Instructions1. AfterloggingintoSplunk,clickon"ProductTour"
2. Clickthroughtheslidestopreviewservices,entities,KPIs,thresholding,DeepDives,Multi-KPIAlerts,NotableEvents
andtheServiceAnalyzer3. Thesetopics,andmore,arecoveredinmoredetailinthefollowingchapters
Page 4
4
2-PreparefortheJourney:CoreConceptsBeforewebeginthejourney,itishelpfultounderstandafewcoreconceptsofITServiceIntelligence.
ITSICoreConcepts–Services
DNS RequestsResponses
TechnicalServices
CustomerTransac6ons
RequestsResponses
BusinessServices
AuthRequestsResponses
WebRequestsResponses
SupportDesk RequestsResponses
Conceptually,aServiceisa“blackbox”whichwesendrequestsandexpectresponses.Includestechnical(lower-level)andbusiness(higher-level)
8
Page 5
5
ITSICoreConcepts–Services
PacketNetwork
HypervisorandHosts
RDBMSs
StorageTier
APIServices
WebServices
CustomerTransac4ons
MobileAPI/
Middlew
are
PartnerPortal
DNS
9
ServicescanencompassmulCpleCersoftheITdomain.Servicesmayalsodependuponotherservices
Page 6
6
ITSICoreConcepts–KPIs&HealthScores
DNS RequestsResponses
KPI:Numberofrequests
KPI:Errorrate
KPI:Averageresponse9me
KPI:ServerCPUload
KPI:ServernetworkI/Ferrors
CustomerTransac:ons
RequestsResponses
KPI:Numberoftransac9ons
KPI:Errorrate
KPI:Averageresponse9me
KPI:CountofIncidentTickets
KPI:Synthe9cTransxHealth
AKeyPerformanceIndicator(KPI)isaSplunksavedsearchthatproducesametriclikeCPU%,AvgResponseTime,ErrorRate,etc.KPIsarecontainedwithinServices.AHealthScoreisascorefrom0-100thathelpsdeterminethehealthofaservice.Itiscalculatedbasedonimportanceandstatus
(e.g.,green,orange,red)ofallKPIs,onceeveryminute
12
Page 7
7
ITSICoreConcepts–ServiceAnalyzer
ServiceAnalyzerisanauto-generated,filterable,8ledviewofServicesandKPIs.ItisalaunchingpointforexploringServiceandEn8tyHealthindetail,aswellas
crea8ngad-hocDeepDives
13
Page 8
8
ITServiceIntelligence–CoreConcepts
27
AGlassTableisacustomizablefreeformdrawingdashboardstoviewHealthscoresand
KPIsofchoicewithvisualtoolstocreatecontextwithlivewidgets
GoDeepertoaDeepDiveView
Page 9
9
ITServiceIntelligence–CoreConcepts
28
DeepDive–SwimlaneanalysisdashboardtoshowKPIindicators
over:meforinves:ga:ons
Page 10
10
ITServiceIntelligence–CoreConcepts
29
Mul5KPIAlerts–Visualtooltocreatecorrela0onsearchesbasedonKPIs
Page 12
12
ITSIrepresentsanewwayofdealingwithITServicechallenges:
• Data-drivenapproachusesALLITData-events,metrics,logs,structured,
unstructured,from-the-device,from-the-wire,etc.
• Service-awarenessprovidesactionableinsightsintohigh-visibilityservices
• Customizedcontextualvisualizationscanbetailoredforanypersonorgroup:
highlytechnicaltobusiness-oriented
• Mitigateproblemsbeforetheyimpactcustomers
Page 13
13
3-TourtheGlassTablesGlassTablesareanewtypeofdashboard,whichallowITSIservices,KPIsandhealthscorestobevisualizedinhighlycustomizableways.GlassTablescanbetailoredtoshowverydetailedtechnicalviews,orhigher-levelbusinessviewswithcustomer/revenue-relevantmetrics.Fromthetechnical"soldiersinthetrenches"toexecutivemanagement,GlassTablescanbecraftedtoshowservices,servicerelationships,transactionflows,healthscores,keybusinessmetricsandothercontentwhicharerelevanttotheusers.Andthey'realotoffuntobuild,too!ThissectionshowsanumberofexampleGlassTables.
Instructions1. NavigatetotheGlassTablelistbyclickingon'GlassTables'inthetopmenubar
2. FromthelistofGlassTables,clickonaTitletoviewthatGlassTable
Page 14
14
3. SelectButtercupGamesBusinessProcess
ThisGlassTableshowsthehigh-levelbusinessprocessstatusButtercupGames.Itcouldbeusedbyserviceowners,executivemanagementorotherswhoneedtoquicklyunderstandthe"bigpicture".
4. SelectOnLineTransactionService
ThisGlassTableshowsadetailedviewofacustomer-facingservice,includingtransactionflow,componentrelationshipsanddependencies,andcriticalhealthscoresandmetricsofkeyservicepointsalongtheway.Itmakesexcellentuseofapre-existingdrawing,withliveITSI"widgets"placedstrategicallyontop.ThisGlassTablewouldhelpfulforNOC,Tier1&2andsimilarsupportpersonnelwhoneedtounderstandthecomplexrelationshipsofalltheservicecomponentssupportinganimportantbusinessservice.
5. SelectButtercupGamesOnlineStore
ThisGlassTableshowsastreamlinedviewofButtercupGames'customer-facingservice--the"onlinestore"summarizedinthe"ButtercupGamesBusinessProcess"GlassTable.Thisviewprovidesmoredetailoftheunderlyingtechnicalservices,theirdependencies,andtheoveralltransactionflow.ItusesnativeGlassTabledrawingtools,aswellasserviceandKPIwidgets,whichdisplayhealthandmetricvalueslive(updatingovertime).Thesewidgetshaveconfigurabledrill-downcapabilities,includingtheabilitytonavigatetoother,even-more-detailedGlassTables.Forexample,ifyouclickonthewidgetnexttoWebTier,youwillnavigateto...
6. WebTier
ThisGlassTablerepresentsamoredetailedvisualizationoftheKPIs,overallWebTierhealthscore,andthehealthscoreofitsdependentservice,Middleware.SuchGlassTablesallowtechnicalpersonneltoquicklytroubleshootproblemsbybeingabletodrilldowntothedetailedtechnicalmetricswhichmatter.
7. SelectButtercupGamesOnlineStore(again)
Severaldrill-downoptionsareavailablewhenawidgetisclicked.ClickonthewidgetnexttoDatabase;thiswillnavigatetoaDeepDive.
Page 15
15
GlassTablesallowservices,dependencies,healthscores,KPIsandothercriticalinformationtobevisualizedinacontextualwaythatistrulymeaningfultothetargetedaudience.Thisallowsuserstoquicklysize-upservicedeliveryhealthandwhennecessary,efficientlyisolateproblems.
Page 16
16
4-TroubleshootingTourwithGlassTablesandDeepDivesThissectiondescribesapossibleproblemscenario,andhowITSIcouldbeusedtoefficientlytroubleshoottofindrootcause.ThiswouldtypicallybedrivenbyaNOCorTier1orTier2supportperson.We'regoingto"setup"thefailurescenarioandfirstseehowGlassTablescanacceleratethetroubleshootingprocess,thencontinueisolatingrootcausewithDeepDives.
Pre-RequisitesYoushouldalreadybefamiliarwith:
• CoreConcepts(Ch.2)• GlassTables(Ch.3)
Abouttheeventgenerator...InordertomaketheITSISandboxmoreinterestingtoplayin,aneventgeneratorisincludedwhichcontinuouslygeneratesasimulatedstreamofrealisticmachineevents,includingwebaccess,database,Linuxmetrics(fromthe*nixTechnologyAdd-on)andothers.Includedinthisstreamofeventsaretwofailurescenarios,showingasequenceoffailuresandresultingservicedegradations,eachscenariorepeatinghourly.Typically,theinitialfailuresforeachscenariooccuratthetopofthehour,andresetbackto"OK"aroundthetopofthenexthour.However,theeventgenerator(eventgen)timingmaynotbeprecise.Thefailurescenariosmayoccuratslightlydifferenttimesfromhourtohour,andmayvaryfromsandboxtosandbox.Thus,withintheSandbox,itisimpossibletopredictexactlyhowthehealthscoresandKPIswillappear,duringanyspecifichour.Thismakesitdifficulttosetupa"clean"failuresimulation.Pleasepardonanyeventgeninconsistencies.WedecidedtoputmostofoureffortintodevelopingITSI--notaneventgenerator.
Page 17
17
Instructions1. NavigatetotheGlassTablecalled,ButtercupGamesOnlineStore:
a. ClickonGlassTablesintheuppermenubartonavigatetothepage,SavedGlassTablesb. ClickonButtercupGamesOnlineStoretonavigatetothisGlassTable
2. Modifytheviewtimebyclickingonthetimepickerintheupperrightcorner.Inthepop-upwindow,typeinanexplicittimefromthepast,suchasXX:15.0fromtheprevioushour(orthehourbeforethat,etc).BesuretousethecorrectHH:MM:SS.sssformat(example:"10:15:00.0")
Page 18
18
3. Inafewseconds,thecolorsofthewidgetswillchange,toindicatetheirstatesatthatparticulartimeinthepast.Asnoted
earlier,thetwodifferentfailurescenariostoggleeachhour.Trydifferenteven&oddhoursinthepasttoseethis.4. Forthepurposesofthistroubleshootingexercise,imaginethatyourGlassTablelookslikethefollowing:
Page 19
19
5. Thescenario:CustomerCarehasinformedusthatcustomersarecallingtocomplainwhentheytrytopurchasethroughtheOnlineStore;theyareseeingslowresponseandoccasionalerrors.Theproblemsseemtobeaffectingbothweb-basedandmobile-basedcustomers.
6. Basedonjustthereportsthatthecustomer-facingweb-basedserviceishavingproblems,mostsupportpersonswouldbegintroubleshooting"fromthetop"--thewebandmobiletiersinthiscase.Ifnoobviousproblemswerefound,theywouldproceeddowntheservicedependencytree--tothemiddlewaretier,etc.
7. ButusingaGlassTablesuchas"ButtercupGamesOnlineStore"providesinstantandcontext-relevantvisibilityintoservicehealthscoresandimportantKPIs,allinoneplace.Intheaboveexample,whichsupportingtierseemstobeindistress?(Database)Bybeingabletovisualizetherelevantservicesandtheirhealthscores,wehavetheabilitytoimmediatelyfocusourtroubleshootingontheareasthataredegraded.Thiscansavehugeamountsoftimeandgreatlyreducethetimerequiredtofindrootcause.
8. OnyourSandboxGlassTable,clickonthewidgetbeneathDatabasetodrilldownintotheDatabasetiertocontinuethetroubleshootingexercise.(SelectLeaveThisPageifprompted)
Page 20
20
(NowinDBDeepDive)9. Clickonthe>nexttoFocustocollapsetheservicetreenavigatorpanelontherightside;wewillexplorethisfeaturelater.10. ChangethePrimaryTimeRangetoLast2Hoursbyclickingonthetimepickerinthelowerleftcorner:
a. IntheRelativesection,typein"2"andselectHoursAgob. ClickApply
Page 21
21
(TheDeepDiveshouldnowdisplayssome"mostlygreen"andsome"mostlyred"–yourscreenmaynotlookexactlylikethebelow,butyoushouldseeapointwherethingsgofromgreentoredandredtogreen)
11. WearelookingattheAggregatedHealthScore(topswimlane)andKPIsfortheDBService,acrossatimerangewhich
showstheservicemovingfrom"healthy"to"nothealthy".12. Slowlymouseovertheswimlanestocomparevaluesatvariouspointsintime.13. Clickthecheckboxintheupperlefttoselectallswimlanes,andusethe“BulkActions”menuto“ShowStateThresholds”or
"HideThresholds",togglingtocomparetheswimlaneswithandwithoutthethresholdcolors/statesoverlaid.14. NotethattheServiceHealthScoreinthetopswimlaneisanaggregationoftheservice'sKPIsanddependentservices,
rangingfrom100-0.Whendidthehealthscorebegintodeteriorate,andwhichKPI(s)mayhavebeenpartoftherootcause?
Page 22
22
15. Clickonthename-boxforStorageFreeSpace:%System,thendragitupwardstorepositionthisswimlane.16. Afewoftheswimlanesarecontinuouslygreen,indicatingthattheyarenotparticularlyhelpfulinourtroubleshooting
exercise("CPUUtilization","MemoryFree",etc).Clickonthecheckboxintheupperleftcornertounselectallswimlanes,thenselectthecheckboxforCPUUtilization:%UserandMemoryFree.SelectBulkActions->Deleteto(temporarily)removethisswimlanefromourDeepDive.
17. ClickonthedarkerbluetilewithintheDBServiceErrorsswimlanetoreveal"rawerrors"fromtheunderlyingSplunk
search.ClickonHideEventstodismiss.
Page 23
23
18. MouseovertheStorageFreeSpace:%Systemswimlane,intheplacewhereitgoesfromgreentored.Notethehigh&lowmetricvaluesshownfortheswimlane,andthatthismetrichasgoneto0%,indicatingthatafilesystemisfull.
19. ClickanywherewithintheStorageFreeSpace:%Systemswimlanetorevealanoptionspopup.SelectAddOverlayasLane.
(Threenewswimlanesareaddedatthebottom,representingtheseparateKPIvaluesfortheindividualentities(hosts)whichcomprisethisKPI)20. Whichhost/serverissufferingfromafilesystem-fullcondition?(mysql-02)
Page 24
24
OverallservicehealthcanbeeffectivelyandefficientlyvisualizedinGlassTables,allowingsupportpersonneltoquicklyfindlikelyhotspots.TheycandrilldowntomoredetailedlayoutsandultimatelycompareandcorrelateKPI&ServicetrendsinparallelswimlaneswithinDeepDive.Fasterrootcauseanalysis(RCA)leadstosubstantialreductionsinMeanTimeToRepair(MTTR).
Page 25
25
5-DiveintoDeepDiveDeepDivesallowKPImetricsandhealthscorestobecomparedinside-by-sideswimlanes,whichallowstrendsandcorrelationstobemoreeasilyandquicklydiscovered.ThischapterexploresDeepDivesandhowtheycanbeused.
Pre-RequisitesYoushouldalreadybefamiliarwith:
• CoreConcepts(Ch.2)• Troubleshooting(Ch.4)alsogoesintoDeepDives
Instructions1. NavigatetotheDeepDivecalled,DBDeepDive:
a. ClickonDeepDivesintheuppermenubartonavigatetothepage,SavedDeepDivesb. ClickonDBDeepDivetonavigatetothisDeepDive
2. Clickonthe>nexttoFocustocollapsetheservicetreenavigatorpanelontherightside;wewillexplorethisfeaturelater.3. SelectanarbitrarytimerangebyclickingonthePrimaryTimeRangemenuoptionatthebottomright;itfunctionslikea
standardSplunksearchbartimepicker4. Zoomintoatightertimerangeinthecurrentviewbyclick-holdinganywhereintheswimlanes,thendragginghorizontally
toselecttherange.5. Togglethethresholdhealthscorecolorsbyclickingonthecheckboxintheupperleftcornertoselectallswimlanes,then
BulkActions->ShowStateThresholds/ShowLevelThresholds/HideThresholds.6. Clickonthe>nexttoFocustoopentheservicetreenavigatorpanelontherightside.
a. Clickonaservicenodetonavigateupanddownthedependencytreeofservicesb. Afterclickingonaservicenode,notethatthoseservice'sKPIsarelistedbelow.c. Clickonthe+onalistedKPItoaddittothecurrentswimlanesd. Clickonthe>nexttoFocustocollapsetheservicetreenavigatorpanelontherightside
7. Mouse-overthename-boxforanyswimlanetorevealthe"optionswheel",thenselectittoviewavailableoptions:
Page 26
26
8. Thestudentisencouragedtoexploretheseoptions,whicharecoveredinmoredetailat
http://docs.splunk.com/Documentation/ITSI/latest/User/DeepDives9. Click-holdonthename-boxforanyswimlane,andthendragitverticallytorepositionthisswimlane.10. ClickonthedarkerbluetilewithintheDBErrors(orany"event"-style)swimlanetoreveal"rawerrors"fromthe
underlyingSplunksearch.ClickonHideEventstodismiss.11. TosaveaDeepDiveaftermodifyingthelayoutand/orvisualizationoptions,clickontheEditmenuoptionintheupper
rightcorner,thenselectSave12. Tocomparethecurrenttimerangeagainstadifferenttimerange,clickonCompareto...inthelowerleftcorner,then
selectacomparisontimerange.ThiscauseseachKPItodisplaytwinswimlanes:primarytimerangeabovecomparisontimerange.Notethatwhenmousingovertheswimlanes,thetimedisplayatthetopnowshowsbothtimes.
13. Todismissthe"twin"lanesdisplay,deselectthecheckboxnexttoCompareto...inthelowerleftcorner
Page 27
27
DeepDiveallowsanyKPIsandServicestobecomparedandcorrelatedinaside-by-sidefashion,acrossmultipletimeranges,usingavarietyofvisualizations.Itisintendedtogreatlyenhanceandstreamlinethetroubleshootingprocessforfindingrootcause,significantlydecreasingMeanTimeToRepair(MTTR).
Page 28
28
7-TourMulti-KPIAlerts
Multi-KPIAlertsareCorrelationSearcheswhichcancombineanyKPIstocreatemeaningful,actionablealerts,usingmultiplecorrelationfactorssuchKPIthresholdindications,lengthoftimeinthisstate,time-of-day,andothers.Multi-KPIalertscanfindnotjust"failures",butearly"canaryinthecoalmine"indicationsthattheserviceisbecomingunstable;itispossibletofindproblemsBEFOREtheyimpactcustomer-facingservices.WhenaMulti-KPIAlertfires,itcreatesaNotableEvent;itcouldalsoexecuteascriptand/orsendemail.
Pre-RequisitesYoushouldalreadybefamiliarwith:
• CoreConcepts(Ch.2)• TroubleshootwithGlassTablesandDeepDives(Ch.4)
Instructions1. NavigatetotheDeepDivecalled,DBDeepDive:
a. ClickonDeepDivesintheuppermenubartonavigatetothepage,SavedDeepDivesb. ClickonDBDeepDivetonavigatetothisDeepDive
2. Clickonthe>nexttoFocustocollapsetheservicetreenavigatorpanelontherightside.3. ChangethePrimaryTimeRangetoLast2Hoursbyclickingonthetimepickerinthelowerleftcorner:
a. IntheRelativesection,typein"2"andselectHoursAgob. ClickonApply
Page 29
29
(TheDeepDiveshouldnowdisplayssome"mostlygreen"andsome"mostlyred"–yourscreenmaynotlookexactlylikethebelow,butyoushouldseeapointwherethingsgofromgreentoredandredtogreen)
Page 30
30
4. WearelookingattheAggregatedHealthScore(topswimlane)andKPIsfortheDBService,acrossarangeoftimewhich
showstheservicemovingfrom"healthy"to"nothealthy".5. Click/dragacrossanarrowerrangeoftimewhentheservicetransitionsfromgreentoyellow/orange.6. Clickonthecheckboxintheupperlefttounselectallswimlanes,thenselectthecheckboxesnexttotheKPIswimlanes
whichwereinvolvedinthisoutage(turnedred)duringthisperiod,suchasStorageFreeSpace,DBServiceQueries&DBServiceResponseTime.
7. Intheupperleft,selectBulkActions->CreateMultiKPIAlert
Page 31
31
(ThiswillopentheMultiKPIAlertconfigurationworkflowpage)
Page 32
32
ITSIprovidesasophisticatedarrayofoptionsforsettingupMulti-KPIAlerts,alsoknownasCorrelationSearches.ThegoalistoallowthecreationofusefulalertsbasedoncorrelationsofseveralKPIs--fewer"noise"alerts,moreactionablealerts.Herearesomeofthefeaturesandcapabilities:
• ControltherangeoftimetocorrelatetheKPIsacross(time-pickerintheupperrightcorner)• AddKPIsfromanyservice• CreateaKPIbasedontheaggregatehealthscoreoftheKPIs,oronStatusovertime(upperrightcorner)• Re-weighttheKPIsusingtheImportancesliders(lowerrightcorner)
Page 33
33
• Controlalertactions,suppression,andotherdetails(laterinthecreationworkflow,afterhitting'Save'inthelowerrightcorner)
OneofthemostimportantthingswhichhappenswhenaMulti-KPIAlertfires,isthecreationofaNotableEvent.NotableEventsareexploredinalaterchapter.ExistingsampleMulti-KPIalertscanbeexaminedbyclickingonConfigure->CorrelationSearches,thenselectingacorrelationsearchfromthelist.Moredetailsareavailablehere:http://docs.splunk.com/Documentation/ITSI/latest/User/CreateMulti-KPIAlertsMulti-KPIAlertscancombineanyKPIstocreateuseful,actionablealerts(lessalert"noise")."Canaryinthecoalmine"problemscanbedetectedearly,potentiallybeforetheyaffectcustomers,revenueorSLAs.
Page 34
34
6-DiveintotheNotableEventsReviewWhenaMulti-KPIAlertfires,itcreatesaNotableEvent.TheNotableEventsReviewisSplunk'snext-generationeventmanagementconsole.NotableEventsReviewprovidesaquickwayview,siftandorganizeevents,allowingustotriage,manageandstreamlineworkflowmoreeffectively.IthastheabilityfilterNotableEventsandeventsfromothereventmanagementsources,basedonvariouscriteria,suchasSeverity,Status,Serviceandothers.Italsoallowseventstobemodified,tochangeOwner,Severity,Status,and/oraddcomments.Eventscanalsohaveworkflowactionsassociatedwiththem,toallowanoperatortheabilitytoquicklyhittroubleshootingoptions,executemitigationscripts,oropena"real"IncidentManagementtrouble-ticket.
Pre-RequisitesYoushouldalreadybefamiliarwith:
• CoreConcepts(Ch.2)• TourMulti-KPIAlerts(Ch.7)
Instructions1. NavigatetotheNotableEventsReviewbyclickingonNotableEventsReviewintheuppermenubar2. ClickonShowTimelinetorevealthetimeline3. Seedetailsforanevent:ClickonanyeventtoopentheDetailspanelontheright.
DetailsincludewhichKPIscontributed,andwhichservicesmightbeaffected,aswellastheabilitytoexaminetheseinmoredetailinaDeepDive.Severity,StatusandAssignmentcanalsobechangeddirectly.
4. ModifySeverityforanevent:ClickontheSeveritydropdownatupperleftoftheDetailspanel,chooseadifferentSeverity5. Chooseaworkflowaction:Clickon</>iconinupperrightcornerofDetailspaneltorevealtheworkflowoptions
CustomworkflowactionscanbecreatedforeachtypeofNotableEvent,tostreamlineworkflowactions.Thesecanbeadditionaltroubleshootingormitigationscripts,orsomethingasbasicasopeninga'real'incidentticket.
6. DismissDetails:ClickontheXintheupperrightcornerofDetailspaneltodismiss7. FiltertheNotableEventsbySeverity:Clickonthe"gearwheel"intheupperrightcorner,thenchooseEditFilterSettings.
ClickAddFilter,andthenSeverity.ClickintheSeverityboxtoseeandchoosefromalistoftheavailableseveritylevels.
Page 35
35
8. FilterbyStatus,Owner,Service,TimeRange,Name("Title")orfreeformsearchcriteriabyaddingotherfilterstoyourfiltersettings.
9. ClickDoneto(re)applysearchfiltercriteria10. Changeviewoptions:Clickonthe"gearwheel"intheupperrightcorner,thenchooseEditViewSettings.SelectViewing
Option->ProminentandDeduplication->On,thenDoneAnEventCountcolumnhasnowbeenaddedfordeduplicatedevents,andSeveritycolorisnowmore'prominent'
11. Add,removeorre-ordercolumns:Clickonthe"gearwheel"intheupperrightcorner,thenchooseEditViewSettings.InColumnsShown,clickXtoremoveacolumn,click+AddColumntoaddacolumn,orclick/dragacolumntore-orderhowitisviewed.
12. Tosorttheeventrows:ClickontheVchevronnexttoSortBy:(leftside,aboverows),thenselectacolumntosortby.Togglethesortorder(ascending/descending)byclickingontheverticalarrownexttoSortBy:
Moredetailsareavailablehere:http://docs.splunk.com/Documentation/ITSI/latest/User/NotableEventsReviewTheNotableEventsReviewallowsanoperatorto:• Quicklyandeffectivelyfind,deduplicateandmanagejusttheeventstheywant• Tieworkflowactionstoevents,tostreamlineoperations• ManageITSINotableEventsandeventsfromothersources
Page 36
36
7-DiveintotheServiceAnalyzerTheServiceAnalyzerisa"BigPicture"viewofallservices,andthe"mostinteresting"KPIs(i.e.,KPIswithdegradedhealthscores).Itis"nofrills",designedforNOCs,Tier1or2support,andotherswhoneedahighlevelviewofallservices/KPIs,orasubset.ItalsoprovidesalaunchingpointforexploringServices,KPIsandEntitiesinmoredetail.
Pre-RequisitesYoushouldalreadybefamiliarwith:
• CoreConcepts(Ch.2)
Instructions1. NavigatetotheServiceAnalyzerbyclickingonServiceAnalyzerintheuppermenubar,thenchoosingDefaultService
Analyzer.2. ClickonMiddlewareServicetonavigatetoitsservicehealthpage.Hereyoucanseetheservicetreeontheleft,theKPIs
inthecenter,andtheentitiesassociatedwithaselectedKPIontheright.3. ClickonDBServiceintheleftservicetreepaneltonavigatetothisservice4. FromServiceHealth,youcanalsonavigatetoadeepdivecontainingtheKPIsforthatserviceusingthelinkatthetopof
theKPItableinthecenterofthepage.a. Noticethedeepdivehasbeenbuiltforyouonthefly,containingalltheKPI’sassociatedwiththatservice
5. ClickonStorageFreeSpace:%SystemandnoticethatyounowhaveatableontherightthatshowstheentitiesassociatedwiththisKPI.
6. Clickonmysql-02intheentitylisttonavigatetoitsEntityHealthpage.a. Thisisanentity-centricview,showinginformationaboutaspecificentity,includingwhichservicesandKPIsit
supports.b. Clickingonaservicenamewillnavigatetothatservicehealthpage
Page 37
37
8-SideTriptoOSHostDetails7. IfyouareusingoneormoreITSImodules,relevantmoduledashboardsforthisentitywillshowupintheleft-sideModules
panel.Inthiscase,"OSHostDetails"islisted.Moredetailsaboutmodulesareavailablehere:http://docs.splunk.com/Documentation/ITSI/latest/IModules/AboutITSIModules
8. ClickonOSHostDetailstonavigatetothispage.a. TheOSHostDetailssectionoffersseveraldashboardswithdetailedstatus,performanceandeventreports.b. OSHostDetailscanalsobeaccessedinDeepDive.c. MoredetailsabouttheOperatingSystem(OS)Moduleareavailablehere:
http://docs.splunk.com/Documentation/ITSI/latest/IModules/AbouttheOperatingSystemModule
7b-AndBacktoServiceAnalyzer9. NavigatebacktoServiceAnalyzer10. ClickintheSelectservice(s)tomonitorboxtoselect&showonlycertainservices11. Clickonthe"OptionWheel"nexttoTop...Servicestocontrolhowmanyservicesareshown12. Clickonthe"OptionWheel"nexttoTop...KPIstocontrolhowmanyKPIsareshown,andtoselectwhichKPIsareshown13. Tocreateanad-hocDeepDive:
a. MouseoveroneormoreServiceorKPItiles,thenselectthecheckboxintheupperrightcornerofthetileb. ClickDrilldowntoDeepDive
ServiceAnalyzerprovidesa"BigPicture"viewofallservicesandthe"mostinteresting"(notgreen)KPIs.ItisalsoalaunchingpointforexploringServices,KPIsandEntitiesinmoredetail,aswellasforcreatingad-hocDeepDiveswithselectedKPIs.
