Copyright©2016Splunk Inc.
AgencyChargebackModelstoEnableEnterpriseSplunkDeployments
MikeWilson- PrincipalArchitect,PublicSectorSalesEngineeringAdilson Jardim - AreaVP,PublicSectorSalesEngineering
AFrameworkforquantifyingITservicecosts
DisclaimerDuringthecourseofthispresentation,wemaymakeforwardlookingstatementsregardingfutureeventsortheexpectedperformanceofthecompany.Wecautionyouthatsuchstatementsreflectourcurrentexpectationsandestimatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthosecontainedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthethispresentationarebeingmadeasofthetimeanddateofitslivepresentation.Ifreviewedafteritslivepresentation,thispresentationmaynotcontaincurrentoraccurateinformation.Wedonotassumeanyobligationtoupdateanyforwardlookingstatementswemaymake.Inaddition,anyinformationaboutourroadmapoutlinesourgeneralproductdirectionandissubjecttochangeatanytimewithout notice.Itisforinformationalpurposesonlyandshallnot,beincorporatedintoanycontractorothercommitment.Splunkundertakesnoobligationeithertodevelopthefeaturesorfunctionalitydescribedortoincludeanysuchfeatureorfunctionality inafuturerelease.
3
Whobearsthecostofprovidingaservice?
4
Finance
Security
Sales
HR
Splunkisaservicetoo!
5
It’snotjustchargeback…
Frameworkforapplyinga"totalcost"torunninganyservice(license+compute+
storage+...)
6
Doyouneedthehelp?
7
Butfirst– Vocabulary
8
Showback Providing metricsanddataregardingresourceutilization(withoutcharging)
Chargeback Implementingformalaccountingpracticestocross-chargedepartmentsforresourceorapplicationutilization
Multi-tenant Architecturesupportingmultipledifferentcustomersononeimplementation
Resources Inthiscontext,allelementsofasystem, including:CPU,memory,storage,virtualenvironments/machines,applications
Term Definition
DefiningCostBasis
9
Type Definition
Fixed Servicetypewithfixedcosting,e.g.#ofusers
Allocation based • Variablecostsperenvironment like#ofSearchHeads,Indexers,Apps,etc.
• VMBased– small,medium,largeVMcosting
UtilizationBased Variablecosts– CPU,Memory,Storage,I/O
CostBasis
Whatwewilladdress
10
Requirementsforchargeback
Structuringateam
Business
Finding: I/O,CPU,SearchCosts,Storage
Whatelseyou canuse
Metrics
Architecture&topology discussion
On-Premise
Cloud asaService
AWSMetrics
ApplicationServices
Considerations forcharging/showback
Charging
DefiningaSplunkasaService
ASplunkCenterofExcellence
12
Engineering Operations
RequirementsLead Admin
KnowledgeAdmin Systems/StorageAdmin
Developer KnowledgeAdmin
Analyst
AnalyticsLead
Defineconsumerorganizations
Defineyourservices:• Design&Development• Analytics,dashboards,API’s,alerts• TieredServicePacks• Howaretheymetered&charged?
CenterofExcellence Team
Let’sdiveintomeatystuff…
SplunkInternalsOverview
14
SearchHead Ina distributedsearch environment,aSplunkEnterpriseinstancethathandles searchmanagement functions,directingsearchrequeststoasetof searchpeersandthenmergingtheresultsback totheuser.
Indexer ASplunkEnterpriseinstancethatindexesdata,transformingrawdatainto events andplacingtheresultsintoan index.Italsosearchestheindexeddatainresponse tosearchrequests.
Forwarder ASplunkEnterpriseinstancethatforwardsdatatoanotherSplunkEnterpriseinstance,suchasan indexer oranotherforwarder,ortoathird-party system.
Application AnapplicationthatrunsonSplunkEnterpriseandtypicallyaddressesseveralusecases.Anappcontainsoneormore views.AnappcanincludevariousSplunkEnterprise knowledgeobjects suchas reports, lookups, scriptedinputs,and modularinputs.
Index Whenyou adddata,theindexerprocessesitandstoresitinan index.Bydefault,datayou feedtoanindexerisstoredinthe main index,butyoucancreateandspecifyother indexesfordifferentdatainputs.
Component Purpose
TamingtheBeast
15
§ InternalSplunkmetricswillassistinunderstanding resourceusageacrosstheinfrastructure
§ Youcanchoosewhentochargeandhowtoreportagainstcustomerusage
§ Splunkarchitectureisflexible,butconsidering howtochargebackmayhelptodefine indexlayoutsornamingconventions
DMC
On-Premise:MonitoringConsole
16
ThedatanecessaryforchargebacksisavailableviaSplunkCoreandeasilyattainedthroughtheSplunkMonitoringConsole(akaDistributedManagementConsole).
SplunkUtilizationMetricsOverview
17
Search Searchstatisticscanbeusedtocalculatecumulativeruntimeperuser orforgroupsofusers
index=_audit sourcetype=audittrailOR
`dmc_audit_get_searches_for_groups(*)`
License LicenseUsage statisticscanbesplitbylicensepool,host,source,sourcetype,orindexindex=_internalsource=*license_usage.log type=Usage
OR`dmc_licensing_base_usage(*,"")`
Storage Diskutilizationmayvaryversusindexingrate. IndexsizescanbecapturedviaRESTcalls.
|restsplunk_server_group=* /services/data/indexes
Review Review
LicenseUtilization
18
§ Monitoring theindexing ratewithintheinfrastructure§ DirectrecoupmentofSplunk licensefees§ Usefulwheninfrastructuregrowthcanbeattributedprimarilytodataingest§ Flexibleandgranularabilitytosplitlicensecostagainst:
§ typesofdata§ hosts§ specificdatasources§ assignedindexers§ somecombinationofthosefactors
LicenseUtilization
19
StorageUtilization
20
§ Monitoring individual oraggregateindexsizesand/orcapacity§ Providesunderstandingof impactonstoragecost§ Usefulwhenindexescanbemapped tospecificcustomers§ Accountsforusageoutsideofjustingest:
§ Accelerateddatamodels§ Summary indexing§ Dataretentionneeds§ Replicatedbuckets
StorageUtilization
21
SearchUtilization
22
§ Monitoring searchesagainstavailablecapacity§ RecoupmentofSplunk infrastructurecosts§ Workswellforcustomerswithlowindexing rates,buthighsearchvolume§ Providesadepictionofsystemresourceutilizationwithin theenvironment§ Costcanbeassociatedwiththenumber ofsearchesorofsearchruntimeasa
flatfeeorasapercentageoftotalorrelativecapacity§ Costcanbeincreasedforpeakhourworkloads (ordiscounted inoff-peak
hours) inordertodiscouragenon-essentialworkloads.§ Maytargetusers,apps,orsearchname
SearchUtilization
23
GroupingtoCostCenters
24
§ Utilizinglookupsallowsyoutogroupunitsofusagetogetherandassigncosts:§ mapusers,datasources,indexes,andotherunits tochargeable
organizationsandusageallocations§ Associateadollarvaluewithsearch,license,orstorageusage
§ Example:associateindexwithgroup andlicensevolume
Customer-wideServices
25
§ Someusagewillbeinthebestinterestoftheentirecustomerbase§ Summary indexing§ Lookupgeneration§ Knowledgeobjectcreation(e.g.macros)§ Customintegrations
§ Considerwhetheryou’dwanttochargefortheseornot§ Willyouincreasecostunitsacrosstheboardtoaccountforthis?§ Isthisabeneficialservicewhichactuallyhelpstosaveresources(e.g.
increasessearchspeed,alleviatessupportburden, etc)?
BestPractices
26
§ Summary indexingshouldbeused inorder tocapturemetrics,thisenables:§ accesstoawiderinternalaudienceviaRBAC§ long termreporting withouttheneedtoextendinternallogretention§ theabilitytosavedataandcalculategrowthforstoragemetricsgathered
viaREST§ Usedataenrichment (e.g.lookups) tofurthergroup tocostcenters§ EnsurethattheDMC,LicenseMaster,andSearchheadsareforwardingdata
backtotheindexing tier.§ Chargebacksshouldnotdiscourageuse,valuederivedcouldbeusedas
showback/credits§ Useallthreemethods together
On-Premise:ChargebackApp
27
ChargebacksforOtherServices
28
§ Identifywherecostisincurred§ e.g.licenses,hardware,services
§ Identifymeasurablemetricswhichrelatedirectlytotheutilizationandcostofthesystem§ e.g.cpu cores,users,countofqueries, servers,etc
§ CapturemetricswithinSplunk§ Usedataenrichment toassociateresponsiblegroupswithcostcenters
29
MeasuringCloudServicesSplunkAppforAWS
Cloud:AWSApp
31
QuantifyingUsage
32
EC2 Compute&application
EBS Storage
ELB LoadBalancing
CapacityPlanner
Intendedgrowth/chargebackmodeling
DatabaseService
Storage/Datamanagement
MeteredComponent MeteredComponent
33
Chargingforservice
TranslatingthistoAccounting
35
Dashboard Fixed #ofusers Actualcost+servicefee
Storage Variable DiskStoragebyhot/cold/indexesbuckets
$/GB +servicefee
Service Type Metrics CostBasis
Searches
SearchAppendix
37
§ SearchstatisticsfromDMC`dmc_audit_get_searches_for_groups(*)`
| stats min(_time) as _time, values(user) as user, max(total_run_time) as total_run_time, first(search) as search, first(search_type) as search_type, first(apiStartTime) as apiStartTime, first(apiEndTime) as apiEndTime by search_id, host
| where isnotnull(search)
| stats sum(total_run_time) as runtime, count(search) as search_count by user, host
| join host type=left [rest splunk_server_group=* /services/server/info | evaltotal_core_time = 60 * 60 * 24 * (numberOfCores + 6) | fields host, total_core_time]
| stats sum(total_core_time) as total_core_time, sum(runtime) as runtime, dc(host) as sh_count by user
| eval user_core_perc = round(runtime / total_core_time * 100, 3)
| rename user as User, total_core_time as "Available CPU Time", runtime as "Total Search Runtime", sh_count as "Search Head Count", user_core_perc as "Percentage of Available Search Time Used"
SearchAppendix
38
§ Licenseusagestatisticsfromcoreindex=_internal source=*license_usage.log type="Usage"
| stats sum(b) as bytes_indexed by idx, h, s, st, pool
§ LicenseusagestatisticsfromDMC`dmc_licensing_base_usage(*, "")`
SearchAppendix
39
§ StoragestatisticsfromDMC| rest splunk_server_group=* /services/data/indexes
| join title splunk_server type=outer [rest splunk_server_group=* /services/data/indexes-extended]
| `dmc_exclude_indexes`
| eval indexSizeGB = if(currentDBSizeMB >= 1 AND totalEventCount >=1, currentDBSizeMB/1024, null())
| eval maxSizeGB = maxTotalDataSizeMB / 1024
| eval sizeUsagePerc = indexSizeGB / maxSizeGB * 100
| stats dc(splunk_server) AS Instances count(indexSizeGB) as "Non-Empty Instances" sum(indexSizeGB) AS totalSize avg(indexSizeGB) as averageSize avg(sizeUsagePerc) as averageSizePerc by title
| eval totalSize = if(isnotnull(totalSize), round(totalSize, 2), 0)
| eval averageSize = if(isnotnull(averageSize), round(averageSize, 2), 0)
| rename title as "Index" totalSize as "Total Size (GB)" averageSize as "Average Size (GB)"
Questions
40
ThankYou
41
Announcements
AWSAgility+SplunkVisibilityLearnmoreaboutourintegratedsolution
Stopbybooth#206tospeakwithoneofourexperts
Attendthesession,YouCan’tSecureWhatYouCan’tSee,tolearnhowAdobeisensuringsecurityandcomplianceinAWSenvironments
RegisterforourSecurityJam– ahands-onworkshopabouthowtokeepyourAWSenvironmentsecure
www.splunk.com/en_us/about-us/events/aws-reinvent.html
Checkouton-demandsessionsfrom.conf2016:
SplunkingAWSforEnd-to-endVisibility
TheChargebackApp
http://conf.splunk.com
.conf2017iscomingtoWashington,D.C.!
44
September25-28,2017WalterE.WashingtonConventionCenter
Reserveyourseatfor.conf2017nowthroughNovember30th togetthesupersaverdiscount!
Reserveyourspottoday,paylater!
SignUpToday:http://live.splunk.com/LP=1822
Afterregistrationopens, youwillhave60daystocompleteyourregistrationtosecurethesupersaverrate.
VisittheInformationKioskintheSolutionPavilion!
SupportOperationHomefront!
45
EarnYour6SponsorBadges!Splunk willdonate$10Dollarsto OperationHomefront’s HolidayMealsforMilitaryFamiliesProgram foreveryattendee thatcompletes theirmissionofearning6sponsorbadges.Theprogramwillprovidemeals
toourlocalmilitaryfamiliesthisholidayseason.Plusabonus ifwehit350 numberofcompletedmissions.Splunkwilldoublethe$3,500donationto
$7,000!
Workshops:GetSplunkHands-onExperienceAttendaSplunkWorkshop
UpcomingScheduleDecember1:IntroductiontoSplunk Enterprise
December14:IntroductiontoSplunk ITTroubleshooting
January11:IntroductiontoSplunk EnterpriseSecurity
January11:NEW! DatabasePerformanceTuningandCapacityPlanningWorkshop
January25:IntroductiontoSplunk ITServiceIntelligence
January25:NEW! Splunk forApplicationDevelopers
LocationSplunkOfficeMcLean,VA
Visithttp://www.doyouknowsplunk.com/workshops
VisittheInformationKioskintheSolutionPavilion!
SplunkUserGroups- ConnectwithLocalSplunkers
NorthernVirginiaMeetsthelast3rd Thursdayofeverymonthhttps://usergroups.splunk.com/group/northern-virginia-splunk-user-group.html
DCMeetsthelastWednesdayofeverymonthhttps://usergroups.splunk.com/group/washington-dc-splunk-user-group.html
BaltimoreMeetsthe3rdMondayofeverymonthhttps://usergroups.splunk.com/group/baltimore-splunk-user-group.html
VisittheInformationKioskintheSolutionPavilion!
TaketheGovSummit PostEventSurvey!
48
Wevalueyourfeedback!Taketheposteventsurvey ontheiPadsinthefoyerstartingat2:30pm!