Date post: | 15-Nov-2014 |
Category: |
Documents |
Upload: | teamquest-corporation |
View: | 285 times |
Download: | 0 times |
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
TeamQuest and the TeamQuest logo are registered trademarks in the US, EU and elsewhere.All other trademarks and service marks are the property of their respective owners.
Optimizing IT Costs & Services with Big Data, Little
Effort…David Wagner
TeamQuest Advocate
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
• Why?• Big Data: conceptual overview • 2013 Capacity Management 101: – History– Goals– Obstacles
• New “Big Data” approaches– Concepts– Case Study Value
Agenda
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
• We passionately believe always having and using the right amount of resources is a societal imperative – Anything less is failure– Anything more is wasteful
• 20+ years sole focus– ensuring our customers can continuously and
automatically perform at their utmost level of efficiency
– ensuring business service performance, conserving scarce resources, saving money and improving productivity
• We call this: IT Service Optimization
Why does TeamQuest Exist?
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
• IS:– Applying Big data approaches to Capacity
Management• Faster and larger value• More scalable
– New ways to think about optimization beyond ITIL Capacity Management• Include ITIL Service Management and Delivery• Not just technology anymore
• Is NOT!– A Primer on Big Data or a Big Data “how to”
Presentation• Hadoop ecosystem deep dive, etc…
What this Presentation is… and is not!
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
• Big Data is about: data actionable information– Plethora of existing sources
• Technology• Business (Sales, Marketing, …)• Service (Transactions, SLAs, …)
– Learning new insights from “old” data– Key is Analytics
• Deep• Wide• Adaptable
• But… Optimizing costs with Capacity Management?
Big Data at 50,000 feet…
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
• Data Access and Aggregation– Build huge “data marts” (aka: Data Warehousing)– Integrate with multiple different data sources
• Technology (e.g. Server, Network, Storage, etc.)• Service (Catalog, Metrics, Tickets, etc.)• Business (KPIs, Plans, Transactions, etc.)
• Implement Analytics against/across– Flexible and adaptive– Turn data within, into actionable information across
• But… Capacity Management???
Technology Approaches
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
• Answering “what if” questions…– Change in technology, demand, etc… impact?– Focus on Optimizing Server Cost versus
Performance• Extremely Technology-centric– Servers, Mainframes– Occasionally Storage or Network – in isolation
• Big Value and Return, but also effort– Highly trained staff– Required building a central, massive datamart
(CMIS)– Scalability of Staff, Tools, …, Politics
2013 Capacity Management 101 - History
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
• Maintain traditional value, and add– Optimize– Amplify– Accelerate
• Increase Business relevance– Valuable predictive analytics in business and
service context– Optimize Efficiency
• Virtualization and Cloud Scale to everything– Many to many inter-relationships; Capacity critical
2013 Capacity Management – Goals: What
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
• Integrate and Analyze across multiple sources– Technology (e.g. Server, Network, Storage, etc.)– Service (Catalog, Metrics, Tickets, etc.)– Business (KPIs, Plans, Transactions, etc.)
• Single pane of “Analytic Glass”– Ability to tie together, correlate, and operate across
• Tear down the wall!– Don’t force reinvention… or data duplication!– Flexible and adaptive– Turn data within, into actionable information across
2013 Capacity Management 101 – Goals: How
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
• Data Access and Aggregation– Building huge “data marts” (fka: Data
Warehousing)• Complexity = (data ETL) x (# sources) x (maintenance
effort)• Compliance: Data duplication, privacy, audit, etc…• Costly and time consuming
• Implementing Analytics against/across– General purpose BI Analytics for Capacity?– Traditional Performance/Capacity for General
Purpose?• “Big Data” + ITIL = Optimized Capacity
Management?
2013 Capacity Management 101 - Obstacles
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
• Service Strategy– Financial management
• Service Design– Service Level and Availability management
• Service Transition– Asset, Change and Configuration Management
• Service Operations– Service Desk– Application and IT operations– Event, Incident, Problem
• Or, in simpler terms… – Integrate Capacity across ITIL V2: Service Support and Service Delivery!
Capacity Management with ITIL 2011
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
• Leverage the data (and tools) you have!– Don’t reinvent or reimplement
• Quickly and easily with True Federation– Use existing data/tools already in place– Don’t force data duplication, ETL– Capacity Analysis across data sources
• Key ITIL discipline metrics amplify Capacity Management Value– Strategy factor financials– Design factor Service Levels, technology performance– Transition track business and technology changes– Operations factor Service risks, multiple technologies
Optimized Capacity Management
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
• ITIL: Strategy– Capacity Management integrated with Financial
costing/reporting• ITIL: Design
– Capacity Management integrated with Risk Registry• ITIL: Transition
– Includes integration with Asset and Configuration Management
• ITIL: Operations– Integration with Service Desk– Operations factor Service risks, multiple technologies
Integrated Case Study Walkthrough
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Very Large BankAs an IT Shop:• Operate tens of thousands of servers• Every server platform under the sun• Manage dozens of data centers• Huge mainframe with many thousands
of MIPS• Thousands of VMs• Thousands of VDIs & Citrix• Many Petabytes of storage
14
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Seamless data integration & analysis
1. All capacity/performance data2. All platforms, OS’s, …3. Configuration data4. Change records5. Risk registry
15
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Deliverable: Fully Automated Application Report
We need:1. Risk detection and tracking2. Risk reporting3. Actionable information
Reporting has to be:• Automated• Repeatable• Human-readable – financials, business terms, not “speeds and
feeds”
16
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Analysis Overview
17
Application and Configuration from Service Catalog and CMDB
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Usage Patterns
18
Time Series data from Performance and Event
Management
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Service Desk and Risk management
19
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Scal
eabl
y ID
Pos
sible
Ex
istin
g Ca
pacit
y Iss
ues
20
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
ID P
ossib
le Fu
ture
Ca
pacit
y Ri
sks
21
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Fixed Costs / Variable Costs - Method
22
Source: wikipedia.org
Variable Costs
Fixed Costs
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Capacity Management + Strategy (Financials)
Fixed/Variable Cost
23
server0009b01a - Excess Capacity ReportProduced by the Server Capacity & Performance Management (SCPM) Team
Analysis Period: August 01 2010 to August 31 2010Run Time: 4:09 PM September 27 2010 (8 seconds)
Purpose: To analyze the system's current resource consumption and compute the available headroom based on a fixed/variable costs methodology and our rules-of-thumb. This report also attempts to determine the nearest bottlenecks, from a consumption perspective.
server0009b01a: Maximum Growth Capability by ResourceName Growth Vaule
CPU RunQ Length Growth 2.15Disk - 0 4.41Memory Utilization Growth 5.48FS - C: 10.61Virtual Memory Growth 20.27CPU Growth 38.10Net In 100MB - NIC1 260.82Net Out 100MB - NIC1 349.39Net In 1GB - NIC1 2608.18Net Out 1GB - NIC1 3493.92
server0009b01a: Top 10 PIDsNAME PIDGROWTH SLOPE MINCPU AVGCPU MAXCPU
System:4 17.54 0.00 0.06 0.09 5.18NTRtScan:1660 29.88 -0.00 0.00 0.02 3.01beasvc:1080 47.72 -0.00 0.00 0.14 1.89svchost:840 184.41 -0.00 0.03 0.07 0.52svchost:872 213.22 0.00 0.05 0.08 0.47TmListen:2160 763.86 -0.00 0.00 0.01 0.12python:1788 848.86 0.00 0.00 0.03 0.11wmiprvse:268 987.95 -0.00 0.00 0.04 0.09wmiprvse:2228 1322.98 0.00 0.02 0.04 0.09wmiprvse:2044 1328.88 -0.00 0.02 0.05 0.09
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Capacity Management + Strategy (Financials)
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Capacity Management + Strategy (Financials)
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
VM Optimization Analysis• Thousands of VMs• Some too small• Some too big• Some idle• Which ones?• What size should they be?
26
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Physical to Virtual Analysis
27
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
28
Total Virtual Machines Idle Virtual Machines Oversized Virtual Machines22 3 13
vCPUs Max % CPU Max % CPU Ready
Avg % CPU
Avg % CPU
ReadyTotal
MemoryMax
Memory Used
Max % Memory
Util
Average Memory
Used
Avg % Memory
UtilRecommended SSO vCPUs
Recommended SSO
Memory in GB
CLUSTER0019V019 4 0 0 0 0 4096 0 0 0 0 1 2CLUSTER0019V024 2 0 0 0 0 2000 0 0 0 0 1 2CLUSTER0019V029-OLD_DO_NOT_USE 4 0 0 0 0 4096 0 0 0 0 1 2
vCPUs Max % CPU Max % CPU Ready
Avg % CPU
Avg % CPU
ReadyTotal
MemoryMax
Memory Used
Max % Memory
Util
Average Memory
Used
Avg % Memory
Util
Recommended SSO vCPUs
Recommended SSO Memory in
GBCLUSTER0019V001 2 40 5 9 1 2044 1921 94 1729 85 1 4CLUSTER0019V003 2 30 10 7 2 2048 1895 93 1737 85 1 4CLUSTER0019V004 2 45 51 3 3 2048 1914 93 1333 65 2 4CLUSTER0019V005 2 41 17 8 2 2048 1955 95 1716 84 2 4CLUSTER0019V006 2 45 41 2 2 2048 1963 96 1510 74 2 4CLUSTER0019V008 2 27 23 2 2 2048 1860 91 1232 60 1 4CLUSTER0019V013 2 30 40 3 3 2048 1845 90 1326 65 1 4CLUSTER0019V014 2 30 36 3 2 2048 1834 90 1286 63 1 4CLUSTER0019V018 2 32 30 3 2 2000 1843 92 1581 79 1 4
CLUSTER0019V029-REAL 4 42 30 5 4 4096 3612 88 2951 72 4 8
CLUSTER0019V030 2 47 23 2 2 2048 1881 92 1387 68 2 4CLUSTER0019v009 2 43 14 2 1 4096 3117 76 2258 55 2 4CLUSTER0019v010 2 30 18 2 1 4096 3102 76 2151 53 1 4
Capacity Optimization Candidates
Idle Virtual Machines
Oversized Virtual Machines
Copyright © 2012 TeamQuest Corporation. All Rights Reserved.
Delivered:• Repeatable processes• Quicker analysis• Powerful • Flexible
29