Measuring User Experience in the Cloud: Which Tactics are Worth the Time?
Julie Craig
Research Director, ApplicationsEnterprise Management Associates
5777 Central Avenue, Suite 105
Boulder, CO 80301
www.enterprisemanagement.com
Agenda
• Cloud Snapshot• Cloud Characteristics Create New Management Challenges• End User Experience Monitoring (EUE) and the Cloud• Summary and Closing
Slide 2
Which needs are effectively being addressed by cloud computing solutions for your organization?
Expand revenue channels by reselling Cloud-based services
Business model enhancements
Free up resources for strategic projects instead of routine op-erations/firefighting
Improve service resilience/ more consistent service per-formance
Enable disaster recovery/business continuity
Reduce complexity of IT management
Faster time to create new services
Increase infrastructure flexibility and agility (data center consol-idation, etc.)
Reduce the operational costs of IT management (e.g. staff, power, rent, maintenance, etc.)
Faster time to deploy existing services
Reduce the capital costs of IT management (e.g. hardware, facil-ities, licenses, etc.)
0% 10% 20% 30% 40% 50% 60%
25%
30%
39%
43%
45%
47%
48%
51%
53%
54%
55%
CapEx Reductions, Fast Deployment, OpEx Reductions are Top Three Reasons for Cloud Adoption
3
What cloud computing service model(s) has your organization deployed, or is planning to deploy?
Platform as a Service (PaaS) – an environment for application development and deployment is delivered by the (internal/ex-
ternal) cloud provider
Infrastructure as a Service (IaaS) – only basic system or machine-level resources are delivered by the (internal/ex-
ternal) cloud provider
Software as a Service (SaaS) – a fully-functional pre-built ap-plication environment is delivered by the (internal/external)
cloud provider
0% 20% 40% 60% 80% 100%
39%
39%
77%
SaaS Leads Public Cloud Deployments
4
31%
35%
75%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Public cloud – openlyavailable to any
organization or thegeneral public
Community cloud –shared by a few
selectedorganizations withcommon interests
Private cloud –operated solely for
your ownorganization
What cloud computing deployment model(s) has your organization deployed, or is planning to deploy?
5
Private Cloud Leads Public Cloud
Agenda
• Cloud Snapshot• Cloud Characteristics Create New Management Challenges• End User Experience Monitoring (EUE) and the Cloud• Summary and Closing
Slide 6
Which does your organization measure in order to quantify the success of cloud-related initiatives?
We don’t quantify the success of cloud-related services
Quantity of new application services
User productivity-specific metrics
Revenue-generation through cloud enabled services
Responsiveness of new application services to user requests
Business-productivity metrics (other than revenue)
Cost savings opex
Cost savings capex
Improved service quality (performance, availability)
0% 20% 40% 60% 80%
10%
29%
31%
33%
36%
39%
45%
50%
57%
Measuring Success
7
Rank Type of Tool
1 Infrastructure performance & availability monitoring
2 End-to-end application availability
3 End-to-end application response time
4 Service Desk/Help Desk Incident/problem management
5 Business impact management
6 Service Level Agreement (SLA) Reporting
7 Transaction aware User Experience Management/ Monitoring
8
Ranked Importance of Current Monitoring/ Management Tools
Please rate the importance of the following management technologies.
Which technologies does your org currently have deployed/full commitment to deploy in your environment?
BSM or service management portal
Application dependency mapping
Usage-based accounting and/or chargeback
End User Experience (EUE)
Service catalog
Run-book or IT Process Automation
CMDB/CMS
Integrated service dashboard
Business Intelligence or BAM
Unified Service Desk
Data warehouse IT analytics
Service level management
0% 10% 20% 30% 40% 50% 60%
23%
23%
25%
28%
29%
30%
33%
35%
41%
45%
46%
52%
End User Experience Management Under-represented
9
Agenda
• Cloud Snapshot• Cloud Characteristics Create New Management Challenges• End User Experience Monitoring (EUE) and the Cloud• Summary and Closing
Slide 10
“End user Experience”:Catch All Term for Multiple Types of Products
• Network tracing or capture• Deep visibility into network flows and packets
• Active or synthetic transaction monitoring• Script-based tests that run against an application at regular intervals.
• Passive or observed transaction monitoring• “Hard” or “soft” appliances scan protocol-based traffic such as web interactions
• End to end transaction reconstruction• Consolidate metrics from across the data center to deliver a reconciled end to
end view of transaction execution
• Client-side EUE• Monitor performance and availability from the perspective of the user workstation
Slide 11
Agentless, Client-Side EUE
• Deep endpoint monitoring for both “internal” and “external” users • Instrumentation at the endpoint via the browser
• “Watch” activity entering, exiting, and on the workstation or device• Diagnose application-related issues, identify delay sources
• Instrumentation for the Cloud• Monitoring can be embedded in VMs• “Cloud Edition” can be installed in Amazon EC2 instances• Central repository and collector monitors EC2 environments
• Instrumentation for Mobile
Slide 12
Correlsense CEO, Oren Elias
Slide 13 End-User-Monitoring.com
The Big Question: How do we Manage Today’s Ever Changing Environments?
Slide 14
Public
Private SOA
Virtualization
Agile
Lots of Moving Parts
End-User-Monitoring.com
With all of the Complexity and Moving Parts…
Slide 15
What Does Not Change?
Public
PrivateVirtualization SOAAgile
End-User-Monitoring.com
The Two Axioms
Slide 16
1. Transactions Drive the Business
2. User’s Expectation of Performance
End-User-Monitoring.com
Public, Private, Hybrid Datacenter
WEB
MQ/ESB
DCOM
CORBA
SERVER
APP SVR
ElasticLB
LDAP
WebServices
‘Under The Hood’
End-User-Monitoring.com
Cloud ≠ ‘Cure All’
Slide 18 End-User-Monitoring.com
Four New Challenges in the Cloud
“Old world” monitoring paradigms no longer fit
Slide 19 End-User-Monitoring.com
Challenge #1 – Dynamic Environments
Highly dynamic environment• Out: Auto-discovery• In: Auto-detection
Slide 20 End-User-Monitoring.com
Challenge #2 - Highly Virtualized Infrastructure
Infrastructure is highly virtualized• Out: Focus solely on Infrastructure and
resource monitoring• In: Focus on end-user and applications
Slide 21 End-User-Monitoring.com
More Demanding Applications
Customer facing applications are more demanding• Out: Focus solely on Availability• In: Focus on performance and service levels
Slide 22 End-User-Monitoring.com
Multi-Tenant Environment
Shared environment• Out: Hardware appliances• In: Software-based solutions
Slide 23 End-User-Monitoring.com
Top 3 Tips For Cloud Application Management
You Can’t Truly Understand Your Web Performance Without Monitoring the Experience of your Real Users
Here’s What You Need to Do:
1) Be Paranoid
• Watch Everything
2) Be Smart
• Watch Before and After
3) Be Lazy
• Use Automation to Make your Life Easier
Datacenter
WEB
MQ/ESB
DCOM
CORBA
SERVER
APP SVR
PROXY
LDAP
WebServices
1) Be ParanoidWatch Every Transaction from Every User
End-User-Monitoring.com
Total End-User Response TimeBrowser Rendering Time
Get the Key User-Experience Metrics
Ren
deri
ng
Real User Monitoring
Datacenter TimeNetwork Latency
Look Across Space
Houston, We Have A Problem
2) Be Smart.From Transactions to Performance Models
End-User Response
Time Breakdown
Transactions Modeled with Transaction
Profile
Compare Between Application Models
Slide 30
Before vs. After
Within SLA vs. Exceeded SLA
End-User-Monitoring.com
Insurance in Your Changing WorldAgile Roll-Outs, Virtual Migrations, Private Cloud Migrations
• Compare “new” and “old” production behavior• Prove response times are better or improved• Realize causes of degradations to avoid rollback
End-User-Monitoring.com
3) AutomateEnforce SLAs on End-User Experience
Application End-User SLAS
Monitor Service Levels
Detect Degradation
Automated Action
Auto-Scale Out
Measure Impact
Alert and Automate
Slide 34 End-User-Monitoring.com
Get Started with SharePath RUM Express for Free
http://www.real-user-monitoring.com
End-User-Monitoring.com
Agenda
• Cloud Snapshot• Cloud Characteristics Create New Management Challenges• End User Experience Monitoring (EUE) and the Cloud• Summary and Closing
Slide 36
Summary
• Cloud computing is a “hot topic” with a variety of potential benefits
• Cloud also brings its own set of challenges • How do you measure success of Cloud adoption?• How do you monitor performance and availability?• How do you get visibility to End User performance?• Which types of EUE tools are most useful for Cloud monitoring?
• Each type of product has its strengths
• Stretch your tools budget by making sure to match tools acquisitions with desired monitoring metrics
Slide 37
For more information on Enterprise Management Associates, please go to www.enterprisemanagement.com, or call +1 303-543-9500
Thank you for attendingtoday’s presentation