Date post: | 02-Jul-2015 |
Category: |
Technology |
Upload: | cliff-crocker |
View: | 562 times |
Download: | 0 times |
Who I Am
Cliff Crocker
VP Product
SOASTA, Inc.
@cliffcrocker
http://www.flickr.com/photos/luc/
Real Users
http://www.flickr.com/photos/spullara/
Happy Users
http://www.flickr.com/photos/sybrenstuvel/
Not SoHappy
RUM 101Real User Measurement (RUM) is a technology for collecting performance metrics directly from the browser of an end user.
• Involves instrumentation of the page with JavaScript
• Measurements are fired across the network to a collection point through a small request object (beacon)
<JS>MySite.com
<beacon>
Origins of RUM2005-6: Steve S. builds
JS tool to measure page load from the browser during a Yahoo! hackday
2008: Steve open sources ‘Episodes’. White pages open
sources ‘Jiffy’
2010: Yahoo! Open sources
‘boomerang.js’ the same day Navigation Timing is introduced
in IE9 beta
• http://stevesouders.com/episodes/• https://code.google.com/p/jiffy-web/• http://lognormal.github.com/boomerang/doc/• https://dvcs.w3.org/hg/webperf/raw-
file/tip/specs/NavigationTiming/Overview.html
This helped…
Navigation Timing API
• Exposes network level timing details to JavaScript
• DNS lookup, TCP connect, SSL handshake, First Byte time, and more
• For a given web request, you can tell exactly where the time was spent
• Supported by IE >= 9, Chrome >=6, Firefox >=7, Android Browser .=4.0, others
• No exposure into rendering times
C’mon Safari, be cool…
http://caniuse.com/nav-timing
~70% Browser Support
Why RUM is Awesome
• Reduces the level of convincing we have to do within our organization that ‘faster is better’
• The only way to measure perceived performance from your end user
• Easy to maintain
• Only getting better…– Check out:http://blog.patrickmeenan.com/2013/07/measuring-
performance-of-user-experience.html• User Timing for RUM and Synthetic
– Fun with resource timing: https://github.com/andydavies/waterfall
What gives??
No RUM
RUM • According to a study done in 2012* only 1 in 5 of the top ecommerce sites are using some form of RUM
*http://www.webperformancetoday.com/2012/11/13/real-user-monitoring-rum-ecommerce-sites/
Challenges with RUM
1. It’s a “data” problem
2. People don’t like change
3. Hard to “sell” internally
For a modest sized site, in a given day
1-3 Million Page Views
Perf Timing
User Agent
OS/Device
Geography
Page Segmentation
Bandwidth Classification
A/B Bucket
‘Other’
Many have advocated aggressive sampling of data…
~1% sample, looking at a period of about 3 hours for a given page segment (Checkout)
1% > checkout
Filtered to the United States…
1% > > checkout
Looking specifically at California…
1% > > > checkout
IE9 Browsers only.
1% > > > checkout
Looking at all traffic.
100% > checkout
All filters applied, identified bimodal distribution driven by backend time.
100% > > > checkout
> backend time
Sampled at 1%
Not sampled
To be fair…
Read: http://www.webperformancetoday.com/2013/03/06/big-
data-vs-big-enough-data/
Many use a batch solution…
• Batch processing large amounts of data has been a traditional approach for RUM
– Great for looking at historical data, trending, etc.
– Big Data is not necessarily Fast Data
• Not ‘Operational’
In order for RUM to be operationally relevant, you need the data in real-time
Synthetic vs. RUM
http://www.flickr.com/photos/myvector/
It is Not Either Or
• Synthetic and RUM coexist
• Important to understand:
– Geographic performance
– Object level detail (waterfall)
– Browser performance
– Competitive positioning
– Site coverage
Performance by Geography
RUM
????
Performance by Geography
Synthetic
Understanding Performance by Geography
• RUM provides better visibility into geographic variability (simply more data points from more locations)
RUM 1
Synthetic 0
Object Level Detail
RUM 1
Synthetic 1
Waterfall available for most major browsers available for synthetic monitoring (IE, FF (Commercial), Chrome, Safari)
Limited support for waterfall using resource timing (IE10 & Chrome stable) – using waterfall.js
Browser Coverage
RUM 2
Synthetic 1
SyntheticSupport for major browsers
Commercial (mainstream synthetic)
WebPagetestsupport
RUMSupport for any browser that executes JS
Competitive Positioning
RUM 2
Synthetic 2
Synthetic
Ability to measure competitors easily
RUM
More complex, no standard benchmarks (yet!)
Commercial offerings
WebPagetest.org
Site Coverage
RUM 3
Synthetic 2
Synthetic
Choose a ‘happy path’ or spot check
RUM
No blind spots!
http://scoutmastercg.com/wp-content/uploads/typimg/green_laser.png
http://www.flickr.com/photos/areese/with/4002609608/
Synthetic Monitoring
• Pros
– Technology is great
– Real Browsers (IE, Chrome, FF)
– Built in Alerting
– Waterfall analysis
– Screen shots & headers
– Object level detail
• Cons
– False Positives
– Impossible/impractical to measure everything
– Fixed number of browser/OS combinations
– Simulated bandwidth constraints at best
– Too few data points for statistical relevance
RUM
Cast a wide net• Identify key areas of concern• Understand real user impact• Tie this back to $$ (next)
Synthetic
Diagnostic tool• Identify issues in a ‘lab’• Reproduce a problem found with
RUM
http://www.flickr.com/photos/84338444@N00/with/3780079044/
http://www.flickr.com/photos/ezioman/
RUM in the Workplace
Studies show…
However, you have to look at your own data
Understand your KPIs and How to Track Them
• Retail: Order Conversion, Add to Cart Conversion, Revenue
• Media: In Line Display Ad Views, Comments,
• Other: Facebook likes/Social Media sharing
Bottom Line, if you want your business to care – then make it relevant for
them (speak their language)
Strategy for Success – RUM First
RUM First
• All the data (or statistically relevant sample)
• Real-time and historical
Supplement with
Synthetic
• Reduce measurements based on coverage with RUM
• Use for adhocdiagnostic & competitive benchmarking
Map to business
KPIs
• YOUR goals
• YOUR data
• Single source of truth
mPulse Litehttp://www.soasta.com/free/