A Modern Approach to Performance Monitoring

Cliff Crocker@cliffcrockerSOASTAVP Product, mPulse

1. How fast am I?

2. How fast should I be?

3. How do I get there?

Synthetic vs. RUM

What is RUM?Real User Measurement (traditional definition)Synthetic 101Synthetic monitoring (for purposes of this discussion) refers to the use of automated agents (bots) to measure your website from different physical locations.

• A set ‘path’ or URL is defined• Tests are run either adhoc or scheduled and data is collected

What is RUM?Real User Measurement (traditional definition)RUM 101Real User Measurement (RUM) is a technology for collecting performance metrics directly from the browser of an end user.

• Involves instrumentation of your website via JavaScript

• Measurements are fired across the network to a collection point through a small request object (beacon)

<JS>

<beacon>

RUM

Cast a wide net• Identify key areas of concern• Understand real user impact• Map performance to human behavior

& $$

Synthetic

Diagnostic tool• Identify issues in a ‘lab’/remove

variables• Reproduce a problem found with RUM http://www.flickr.com/photos/84338444@N00/with/3780079044/

http://www.flickr.com/photos/ezioman/

The Early Days of RUM

• Round-trip time• Start/stop timers via JavaScript• Early contributors:• Steve Souders/Episodes• Philip Tellis/Boomerang.js• Both widely in use today

Navigation TimingWhat gives RUM a kick

Browser Support for Navigation Timing

!

11© 2014 SOASTA CONFIDENTIAL - All rights reserved.

Navigation TimingExamples

DNS: Domain Lookup Time

function getPerfStats() { var timing = window.performance.timing; return { dns: timing.domainLookupEnd - timing.domainLookupStart};}

12© 2014 SOASTA CONFIDENTIAL - All rights reserved.

Navigation TimingExamples

TCP: Connection Time to Server

function getPerfStats() { var timing = window.performance.timing; return { connect: timing.connectEnd - timing.connectStart};}

13© 2014 SOASTA CONFIDENTIAL - All rights reserved.

Navigation TimingExamples

TTFB: Time to First Byte

function getPerfStats() { var timing = window.performance.timing; return { ttfb: timing.responseStart - timing.connectEnd};}

14© 2014 SOASTA CONFIDENTIAL - All rights reserved.

Navigation TimingExamples

Base Page

function getPerfStats() { var timing = window.performance.timing; return { basePage: timing.responseEnd - timing.responseStart};}

15© 2014 SOASTA CONFIDENTIAL - All rights reserved.

Navigation TimingExamples

Front-end Time

function getPerfStats() { var timing = window.performance.timing; return { frontEnd: timing.loadEventStart - timing.responseEnd};}

16© 2014 SOASTA CONFIDENTIAL - All rights reserved.

Navigation TimingExamples

Page Load Time

function getPerfStats() { var timing = window.performance.timing; return { load: timing.loadEventStart - timing.navigationStart};}

Measuring AssetsYour page is composed of many different parts

• Strength of synthetic• Full visibility into asset performance• Images• JavaScript• CSS• HTML

• A lot of which is served by third-parties• CDN!

Object Level RUMGoing beyond page level metrics

Browser Support for Resource Timing

CORS: Cross-Origin Resource Sharing

Timing-Allow-Origin = "Timing-Allow-Origin" ":" origin-list-or-null | "*"

Start/End time only unless Timing-Allow-Origin HTTP Response Header defined

Resource Timing

var rUrl = ‘http://www.akamai.com/images/img/cliff-crocker-dualtone-150x150.png’;var me = performance.getEntriesByName(url)[0];var timings = { loadTime: me.duration, dns: me.domainLookupEnd - me.domainLookupStart, tcp: me.connectEnd - me.connectStart, waiting: me.responseStart - me.requestStart, fetch: me.responseEnd - me.responseStart}

Measuring a single resource

Resource TimingOther uses

• Slowest resources• Time to first image• Response time by domain• Time a group of assets• Response time by initiator type (element type)• Cache-hit ratio for resources

For examples see: http://www.slideshare.net/bluesmoon/beyond-page-level-metrics

Resource TimingCool projects

• PerfMap - https://github.com/zeman/perfmap• Mark Zeman

• Waterfall.js - https://github.com/andydavies/waterfall• Andy Davies

1. How fast am I?

2. How fast should I be?

3. How do I get there?

Picking a NumberHow do you set front-end performance goals?

• Industry benchmarks?

• Apdex?• Analyst report?• Case studies?

“Synthetic monitoring shows you how you relate to your competitors, RUM shows you how you relate to your customers.”

– Buddy Brewer

BenchmarkingHow you relate to the competition

BenchmarkingSOASTA mPulseUX

• http://soasta.com/mpulseUX

BenchmarkingImportant to remember that it’s not always about Page Load

• Page construction• Requests• Images• Size

• Other important metrics• Speedindex• Start Render• PageSpeed/Yslow scoring

Performance is a Business Problem

Yahoo! - 2008Slower pages cause increase in abandonment

Increase of 400ms causes 5-9% increase in user abandonment

http://www.slideshare.net/stubbornella/designing-fast-websites-presentation

Shopzilla - 2009Faster pages increase site conversion rate

A reduction in Page Load time of 5s increased site conversion 7-12%!

http://assets.en.oreilly.com/1/event/29/Shopzilla%27s%20Site%20Redo%20-%20You%20Get%20What%20You%20Measure%20Presentation.ppt

Walmart - 2012Impact of slow pages on conversion rate

http://minus.com/msM8y8nyh#1e

SF WebPerf – 2012Up to 2% conversion drop for every additional second

So What?

SIMULATION

How Fast Should You Be?Key takeaways

• Use synthetic measurement for benchmarking your competitors

• Understand how fast your site needs to be to reach business goals/objectives with RUM

• You must look at your own data

1. How fast am I?

2. How fast should I be?

3. How do I get there?

Real users are not normal

Page Load TimesAll Countries, User Agents, Carriers, Operating Systems

2.76s – Median

10.45s – p95

17.26s – p98

Platform Median 95th Percentile

98th Percentile

Mobile 3.62s 12.53s 20.04s

Desktop 2.44s 9.31s 15.86s

Page Load TimesBy Platform

Page Load TimesBy Operating System

OS Median 95th Percentile

98th Percentile

Windows 7 2.41s 9.29s 15.89s

Mac OS X/10 2.30s 8.11s 13.45s

iOS7 3.27s 10.64s 15.79s

Android 4 4.06s 14.30s 27.93s

iOS8 3.53s 11.54s 19.72s

Windows 8 2.67s 10.75s 18,74s

Other FactorsAll Countries, User Agents, Carriers, Operating Systems

• Geography• User Agent• Connection Type• Carrier/ISP• Device Type

Not All Pages are Created EqualTop of funnel impact (Browse Pages)

For a typical eCommerce site, conversion rate drops by up to

50% when “browse” pages increase from 1

to 6 seconds

Not All Pages are Created EqualBottom of funnel impact (Checkout pages)

However, there is much less

impact to conversion

when “checkout”

pages degrade

How Do I Get There?Key takeaways

• Focus on the highest value opportunities/demographics

• Identify key pages that have the most impact on your KPIs

• Prioritize based on reducing friction within the funnel or critical path

Thank You!

AttributionsImages

1. Modern (Title Slide): https://www.flickr.com/photos/looking4poetry/3477854720/in/photolist-6ijV3q-9t3sjb-cj8VWE-MNjPA-4yBqug-2rk5he-4nZVzJ-4yE45J-9t3Fvj-2rptr5-69ymdr-6XVTH-6QZ4dg-9t3HN5-8geUkB-6oEXs5-eH9zS-H2XTt-cKdJvb-7GNKWx-cKdKZd-5Rw3qt-jfT5Dx-Jfiuy-bTop6R-2m8kAB-Jfiuu-7E8eMf-9jqD6-9hYAvd-Jfius-gsqr7U-7KiNAZ-8kzG9V-euXdb-44qJNN-47jBuY-nsxZrZ-7cGjQL-4cfHKq-cGzbSC-aQtWPR-8y3sR3-6okjAW-5A7BSC-6aRsvh-eSMHkX-kxvdyK-9t3udu-c8EkYu

2. Man vs. Machine: https://www.flickr.com/photos/eogez/3289851965/in/photolist-61HmnB-9nkQu3-dfinF1-4GjK24-4GjL6Z-4GoV6C-dibVCU-4GoTXo-4GjF3k-4GjH7Z-4GoR6h-ek6eT9-ek6eUU-4fADUx-4fEDV9-65JA6W-65Jsxu-65JDnj-4fAE12-65JDTh-65JBZu-65Ek9M-65JtDu-65JuiN-65JzFS-65EfiH-65Jv63-65JxX7-65Enyc-65Eh4c-65Jxjm-65JwmU-65EhUP-65JCML-65EjxP-65JBQf-65Ee4Z-65JD4b-65JBF1-65Ec1r-65EkPB-65Jy4o-65Eg96-65JAjb-65Emfe-65Efoz-65JBam-65Eniv-65JCrU-65Eed2

3. https://www.flickr.com/photos/bradybd/2818154005/

4. https://www.flickr.com/photos/perspective/149321089/

5. Dartboard: http://en.wikipedia.org/wiki/Darts#mediaviewer/File:Darts_in_a_dartboard.jpg

References:

6. http://calendar.perfplanet.com/2011/a-practical-guide-to-the-navigation-timing-api/

7. http://www.slideshare.net/bluesmoon/beyond-page-level-metrics