B E S T O F B O T H W O R L D SS Y N T H E T I C A N D R U M

Mark Zeman @SpeedCurve

Cliff Crocker @cliffcrocker

A C T I V E M O N I T O R I N G

• Simulated health-checks of the system and it’s parts

• Scripted journeys through an application

• “Creation” of a singular or set of events and/or actions

• Synthetic is a form of active monitoring AP Photo/ Eric Risberg

PA S S I V E M O N I T O R I N G

• Listening to actual traffic as it moves through a system

• Observing behavior of the system and/or its operator

• “Measurement” of actions or events

• RUM is a form of passive monitoring

Our focus is on the experience of the crowd

allhumorpic.com

What is RUM?Real User Measurement (traditional definition)RUM 101Real User Measurement (RUM) is a technology for collecting performance metrics directly from the browser of an end user.

• Involves instrumentation of your website via JavaScript

• Measurements are fired across the network to a collection point through a small request object (beacon)

<JS> <beacon>

T H E G O O D PA R T S - R U M

• Always on

• Every user, every browser, every network, anywhere in the world

• Ability to capture human behavior/events with performance data

• Only getting better (waterfalls)

What is RUM?Real User Measurement (traditional definition)Synthetic 101Synthetic monitoring (for purposes of this discussion) refers to the use of automated agents (bots) to measure your website from different physical locations.

• A set ‘path’ or URL is defined• Tests are run either adhoc or scheduled and data is collected

T H E G O O D PA R T S - S Y N T H E T I C

• Rich data collected (waterfall, videos/filmstrip, HTTP Headers)

• Consistent “clean room” baseline

• Nothing to install

• Doesn’t require users/ability to measure pre-production, competition

C O M M O N T H I N G S W E H E A R A B O U T B O T H R U M A N D S Y N T H E T I C

• “Why are these numbers so different?”

• “I don’t trust your data. Your numbers are wrong.”

• “How are you calculating Page Load time?”

• “I can’t share two sets of numbers with the business.”

Real users are not normal

W H Y A R E T H E T W O N U M B E R S S O FA R O F F ?

Page Load TimesAll Countries, User Agents, Carriers, Operating Systems

3.23s – Median

Page Load TimesAll Countries, User Agents, Carriers, Operating Systems

10.45s – p95

Page Load TimesAll Countries, User Agents, Carriers, Operating Systems

17.26s – p98

Page Load TimesBy Operating System

OS Median 95th Percentile 98th Percentile

Windows 7 2.41s 9.29s 15.89s

Mac OS X/10 2.30s 8.11s 13.45s

iOS7 3.27s 10.64s 15.79s

Android 4 4.06s 14.30s 27.93s

iOS8 3.53s 11.54s 19.72s

Windows 8 2.67s 10.75s 18.74s

0 1 2 3 5 6 7 8 9 10 11 12 14 15 16 17 18 19

Median: 3.23s

R U MS Y N T H E T I C

F I R S TS Y N T H E T I C

R E P E AT

3 . 2 3 4 . 1 2 . 2

W H Y A R E T H E T W O N U M B E R S S O FA R O F F ?

“ B U T, I T L O A D S S O M U C H FA S T E R F O R M E ? ? ! ! ”

• 2015 Macbook Pro

• Warm browser cache

• FIOS

vs.

• x86 - Windows 7 VM

• Completely cold cache/dns

• Throttled bandwidth

4 seconds 20 seconds

R U M S Y N T H E T I C

• Primary source of truth

• User behavior/conversion

• Complete coverage

• Diagnostics

• Page Construction

• Pre-production & Competitive benchmarking

R U M S Y N T H E T I C

• Primary source of truth

• User behavior/conversion

• Complete coverage

• Diagnostics

• Page Construction

• Pre-production & Competitive benchmarking

You need both

S TA R T R E N D E R 1 . 5 8 S

D N S 1 1 0 M S

T C P 4 0 M S

T T F B 6 5 0 M S

D O M L O A D I N G 6 8 0 M S

D O M R E A D Y 2 . 7 5 S

PA G E L O A D 3 . 0 5

F U L LY L O A D E D 1 9 . 6 8 S

U S E R T I M I N G 1 . 9 5 S

R E S O U R C E T I M I N G 2 3 7 M S

R E Q U E S T S 1 1 6

B Y T E S I N 2 , 1 4 3 K B

S P E E D I N D E X 3 0 8 6

PA G E S P E E D S C O R E

8 31 S = $ 2 7 M

D O M E L E M E N T S

3 9 3 1

D O M S I Z E 3 , 1 4 3 K B

V I S U A L LY C O M P L E T E

1 5 . 2 S

R E D I R E C T 0 S

S S L N E G O T I AT I O N

9 3 M S

T H I S I S Y O U R “ O N E N U M B E R ”

T H E M A N Y FA C E S O F P E R F O R M A N C E

C-Level Operations Developer Designer

thenounproject.com

S TA R T R E N D E R D N S T C P T T F B

D O M L O A D I N G D O M R E A D Y PA G E L O A D F U L LY L O A D E D

U S E R T I M I N GR E S O U R C E

T I M I N GR E Q U E S T S B Y T E S I N

S P E E D I N D E XPA G E S P E E D

S C O R E1 S = $ $

D O M E L E M E N T S

D O M S I Z EV I S U A L LY

C O M P L E T ER E D I R E C T

S S L N E G O T I AT I O N

S TA R T R E N D E R D N S T C P T T F B

D O M L O A D I N G D O M R E A D Y PA G E L O A D F U L LY L O A D E D

U S E R T I M I N GR E S O U R C E

T I M I N GR E Q U E S T S B Y T E S I N

S P E E D I N D E XPA G E S P E E D

S C O R E1 S = $ $

D O M E L E M E N T S

D O M S I Z EV I S U A L LY

C O M P L E T ER E D I R E C T

S S L N E G O T I AT I O N

S TA R T R E N D E R D N S T C P T T F B

D O M L O A D I N G D O M R E A D Y PA G E L O A D F U L LY L O A D E D

U S E R T I M I N GR E S O U R C E

T I M I N GR E Q U E S T S B Y T E S I N

S P E E D I N D E XPA G E S P E E D

S C O R E1 S = $ $

D O M E L E M E N T S

D O M S I Z EV I S U A L LY

C O M P L E T ER E D I R E C T

S S L N E G O T I AT I O N

S TA R T R E N D E R D N S T C P T T F B

D O M L O A D I N G D O M R E A D Y PA G E L O A D F U L LY L O A D E D

U S E R T I M I N GR E S O U R C E

T I M I N GR E Q U E S T S B Y T E S I N

S P E E D I N D E XPA G E S P E E D

S C O R E1 S = $ $

D O M E L E M E N T S

D O M S I Z EV I S U A L LY

C O M P L E T ER E D I R E C T

S S L N E G O T I AT I O N

S TA R T R E N D E R 1 . 5 8 S

D N S 1 1 0 M S

T C P 4 0 M S

T T F B 6 5 0 M SD O M L O A D I N G

6 8 0 M SD O M R E A D Y

2 . 7 5 SPA G E L O A D

3 . 0 5F U L LY L O A D E D

1 9 . 6 8 SU S E R T I M I N G 1 . 9 5 S

R E S O U R C E T I M I N G 2 3 7 M S

R E Q U E S T S 1 1 6

B Y T E S I N 2 , 1 4 3 K BS P E E D I N D E X

3 0 8 6PA G E S P E E D S C O R E

8 31 S = $ 2 7 M D O M E L E M E N T S

3 9 3 1D O M S I Z E 3 , 1 4 3 K B

V I Z C O M P L E T E 1 5 . 2 S

R E D I R E C T 0 S

S S L N E G O T I AT I O N 9 3 M S

Analysis

S TA R T R E N D E R 1 . 5 8 S

D N S 1 1 0 M S

T C P 4 0 M S

T T F B 6 5 0 M SD O M L O A D I N G

6 8 0 M SD O M R E A D Y

2 . 7 5 SPA G E L O A D

3 . 0 5F U L LY L O A D E D

1 9 . 6 8 SU S E R T I M I N G 1 . 9 5 S

R E S O U R C E T I M I N G 2 3 7 M S

R E Q U E S T S 1 1 6

B Y T E S I N 2 , 1 4 3 K BS P E E D I N D E X

3 0 8 6PA G E S P E E D S C O R E

8 31 S = $ 2 7 M D O M E L E M E N T S

3 9 3 1D O M S I Z E 3 , 1 4 3 K B

V I Z C O M P L E T E 1 5 . 2 S

R E D I R E C T 0 S

S S L N E G O T I AT I O N 9 3 M S

Analysis

– C T O

“What is the right number for me to communicate to the organization?”

R E A L I S T I C A L LY, T H E R E I S N O T O N E N U M B E R

1s=$27M Page Load 3.05s

Start Render 1.58s

Speed Index 3086

– C E O

“How is site performance impacting revenue?”

N O T A L L PA G E S A R E C R E AT E D E Q U A LT O P O F F U N N E L I M PA C T ( B R O W S E PA G E S )

For a typical eCommerce site, conversion rate

drops by up to 50% when “browse”

pages increase from 1 to 6 seconds

N O T A L L PA G E S A R E C R E AT E D E Q U A LB O T T O M O F F U N N E L I M PA C T ( C H E C K O U T PA G E S )

However, there is much less impact

to conversion when “checkout” pages degrade

H O W FA S T S H O U L D I B E ?

1s = $27M

– C E O

“How do we compare to the competition?”

– C E O

“How is our site doing in Spain? Are we winning there?”

S I T E P E R F O R M A N C E - S PA I N

• Represents 1% of our overall traffic

• 3rd overall in our competitive index

3.91s Competitive Index

Site 1 Site 2 Site 3 Site 4 Site 5

– V P E C O M M E R C E

“How are third parties impacting my site?”

Dom

ains

Alexa Top 25 Retail

31

10

35

23

86

41

20

37

303131

4

48

3842

16

11

6

28

68

31

36

4

198

2125133511225324323517

37

17

Primary domains 3rd Party domains

3 R D PA R T Y S T U D Y - U N I Q U E D O M A I N S P E R PA G E

Domains: Median - 32 Min - 25 Max - 87

91%

9%

91% of domains are 3rd parties

3 R D PA R T Y S T U D Y

H U F F I N G T O N P O S T - T H I R D PA R T Y I M PA C T U S I N G P E T T Y C A S H

S E T T I N G M E A N I N G F U L S L A S

Example SLA: Response time measured using resource timing from Chrome browsers in the United States should not exceed a median (50th percentile) of 100ms or a 95th percentile of 500ms for a population of more than 500 users in a 24 hour period

– F R O N T E N D D E V E L O P E R

“How should I measure my site in pre-production?”

– V P E C O M M E R C E

“Should I optimize my site for mobile?”

Platform Median 95th Percentile 98th Percentile

Mobile 3.62s 12.53s 20.04s

Desktop 2.44s 9.31s 15.86s

Page Load TimesDesktop vs. Mobile (Mobile and Tablet Optimized)

0 1.5 3 4.5 6 7.5 9 10.5 12 13.5 15 16.5 18 19.5

Platform Median 95th Percentile 98th Percentile

Mobile 4.70s 17.87s 34.43s

Desktop 3.81s 25.15 55.34s

Page Load TimesDesktop vs. Mobile (Full site on Tablet)

1024 1366320 1920

IMG

JSCSS

35303530

768 1280

1770

637

320 1920

IMG

JS

– O P E R AT I O N S

“How can I determine the effectiveness of my CDN provider?”

W H I C H S I T E I S U S I N G A C D N ?

Site A Site B

– F R O N T E N D D E V E L O P E R

“How has page construction changed over the last month, and what impact has that had on performance?”

– F R O N T E N D D E V E L O P E R

“What is the performance of my single page web application?”

T H E C H A L L E N G E O F S I N G L E - PA G E A P P L I C AT I O N S

• Challenges with synthetic: Only measure first page load, not feasible to measure secondary loads via XHR

• Challenges with RUM: Determining whether call is a “page” or service call;Risk of diluting overall numbers

– D E S I G N E R

“How do I know how fast my page becomes usable?”

Start Render Loaded

Usable

• Other questions?

S U M M A R Y

• You need rich sets of data

• There is not one number

• Choose the right number(s) for the right audience