1© 2012 SOASTA. All rights reserved.
RUM Roundtable
Webinar
Presents
Real User Measurement Past, Present & Future
2© 2012 SOASTA. All rights reserved. October 30, 2012
Real user measurement – then and now…and….In This Webinar
TODAY’S PRESENTERS
Aaron Kulick: – Speed Freak @WalmartLabs - @GoFastWebBuddy Brewer: VP Engineering, SOASTA - @bbrewerPhillip Tellis: Chief Architect, SOASTA - @bluesmoonCliff Crocker: VP Product Management, SOASTA mPulse -
@cliffcrocker
Agenda: • Roundtable discussion• Moderated QA with Panelists
Questions: Submit in the question box during event
• VP Product SOASTA
• Real User Measurement (mPulse)
• @cliffcrocker
Cliff Crocker
• VP Engineering SOASTA
• Co-founder of LogNormal
• @bbrewer
Buddy Brewer
• Chief Architect SOASTA
• Co-founder of LogNormal
• @bluesmoon
Philip Tellis
• Guerilla Engineer @WalmartLabs
• Founder SF Web Perf Meetup and Web Perf Days
• @GoFastWeb
Aaron Kulick
4© 2012 SOASTA. All rights reserved. October 30, 2012
2001 – 2005 (Five Years)This monitoring stuff is easy
Internet Explorer 6
5© 2012 SOASTA. All rights reserved. October 30, 2012
2006 – 2009Browser diversity, the mobile web catches on
Internet Explorer 7, 8
Firefox 2, 3
iPhone 1st Gen, 3G
Chrome 1, 2, 3
6© 2012 SOASTA. All rights reserved. October 30, 2012
2010 – 2012This isn’t going to scale
Internet Explorer 9
Chrome 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,18, 19
Firefox 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
Android Cupcake, Donut, Éclair, Froyo, Gingerbread, Honeycomb, Jelly Bean
iPhone 3GS, 4, 4S, 5
iPad 1, 2, 3, 4, Mini
7© 2012 SOASTA. All rights reserved. October 30, 2012
• When we speak about performance, we're really talking about latency
• Humans have measured latency for as long as we've known large distances
Measuring Performance
8© 2012 SOASTA. All rights reserved. October 30, 2012
• “How long does it take to get a package from NY to SF?”
• “How fast can a man run 100 metres?”
• “What's the fastest laptime at the Indy 500?”
Measuring Performance
9© 2012 SOASTA. All rights reserved. October 30, 2012
In general, fast things excite us and slow things frustrate us.
Measuring Performance
10© 2012 SOASTA. All rights reserved. October 30, 2012
So what should we measure, and how should we measure it?
Measuring Performance
11© 2012 SOASTA. All rights reserved. October 30, 2012
• The traditional way is to use robots
• This is good to tell us about infrastructure - but what we really care about is user experience
Measuring Performance
12© 2012 SOASTA. All rights reserved. October 30, 2012
That's akin to asking a Robot to drive Route 66 and tell you if it wasexciting or boring
… and you have to do this with 374 different cars!
Measuring Performance
13© 2012 SOASTA. All rights reserved. October 30, 2012
So let's bring out the RUM!
Measuring Performance
14© 2012 SOASTA. All rights reserved. October 30, 2012
• Measure the real experience of a real user
• How fast was their experience?• Were they engaged, did they take out their credit cards, do
they stick around?
Real User Measurement
15© 2012 SOASTA. All rights reserved. October 30, 2012
A Brief History of RUM …
Real User Measurement
16© 2012 SOASTA. All rights reserved. October 30, 2012
At a Yahoo! internal hack day, Steve Souders built aJavaScript tool to measure page load time within the browser.
RUM Timeline 2005 – 2006
17© 2012 SOASTA. All rights reserved. October 30, 2012
RUM Timeline 2006 – 2010
Several Improvements were made: • Bandwidth measurements added• Creation of Yslow• 100s of billions of datapoints were analysed (a tiny sample of
Yahoo!’s traffic at the time)
18© 2012 SOASTA. All rights reserved. October 30, 2012
RUM Timeline 2008
Steve Souders open sources Episodes• http://stevesouders.com/episodes/paper.php
WhitePages.com published jiffy• http://code.google.com/p/jiffy-web/
19© 2012 SOASTA. All rights reserved. October 30, 2012
RUM Timeline 2010
Yahoo! open sourced the JavaScript library, calledboomerang, under the BSD license• http://lognormal.github.com/boomerang/doc/
At the same time (on the same day actually), Internet Explorer 9beta announced support for the Navigation Timing API.• http://www.w3.org/TR/navigation-timing/
20© 2012 SOASTA. All rights reserved. October 30, 2012
Navigation Timing API
Let's talk a bit about NavTiming
• Exposes network level timing details to JavaScript
• DNS lookup, TCP connect, SSL handshake, First Byte time, and more
• For a given web request, you can tell exactly where the time was spent
• Supported by IE >= 9, Chrome >=6, Firefox >=7
• No exposure into rendering times
22© 2012 SOASTA. All rights reserved. October 30, 2012
Navigation Timing API
The level of detail available in NavTiming made RUM performance data comparable to what you'd get from a bot.
boomerang integrates well with it, and works around browser bugs when needed
23© 2012 SOASTA. All rights reserved. October 30, 2012
RUM Timeline 2011
Buddy and I started LogNormal to pull in data from boomerang,analyze it, and present it back to site owners in a useful way.
24© 2012 SOASTA. All rights reserved. October 30, 2012
RUM Timeline 2012
In 2012, the area has exploded with many new companies getting in tothe playing field
LogNormal is now part of SOASTA and the product is called mPulse
25© 2012 SOASTA. All rights reserved. October 30, 2012
RUM Timeline 2012
At SOASTA, we'd like mPulse to look deeper into the human aspect ofweb and mobile performance
26© 2012 SOASTA. All rights reserved. October 30, 2012
RUM at Walmart
Operational RUM
Real User Measurement and the ‘Tribe’
http://sfwebperf.org@SFWebPerf
http://webperfdays.org@WebPerfDays
29© 2012 SOASTA. All rights reserved. October 30, 2012
Panelist Question
Thinking beyond the current horizon, where do you see real user measurement over the next 10 years?
Listen to Your UsersSource: Flickr – jamescridland
Q&A
31© 2012 SOASTA. All rights reserved. October 30, 2012
Thank You!
Contact SOASTA:[email protected] us:
twitter.com/cloudtest
facebook.com/cloudtest
Next Webinar: Nov. 15, 2010 - 10 a.m. PST“Automated Testing & Continuous Integration in the
Cloud”* Kohsuke Kawaguchi – Jenkins CI creator & SOASTA TouchTest mobile CI
demo *
Register at www.soasta.com/knowledge-center/webinars
Contact Us: [email protected]