Paruj RatanaworabhanKasetsart University, Thailand
Ben Livshits and Ben ZornMicrosoft Research, Redmond
JSMeter: Characterizing the Behavior of JavaScript Web Applications
1
in collaboration withDavid Simmons, Corneliu Barsan, and Allen Wirfs-Brock
Background• Who is this guy?
– Current: Principal Researcher, Microsoft – Past: Professor, U of Colorado; PhD, UC Berkeley
• Research in Software Engineering (RiSE)– 30 researchers and developers– Notables
• Code Contracts (Visual Studio 2010)• Static Driver Verifier in Windows• Peli’s Video Blog at MSDN Channel 9
2
Why Measure JavaScript?
• Standardized, de facto language for the web
– Support in every browser, much existing code
• Browser and JavaScript performance is important
– Are current JavaScript benchmarks representative?
– Limited understanding of JavaScript behavior in real sites
• Who cares?
– Users, web application developers, JavaScript engine developers
3
4
ZDNet 29 May 2008
ghacks.net Dec. 2008
BrowserWars!
Artificial Benchmarks versus Real World Sites
5
Maps Maps
7 V8 programs:
• richards• deltablue• crypto• raytrace• earley-
boyer• regexp• splay
8 SunSpider programs:• 3-draytrace• access-nbody• bitops-nsieve• controlflow• crypto-aes• date-xparb• math-cordic• string-
tagcloud
JSMeter 11 real sites:
Goals of JSMeter Project• Instrument JavaScript execution and measure behavior• Compare behavior of JavaScript benchmarks against real sites• Consider how benchmarks can mislead design decisions
6
How We Measured JavaScript
\ie\jscript\*.cpp
Source-level instrumentation
custom jscript.dll
custom trace fileswebsite visits
Offline analyzers
custom trace files0%
10%20%30%40%50%60%70%80%90%
100%
Constant
Other Str Ops
Concat Op
Visiting the Real Sites
• Getting past page load performance• Attempted to use each site in “normal” way:
7
amazon Search a book, add to shopping cart, sign in, and sign outbing Type in a search query and also look for images and newsbingmap Search for a direction from one city to anothercnn Read front page newsebay Search for a notebook, bid, sing in, and sign outeconomist Read front page news, view comments
facebookLog in, visit a friend pages, browse through photos and comments
gmail Sign in, check inbox, delete a mail, and sign outgoogle Type in a search query and also look for images and newsgooglemap Search for a direction from one city to anotherhotmail Sign in, check inbox, delete a mail, and sign out
Understanding JavaScript Behavior
Code
Events
JavaScript
Objects
8
Code Behavior
Code
Events
JavaScript
Objects
9
• Function size
• Instructions/call
• Code locality
• Instruction mix
Total Bytes of JavaScript Source
10
amazo
n
bingmapeb
ay
faceb
ook
hotmail
richard
scry
ptoea
rley
splay
3d-raytr
ace
bitops-n
sieve
crypto-
aes
math-co
rdic
string-ta
gcloud
0
500
1000
1500
2000
2500
Sour
ce s
ize
(kilo
byte
s)
Real Sites V8 SunSpider
Code|Objects|Events
Static Unique Functions Executed
11
amazo
n
bingm
ap ebay
faceb
ook
goog
le
hotm
ail
richa
rdscry
ptoea
rley
splay
3d-ra
ytrace
bitop
s-nsie
ve
crypto
-aes
math-co
rdic
string
-tagcl
oud
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
Uni
que
Func
tion
s Ex
ecut
ed
Real Sites V8 SunSpider
Code|Objects|Events
Bytecodes / Call
12am
azon
bingmapeb
ay
faceb
ook
hotmail
richard
scry
ptoea
rley
splay
3d-raytr
ace
bitops-n
sieve
crypto-
aes
math-co
rdic
string-ta
gcloud
0.00
50.00
100.00
150.00
200.00
250.00
300.00
Byte
code
s / C
all
function(a,b) { var i=0,elem,pos=a.length; if(D.browser.msie) { while(elem=b[i++]) if(elem.nodeType!=8) a[pos++]=elem; } else while(elem=b[i++]) a[pos++]=elem; return a}
Real Sites V8 SunSpider
Code|Objects|Events
Fraction of Code Executed
13
amazo
n
bingmapeb
ay
faceb
ookgoo
gle
hotmail
richard
scry
ptoea
rley
splay
3d-raytr
ace
bitops-n
sieve
crypto-
aes
math-co
rdic
string-ta
gcloud
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%Fr
acti
on o
f Co
de E
xecu
ted Most code
not executed
Real Sites V8 SunSpider
Code|Objects|Events
Hot Function Distribution
14
Real Sites V8 Benchmarks
80% of time in < 10 functions80% of time in 100+ functions
80% 80%
Code|Objects|Events
Object Allocation Behavior
Code
Events
JavaScript
Objects
15
• Allocation by types
• Live heap composition
• Lifetime distribution
Total Bytes Allocated
16
amazo
n
bingmapeb
ay
faceb
ook
hotmail
richard
scry
ptoea
rley
splay
3d-raytr
ace
bitops-n
sieve
crypto-
aes
math-co
rdic
string-ta
gcloud
0
5000
10000
15000
20000
25000
30000
35000
40000
45000To
tal h
eap
data
(ki
loby
tes)
Real Sites V8 SunSpider
Code|Objects|Events
Heap Data by Type
17
amazo
n
bingmap
ebay
hotmail
richards
crypto
earle
ysplay
access-
nbody
contro
lflow
date-xp
arb
regexp
-dna
aggregate
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
ObjectsEnumaratorRegExpMathErrorDateNative FuncStringArraysScript FuncHe
ap D
ata
by T
ype
Real Sites V8 SunSpider
Many functionsRest are strings
Few benchmarksallocate much data
Code|Objects|Events
Live Heap Over Time (gmail)
18
Functions growsteadily
Code|Objects|Events
GC reducessize of heap
Objects growsteadily too
Live Heap over Time (ebay)
19
Heaps repeatedly created, discarded
Heap containsmostly functions
Code|Objects|Events
Heap drops to 0 on page load
2 Search Websites, 2 Architectures
Bing Google
Code|Objects|Events
20
You stay on the same page during your entire visitCode loaded onceHeap is bigger
Every transition loads a new pageCode loaded repeatedlyHeap is smaller
Event Handlers in JavaScript
Code
Events
JavaScript
Objects
21
• Number of events• Sizes of handlers
Event-driven Programming Model• Single-threaded, non-preemptive event
handlers• Example handlers: onabort, onclick, etc.• Very different from batch processing of
benchmarks• Handler responsiveness critical to user
experience
22
Code|Objects|Events
Total Events Handled
23
amazo
nbing
bingmap cnn
ebay
econom
ist
faceb
ookgmail
googlem
ap
hotmail
richard
s
deltab
luecry
pto
raytra
ceea
rleyreg
expsp
lay0
1,000
2,000
3,000
4,000
5,000
6,000
7,000To
tal E
vent
s H
andl
ed
Real Sites V8
Almost no events
Code|Objects|Events
Median Bytecodes / Event Handled
24am
azon
bing
bingm
ap cnn ebay
econo
mist
faceb
ook
gmail
goog
le
goog
lemap
hotm
ail0
50
100
150
200
250
300
350
400
450
500M
edia
n By
teco
des
/ Eve
nt H
andl
ed
Code|Objects|Events
506 2137
Sure, this is all good, but…
• Everyone knows benchmarks are unrepresentative
• How much difference does it make, anyway?• Wouldn’t any benchmarks have similar issues?
25
Cold-code Experiment• Observation
– Real web apps have lots of code (much of it cold)– Benchmarks do not
• Question: What happens if the benchmarks have more code?– We added extra, unused to code to 7 SunSpider
benchmarks– We measured the impact on the benchmark
performance
26
Performance Impact of Cold Code
27
3d-raytr
ace
access
-nbody
bitops-n
sieve
controlflow
crypto-ae
s
math-co
rdic
string-t
agcloud
0
100
200
300
400
500
600
700
800
900
0K200K400K800K1M2M
Tim
e (m
sec)
3d-raytr
ace
access
-nbody
bitops-n
sieve
controlflow
crypto-ae
s
math-co
rdic
string-t
agcloud
0
100
200
300
400
500
600
700
800
900
0K200K400K800K1M2M
Tim
e (m
sec)
Chrome3.0.195.38
IE 88.0.601.18865Cold code has
non-uniform impacton execution time
Cold code makesSunSpider on Chromeup to 4.5x slower
Impact of Benchmarks• What gets emphasis
– Making tight loops fast– Optimizing small amounts of code
• Important issues ignored– Garbage collection (especially of strings)– Managing large amounts of code– Optimizing event handling – Considering JavaScript context between page loads
28
Conclusions• JSMeter is an instrumentation framework
– Used to measure and compare JavaScript applications– High-level views of behavior promote understanding
• Benchmarks differ significantly from real sites– Misleads designers, skews implementations
• Next steps– Develop and promote better benchmarks– Design and evaluate better JavaScript runtimes– Promote better performance tools for JavaScript
developers
29
Additional Resources
• Project: http://research.microsoft.com/en-us/projects/jsmeter/ • Video: Project JSMeter: JavaScript Performance Analysis in the Real World
" - MSDN Channel 9 interview with Erik Meier, Ben Livshits, and Ben Zorn
• Paper:– “JSMeter: Comparing the Behavior of JavaScript Benchmarks with Real Web
Applications”, Paruj Ratanaworabhan, Benjamin Livshits and Benjamin G. Zorn, USENIX 2010 Conference on Web Application Development (WebApps’10), June 2010.
• Additional measurements:– "JSMeter: Characterizing Real-World Behavior of JavaScript Programs", Paruj
Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2009-173, December 2009 (49 pages), November 2009.
30
Additional Slides
31
Related Work• JavaScript
– Richards, Lebresne, Burg, and Vitek (PLDI’10)– Draw similar conclusions
• Java– Doufour et al. (OOPSLA’03), Dieckmann and U.
Hölzle (ECOOP’99)• Other languages
– C++: Calder et al. (JPL’95)– Interpreted languages: Romer et al. (ASPLOS’96)
32
