Paruj RatanaworabhanKasetsart University, Thailand

Ben Livshits and Ben ZornMicrosoft Research, Redmond

JSMeter: Characterizing the Behavior of JavaScript Web Applications

1

in collaboration with

David Simmons, Corneliu Barsan, and Allen Wirfs-Brock

Background• Who is this guy?

– Current: Principal Researcher, Microsoft – Past: Professor, U of Colorado; PhD, UC Berkeley

• Research in Software Engineering (RiSE)– 30 researchers and developers– Notables

• Code Contracts (Visual Studio 2010)• Static Driver Verifier in Windows• Peli’s Video Blog at MSDN Channel 9

2

Why Measure JavaScript?

• Standardized, de facto language for the web

– Support in every browser, much existing code

• Browser and JavaScript performance is important

– Are current JavaScript benchmarks representative?

– Limited understanding of JavaScript behavior in real sites

• Who cares?

– Users, web application developers, JavaScript engine developers

3

4

ZDNet 29 May 2008

ghacks.net Dec. 2008

BrowserWars!

Artificial Benchmarks versus Real World Sites

5

Maps Maps

7 V8 programs:

• richards

• deltablue

• crypto

• raytrace

• earley-boyer

• regexp

• splay

8 SunSpider programs:• 3-draytrace

• access-nbody

• bitops-nsieve

• controlflow

• crypto-aes

• date-xparb

• math-cordic

• string-tagcloud

JSMeter 11 real sites:

Goals of JSMeter Project• Instrument JavaScript execution and measure behavior• Compare behavior of JavaScript benchmarks against real sites• Consider how benchmarks can mislead design decisions

6

How We Measured JavaScript

\ie\jscript\*.cpp

Source-level instrumentation

custom jscript.dll

custom trace fileswebsite visits

Offline analyzers

custom trace files0%

10%20%30%40%50%60%70%80%90%

100%

Constant

Other Str Ops

Concat Op

Visiting the Real Sites

• Getting past page load performance• Attempted to use each site in “normal” way:

7

amazon Search a book, add to shopping cart, sign in, and sign outbing Type in a search query and also look for images and newsbingmap Search for a direction from one city to anothercnn Read front page newsebay Search for a notebook, bid, sing in, and sign outeconomist Read front page news, view comments

facebookLog in, visit a friend pages, browse through photos and comments

gmail Sign in, check inbox, delete a mail, and sign outgoogle Type in a search query and also look for images and newsgooglemap Search for a direction from one city to anotherhotmail Sign in, check inbox, delete a mail, and sign out

Understanding JavaScript Behavior

Code

Events

JavaScript

Objects

8

Code Behavior

Code

Events

JavaScript

Objects

9

• Function size

• Instructions/call

• Code locality

• Instruction mix

Total Bytes of JavaScript Source

10

amaz

on

bingm

apeb

ay

face

book

google

hotm

ail

richar

ds

cryp

to

earle

y

splay

3d-rayt

race

bitops-

nsieve

cryp

to-a

es

mat

h-cor

dic

strin

g-tagclo

ud

0

500

1000

1500

2000

2500

So

urc

e s

ize

(k

ilo

byte

s)

Real Sites V8 SunSpider

Code|Objects|Events

Static Unique Functions Executed

11

amaz

on

bing

map

ebay

face

book

goog

le

hotm

ail

richa

rds

cryp

to

earle

ysp

lay

3d-ra

ytra

ce

bito

ps-n

sieve

cryp

to-a

es

mat

h-co

rdic

strin

g-ta

gcloud

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

Uniq

ue F

unct

ions E

xecu

ted

Real Sites V8 SunSpider

Code|Objects|Events

Bytecodes / Call

12am

azon

bingm

apeb

ay

face

book

google

hotm

ail

richar

ds

cryp

to

earle

y

splay

3d-rayt

race

bitops-

nsieve

cryp

to-a

es

mat

h-cor

dic

strin

g-tagclo

ud

0.00

50.00

100.00

150.00

200.00

250.00

300.00

Byte

co

de

s /

Ca

ll

function(a,b) {    var i=0,elem,pos=a.length;    if(D.browser.msie) {        while(elem=b[i++])           if(elem.nodeType!=8)                 a[pos++]=elem;    }  else            while(elem=b[i++])                 a[pos++]=elem;    return a}

Real Sites V8 SunSpider

Code|Objects|Events

Fraction of Code Executed

13

amaz

on

bingm

apeb

ay

face

book

google

hotm

ail

richar

ds

cryp

to

earle

y

splay

3d-rayt

race

bitops-

nsieve

cryp

to-a

es

mat

h-cor

dic

strin

g-tagclo

ud

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

Fra

cti

on

of

Co

de

Exe

cu

ted

Most codenot executed

Real Sites V8 SunSpider

Code|Objects|Events

Hot Function Distribution

14

Real Sites V8 Benchmarks

80% of time in < 10 functions80% of time in 100+ functions

80% 80%

Code|Objects|Events

Object Allocation Behavior

Code

Events

JavaScript

Objects

15

• Allocation by types

• Live heap composition

• Lifetime distribution

Total Bytes Allocated

16

amaz

on

bingm

apeb

ay

face

book

google

hotm

ail

richar

ds

cryp

to

earle

y

splay

3d-rayt

race

bitops-

nsieve

cryp

to-a

es

mat

h-cor

dic

strin

g-tagclo

ud

0

5000

10000

15000

20000

25000

30000

35000

40000

45000Tota

l h

eap

data

(kilob

yte

s)

Real Sites V8 SunSpider

Code|Objects|Events

Heap Data by Type

17

amazon

bingmapebay

facebook

google

hotmail

richard

s

crypto

earley

splay

access-

nbody

contro

lflow

date-xparb

regexp-dna

aggregate0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ObjectsEnumaratorRegExpMathErrorDateNative FuncStringArraysScript FuncH

eap

Dat

a by

Typ

e

Real Sites V8 SunSpider

Many functionsRest are strings

Few benchmarksallocate much data

Code|Objects|Events

Live Heap Over Time (gmail)

18

Functions growsteadily

Code|Objects|Events

GC reducessize of heap

Objects growsteadily too

Live Heap over Time (ebay)

19

Heaps repeatedly created, discarded

Heap containsmostly functions

Code|Objects|Events

Heap drops to 0 on page load

2 Search Websites, 2 Architectures

Bing Google

Code|Objects|Events

20

You stay on the same page during your entire visitCode loaded onceHeap is bigger

Every transition loads a new pageCode loaded repeatedlyHeap is smaller

Event Handlers in JavaScript

Code

Events

JavaScript

Objects

21

• Number of events• Sizes of handlers

Event-driven Programming Model• Single-threaded, non-preemptive event

handlers• Example handlers: onabort, onclick, etc.• Very different from batch processing of

benchmarks• Handler responsiveness critical to user

experience

22

Code|Objects|Events

Total Events Handled

23

amaz

onbin

g

bingm

ap cnn

ebay

econ

omist

face

book

gmail

google

google

map

hotm

ail

richar

ds

delta

blue

cryp

to

rayt

race

earle

y

regex

psp

lay

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000T

ota

l E

ve

nts

Ha

nd

led

Real Sites V8

Almost no events

Code|Objects|Events

Median Bytecodes / Event Handled

24am

azon

bing

bing

map cn

neb

ay

econ

omist

face

book

gmail

goog

le

goog

lem

ap

hotm

ail

0

50

100

150

200

250

300

350

400

450

500M

ed

ian

Byte

co

de

s /

Eve

nt

Ha

nd

led

Code|Objects|Events

506 2137

Sure, this is all good, but…

• Everyone knows benchmarks are unrepresentative

• How much difference does it make, anyway?• Wouldn’t any benchmarks have similar issues?

25

Cold-code Experiment• Observation

– Real web apps have lots of code (much of it cold)– Benchmarks do not

• Question: What happens if the benchmarks have more code?– We added extra, unused to code to 7 SunSpider

benchmarks– We measured the impact on the benchmark

performance

26

Performance Impact of Cold Code

27

3d-raytr

ace

access

-nbody

bitops-n

sieve

contro

lflow

crypto-ae

s

math-co

rdic

string-t

agclo

ud0

100

200

300

400

500

600

700

800

900

0K200K400K800K1M2M

Tim

e (m

sec)

3d-raytr

ace

access

-nbody

bitops-n

sieve

contro

lflow

crypto-ae

s

math-co

rdic

string-t

agclo

ud0

100

200

300

400

500

600

700

800

900

0K200K400K800K1M2M

Tim

e (m

sec)

Chrome3.0.195.38

IE 88.0.601.18865

Cold code has non-uniform impacton execution time

Cold code makesSunSpider on Chromeup to 4.5x slower

Impact of Benchmarks

• What gets emphasis– Making tight loops fast– Optimizing small amounts of code

• Important issues ignored– Garbage collection (especially of strings)– Managing large amounts of code– Optimizing event handling – Considering JavaScript context between page loads

28

Conclusions• JSMeter is an instrumentation framework

– Used to measure and compare JavaScript applications– High-level views of behavior promote understanding

• Benchmarks differ significantly from real sites– Misleads designers, skews implementations

• Next steps– Develop and promote better benchmarks– Design and evaluate better JavaScript runtimes– Promote better performance tools for JavaScript

developers

29

Additional Resources

• Project: http://research.microsoft.com/en-us/projects/jsmeter/ • Video: Project JSMeter: JavaScript Performance Analysis in the Real World

" - MSDN Channel 9 interview with Erik Meier, Ben Livshits, and Ben Zorn

• Paper:– “JSMeter: Comparing the Behavior of JavaScript Benchmarks with Real Web

Applications”, Paruj Ratanaworabhan, Benjamin Livshits and Benjamin G. Zorn, USENIX 2010 Conference on Web Application Development (WebApps’10), June 2010.

• Additional measurements:– "JSMeter: Characterizing Real-World Behavior of JavaScript Programs", Paruj

Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2009-173, December 2009 (49 pages), November 2009.

30

Additional Slides

31

Related Work• JavaScript

– Richards, Lebresne, Burg, and Vitek (PLDI’10)– Draw similar conclusions

• Java– Doufour et al. (OOPSLA’03), Dieckmann and U.

Hölzle (ECOOP’99)• Other languages

– C++: Calder et al. (JPL’95)– Interpreted languages: Romer et al. (ASPLOS’96)

32