Paruj Ratanaworabhan,

Ben Livshits, and Ben Zorn

JSMeter: Characterizing the Behavior of JavaScript Web Applications

1

In collaboration with

David Simmons, Corneliu Barsan, and Allen Wirfs-Brock

Why Measure JavaScript?

• Standardized, de facto language for the web

– Support in every browser, much existing code

• Major Microsoft investment

– Web Application Companion based on Script#

• Benchmarks are driving browser marketing: “browser wars”

– Are benchmarks representative?

– Limited understanding of its behavior in real sites

• Who cares?

– Users, web site designers, JavaScript engine developers

2

3

ZDNet 29 May 2008

ghacks.net Dec. 2008

Artificial Benchmarks versus Real World Sites

4

Maps Maps

7 V8 programs:

• richards

• deltablue

• crypto

• raytrace

• earley-boyer

• regexp

• splay

8 SunSpider programs:

• 3-draytrace

• access-nbody

• bitops-nsieve

• controlflow

• crypto-aes

• date-xparb

• math-cordic

• string-tagcloud

JSMeter

Goals of this Talk

• Show behavior of JavaScript benchmarks versus real sites

• Consider how benchmarks can mislead design decisions

11 real sites

5

Instrumentation Framework

\ie\jscript\*.cpp

Source-level instrumentation

custom jscript.dll

custom trace fileswebsite visits

Offline analyzers

custom trace files0%

10%20%30%40%50%60%70%80%90%

100%

Constant

Other Str Ops

Concat Op

Visiting the Real Sites

• Attempted to use each site in “normal” way:

6

amazon Search a book, add to shopping cart, sign in, and sign outbing Type in a search query and also look for images and newsbingmap Search for a direction from one city to anothercnn Read front page newsebay Search for a notebook, bid, sing in, and sign outeconomist Read front page news, view comments

facebookLog in, visit a friend pages, browse through photos and comments

gmail Sign in, check inbox, delete a mail, and sign outgoogle Type in a search query and also look for images and newsgooglemap Search for a direction from one city to anotherhotmail Sign in, check inbox, delete a mail, and sign out

Understanding JavaScript Behavior

Code

Events

JavaScript

Objects

7

Code Behavior

Code

Events

JavaScript

Objects

8

• Function size

• Instructions/call

• Code locality

• Instruction mix

Total Bytes of JavaScript Source

9

amaz

on

bingm

apeb

ay

face

book

google

hotm

ail

richar

ds

cryp

to

earle

y

splay

3d-rayt

race

bitops-

nsieve

cryp

to-a

es

mat

h-cor

dic

strin

g-tagclo

ud

0

500

1000

1500

2000

2500

So

urc

e s

ize

(k

ilo

byte

s)

Real Sites V8 SunSpider

Code|Objects|Events

Static Unique Functions Executed

11

amaz

on

bing

map

ebay

face

book

goog

le

hotm

ail

richa

rds

cryp

to

earle

ysp

lay

3d-ra

ytra

ce

bito

ps-n

sieve

cryp

to-a

es

mat

h-co

rdic

strin

g-ta

gcloud

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

Uniq

ue F

unct

ions E

xecu

ted

Real Sites V8 SunSpider

Code|Objects|Events

Total Bytecodes Executed

12

amaz

on

bingm

apeb

ay

face

book

google

hotm

ail

richar

ds

cryp

to

earle

y

splay

3d-rayt

race

bitops-

nsieve

cryp

to-a

es

mat

h-cor

dic

strin

g-tagclo

ud

0

5000

10000

15000

20000

25000

30000Tota

l B

yte

codes E

xcu

ted (

thousands)

90395

Real Sites V8 SunSpider

Code|Objects|Events

Bytecodes / Call

14am

azon

bingm

apeb

ay

face

book

google

hotm

ail

richar

ds

cryp

to

earle

y

splay

3d-rayt

race

bitops-

nsieve

cryp

to-a

es

mat

h-cor

dic

strin

g-tagclo

ud

0.00

50.00

100.00

150.00

200.00

250.00

300.00

Byte

co

de

s /

Ca

ll

function(a,b) {    var i=0,elem,pos=a.length;    if(D.browser.msie) {        while(elem=b[i++])           if(elem.nodeType!=8)                 a[pos++]=elem;    }  else            while(elem=b[i++])                 a[pos++]=elem;    return a}

Real Sites V8 SunSpider

Code|Objects|Events

Fraction of Code Executed

15

amaz

on

bingm

apeb

ay

face

book

google

hotm

ail

richar

ds

cryp

to

earle

y

splay

3d-rayt

race

bitops-

nsieve

cryp

to-a

es

mat

h-cor

dic

strin

g-tagclo

ud

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

Fra

cti

on

of

Co

de

Exe

cu

ted

Most codenot executed

Real Sites V8 SunSpider

Code|Objects|Events

Hot Function Distribution

16

Real Sites V8 Benchmarks

80% of time in < 10 functions80% of time in 100+ functions

80% 80%

Code|Objects|Events

Opcode Distribution

17Green = SunSpider | Blue= Real Web Apps | Red = V8

Real Apps

Outliers

Code|Objects|Events

Object Allocation Behavior

Code

Events

JavaScript

Objects

18

• Allocation by types

• Lifetime distribution

• Live heap composition

Total Bytes Allocated

19

amaz

on

bingm

apeb

ay

face

book

google

hotm

ail

richar

ds

cryp

to

earle

y

splay

3d-rayt

race

bitops-

nsieve

cryp

to-a

es

mat

h-cor

dic

strin

g-tagclo

ud

0

5000

10000

15000

20000

25000

30000

35000

40000

45000Tota

l h

eap

data

(kilob

yte

s)

Real Sites V8 SunSpider

Code|Objects|Events

Heap Data by Type

20

amazon

bingmapebay

facebook

google

hotmail

richard

s

crypto

earley

splay

access-

nbody

contro

lflow

date-xparb

regexp-dna

aggregate0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ObjectsEnumaratorRegExpMathErrorDateNative FuncStringArraysScript FuncH

eap

Dat

a by

Typ

e

Real Sites V8 SunSpider

Many functionsRest are strings

Few benchmarksallocate much data

Code|Objects|Events

Object Type Distribution

21

Real Apps

Code|Objects|Events

economistis an outlier(arrays)

Live Heap Over Time (gmail)

22

Functions growsteadily

Code|Objects|Events

GC reducessize of heap

Objects growsteadily too

Live Heap over Time (ebay)

23

Heaps repeatedly created, discarded

Heap containsmostly functions

Code|Objects|Events

Heap drops to 0 on page load

Two Search Websites

24

Bing(Web 2.0)

Google(Web 1.0)

Code|Objects|Events

Two Mapping Websites

25

Bing Maps Google Maps

Code|Objects|Events

Object Lifetimes (V8)

26

earley splay

Code|Objects|Events

Lives forever

Dies immediately

Lives untilprogram ends

Object Lifetimes (gmail)

27

Strings areshortest lived

Functions arelongest lived

Many objectslive a major fraction ofpage lifetime

This is as oldas an objectcan be

Code|Objects|Events

Event Handlers in JavaScript

Code

Events

JavaScript

Objects

28

• Number of events• Sizes of handlers

Event-driven Programming Model

• Single-threaded, non-preemptive event handlers

• Example of some standard event-handler attributes:

– onabort, onclick, and onmouseover

• Very different from batch processing of benchmarks

• Handler responsiveness critical to user experience

29

Code|Objects|Events

Total Events Handled

30

amaz

onbin

g

bingm

ap cnn

ebay

econ

omist

face

book

gmail

google

google

map

hotm

ail

richar

ds

delta

blue

cryp

to

rayt

race

earle

y

regex

psp

lay

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000T

ota

l E

ve

nts

Ha

nd

led

Real Sites V8

Almost no events

Code|Objects|Events

Median Bytecodes / Event Handled

32am

azon

bing

bing

map cn

neb

ay

econ

omist

face

book

gmail

goog

le

goog

lem

ap

hotm

ail

0

50

100

150

200

250

300

350

400

450

500M

ed

ian

Byte

co

de

s /

Eve

nt

Ha

nd

led

Code|Objects|Events

Maximum Bytecodes / Event Handled

33am

azon

bing

bing

map cn

neb

ay

econ

omist

face

book

gmail

goog

le

goog

lem

ap

hotm

ail

0

200,000

400,000

600,000

800,000

1,000,000

1,200,000M

axim

um

Byte

co

de

s /

Eve

nt

Ha

nd

led

Code|Objects|Events

Distribution of Time in Handlers

35

Code|Objects|Events

A Simple Experiment• Observation

– Real web apps have lots of code (much of it cold)– Benchmarks do not

• Question: What happens if the benchmarks have more code?– We added extra, unused to code to 6 SunSpider

benchmarks– We measured the impact on the benchmark

performance

36

Performance Impact of Cold Code

37

3d-raytr

ace

access

-nbody

bitops-n

sieve

contro

lflow

crypto-ae

s

math-co

rdic

string-t

agclo

ud0

100

200

300

400

500

600

700

800

900

0K200K400K800K1M2M

Tim

e (m

sec)

3d-raytr

ace

access

-nbody

bitops-n

sieve

contro

lflow

crypto-ae

s

math-co

rdic

string-t

agclo

ud0

100

200

300

400

500

600

700

800

900

0K200K400K800K1M2M

Tim

e (m

sec)

Chrome IE 8

Cold code has non-uniform impacton execution time

Without cold code, Chrome12x faster than IE8With 2M cold code,Chrome 4.7x faster than IE8Which comparison is more meaningful?

Cold code makesSunSpider on Chromeup to 4.5x slower

Impact of Benchmarks

• What gets emphasis– Making tight loops fast– Optimizing small amounts of code

• Important issues ignored– Garbage collection (especially of strings)– Managing large amounts of code– Optimizing event handling – Considering JavaScript context between page loads

38

Conclusions

• JSmeter is an instrumentation framework

– Used to measure and compare JavaScript applications

– High-level views of behavior promote understanding

• Behavior in benchmarks significantly differs from real sites

– Misleads designers, skews implementations

• Next steps

– Develop and promote better benchmarks

– Design and evaluate better JavaScript runtimes

– Promote better performance tools for JavaScript developers

39

Additional Resources• Project

– http://research.microsoft.com/en-us/projects/jsmeter/ • Papers

– "JSMeter: Measuring JavaScript Behavior in the Wild", Paruj Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2010-8, January 2010 (17 pages)

– "JSMeter: Characterizing Real-World Behavior of JavaScript Programs", Paruj Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2009-173, December 2009 (49 pages)

– “An Analysis of the Dynamic Behavior of JavaScript Programs”, Gregor Richards, Sylvain Lebresne, Brian Burg, Jan Vitek (to appear in PLDI 2010)

40