+ All Categories
Home > Documents > JSMeter: Characterizing the Behavior of JavaScript Web ... file– Support in every browser,...

JSMeter: Characterizing the Behavior of JavaScript Web ... file– Support in every browser,...

Date post: 12-Feb-2018
Category:
Upload: ngothu
View: 222 times
Download: 0 times
Share this document with a friend
36
Paruj Ratanaworabhan, Ben Livshits, and Ben Zorn JSMeter: Characterizing the Behavior of JavaScript Web Applications 1 In collaboration with David Simmons, Corneliu Barsan, and Allen Wirfs-Brock
Transcript

Paruj Ratanaworabhan,

Ben Livshits, and Ben Zorn

JSMeter: Characterizing the Behavior of JavaScript Web Applications

1

In collaboration with

David Simmons, Corneliu Barsan, and Allen Wirfs-Brock

Why Measure JavaScript?

• Standardized, de facto language for the web

– Support in every browser, much existing code

• Major Microsoft investment

– Web Application Companion based on Script#

• Benchmarks are driving browser marketing: “browser wars”

– Are benchmarks representative?

– Limited understanding of its behavior in real sites

• Who cares?

– Users, web site designers, JavaScript engine developers

2

3

ZDNet 29 May 2008

ghacks.net Dec. 2008

Artificial Benchmarks versus Real World Sites

4

Maps Maps

7 V8 programs:

• richards

• deltablue

• crypto

• raytrace

• earley-boyer

• regexp

• splay

8 SunSpiderprograms:

• 3-draytrace

• access-nbody

• bitops-nsieve

• controlflow

• crypto-aes

• date-xparb

• math-cordic

• string-tagcloud

JSMeter

Goals of this Talk

• Show behavior of JavaScript benchmarks versus real sites

• Consider how benchmarks can mislead design decisions

11 real sites

5

Instrumentation Framework

\ie\jscript\*.cpp

Source-level instrumentation

custom jscript.dll

custom trace fileswebsite visits

Offline analyzers

custom trace files

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Constant

Other Str Ops

Concat Op

Visiting the Real Sites

• Attempted to use each site in “normal” way:

6

amazon Search a book, add to shopping cart, sign in, and sign out

bing Type in a search query and also look for images and news

bingmap Search for a direction from one city to another

cnn Read front page news

ebay Search for a notebook, bid, sing in, and sign out

economist Read front page news, view comments

facebook Log in, visit a friend pages, browse through photos and comments

gmail Sign in, check inbox, delete a mail, and sign out

google Type in a search query and also look for images and news

googlemap Search for a direction from one city to another

hotmail Sign in, check inbox, delete a mail, and sign out

Understanding JavaScript Behavior

Code

Events

JavaScript

Objects

7

Code Behavior

Code

Events

JavaScript

Objects

8

• Function size

• Instructions/call

• Code locality

• Instruction mix

Total Bytes of JavaScript Source

9

0

500

1000

1500

2000

2500

am

azon

bin

g

bin

gm

ap

cnn

ebay

econom

ist

facebook

gm

ail

google

google

map

hotm

ail

richard

s

delt

ab

lue

cry

pto

raytr

ace

earl

ey

regexp

spla

y

3d-ra

ytr

ace

access-nbody

bit

ops-nsie

ve

contr

olf

low

cry

pto

-aes

date

-xparb

math

-cord

ic

regexp-dna

str

ing-ta

gclo

ud

Sourc

e s

ize (kilob

yte

s)

Real Sites V8 SunSpider

Code|Objects|Events

Static Unique Functions Executed

11

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

am

azon

bin

g

bin

gm

ap

cnn

ebay

econom

ist

facebook

gm

ail

google

google

map

hotm

ail

richard

s

delt

ablu

e

cry

pto

raytr

ace

earl

ey

regexp

spla

y

3d-ra

ytr

ace

access-nbody

bit

ops-nsie

ve

contr

olf

low

cry

pto

-aes

date

-xparb

math

-cord

ic

regexp-dna

str

ing-ta

gclo

ud

Uniq

ue F

uncti

ons E

xecute

d

Real Sites V8 SunSpider

Code|Objects|Events

Total Bytecodes Executed

12

0

5000

10000

15000

20000

25000

30000

am

azon

bin

g

bin

gm

ap

cnn

ebay

econom

ist

facebook

gm

ail

google

google

map

hotm

ail

richard

s

delt

ab

lue

cry

pto

raytr

ace

earl

ey

regexp

spla

y

3d-ra

ytr

ace

access-nbody

bit

ops-nsie

ve

contr

olf

low

cry

pto

-aes

date

-xparb

math

-cord

ic

regexp

-dna

str

ing-ta

gclo

ud

Tota

l Byte

codes

Excute

d(t

housands)

90395

Real Sites V8 SunSpider

Code|Objects|Events

Bytecodes / Call

14

0.00

50.00

100.00

150.00

200.00

250.00

300.00

am

azon

bin

g

bin

gm

ap

cnn

ebay

econom

ist

facebook

gm

ail

google

google

map

hotm

ail

richard

s

delt

ab

lue

cry

pto

raytr

ace

earl

ey

regexp

spla

y

3d-ra

ytr

ace

access-nbody

bit

ops-nsie

ve

contr

olf

low

cry

pto

-aes

date

-xparb

math

-cord

ic

regexp

-dna

str

ing-ta

gclo

ud

Byte

codes /

Call

function(a,b) {var i=0,elem,pos=a.length;if(D.browser.msie) {

while(elem=b[i++])if(elem.nodeType!=8)

a[pos++]=elem;} else

while(elem=b[i++])a[pos++]=elem;

return a}

Real Sites V8 SunSpider

Code|Objects|Events

Fraction of Code Executed

15

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

am

azon

bin

gbin

gm

ap

cnn

ebay

econom

ist

facebook

gm

ail

google

google

map

hotm

ail

richard

sdelt

ab

lue

cry

pto

raytr

ace

earl

ey

regexp

spla

y

3d-ra

ytr

ace

access-nbody

bit

ops-nsie

ve

contr

olf

low

cry

pto

-aes

date

-xparb

math

-cord

icre

gexp-dna

str

ing-ta

gclo

ud

Fra

cti

on o

f C

od

e E

xecute

d

Most code

not executed

Real Sites V8 SunSpider

Code|Objects|Events

Hot Function Distribution

16

Real Sites V8 Benchmarks

80% of time in < 10 functions80% of time in 100+ functions

80% 80%

Code|Objects|Events

Opcode Distribution

17Green = SunSpider | Blue= Real Web Apps | Red = V8

Real Apps

Outliers

Code|Objects|Events

Object Allocation Behavior

Code

Events

JavaScript

Objects

18

• Allocation by types

• Lifetime distribution

• Live heap composition

Total Bytes Allocated

19

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

am

azon

bin

g

bin

gm

ap

cnn

ebay

econom

ist

facebook

gm

ail

google

google

map

hotm

ail

richard

s

delt

ab

lue

cry

pto

raytr

ace

earl

ey

regexp

spla

y

3d-ra

ytr

ace

access-nbody

bit

ops-nsie

ve

contr

olf

low

cry

pto

-aes

date

-xparb

math

-cord

ic

regexp

-dna

str

ing-ta

gclo

ud

Tota

l heap

data

(kilobyte

s)

Real Sites V8 SunSpider

Code|Objects|Events

Heap Data by Type

20

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

amaz

on

bin

gb

ingm

ap cnn

ebay

eco

no

mis

tfa

ceb

oo

kgm

ail

goo

gle

goo

glem

aph

otm

ail

rich

ard

sd

elta

blu

ecr

ypto

rayt

race

earl

eyre

gexp

spla

yV

8 a

ggra

gate

3d

-ray

trac

eac

cess

-nb

od

yb

ito

ps-

nsi

eve

con

tro

lflo

wcr

ypto

-aes

dat

e-xp

arb

mat

h-c

ord

icre

gexp

-dn

ast

rin

g-ta

gclo

ud

aggr

egat

e

He

ap D

ata

by

Typ

e

Objects

Enumarator

RegExp

Math

Error

Date

Native Func

String

Arrays

Script Func

Real Sites V8 SunSpider

Many functions

Rest are strings

Few benchmarks

allocate much data

Code|Objects|Events

Object Type Distribution

21

Real Apps

Code|Objects|Events

economist

is an outlier

(arrays)

Live Heap Over Time (gmail)

22

Functions grow

steadily

Code|Objects|Events

GC reduces

size of heap

Objects grow

steadily too

Live Heap over Time (ebay)

23

Heaps

repeatedly

created,

discarded

Heap contains

mostly functions

Code|Objects|Events

Heap drops to 0

on page load

Two Search Websites

24

Bing

(Web 2.0)

Google

(Web 1.0)

Code|Objects|Events

Two Mapping Websites

25

Bing Maps Google Maps

Code|Objects|Events

Object Lifetimes (V8)

26

earley splay

Code|Objects|Events

Lives forever

Dies

immediately

Lives until

program ends

Object Lifetimes (gmail)

27

Strings are

shortest lived

Functions are

longest lived

Many objects

live a major

fraction of

page lifetime

This is as old

as an object

can be

Code|Objects|Events

Event Handlers in JavaScript

Code

Events

JavaScript

Objects

28

• Number of events• Sizes of handlers

Event-driven Programming Model

• Single-threaded, non-preemptive event handlers

• Example of some standard event-handler attributes:

– onabort, onclick, and onmouseover

• Very different from batch processing of benchmarks

• Handler responsiveness critical to user experience

29

Code|Objects|Events

Total Events Handled

30

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000T

ota

l Events

Hand

led

Real Sites V8

Almost no events

Code|Objects|Events

Median Bytecodes / Event Handled

32

0

50

100

150

200

250

300

350

400

450

500M

edia

n B

yte

codes /

Event

Handle

d

Code|Objects|Events

Maximum Bytecodes / Event Handled

33

0

200,000

400,000

600,000

800,000

1,000,000

1,200,000M

axim

um

Byte

codes /

Event

Handle

d

Code|Objects|Events

Distribution of Time in Handlers

35

Code|Objects|Events

A Simple Experiment

• Observation

– Real web apps have lots of code (much of it cold)

– Benchmarks do not

• Question: What happens if the benchmarks have more code?

– We added extra, unused to code to 6 SunSpiderbenchmarks

– We measured the impact on the benchmark performance

36

Performance Impact of Cold Code

37

0

100

200

300

400

500

600

700

800

900

Tim

e (

mse

c)

0K

200K

400K

800K

1M

2M

0

100

200

300

400

500

600

700

800

900

Tim

e (

mse

c)

0K

200K

400K

800K

1M

2M

Chrome IE 8

Cold code has

non-uniform impact

on execution time

Without cold code,

Chrome12x faster than IE8With 2M cold code,

Chrome 4.7x faster than IE8

Which comparison is more meaningful?

Cold code makes

SunSpider on Chrome

up to 4.5x slower

Impact of Benchmarks

• What gets emphasis

– Making tight loops fast

– Optimizing small amounts of code

• Important issues ignored

– Garbage collection (especially of strings)

– Managing large amounts of code

– Optimizing event handling

– Considering JavaScript context between page loads

38

Conclusions

• JSmeter is an instrumentation framework

– Used to measure and compare JavaScript applications

– High-level views of behavior promote understanding

• Behavior in benchmarks significantly differs from real sites

– Misleads designers, skews implementations

• Next steps

– Develop and promote better benchmarks

– Design and evaluate better JavaScript runtimes

– Promote better performance tools for JavaScript developers

39

Additional Resources

• Project– http://research.microsoft.com/en-us/projects/jsmeter/

• Papers– "JSMeter: Measuring JavaScript Behavior in the Wild", Paruj Ratanaworabhan,

Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2010-8, January 2010 (17 pages)

– "JSMeter: Characterizing Real-World Behavior of JavaScript Programs", Paruj Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2009-173, December 2009 (49 pages)

– “An Analysis of the Dynamic Behavior of JavaScript Programs”, Gregor Richards, Sylvain Lebresne, Brian Burg, Jan Vitek (to appear in PLDI 2010)

40


Recommended