Presented by,MySQL AB® &

O’Reilly Media, Inc.

Web Workloads for Comparing, Testing and Tuning MySQL Performance , SPECjAppServer2004, EAStress and Faban

Tom Daly Senior Performance Engineer Sun Microsystems

Agenda

MySQL and SPECjAppServer2004resources you probably never knew were there

MySQL and EAStresshow you can run SPECjAppServer and why you would

MySQL and Faban , create your own web based transaction oriented benchmarks

Aims/Objectives

To introduce you to some tools and resources for testing and tuning MySQL with web applications that you might find useful To give you an idea of the sorts of analysis you can do with these tools

We work from least work to most work

Advantages

typical and realistic often SQL is generated / is typical of modern

web based transactions tests DB and MySQL dependencies like

network and JDBC Tests connector, connection poolingstresses system with many usersCan test usage patterns available from web

statistics reports directly

Testing MySQL using web workload

Due to the above factors testing at the web tier will more often give more representative results for DBMSs used for web applications

Disadvantages

need to ensure that middle-tier is not clouding results

some middle ware expertise required require hardware for the middle tiercan be more work

Testing MySQL using web workload

Benchmark =

workload + rules and basis for comparison + process to ensure rule compliance

Benchmarks can be useful for end user comparisons and sizing!

Benchmarks are potentially much more useful for comparisons than workloads

SPECjAppServer2004

Is an industry standard benchmark

Models a web + mfg business

Extensive run rules

No changes to source code

No benchmark specials

All products available and supported

lots more

Peer review of all results prior to publishing

SPECjAppServer2004

SPECjAppServer2004

So it turns out ...

SPECjAppServer2004 is a reasonably useful benchmark for comparing and improving database (MySQL performance)It presents an OLTP workload to MySQLUses a DB that may or may not be in DB cache Representative to MySQL of loads from applications developed in Java,Ruby on Rails, web services PhP etc.

SPECjAppServer2004

SPECjAppServer2004 typical configuration

Sun 720 SPECjAppServer2004 JOPS@Standard

models > 5500, concurrent userswith 90% of results under 2 secs (or 5 secs) MySQL 5.0.3

approx. 800 writes per sec to the log, approx. 4kw sec

Results list is a useful database sizing and comparison resource

Also valuable resource for tuning information

SPECjAppServer2004

For instance if your application today requires less than say 6000 users and 8GB Ram then perhaps MySQL on 4 (modern) cores will meet your OLTP performance requirements

Demo – Benchmark Results Page walk through (FDR and FDA)

Run rules require that all submissions include “complete bill of materials for the SUT”

SPECjAppServer2004

Supplier Description Product # Qty Unit Price Price-------- ---------------------------------------- ------------------ --- ----------- --------------Sun Sun Fire X4100 (2x285,4x2GB,2X73GB) A64-EGB2-2H-8G-CB7 3 $8995 $26,985Sun Solaris 10 RTU 3 $0 $0Sun SunSpectrum Upgrade: 3YGOLD, 24x7 W9D-A64-24-3G 3 not priced not priced Sun Sun Fire X4100 (2x285,4x2GB,2X73GB) A64-EGB2-2H-8G-CB7 1 $8995 $8995Sun Solaris 10 RTU 1 $0 $0Sun Single-Port PCI Ultra320 SCSI HBA SGXPCI1SCSILM320-Z 1 $340 $340 Sun SunSpectrum Upgrade: 3YGOLD, 24x7 W9D-A64-24-3G 1 not priced not pricedSun Sun StorEdge 3320, 12x73GB, 1 RAID CONT XTA3510R01A1R876 1 $22,495   $22,495Sun Single-Port PCI Ultra320 SCSI HBA SGXPCI1SCSILM320-Z 1 $445 $445Sun SunSpectum Upgrade: 3Y GOLD, 24x7 W9D-SE3510-24-3G 1 not priced not pricedSun 17" Entry Color Monitor X7147A 1Sun PS/2 Keyboard & Mouse #320-1261 1Sun Sun Java System Application Server $0 $0Sun Sun Java System Application support cpu/1 yr SJSAS-PE9F-1PR 9 not priced not priced

MySQL MySQL Database 5.0 $0 $0MySQL MySQL Network Gold Support 1 yr 3 not priced not priced ======= $59,260 =======

MySQL PostgreSQL Oracle0

100

200

300

400

500

600

700

800

900

1000

720

813

874

JOPS Througput

JOP

S

MySQL PostgreSQL Oracle0

20000

40000

60000

80000

100000

120000

140000

160000

180000

200000

Acquisition Cost Compar-ison

Co

st in

US

Do

llars

SPECjAppServer2004 JOPS@standardSPEC, SPECjAppServer reg tm of Standard Performance Evaluation Corporation.Sun 813.73 SPECjAppServer2004 JOPS@Standard (Sun Fire X4200, 4 chips / 8 cores, T2000 1 chip / 8 cores)Sun 720.56 SPECjAppServer2004JOPS@Standard(Sun Fire, X4100, 4 chips/8 cores, X4100 1 chip / 4 cores ) HP 874.17 SPECjAppServer2004 JOPS@standard (rx2660 , 4 cores, 2 chips, rx2660 4 cores. 2chips)

performance and cost example

Models an enterprise class web application

not useful for older style “stored procedure code”

not a good BIDW application model

not necessarily a good Web 2.0 model, i.e. wouldn't be a good workload to test MySQL use by facebook (see EAStress or Faban for this)

SPECjAppServer2004

Is in active development at SPEC

SPECjAppServer

SPECjAppServer2004 research mode “EAStress”

SPECjAppServer2004 research mode “EAStress2004”

More open and easy to use version of SPECjAppServer2004 intended for R&D

can use results w/o going to SPEC for approval

Very similar workload but changed enough so as to ensure non comparability with published results

“EAStress2004” - mandatory notification

SPECjAppServer is a trademark of the Standard Performance Evaluation Corp. (SPEC). The EAStress workload results or findings in this

publication have not been reviewed or accepted by SPEC, therefore no comparison nor performance inference can be made against any published SPEC result. The official web site for SPECjAppServer2004 is located at

http://www.spec.org/osg/jAppServer2004.

Demo EAStress2004 – reports walk through

“EAStress” - why you might care

Good representative pre-written web application you can use to tune, test, size and even compare MySQL performance

Allows testing of many what-if scenarios

Examples follow

“EAStress” - experiments / results

cache=0 cache=256mb600

620

640

660

680

700

720

701

638

query_cache_size

EAStress2004 HASOPM

Note difference is understated as response criteria failed for cache=256mb Mfg = 7 secs

“EAStress” - experiments / results

Bug #15815 fixed in MySQL 5.1.24rc IR=718 for both runs but 5.1.23 failed to meet response time criteria

5.1.23rc 5.1.24rc

580

600

620

640

660

680

700

720

631

701

Bug fix #15815

HASOPM

“EAStress” - experiments / results

/data mounted forcedirectio or ufsNote /data as UFS results did not meet the response time criteria

/data = UFS /data = directio

670

675

680

685

690

695

700

705

682.58

701

directio

HASOPM

EAStress Observations query cachetxn log tables beware single contention pointcompiler optionsBig impact of Bug #158155.0, 5.1 performance improvementUseServerPrepared Statements caused errors Supported > 9000 of concurrent users with

Sun 8 core Intel server as the DB

“EAStress” - experiments / results

EAStress

You can purchase EAStress (SPECjAppServer2004) from http://www.spec.org/order.html (and there are academic and research prices )

Faban – building your own web workload for MySQL

Faban : create your own benchmarks

Faban is transaction oriented pert testing tool, closely related to the SPECjAppServer work Can use for MySQL/Java or anything web based

Faban

Infrastructure for developing transaction oriented web benchmarks Written by Akara Sucharitakul of Sun

Faban components

• Harness – benchmark process automation, collect statistics, display results

• Driver Framework - benchmark development framework and component model

• Analysis tools - graph, tabulate, and compare results

The Faban Harness

Provides a Web UI for managing benchmark runsAllows users to ...

Submit/queue up benchmark runsView progress and centralized log of runView/compare results, statistics, graphs, etc.

Also ...Configures/prepares the benchmarking rigAutomatically collects all system configurationStarts/restarts processes/serversLoad dataCollect stats across rig

Faban Harness Architecture

Web Interface(Tomcat)

Log Server

Master

Registry

Agent Agent (Remote)

Benchmark

Run Daemon

The Faban Driver FrameworkA driver developer's component model

Container/driver distributable across multiple systemsDeveloper defines driver operationsOperation: a unit of work to be measuredOperation defines data generation and call to SUT

Utilizes Java SE 5 annotations to describe benchmark spec, policy, stochastic modelXML-based configuration for individual runMeasures throughput at latency limit (SLA)100% pure Java, platform independent - runs on J2SE5.0 or later

Specifying the Policies

MixDetermines the statistical model to call the operations

CycleDetermines the time duration between operation calls

Mixes

Detemines the operation to selectCurrent mixes:

FlatMixMatrixMixFlatSequenceMixFixedSequence

Mixes are extensible – new mix policies can be added as neededWell defined interface for adding mixes

@MatrixMix( operations = {“Operation1”, “Operation2”, “Operation3”}, mix = {@Row({10, 50, 40}), @Row({30, 10, 60}), @Row({45, 45, 10})}, deviation = 2)

Cycles

Determines the time to fire an operationCan be set to think or cycle timeCurrent cycle policies:

FixedTimeUniformNegative Exponential

Cycles are extensible – new cycle policies can be added as neededWell defined interface for adding cycles

@NegativeExponential( cycleType = CycleType.CYCLETIME, cycleMean = 5, cycleMax = 25, cycleDeviation = 1)

Metric Recording

AutomaticNo timer call in code, cleaner codeMore accurate timing of server response timesRequires TimedSocket installed in transport protocol

ManualTimer calls demarcates critical section of the operationWorks with all protocols, or even benchmarks running on the local system

Faban code

Other Features

Distributed or centralizedTime syncronizationSleep time calibrationTransport protocol handlers@BackgroundMix and cycle extensibility

Open Source

Faban is open source, periodhttp://faban.sunsource.net/Licensed under CDDL

http://www.sun.com/cddl/Same as GlassFish, etc

CVS repository is public, nothing hiddenRepository is usually ahead of buildsYou shall request changes to fit your needsYou shall make the changes if you want it fastThe changes benefit all of us

Summary

Driver framework promotes high quality benchmarks/workloadsHarness provides process automation for

Running the benchmarksCollect resultsMonitor system stats across rig

Use web interface to view the status, results, and analyze the results

Faban - demo

Q&A