www.pha.com.au

Load Testing UNIX Systems

Peter Hardingplh@pha.com.au

www.pha.com.au

System & Load testing

Two main circumstances you might want to test:– Sizing new hardware (system acquisition)– Deploying new application (new or

upgraded software, eg. Y2K)

www.pha.com.au

Two Types of Tests

In either situation there are two types of test we are interested in:– System testing:

• Backups• End of Day/End of Month

– Load testing:• Capacity – throughput/response time• Scalability

www.pha.com.au

A Load Testing Methodology

• Typically want to run a system up with a client’s application and databases in a hardware configuration reflecting actual client production infrastructure.

• Want to find out such things as maximum throughput, expected response times under a designated load.

• May want to validate modelled predictions for performance (& capacity).

www.pha.com.au

Sample RTE Testing Architectures

1) Monolithic (Development of tests)

2) Two tier (Monolithic application environment)

3) Multi-tier (Client-server, PeopleSoft, SAP)

www.pha.com.au

Environments

• ASCII (telnet, rlogin, ssh)• Windows• X Windows (X11, KDE, Gnome, …)• Web

– http– https (SSL)

• Other protocols– ftp, nntp, – Tuxedo– Corba– SQL*Net

• Java– See: www.precise.com

www.pha.com.au

Commercial Tools

• Mercury Interactive (www.mercuryinteractive.com)– WinRunner– LoadRunner

• Rational (now owned by IBM)– Rational Robot

• Seague (www.seague.com)– SilkPerformer V

• CompuWare (www.compuware.com)– QALoad

• Others– Lots – especially in the web space

www.pha.com.au

Single Tier (Insitu) Benchmark Architectures

System Under Test (SUT)+

RTE System(s)(Client - Driver System)

www.pha.com.au

Two Tier Benchmark Architectures

RTE System(s)(Client - Driver System)

System Under Test (SUT)

RTE System(s)(Client - Driver System)

TTY LinesNetwork Connection (X25 or Ethernet)

System Under Test (SUT)

www.pha.com.au

Three Tier Benchmark Architecture

RTE System(s)(Client System)

System Under Test (SUT)

RTE System (Driver System)

www.pha.com.au

The General Approach

• Planning Phase

• Development Phase

• Testing Phase

• Review Phase

• Reporting Phase

www.pha.com.au

Planning Phase

• Conversation with client determine scope of exercise.

• Consultation with client resources are required to identify WHAT is to be tested – ie. The TASKS.

• Further conversation will hopefully yield indications of expected mix of activities (TASKS). This is often subject to wild speculation. This allows us to specify the load scenarios from which we build the workload files.

www.pha.com.au

Development Phase

• Familiarize yourself with the application environment (probably using a development environment for SUT).

• Set up test harness and RTE/data server libraries on driver(s).

• Record basic navigation through the application.

• Record and test each identified TASK. Handling error conditions will require most effort.

www.pha.com.au

Test Phase

• Setup of System Under Test (SUT) environment will be required about now.

• Setup data to be fed to robots during actual tests.

• Build workload files.• Run tests.• Do analysis.• Publish report.

www.pha.com.au

The Basic Components

• Recorder (record)

• RTE Library (librte.so)

• Test Harness (sim.c et al.)

• Controller (start /regulate/review)

• Data Server (dserver)

• Workload setup (wcalc)

• Post-processing scripts (postproc.pl)

www.pha.com.au

The Test Harness

• The test harness consists of a directory containing the C source code for the test framework and each of the robots.

• It also contains:– a Makefile (which links in librte.so and libdcl.so)– Various text files used to configure each run

(.regrc, .rterc, Parameters, SeqNo, RTE, Offset, Host, etc.).

– A directory (or symbolic link to one) in which the tree of log files and other test output are stored.

– Symbolic links into the log directoy of the current and last run. (These are set up by the start script)

www.pha.com.au

The Controller

• The controller is a C executable, regulate, which takes a large number of arguments many of which are passed down into the individual robots.

• It is typically invoked using the start script which crafts these arguments automagically using various config files in working directory (SeqNo, Offset, RTE, Host, etc.).

www.pha.com.au

Controller Operation

• On startup the controller (regulate) reads .regrc file which controls rate of start up.

• It establishes shared memory and semaphores used to provide communication between robots, regulate and review (the viewer).

• It enters a loop which fork/execs the child processes and overlays the executable specified in ‘RTE’

• Once all child processes are initiated it falls through into a loop where it executes wait() until all children have terminated and been cleaned up.

• The controller can optionally restart children as they terminate if desired. This is specified in .regrc file.

www.pha.com.au

Test Monitoring

• Tests are monitored by the review program which reads the global shared memory segment to access status information written by each running robot.

• Review uses a ps command in the background to establish the pid of the version of regulate it should monitor.

• If multiple tests are running the pid of the controlling regulate to be monitored may be specified on the command line.

www.pha.com.au

review

• The command line interface within review mimics that used by vi.

• A text help file is provided which summarises commands available.

• review also provided access to log, stdout, stderr and trace files of running robots.

www.pha.com.au

The Data Server

• The data server uses System V IPC (queues) to pass data from a single server to each of the robots (initiated by each robot).

• Data sources are CSV (text) files customized for the tests.

• Typically CSV files will be crafted from output of SQL on underlying database but may be hand crafted (say by client personnel using Excel)

www.pha.com.au

Data Server Operation

• Starting it up:[plh@deneb plh]$ dserver[dserver] Server working directory is /u/pha/plh/DATAStale dserver pid file![dserver] Server pid is 2227Used 0 MByte of malloc space

• Shutting it down:[plh@deneb plh]$ dserver -T[dserver] Terminating server with pid, 2227[plh@deneb plh]$ [shutdown] dserver shut down Wed Apr 16 15:28:49 2003

• On shutdown unused data in memory is flushed to disk.

• Makefile is used to reset state of data files from master copies matched to databases of system under test (SUT).

www.pha.com.au

Data Server Configuration

• Data server files are contained in a directory which should be defined in the environment as $DSERVER_DIR.

• Configuration is contained in dserver.ini in that directory.

• Data is stored in .dat files, used data in .used files and the master copies in .master files.

www.pha.com.au

dserver.ini

• Format of file is as follows:[plh@deneb DATA]$ cat dserver.ini [Data]Description=Address,CSV#Description=Name,CSV#Description=Note,CSV#Description=UniqNo,Sequence#Description=WhsItem,Keyed#Description=Keyed,Keyed#Description=Indexed,IndexedDescription=CreditorId,CSVDescription=CredActivity,CSVDescription=ItemId,CSVDescription=ItemQryTimes,CSVDescription=ProdCode,CSV…Description=FinCredATB_Out,CSVDescription=FinGLAcctInq,CSVDescription=FinGLAcctInq_Out,CSVDescription=FinBankRecSR,CSVDescription=FinBankRecSR_Out,CSV

www.pha.com.au

Workload Generation

• The various load scenarios are converted into workload files which completely choreograph each run by the workload calculator (wcalc).

• This is a UNIX command line tool which takes a configuration file and produces a workload file. (eg. wcalc –c BuyerLookupPPCE)

• The workload file is read by each of the robots when they start to extract the tasks to be performed and their start-times.

www.pha.com.au

Workload Definition File(eg Scen01_3600.wcd)

## An initial mean load workload definition## Date: Tue Jun 4 12:41:32 EST 1996

[Config]NoUsers=40Duration=3600 # secondsStaggerDelay=0 # secondsRestartTimer=FALSE

[Threads]# Name,NoSessions,NoTx,GivenCycleTimeThread=BuyLookupPPCE,16,3200,0,120Thread=BuyLookupPPCE_PF6,5,3000,0,120Thread=BuyLookupPPCE_PF7,5,3000,0,120Thread=BuyCyberSADD,5,1800,0,120## Very quick financial reports#Thread=FinAPCredEnq,1,600,0,120Thread=FinChqListing,1,240,0,120Thread=FinCredATB,1,600,0,120Thread=FinGLAcctInq,1,120,0,120Thread=FinGLTBF1,1,120,0,120Thread=FinMiscCRDR,1,120,0,120## Long reports#Thread=FinCyberGLTR,1,10,0,120Thread=BuyBuyerPOR,1,10,10,120Thread=BuyCyberPOR,1,20,20,120

www.pha.com.au

Workload File(eg. Scen01_3600.wld)

## Generated: wcalc 2.0 - Thu Feb 6 10:08:58 2003## Duration: 3600 seconds# No Sessions: 40# No Working: 40# Scale Factor: 1## Thread Sessions TPH TPR TPU Delays Start Range Delta# ====== ======== === === === ============== ===== ===== =====## BuyLookupPPCE 16 3200 3200 200 17.4 0 120 0 3480 1.1# BuyLookupPPCE_PF6 5 3000 3000 600 5.8 0 120 0 3480 1.2# BuyLookupPPCE_PF7 5 3000 3000 600 5.8 0 120 0 3480 1.2# BuyCyberSADD 5 1800 1800 360 9.7 0 120 0 3480 1.9# FinAPCredEnq 1 600 600 600 5.8 0 120 0 3480 5.8# FinChqListing 1 240 240 240 14.5 0 120 0 3480 14.5# FinCredATB 1 600 600 600 5.8 0 120 0 3480 5.8# FinGLAcctInq 1 120 120 120 29.0 0 120 0 3480 29.0# FinGLTBF1 1 120 120 120 29.0 0 120 0 3480 29.0# FinMiscCRDR 1 120 120 120 29.0 0 120 0 3480 29.0# FinCyberGLTR 1 10 10 10 348.0 0 120 0 3480 348.0# BuyBuyerPOR 1 10 10 10 347.0 10 120 10 3470 347.0# BuyCyberPOR 1 20 20 20 173.0 20 120 20 3460 173.0# --- ---- ----# 40 12840 12840#0000, 0000, BuyLookupPPCE0000, 0017, BuyLookupPPCE0000, 0034, BuyLookupPPCE0000, 0052, BuyLookupPPCE

www.pha.com.au

Outputs from Robots

Each robot produces four output files:– Log file NNNN.log– Stdout NNNN.stdout– Stderr NNNN.stderr– Trace file NNNN.trc

www.pha.com.au

Log File Format

The log file encapsulates:– Benchmark start and end times– Task elapsed times– Query response times

www.pha.com.au

Log file sample

******** S Mon Feb 17 16:10:51 2003 [1045458651] RTE version 3.2.12

-------> I Identifier "99_40"

-------> I SessionIdx 9 RangeOffset 0

0.000 U Concurrency = 10

0.000 U Display = 0

0.000 U DelayMode = 1

0.000 U UseCycleTimes = 0

0.000 U UseStaggerDelay = 0

0.000 U LoopMax = 1000

0.000 U MaxWait = 240

0.000 U FixedSequence = 1

0.000 U SendRate = 0

0.000 U StaggerDelay = 1

0.000 U TimeoutRestart = 0

0.000 U TimerRestartInterval = 300

0.000 U Tracing = 0

0.004 U **** "170203"

0.004 U **** Fixed sequence initialization (Seed = 9)

0.006 U Adding "BuyLookupPPCE" (idx 12) start 32 sec

0.006 U Adding "BuyLookupPPCE" (idx 12) start 90 sec

www.pha.com.au

Log file sample (Cont) 0.012 U Workload file "Scen04_3600.wld" contains 60 threads. 0.012 U Telnet to "192.168.1.1" 0.062 M 0.000 0.050 0.062 0.062 S "t01" 0.062 S "^J" 0.101 M 0.000 0.039 0.039 0.102 S "tst1ng" 0.102 S "^J" 0.156 M 0.000 0.054 0.054 0.156 S "Ush^M" 0.210 M 0.000 0.054 0.054 0.210 S "^[[M" 0.248 M 0.000 0.038 0.038 0.248 S "FSMENU^M" 0.309 M 0.000 0.061 0.061 51.920 U %Benchmark start at 51.9 sec 51.921 U Idle ... 56.921 U [nextTask] currentThread 12 Task 0 56.921 U Task 0 sleeping with delay 27 83.923 U checkRunning() 83.923 U CMD_RUN

www.pha.com.au

Log file sample (Cont) 83.923 U %Start BuyLookupPPCE at 32.0 sec 83.923 S "^[[P" 83.927 M 0.000 0.004 0.004 83.927 S "^[[T" 83.966 M 0.000 0.039 0.039 83.966 S "^[[O" 84.172 M 0.000 0.206 0.206 84.172 U Looking up "21820328"^M 84.172 S "21820328" 84.172 S "^M" 84.211 M 0.000 0.039 0.039 85.595 M 0.000 1.384 1.423 85.595 U %Query BuyLookupPPCE LookUp 1.4 sec 85.595 U 21820328,1423.0 99.595 S "^[[d" 99.614 M 0.000 0.018 0.019 99.614 S "^[[d" 99.663 M 0.000 0.049 0.049 99.663 S "^[[d" 99.702 M 0.000 0.039 0.039 99.702 S "^[[d" 99.741 M 0.000 0.039 0.039 99.741 U %Finish BuyLookupPPCE (OK) at 47.8 Proc 15.8 Elap 15.8 CSD 0.0 CPD 14 99.742 U [nextTask] currentThread 12 Task 1 99.742 U Task 1 sleeping with delay 43 142.743 U checkRunning() 142.743 U CMD_RUN 142.743 U %Start BuyLookupPPCE at 90.8 sec

www.pha.com.au

Starting a Run

• Use the script start.sh to initiate run.[plh@deneb REJECTSHOP]$ ./start.sh 20Log files will be written to directory results/192.168.69.34/104_20Starting run 104 on hostname 192.168.69.34 at rate 1Duration of run will be 1800 secondsTaking 60 samples at intervals of 30 secondsRunning "regulate -h 192.168.69.34 -e sim -r 1 -W TstThread0.wld -w

results/192.168.69.34/104_20/logs -i 104_20 -D 1800 -o 0 20" Regulate Process PID 2305Logfile Generation ...

• Run the review executable (in working directory).• Wait until all robots have paused (P).• Type ‘:S’ (Start) to initiate test.• Terminate by typing ‘:E’ (End) – robots will complete current

task and then exit gracefully.• Abort by typing ‘:X’ (eXterminate) – robots all terminate on

signal.

www.pha.com.au

Review Screen

Review is a curses application whose screen looks like this:

[V3.2.12] Controlling PID: 2535 Run Duration: 1800 User: plh

Started ramp up at: Wed Apr 16 16:04:54 2003 Elapsed: 00:00:25 --

Sessions: started 20 finished 0 failed 0 concurrent 0 of 20

Ident: 105_20

Sess: 0 1 2 3 4 5 6 7 8 9

0 5P 5P 5P 5P 5P 5P 5P 5P 5P 5P

10 5P 5P 5P 5P 5P 5P 5P 5P 5P 5P

For each session (robot) the number (5) is the send/match count and the letter (P) is the status code. This is set by rteSleep() call in robot [e.g. rteSleep( 15, ‘C’);].

www.pha.com.au

review (once started)

Once running review will display progress:

[V3.2.12] Controlling PID: 2535 Run Duration: 1800 User: plh

Started ramp up at: Wed Apr 16 16:04:54 2003 Elapsed: 00:04:27 S-

Sessions: started 20 finished 0 failed 0 concurrent 0 of 20

Ident: 105_20

Sess: 0 1 2 3 4 5 6 7 8 9

0 13A 13A 13A 5i 13A 5i 13A 5i 5i 5i

10 5i 5@ 5@ 5@ 5@ 5@ 5i 5i 5i 5i

www.pha.com.au

Post-Processing

The running robots each write elapsed and query times into a log file. These files are post processed to produce analysis for reports.The post-processing phase produces:

Histograms of query response time and task elapsed time.

CSV versions of the above data which is folded into an Excel spreadsheet using a macro.

Various graphs (in PostScript using GnuPlot which are converted into PDF files for distribution).

www.pha.com.au

Post-processing Scripts

• Historically a combination of shell and Perl scripts have been utilized for rehashing the log files to produce.

• Most recently Perl has been used in conjunction with GnuPlot and Acrobat.

www.pha.com.au

Response Time Histogram

***** BuyLookupPPCE_PF6-PF6 **************************************

No of queries: 580

Mean: 0.23 sec

Standard Deviation: 0.81

Variance: 0.65

Median: 0.0 sec (no = 280)

Mode: 0.0 sec (no = 280)

Fraction at mode: 0.483 (Mode/Total)

Percentile Distribution:

50% 60% 70% 75% 80% 85% 90% 95% 99%

Time: 0.0 0.1 0.1 0.1 0.1 0.1 0.1 0.2 0.8

Query Response Time Distribution:

Time: 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Freq: 280 216 29 2 3 3 3 5

Time: 0.8 0.9 1.0 1.1 1.2 1.3 1.4 1.5

Freq: 6 3 2 1 3 2 1 1

Time: 3.8 3.9 4.6 5.2 5.8 6.8 8.3 8.6

Freq: 1 1 1 1 1 1 1 1

www.pha.com.au

Analysis

Typically the analysis is performed in conjunction with tools that report on system performance during the test runs:– sar, iostat, netstat, etc.– Various shell and Perl scripts wrapped

around sar– TeamQuest

www.pha.com.au

Reporting

Reports typically consists of two parts– Overall system performance & capability at

various workload levels (scenarios).– Throughput and response time at various

load levels (scenarios).

www.pha.com.au

Conclusions

• Labour intensive.• Complex.• Fun (if you like breaking systems and

software).• Useful for leveling the playing field in

system acquisitions.• Useful for determining load limit and

throughput thresholds and bounds.