DB-5: Simulating User LoadTom BascomVP TechnologyWhite Star [email protected]
DB-5: Simulating User Load - 2
Sanity Check!
This Presentation is NOT about simulating your user interface.
We are going to discuss a method for creating a database load that mimics a target usage profile.
DB-5: Simulating User Load - 3
The Challenge
A Volatile Business Environment System Load expands to consume all
resources… Something is Always changing…
Fighting Misconceptions:– Load testing is too difficult.
– Load simulation tools are expensive.
– Load testing tools don’t support Progress.
DB-5: Simulating User Load - 4
Why Simulate User Load?
Regression Testing Finding the Limits / Capacity Planning Root Cause Analysis Minimizing Cost / Optimizing Resources Quality of Service (QoS) Assurances To support other testing efforts more
realistically
DB-5: Simulating User Load - 5
Considerations of Fidelity
Assumptions Characterizing the Load Where are the Bottlenecks? Users are fickle… How do we (realistically) drive test scenarios?
(Or: Are we really simulating user load?)
DB-5: Simulating User Load - 6
Assumptions
We are primarily interested in the DATABASE. READ activity is, by far, the largest consumer
of database resources. WRITE activity is also important. An ABL session is an ABL session… (in terms
of resource use)
The functionality of the application is not being tested (by these tests).
DB-5: Simulating User Load - 7
Characterizing the Load
Historical data collection is a must. Application Metrics
– Orders Taken– Widgets Produced
Baseline db Performance– Logical IO ops/sec– Disk IO ops/sec– Latch & Resource Waits– TRX, Record Locks & Blocked Clients
IO by User TableStats
DB-5: Simulating User Load - 8
Where are the Bottlenecks? Database
– Latch contention
– Transaction throughput Operating System
– Disk
– CPU
– Memory
– Network
DB-5: Simulating User Load - 9
Users are fickle…
They work on different things… at different rates … pretty much randomly … and sometimes at their leisure ;-)
Our best friends are:– IO by User (PROMON, ProTop)
– _TableStat (VST, ProTop)
DB-5: Simulating User Load - 10
IO Requests by User
0
50
100
150
200
250
300
350
0 10 20 30 40 50 60 70
DB-5: Simulating User Load - 11
IO Requests by Table
0
1000
2000
3000
4000
5000
Reco
rds/s
ec
Ord
er-
Lin
e
Ord
er
Invoic
e
Custo
mer
Item
Sta
te
Ref-
Call
Local-D
efa
ult
Sale
sR
ep
Table
DB-5: Simulating User Load - 12
IO Requests by Table
Create Read Update Delete
5,940,439 1,662,688,907 2,540,165 2,500,292
0.35% 99.34% 0.15% 0.15%
83.8/s 23,454.5/s 35.8/s 35.3/s
Table Name Created % Read % Updated % Deleted %
CHART_ENTRY 0 0.0 1,061,326,658 63.8 0 0.0 0 0.0
BUREAU_QUEUE 0 0.0 241,087,053 14.5 25,150 0.9 0 0.0
APP_INFO 4,711 0.0 82,913,516 4.9 213,944 8.4 26 0.0
BTE_QUEUE 0 0.0 27,612,013 1.6 14,312 0.5 0 0.0
SCORE_EXPWGHT 0 0.0 20,349,269 1.2 0 0.0 0 0.0
APP_REQUEST 4,711 0.0 13,491,724 0.8 45,119 1.7 668 0.0
APP_TRADE 105,260 1.7 11,499,880 0.6 143,465 5.6 20,672 0.8
APP_OPTION 23,791 0.4 11,341,471 0.6 50,809 2.0 9,614 0.3
APP_TRANSTAGG 3,347,503 56.3 8,603,941 0.5 0 0.0 1,919,357 76.7
DB-5: Simulating User Load - 13
Data Distribution
0
250
500
750
1000R
eco
rds
Ord
er-L
ine
Ord
er
Invo
ice
Cus
tom
er
Item
Sta
te
Ref
-Cal
l
Loca
l-Def
ault
Sal
esR
epTable
DB-5: Simulating User Load - 14
Idle Users?
Current Users: 192 Idle Users: 0:01 18 0:02 8 0:03 10 0:04 4 0:05 4 0:06 2 0:07 4 0:08 4 0:09 1 10m - 1hr: 51 Hour+ old: 32
DB-5: Simulating User Load - 15
How do we drive test scenarios?
With some ABL code that reads and writes records as the application does.
Distributed among many sessions. Profiled according to TableStats.
DB-5: Simulating User Load - 17
WARNING
Using the CREATE, UPDATE or DELETE routines will modify your database!
Use these destructive options only with test copies of your database.
Use the READ-ONLY option with production databases!
DB-5: Simulating User Load - 18
A Simulation Controller
You already have everything that you need… A generic Load driver Tuning the driver to your Load Controlling test runs
DB-5: Simulating User Load - 19
Control Program (pacectl.p)
cmd = “mbpro &1 -pf pace.pf -param &2 -U &3".
input from value( "io.usr" ).
do while true:
import io_usr nm_usr.
xcmd = substitute( cmd, pdbname(1), io_usr, nm_usr ).
message xcmd.
if opsys = "unix" then
os-command silent value( xcmd ).
else
run spawn( input xcmd, input "", output pid ).
pause 1.
end.
DB-5: Simulating User Load - 20
Config File (io.usr)
0,500,50,0 jami
0,100,10,0 tucker
0,50,5,0 jami
0,40,4,0 julia
0,30,3,0 peter
0,20,2,0 emily
DB-5: Simulating User Load - 21
pace.p
do while true:
x = random( 1, pickLimit ).
find first tt_pace where tblPick >= x.
run xread( tblName, r_rate ).
t = time - s.
do while (( i / t ) > r ):
pause 1 no-message.
t = time - s.
end.
end.
DB-5: Simulating User Load - 22
xread()
create buffer bh for table tblName.qh:set-buffers( bh ).qh:query-prepare( “for each “ + tblName ). j = random( j / 2, j * 10 ).do while k < j: qh:get-next( no-lock ) no-error. if qh:query-off-end then do: qh:get-first( no-lock ) no-error. next. end. k = k + 1.end.
DB-5: Simulating User Load - 24
Selected Case Studies
Benchmarking OpenEdge 10.0– PSDN Whitepaper:
http://www.psdn.com/library/entry!default.jspa?categoryID=256&externalID=876
Ensuring Disk Array Performance Verifying a Failover Plan Validating Parameter Changes Providing a background load for development
or testing environments. Supporting Demos ;-)
DB-5: Simulating User Load - 25
Benchmarking OE10
PSDN Whitepaper:
http://www.psdn.com/library/entry!default.jspa?categoryID=256&externalID=876
Problem:– How Does OE10 Compare to Progress 9?
DB-5: Simulating User Load - 26
Ensuring Disk Array Performance
Problem:– Will a new disk array configuration support
the production workload? Several previous attempts to go live had
failed. Load simulation was used to prevent a
repeat of that fiasco. Problems were found and solutions were
verified and confidence was repaired prior to a successful implementation.
DB-5: Simulating User Load - 27
Verifying a Failover Plan
Problem:– Redundant (and expensive) systems were
purchased and configured for mutual failover. A method to realistically test a failover with
significant load on the hardware was needed.
The failover worked very well! But the “fail back” failed miserably As a result a serious OS bug was identified
and fixed and downtime avoided.
DB-5: Simulating User Load - 28
Validating Parameter Changes
Problem:– Changing many Progress and OS parameters
can result in unanticipated limits being reached.
# of file handles, semaphores, memory size etc…
Solution:– Use the load simulator to start sessions and
stress these limits.
DB-5: Simulating User Load - 29
Background Load
Problem:– Developers and testers are often working on
over-powered servers with no competition for resources.
– Which leads to a false sense of good performance when testing.
Solution:– Run load simulator to provide a continuous
background load.
DB-5: Simulating User Load - 30
Demos!
Problem:– Single user demos are boring.
Solution:– Create some fake users that do interesting
things.
DB-5: Simulating User Load - 31
Summary
Basic Load Simulation is easier than you think!
And doesn’t necessarily require fancy (and expensive) tools.
It does require fore-thought and planning. And can be immediately useful to you!