Post on 08-Jun-2020
transcript
Tech Talk: Performance benchmark
Tech Talk: Performance benchmark
Key benchmark deliverables➢ Performance benchmark report
➢ Issues detected/fixed in each
iteration
➢ Optimizations performed in
each iteration
Figure: sample performance benchmark report
Number of
times executed
Number of times
executed
Number of times
executed
Avg. Test Time
(sec)Avg. Test Time (sec) Avg. Test Time (sec)
(throughput for
iteration 1)
(throughput for
iteration 2)
(throughput for
iteration 3)(iteration 1) (iteration 2) (iteration 3)
994 executions 1470 executions 194 seconds = ~3 minutes 1 minute 52 seconds
From iteration 1: +190%From iteration 2:
+47%From iteration 1: -64% From iteration 2: -42%
Total gain: +329% Total gain: -79%
951 executions 1471 executions 187 seconds = ~3 minutes1 minute 44
seconds
From iteration 1: +182%From iteration 2:
+54%From iteration 1: -65% From iteration 2: -44%
Total gain: +336% Total gain: -81%
890 executions 1430 executions 180 seconds = 3 minutes 1 minute 44 seconds
From iteration 1: +142%From iteration 2:
+60%From iteration 1: -62% From iteration 2: -42%
Total gain: +289% Total gain: -78%
935 executions 1497 executionsSubmit: 2.0
secondsSubmit: 2.0 seconds End-to-end flow: 9 seconds
From iteration 1: +161%From iteration 2:
+60%
End-to-end flow:
21.0 secondsEnd-to-end flow: 12.0 seconds From iteration 2: -25%
Total gain: +318% From iteration 1: -42% Total gain: -57%
N/A N/A N/A N/A
(run in isolation) (run in isolation) (run in isolation) (run in isolation)#9 Price verification 1461 5 seconds End-to-end flow: 45,7 seconds
#12 Sales order (flow A-1) 368 executions 3 minutes 469 = ~8 minutes
#10 Stock verification 358 executions 5 seconds
541 seconds = ~9
minutes
#8 Sales order normal flow 337 executions 3 minutes 534 = ~9 minutes
Scenario
IDScenario
Maximum
acceptable target
from business
#11 Sales order (flow A-2) 342 executions 3 minutes
Tech Talk: Performance benchmark
- Focus on the performance
requirements
- Keep the objectives reasonable
- Document in BRD/FRD
1. Define
performance
objectives
2. Define
scenarios
3. Design and
develop
system
6. Configure
test
environment
5. Build test
simulations
and data
4. Design
tests
7. Execute
tests
8. Tune and
Retest
iteratively
9. Report
- Focus on all critical components
- Keep them close to reality
- Document in BRD/FRD
- Keep goals and objectives in
mind
- Analyze traces and fix issues
- Maintain constant checks on
progress
User1 loginSO
Headerwait Line wait Post
User2 loginPO
Headerwait Line wait Post
User3 login WaitGL
Headerwait Line wait Post
User4 login waitPO
Headerwait Line wait Post
User5 loginSO
Headerwait Line Line Line Line
User6 login waitGL
Headerwait Line wait Post
User7 loginSO
Servicewait
SO
Servicewait SO Service
ConcurrencyLoad pattern
Transaction Mix
Pacing
Network/Client
Narrow it
down
Troubleshoot
Develop the
solution
Test the
solution
Success or
next iteration
ACTIVITY PARTNER (sample) CUSTOMER (sample)
Define the target/projected business goals I AR
Define the detailed benchmark scenarios RIC A
Take task recordings and document the reproduction steps I AR
Provide the environment artefacts (code build and database to use) I AR
Build the benchmark environment R A
Create test scripts and data scripts R A
Execute the performance benchmark R A
Deliver the performance benchmark report R A
In case of bugs in the standard solution, open a support request to Microsoft C AR
Tune performance (outside Microsoft's scope, e.g. configuration and
customizations)R A
CAPTION R: Responsible A: Accountable C: Consulted I: Informed
Tech Talk: Performance benchmark
the environments
planning Tech Talk
Tech Talk: Performance benchmark
1. Define and
capture scenarios
2. Environment
preparation
3. Create and
validate scripts with
single user
4. Convert to multi-
user tests
5. Performance test
with incremental
workload
6. Conclude and
report
Tech Talk: Performance benchmark
Main article on Docs for PerfSDK
Tech Talk on the performance testing approach
Tech Talk on the troubleshooting tools
Tech Talk on the patterns / anti-patterns
Tech Talk on the RSAT
Considerations for retail performance testing