|
The Seven Habits of Highly Effective (SAP/ERP) Test Management
Pepijn Paap Senior SAP Test Manager Business Line SAP Solutions
Utrecht, October 2016
2
|
SAP HANA TEST Acceptance Criteria
# Exit Criteria Responsible Results (Example)
1 For all functionality, at least, the happy flow (Positive Tests) is tested and approved. Test Manager OK
2 For all critical and high risks specified in the test strategy also negative flows are created, tested and approved. Test Manager OK
3
When User Acceptance Test is finished and test cases are completely executed, no more defects are available with impact:
• High (Blocking, Critical);
• Maximum of 5 Medium;
• Maximum of 25 Low (Cosmetic).
Test Manager
Project Manager
NOK
High = 2
Medium =8
4
For User Acceptance Test an overall percentage of executed test cases is 98%.
• Completeness of test scripts in risk class critical and high is 100%
• Completeness of test scripts in risk class medium is at least 80%
• Completeness of test scripts in risk class low is at least 60%
Test Manager
OK
100%
5 For User Acceptance Test a percentage of positive executed test cases is 95%. Test Manager
OK 97,5% without postponed
97,8% with postponed
6
To finalize UAT phase, the following indication for open defects has to be taken into account:
• Postponed to next release (Wish/ Change)
• Accept (Known-Error) by user organization
• Don’t accept included with reason and remarks (Reject)
• Housed in a different manner to enable the system is able to continue to the next stage (in Software Development Life Cycle).
Test Manager
Project Manager
NOK
Not for all remaining issues a
follow up action is defined
7 A documented work-around is available for Known-Errors. Test Manager N.A.
|
Integral international Test Process
6 Sogeti PowerPoint Referentie 2014
Blueprint/ Funct Specs
Process Flows
Requirement
Knowledge transfer
Replicate KT Test
Case Design
Review
Video Capture Training
Test Automation
Capture
RQ coverage
Test Automation
Test Execution Manual
Test Execution
Automation
Defect Management
Test Closure Test End
Report (Go/ NoGo)
Preparation Specification
Execution Completion
Management & Control & Infrastructure
Management & Control & Infrastructure
Client/ India India India Client India NL
India/ NL India/ Client/ NL India/ NL
Ap
pro
ved
Peer/TA Review
India/NL
Input
Input
Training Team
Client
India/ Client/ NL
Client
Resource cost reduce 25% - 35%
Productivity and reusability increase 25%
|
11 Sogeti PowerPoint Referentie 2014
SAP Process Risk Analyses -Rating Guide Process Complexity
1 Very simple process, main flow + max 2 alternate flows visible to the user (less then 3
Tcodes)
2 Simple process, max 6 alternate flows, max 10 data tables involved (4 to 6 Tcodes)
3 Medium complex algorithm, max 6 alternate flows, max 10 data tables involved (7 to 10
Tcodes)
4 Complex algorithms: Multiple decisions or many dependencies or complex
computations (11 to 15 Tcodes)
5 Very complex (combination of complexity factors above) (more then 16 Tcodes)
Technical Impact
1 Standard SAP, simple parameterizations only
2 Standard SAP, extensive parameterizations
3 Enhancement, simple ABAP, simple
parameterizations and no interfaces
Or data setup that has significant impact
on processing (e.g. basic reference
model setup)
4 Enhancement, simple ABAP, average
parameterizations and a few interfaces
Or processing significant data setup
5 Enhancement, complex ABAP, A lot of
parameterizations and a lot of interfaces
Or high impact (master ) data setup
Business Criticality
1 Limited impact (may be caught up or corrected without impacting related business
processes)
2 Errors will induce significant hours to be
spent on internal corrections or rework
Primary business process not affected
3 Errors are likely to affect primary
business processes (e.g. standstill or
under performance) but impact will be
local and have no external visibility
Or are likely to have internal financial
impact, but financial impact limited to
single transactions
4 Errors may lead to significant financial
losses (normally on local scale)
Or may have external visibility
5 Errors may have structural negative impact on profitability or company image
Frequency of use
1 Used incidentally / on a monthly basis or less
2 Used on Weekly basis Or monthly basis and over 10% of all
users involved
3 Normal use is on a daily basis
4 Used continually (during office hours or normal productive hours)
5 Heavy duty: continual usage by a significant number of users
|
What is PRACTICES
SAP specific characteristics classified to points of intrest
12
• Processes
• Reports
• Authorizations
• Customizations
• Transports
• Interfaces
• Conversions
• Enhancements
• Screens
Relation with: • Quality Attributes • Test base • Test phase (and environment) • Test specification technique
P
R
E
C
C
A
T
I
S
PRACTICES “Successful SAP test
Approach”
|
PRACTICES “Successful SAP test approach”
SAP Best Test PRACTICES 13
Motivation Preventive Measures
(Static Testing) Quality
Attribute Test Base Where
Coverage Based (Formal)
Experienced Based (Informal)
Processes Business processes are leading! Review Requirements,
Process flows, Use cases
Functionality
Continuity
Business Blue Print
Process Models
Functional Design
FAT/ UAT / RT Process Cycle Test
Data Comb. Test
Error guessing
Exploratory testing
Reports Everything that is being sent of the
customer has to be 100% OK Review Requirements
Functionality
Performance Functional Design
UT/ SIT / FAT/
UAT/ PAT Data Comb. Test
Error guessing
Exploratory testing
Check list
Authorizations Authorizations have a important role in
SAP process handling
Review Business Blue Print,
Authorization matrix
Security
Suitability
Business Blue Print
Authorization matrix FAT/ UAT / PAT Process Cycle Test
Error guessing
Exploratory testing
Check list
Customizing Understand the intended behavior that is
achieved through customizing
Review Business Blue Print,
Installation doc Functionality
BBP & BPM Installation doc
UT/ SIT / FAT/
UAT / PAT
Process Cycle Test
Data Comb. Test
Error guessing
Exploratory testing
Transports A mistake in a transport can cause
serious issues
Review naming convention
for transports Functionality Functional Design
UT/ SIT / FAT/
UAT/ PAT
Error guessing
Exploratory testing
Check list
Interfaces Testing interfaces is a multi discipline
activity
Review func. & technical
design, Interface doc
Functionality
Connectivity
Functional Design
Interface doc UT/ SIT / FAT/
UAT/ PAT
Process Cycle Test
Data Comb. Test
Error guessing
Exploratory testing
Check list
Conversions Verify (early) if the processes can handle
the converted data
Review func. & technical
design Functionality
Functional Design
Technical Design
SIT / FAT/ UAT /
PAT Data Comb. Test Check list
Enhancements Enhancements require extra test
attention and effort
Review func. & technical
design
Functionality
Suitability
Functional Design
Technical Design
UT/ SIT / FAT/
UAT/ PAT
Process Cycle Test
Data Comb. Test
Error guessing
Exploratory testing
Check list
Screens SAP Screens can be heavily modified Review func. & technical
design User-Friendliness
Functional Design
Technical Design SIT / FAT/ UAT Check list
| 14
Energy & Utility Automotive Manufacturing Consumer
Product Retail Life Science Telecom Distribution Insurance Media &
Entertainment Aviation
Banking
5 6 33 33 4091 100
141 158 183 198 198
330373 377
471
633711
807866
905954 964
1005 1006 1010 1026 1037 1042 1048
0 11 1 1
2 22 2
2 2 2
22 2
2
2
2
2
23
3 33 3 3 3 3 4 4
0 00 0 0
0 00 0
0 0 0
00 0
0
0
0
0
00
0 00 0 0 0 0 0 0
0 00 0 0
0 00 0
0 0 0
00 0
0
0
0
0
00
0 00 0 0 0 0 0 0
273 273309 309 309
392433
486519
542581
620
695739 748
869
1005
1140
1226
1287
1340
1404 1404 1404 1404 1419 1419 1419 1419 1419
0
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
1400
15006
-ju
n
7-j
un
8-j
un
9-j
un
10
-ju
n
13
-ju
n
14
-ju
n
15
-ju
n
16
-ju
n
17
-ju
n
20
-ju
n
21
-ju
n
22
-ju
n
23
-ju
n
24
-ju
n
27
-ju
n
28
-ju
n
29
-ju
n
30
-ju
n
1-j
ul
4-j
ul
5-j
ul
6-j
ul
7-j
ul
8-j
ul
11
-ju
l
12
-ju
l
13
-ju
l
14
-ju
l
15
-ju
l
Test
Sce
nar
ios
Test Case Execution Plan vs Actuals
POSTPONED
OBSOLETE
BLOCKED
STARTED
FAILED
PASSED
Planned Test Cases
Executed Test Cases
|
Tips for tomorrow
16
Summary 1. Raise the bar
2. Continuous learning
3. Find talent
4. Always see challenges
When you master the hard skills in combination with the soft skill, you master the testing
game. And as soon as you master the game, the stakeholders are changing the rules
Looking Ahead Implement a SAP Product Risk Analysis (PRA)
Use the right expertise and methodology (PRACTICES)
HANA migration – Custom code and integration
Cloud migration – Integration
SAP Stakeholder management
Choice the battles you want to win
Reduce resource costs and increase productivity using different
testing models
5. Be fast
6. Be efficient
7. Be critical