Date post: | 30-Mar-2015 |
Category: |
Documents |
Upload: | jackson-vaughn |
View: | 213 times |
Download: | 0 times |
SoftTest Ireland Copyright 2008 ElectroMind Ltd 1
People Challenges for Test Teams
Stephen K. Allott
SoftTest Ireland Copyright 2008 ElectroMind Ltd 2
SoftTest Ireland Copyright 2008 ElectroMind Ltd 3
SoftTest Ireland Copyright 2008 ElectroMind Ltd 4
SoftTest Ireland Copyright 2008 ElectroMind Ltd 5
Introductions
My background Aims of this workshop
Explain some common testing challenges based on recent experience
Ask for your experiences Agree a list of challenges
we all face as testers Try to come up with some
quick wins & longer term approaches to solutions
SoftTest Ireland Copyright 2008 ElectroMind Ltd 6
My recent clients
Investment bank Airline web site Travel company web site Mobile phone operator Retail web site Travel company back office Recruitment web site
Includes assignments done in association with TestHouse (London) and Neueda Technologies (Antrim & Dublin)
SoftTest Ireland Copyright 2008 ElectroMind Ltd 7
Their technologies
Microsoft Dot (.) NET 1/2/3 Microsoft VSTS / TFS Oracle MS SQL Server Windows XP Citrix Metaframe Quality Centre / QTP
SoftTest Ireland Copyright 2008 ElectroMind Ltd 8
Number of testers
Typically 12 to 14 in the test team A couple of clients had only 1, some have none!
SoftTest Ireland Copyright 2008 ElectroMind Ltd 9
Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13
Key Area Controlled Efficient Optimising
Test strategy A B C D
Life-cycle model A B
Moment of involvement A B C D
Estimating and planning A B
Test specification techniques A B
Static test techniques A B
Metrics A B C D
Test automation A B C
Test environment A B C
Office environment A
Commitment and motivation A B C
Test functions and training A B C
Scope of methodology A B C
Communication A B C
Reporting A B C D
Defect management A B C
Testware management A B C D
Test process management A B C
Evaluation A B
Low-level testing A B C
Composite TPI Test Maturity Matrix from several clients
SoftTest Ireland Copyright 2008 ElectroMind Ltd 10
Summary of TPI results
No documented test process No formal test policy, strategy or plan Many tests, no formal design No metrics No proper test environment
Some tested in “live”
No documented test basis Not involved early enough
SoftTest Ireland Copyright 2008 ElectroMind Ltd 11
Some common observations
Constantly changing business priorities Increasing (frantic) demand for new functionality Weekly, sometimes daily release cycles Attempt to use automation to solve all the problems No numbers to explain testing’s added value Small important teams yet with little influence
SoftTest Ireland Copyright 2008 ElectroMind Ltd 12
People challenges – behaviour
Motivation A few are resigned to how it is, no hope? Some expect spoon feeding Some adopt a negative attitude Many are willing to make improvements, if there was time
Turnover – very high offshore Culture
Some managers get too involved in detail Co-dependency – testers must push back It hasn’t broken yet so . . .
SoftTest Ireland Copyright 2008 ElectroMind Ltd 13
People challenges – skills
Testing SkillsProcessesStandardsTechniquesTools
IT SkillsProgrammingDatabasesInfrastructureWeb
Domain KnowledgeIndustry specificApplication knowledgeE-Business
Soft SkillsCommunicationNegotiationManaging technical teams
SoftTest Ireland Copyright 2008 ElectroMind Ltd 14
People challenges – perception
In general testers are not valued Testers are often blamed
Project delays Poor quality Managing expectations with RBT
Lack of awareness of what the test team does
SoftTest Ireland Copyright 2008 ElectroMind Ltd 15
People challenges – managers
Demand for certified staff Demand for skills & knowledge Pressure to cut costs and use offshore resources Developing good test leads Role of testers in an agile project Selling the value of testing to the organisation
SoftTest Ireland Copyright 2008 ElectroMind Ltd 16
Themes
Facilitated discussion bearing in mind the following themes:
1. Relentless business pressure for new functionality2. Constant and last minute changes to requirements3. Introduction of new development / testing methodologies e.g. agile 4. Very frequent releases, often weekly, sometimes daily!5. Increasing application complexity over time6. Knowledge gap (e.g. what lies beyond the browser?)7. Steep learning curve (business as well as technical)8. Use of contract and offshore resources (loss of in-house expertise)9. Little or no documentation against which to test10. Lack of attention to all testing requirements (e.g. middleware, back
end database, non-functional quality characteristics)
SoftTest Ireland Copyright 2008 ElectroMind Ltd 17
Discussion – how do we respond?
Business Pressures
New functionalityFrequent changeMaturity
TechnologyComplexRapid changeInterfaces
OffshoreCheaperPlentifulExpertise
BehaviourMotivationAttitudeCulture
SkillsAnalysisIT / TechnicalBusinessTestingSoft skills
ManagersCostsTime to marketAgile
SoftTest Ireland Copyright 2008 ElectroMind Ltd 18
Conclusions
In my opinion testers of the future must . . . Measure / appreciate application complexity Be able to accept and manage last minute changes Be flexible but push back with estimates Increase their technical knowledge Learn more about testing, especially techniques Apply the tried and tested techniques Capture metrics to help prove the business case Get involved in the projects much earlier
Recruit / train / certify / develop / coach = Retain Share your views with the group
SoftTest Ireland Copyright 2008 ElectroMind Ltd 19
Final Thoughts / Questions?
SoftTest Ireland Copyright 2008 ElectroMind Ltd 20
People Challenges for Test Teams
+44(0) 773 476 1363
www.electromind.com
SoftTest Ireland Copyright 2008 ElectroMind Ltd 21
People Challenges for Test Teams
Appendix – Bonus Slides
SoftTest Ireland Copyright 2008 ElectroMind Ltd 22
A typical project in trouble
CR Churn by Month - AveCR=47 AveChurn=150 AveChurnCR=3
1427
58 5565
40
116
279
229213
0 0
254
124
263
0
50
100
150
200
250
300
June July August September October
Months
CR
co
un
t
0
50
100
150
200
250
300
Ch
ang
es t
o C
Rs
CRs Churn Bugs
SoftTest Ireland Copyright 2008 ElectroMind Ltd 23
Bug snapshot by priority
Sev 1 Sev 2 Sev 3 Sev 4 Total
Pri 1 57 81 14 152
Pri 2 19 52 12 1 84
Pri 3 6 14 10 30
Pri 4 2 1 3
None 68 288 114 10 480
Total 146 428 154 21 749
SoftTest Ireland Copyright 2008 ElectroMind Ltd 24
Bug snapshot by resolution
Sev 1 Sev 2 Sev 3 Sev 4 Total
By design 10 37 29 4 80
Duplicate 6 33 7 1 47
External 2 16 4 0 22
Fixed 86 254 95 12 447
Not repro 29 56 4 0 89
Postponed 12 17 6 4 39
Will not fix 1 15 9 0 25
Total 146 428 154 21 749
SoftTest Ireland Copyright 2008 ElectroMind Ltd 25
Impact of the improvements
CR Churn by Month - AveCR=47 AveChurn=150 AveChurnCR=3
1427
58 5565
30
61
17 15 10 5
40
116
279
229213
6686
18 15 10 50 0
254
124
263
13
90
12 9 5 10
50
100
150
200
250
300
June Ju
ly
Augus
t
Septe
mbe
r
Octobe
r
Novem
ber
Decem
ber
Janu
ary
Febru
ary
Mar
chApr
il
Months
CR
co
un
t
0
50
100
150
200
250
300
Ch
ang
es t
o C
Rs
CRs Churn Bugs
Events Systems1. Vendor2. User Reg.3. User Shop
Sales and Marketing Systems1. Consulting Practice Mgt.2. Vendor Account Mgt.3. Event Mgt.4. Leads and Referral
Databases1. Profile2. Common3. Event OLTP4. Transaction OLTP5. Contact History OLTP6. Domain Reference
World-wide Marketing DB1. Profile OLAP2. Event OLAP3. Analytical Reporting4. Contract History OLAP5. Domain
Corporate Sales and Marketing Systems
System Interfaces1. Event Orders2. Billing & Financials3. Credit Card (real-time)4. Event Communication / FAX5. Direct email service6. Direct mail Postal interface7. Leads & Referrals8. OLTP Staging Interface9. OLAP Staging Interface10. SAP11. Licensing12. Call Center
Source: Reality Test Inc.
SoftTest Ireland Copyright 2008 ElectroMind Ltd 27
Basic complexity calculation
Characteristic Value Factor Total
Windows 37 8 296
Objects 23 1 23
Tables 47 1 47
Apps Impacted 3 10 30
Factors are weighted from a testing perspective
SoftTest Ireland Copyright 2008 ElectroMind Ltd 28
Comparison across applications
Application UI Back End
Env Test Code
Middle Tier
Total
Sales Mgr. 1360 2250 152 0 0 3762 Call Center 960 824 90 129 0 2003 Web Sales 389 183 109 0 2 681
Middleware component
0 146 23 27 232 428
SoftTest Ireland Copyright 2008 ElectroMind Ltd 29
0
1000
2000
3000
4000
Application Name
Co
mp
lex
ity
Va
lue
Complexity comparison
SoftTest Ireland Copyright 2008 ElectroMind Ltd 30
Actual staffing comparison
0
5
10
15
20
Applications
Nu
mb
er
of
Te
ste
rs
Estimated
Actual
SoftTest Ireland Copyright 2008 ElectroMind Ltd 31
Organisation evaluation
Organisation within the company Process definition and execution Teamwork Ability to do the job
SoftTest Ireland Copyright 2008 ElectroMind Ltd 32
Organisation within the company
Section Item being evaluated Max Score
1.1 Reporting structure 50
1.2 Professionalism 170
1.3 Organisational churn 135
1.4a Staffing – permanence 30
1.4b Staffing – stability 35
1.4c Staffing – morale 50
1.4d Staffing – number of testers 40
1.4e Staffing – organisation type 40
1.5 Budget control 30
SoftTest Ireland Copyright 2008 ElectroMind Ltd 33
Process definition and execution
Section Item being evaluated Max Score
2.1 Software development life-cycle 60
2.2 Design reviews and documentation 70
2.3 Early participation in development 50
2.4 Ability to influence schedule deadlines 60
2.5 Development standards 60
2.6 Defined testing approach & procedures 190
2.7 Metrics reporting 60
2.8 Churn 90
2.9 Patch releases 40
2.10 Post-project reviews 60
SoftTest Ireland Copyright 2008 ElectroMind Ltd 34
Teamwork
Section Item being evaluated Max Score
3.1 Support from production support 60
3.2 Support from management 60
3.3 Correct level of quality 90
3.4 Resource allocation 80
SoftTest Ireland Copyright 2008 ElectroMind Ltd 35
Ability to do the job
Section Item being evaluated Max Score
4.1 Adequate equipment / lab 200
4.2 Automated testing 70
4.3 Programming ability 70
4.4 Non-testing activities 50
SoftTest Ireland Copyright 2008 ElectroMind Ltd 36
Total evaluation score
900
1000
1100
1200
1300
Q1/00 Q2/00 Q3/00 Q4/00 Q1/01 Q2/01 Q3/01
Date
Eva
lua
tion
Total Evaluation Score
SoftTest Ireland Copyright 2008 ElectroMind Ltd 37
Organisational Evaluation
100
200
300
400
500
Q1/00 Q2/00 Q3/00 Q4/00 Q1/01 Q2/01 Q3/01
Date
Eva
lua
tion
Organization
Process
Teaming
Ability
SoftTest Ireland Copyright 2008 ElectroMind Ltd 38
Components of a centre of excellence
Resource pool Test lab
Test Managers SIG
Consultants
Knowledge Base
R&D Centre
Tool Experts
Project 1
Project 2
Project 3
Project 4
SoftTest Ireland Copyright 2008 ElectroMind Ltd 39
Benefits of a centre of excellence
Independent testing Breadth & depth Fewer product recalls Fewer production failures Central reference point Knowledgeable staff Expertise across org. Structured career path
Variety of work Identity, pride Morale, professionalism Extra revenue stream Consistent approach Multi-methodologies Earlier involvement