Date post: | 06-Dec-2014 |
Category: |
Technology |
Upload: | stag-software-pvt-ltd |
View: | 4,252 times |
Download: | 7 times |
Path to reach Level 3 test
process maturity in TMMi
framework
M C Nagaraj
in.linkedin.com/in/mcnagaraj
Copyright STAG Software Pvt. Ltd., 2011
Agenda
TMMi – Level 1 to 5
Requirement interpretation
Linking current process to TMMi Specific practices
All TMMi related information are taken from TMMi framework document V 3.1 created and Copyrights owned by TMMi foundation.www.tmmifoundation.org
Copyright STAG Software Pvt. Ltd., 2011
Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
Defect Prevention
Total Process Optimization
Quality Control
Test Policy and Strategy, Test Planning, Test Monitoring and
Control, Test Design and Execution, Test Environment
Test Organization, Test Training Program, Test Lifecycle and
Integration, Non-functional Testing, Peer Reviews
Level 1 : Initial
TMMi Framework
Five levels of Testing Process Maturity
Level 2 : Managed
Level 3 : Defined
Level 4 : Measured
Level 5 : Optimization
Copyright STAG Software Pvt. Ltd., 2011
Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
Defect Prevention
Total Process Optimization
Quality Control
Test Policy and Strategy, Test Planning, Test Monitoring and
Control, Test Design and Execution, Test Environment
Test Organization, Test Training Program, Test Lifecycle and
Integration, Non-functional Testing, Peer Reviews
Level 1 : Initial
TMMi Framework
Five levels of Testing Process Maturity
Level 2 : Managed
Level 3 : Defined
Level 4 : Measured
Level 5 : Optimization
Copyright STAG Software Pvt. Ltd., 2011
Level-1
• Testing is a chaotic process
• Not distinguished from debugging
• Documented set of specification not available
• Tests are developed in ad-hoc way after coding is completed
• The objective of testing is to show that software works
Copyright STAG Software Pvt. Ltd., 2011
Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
Defect Prevention
Total Process Optimization
Quality Control
Test Policy and Strategy, Test Planning, Test Monitoring and
Control, Test Design and Execution, Test Environment
Test Organization, Test Training Program, Test Lifecycle and
Integration, Non-functional Testing, Peer Reviews
Level 1 : Initial
TMMi Level 2
Five levels of Testing Process Maturity
Level 2 : Managed
Level 3 : Defined
Level 4 : Measured
Level 5 : Optimization
Process areas at TMMi Level 2
2.1 Test Policy and strategy
2.2 Test Planning
2.3 Test Monitoring and Control
2.4 Test Design and Execution
2.5 Test Environment
Copyright STAG Software Pvt. Ltd., 2011
2.1 Test Policy and Strategy
Specific Goals Specific Practices Guidelines to implement
SG 1 - Establish a Test Policy SP 1.1 Define test goalsSP 1.2 Define test policySP 1.3 Distribute the test policy to stakeholders
-Look at organization business goaland role of testing in achieving the same- Policy should address test cycle activities are planned, done by trained engineers, process improve with management support/focus
SG 2 - Establish a Test Strategy SP 2.1 Perform generic product risk assessmentSP 2.2 Define test strategySP 2.3 Distribute the test strategy to stakeholders
-Use organization standard way ofidentifying risks in a project-Define quality levels and objectives
SG 3 - Establish Test Performance Indicators
SP 3.1 Define test performance indicatorsSP 3.2 Deploy test performanceindicators
-Define how you measure success of test process-Set some target for key metrics and measure what you achieved
Copyright STAG Software Pvt. Ltd., 2011
Process areas at TMMi Level 2
2.1 Test Policy and strategy
2.2 Test Planning
2.3 Test Monitoring and Control
2.4 Test Design and Execution
2.5 Test Environment
Copyright STAG Software Pvt. Ltd., 2011
2.2 Test Planning
Specific Goals Specific Practices Guidelines to implement
SG 1- Perform Product Risk Assessment
SP 1.1 Define product risk categoriesSP 1.2 Identify product risksSP 1.3 Analyze product risks
-Look at from end users perspective value & prioritize requirements-Can you predict what can upset end users?
SG 2 - Establish a Test Approach SP 2.1 Identify items and features to be testedSP 2.2 Define the test approachSP 2.3 Define entry criteriaSP 2.4 Define exit criteriaSP 2.5 Define suspension and resumption criteria
- Extract list of features-Extract list of user types and business use cases they use-Extract potential defect types-Quality levels and types of defects to be uncovered at each levels-Entry, exit, suspension and resumption criteria for all levels
SG 3 - Establish Test Estimates SP 3.1 Establish a top-level work breakdown structureSP 3.2 Define test life cycleSP 3.3 Determine estimates for test effort and cost
-Have organization standard test life cycle activity list-Customize this to project specific-Estimate effort, review update and baseline-Track number of cycles of re-estimation done
Copyright STAG Software Pvt. Ltd., 2011
2.2 Test Planning
Specific Goals Specific Practices Guidelines to implement
SG 4 – Develop a Test Plan SP 4.1 Establish the test scheduleSP 4.2 Plan for test staffingSP 4.3 Plan for stakeholders involvementSP 4.4 Identify test project risksSP 4.5 Establish the test plan
-Milestone definition-Detail schedule-Risk assessment and prioritization-Plan document with important sections in place
SG 5 – Obtain Commitment to the Test Plan
SP 5.1 Review test planSP 5.2 Reconcile work and resource levelsSP 5.3 Obtain test plan commitments
-Review plan with all concerned-Update feedback and ensure confidence is high to meet the plan-Team commitments for all dependency will help
Copyright STAG Software Pvt. Ltd., 2011
Process areas at TMMi Level 2
2.1 Test Policy and strategy
2.2 Test Planning
2.3 Test Monitoring and Control
2.4 Test Design and Execution
2.5 Test Environment
Copyright STAG Software Pvt. Ltd., 2011
2.3 Test Monitoring and Control
Specific Goals Specific Practices Guidelines to implement
SG 1 – Monitor Test Progress against Plan SP 1.1 Monitor test planning parametersSP 1.2 Monitor test environment resources provided and usedSP 1.3 Monitor test commitmentsSP 1.4 Monitor test project risksSP 1.5 Monitor stakeholder involvementSP 1.6 Conduct test progress reviewsSP 1.7 Conduct test progress milestone reviews
-Monitor effort/cost, schedule variance against plan-Ensure test environment is same as stated in expectation in document-Milestone review, weekly review etc.-Stakeholders update on key milestone and seek feedback-Risk tracking
SG 2 – Monitor Product Quality against Plan and Expectations
SP 2.1 Check against entry criteriaSP 2.2 Monitor defectsSP 2.3 Monitor product risksSP 2.4 Monitor exit criteriaSP 2.5 Monitor suspension and resumption criteriaSP 2.6 Conduct product quality reviewsSP 2.7 Conduct product quality milestone reviews
-Quality levels exit criteria defined and actual- Monitor suspension and resumptions cycles- Test summary report at key quality levels achievement
Copyright STAG Software Pvt. Ltd., 2011
2.3 Test Monitoring and Control
Specific Goals Specific Practices Guideline to implement
SG 3 – Manage Corrective Action to Closure
SP 3.1 Analyze issuesSP 3.2 Take corrective actionSP 3.3 Manage corrective action
-Track project issues formally-Track action items and ensure closure
Copyright STAG Software Pvt. Ltd., 2011
Process areas at TMMi Level 2
2.1 Test Policy and strategy
2.2 Test Planning
2.3 Test Monitoring and Control
2.4 Test Design and Execution
2.5 Test Environment
Copyright STAG Software Pvt. Ltd., 2011
2.4 Test Design and Execution
Specific Goals Specific Practices Guidelines to implement
SG 1 – Perform Test Analysis and Design using Test Design Techniques
SP 1.1 Identify and prioritize test conditionsSP 1.2 Identify and prioritize test casesSP 1.3 Identify necessary specific test dataSP 1.4 Maintain horizontal traceability with requirements
-Have a clear test case architecture to be followed across organization-Organize test cases at quality level -Maintain requirement traceability to test scenarios/test cases, fault type, types of test applicable
SG 2 – Perform Test Implementation
SP 2.1 Develop and prioritize test proceduresSP 2.2 Create specific test dataSP 2.3 Specify intake test procedureSP 2.4 Develop test execution schedule
-Test cycle schedule-Build health check criteria-Create re-usable test data- Quality level entry/exit criteria
Copyright STAG Software Pvt. Ltd., 2011
2.4 Test Design and Execution
Specific Goals Specific Practices Guidelines to implement
SG 3 – Perform Test Execution SP 3.1 Perform intake testSP 3.2 Execute test casesSP 3.3 Report test incidentsSP 3.4 Write test log
-Verify entry criteria beforestarting any levels of testing- Execute test as per plan and record outcome-Capture defects to track and close
SG 4 – Manage test incident to closure
SP 4.1 Decide deposition of test incidents in configuration control boardSP 4.2 Perform appropriate action to close the test incidentSP 4.3 Track the status of test incidents
-Be part of change control board to resolve all issues reported during testing- Ensure defects fixed are verified and closed in each release- Defect tracking report in various forms so that helps to take right decision
Copyright STAG Software Pvt. Ltd., 2011
Process areas at TMMi Level 2
2.1 Test Policy and strategy
2.2 Test Planning
2.3 Test Monitoring and Control
2.4 Test Design and Execution
2.5 Test Environment
Copyright STAG Software Pvt. Ltd., 2011
2.5 Test Environment
Specific Goals Specific Practices Guidelines to implement
SG 1 – Develop Test Environment Requirements
SP 1.1 Elicit test environments needsSP 1.2 Develop the test environment requirementsSP 1.3 Analyze the test environment requirements
-Capture required environment clearly at one place- Ensure environment available and required are identified
SG 2 – Perform Test Environment Implementation
SP 2.1 Implement the test environmentSP 2.2 Create generic test dataSP 2.3 Specify test environment intake test procedureSP 2.4 Perform test environment intake test
-Set up test environment as needed- Create generic test data and archive for re-use- Verify whether needs of environment required for testing is met formally
Copyright STAG Software Pvt. Ltd., 2011
2.5 Test Environment
Specific Goals Specific Practices Guidelines to implement
SG 3 – Manage and Control Test Environments
SP 3.1 Perform systemsmanagementSP 3.2 Perform test data managementSP 3.3 Co-ordinate the availability and usage of the test environmentsSP 3.4 Report and manage test environment incidents
-Plan and track system requirements-Plan development of test data and ensure it is version controlled- Ensure required co-ordination process in place when environment are shared- If environment issues found capture and track till it is closed
Copyright STAG Software Pvt. Ltd., 2011
Copyright STAG Software Pvt. Ltd., 2011
What you can see in L2 organization?
• Organization test goals and policy published
• There is a clear separation between debugging and testing phase
• It is a planned activity in project plan
• Plan starts after coding is complete (only execution based testing visible)
• Basic testing techniques in place
• Testing is multi-leveled
• Test environment managed as required
Copyright STAG Software Pvt. Ltd., 2011
Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
Defect Prevention
Total Process Optimization
Quality Control
Test Policy and Strategy, Test Planning, Test Monitoring and
Control, Test Design and Execution, Test Environment
Test Organization, Test Training Program, Test Lifecycle and
Integration, Non-functional Testing, Peer Reviews
Level 1 : Initial
TMMi Framework
Five levels of Testing Process Maturity
Level 2 : Managed
Level 3 : Defined
Level 4 : Measured
Level 5 : Optimization
Process areas at TMMi Level 3
3.1 Test Organization
3.2 Test Training Program
3.3 Test Lifecycle and Integration
3.4 Non-functional testing
3.5 Peer Reviews
Copyright STAG Software Pvt. Ltd., 2011
3.1 Test Organization
Specific Goals Specific Practices Guidelines to implement
SG 1 – Establish a Test Organization
SP 1.1 Define the test organizationSP 1.2 Obtain commitment for the test organizationSP 1.3 Implement the test organization
-Clear definition of test team organization- Management commitment to support the test organization- Create team to fill all roles
SG 2 – Establish Test Functions for Test Specialists
SP 2.1 Identify test functionsSP 2.2 Develop job descriptionsSP 2.3 Assign staff members to test functions
-Define key test lifecycle activities-Map role and responsibilities to key test life cycle activities
SG 3 – Establish Test Career Paths
SP 3.1 Establish test career pathSP 3.2 Develop personal test career development plans
-Ensure there is clear definition of career path for test engineers- Ensure individual career plans prepared and tracked by managers
Copyright STAG Software Pvt. Ltd., 2011
3.1 Test Organization
Specific Goals Specific Practices Guidelines to implement
SG 4 – Determine, Plan and Implement Test Process Improvement
SP 4.1 Assess the organization’s test processSP 4.2 Identify the organization test process improvementsSP 4.3 Plan test process improvementsSP 4.4 Implement test process improvements
-Assess organization test process formally- Identify improvement areas- Plan test process improvements by having a team- Ensure team work as per plan and achieve improvements planned
SG 5 – Deploy the Organization Test Process and Incorporate Lessons Learned
SP 5.1 Deploy standard test process and test process assetsSP 5.2 Monitor implementationSP 5.3 Incorporate lessons learnt into the organizational test process
-Define standard process- Capture test process assets- Monitor adherence to process by planned audits- Analyze completed projects and recommend any improvement to process needed
Copyright STAG Software Pvt. Ltd., 2011
3.2 Test Training Program
Specific Goals Specific Practices Guidelines to implement
SG 1 – Establish an Organizational Test Training Capability
SP 1.1 Establish the strategictest training needsSP 1.2 Align the organizational and project test training needsSP 1.3 Establish an organizational test training planSP 1.4 Establish test training capability
-Define role based competency and standard training needs- Identify project specific competency required and plan training- Have clear training plan for team and project specific training on need basis-Ensure trainers internal/external identified and ready for all topics
SG 2 – Provide Necessary Test Training
SP 2.1 Deliver test trainingSP 2.2 Establish test training recordsSP 2.3 Assess test training effectiveness
-Conduct training as per plan- Maintain test training records- Verify the effectiveness of training conducted on job
Copyright STAG Software Pvt. Ltd., 2011
3.3 Test Lifecycle and Integration
Specific Goals Specific Practices Guidelines to implement
SG 1 – Establish Organizational Test Process and Assets
SP 1.1 Establish standard test processSP 1.2 Establish test lifecycle model descriptions addressing all test levelsSP 1.3 Establish tailoring criteria and guidelinesSP 1.4 Establish the organization’s testprocess databaseSP 1.5 Establish the organization's test process asset librarySP 1.6 Establish work environment standards
-Have organization standard test process approved- Define standard test lifecycle activities-Establish tailoring guidelines and criteria-Capture test process followed at project level as data - Capture process assets including tailoring done- Ensure work environment is defined and adhered to as per plan
SG 2 – Integrate Test Lifecycle Models with Development Models
SP 2.1 Establish integrated test lifecycle modelsSP 2.2 Review integrated lifecycle modelsSP 2.3 Obtain commitments on the role of testing within the integrated lifecycle models
-Ensure multiple test cycles are planned with focus on all levels of testing- Review with all concerned on depends across each team and agree on entry/exit criteria
Copyright STAG Software Pvt. Ltd., 2011
3.3 Test Lifecycle and Integration
Specific Goals Specific Practices Guidelines to implement
SG 3 – Establish a Master Test Plan
SP 3.1 Perform product risk assessmentSP 3.2 Establish the test approachSP 3.3 Establish test estimatesSP 3.4 Define the organization for testingSP 3.5 Develop the master test planSP 3.6 Obtain commitment to the master test plan
-Prioritize customer expectation- Define test strategy- Estimate effort required- Define team required -Have master test plan with milestone- Have detail plan for all milestone- Have commitment from all concerned in team on schedule
Copyright STAG Software Pvt. Ltd., 2011
3.4 Non-functional Testing
Specific Goals Specific Practices Guidelines to implement
SG 1 – Perform a Non-Functional Product Risk Assessment
SP 1.1 Identify non-functional product risksSP 1.2 Analyze non-functional product risks
-Identify non-functionalproduct requirements- Analyze what can go wrong here
SG 2 – Establish a Non-Functional Test Approach
SP 2.1 Identify features to be testedSP 2.2 Define the non-functional test approachSP 2.3 Define non-functional exit criteria
-Identify scope for NF testing- Define test approach for NF testing- Define exit criteria for all planned NF tests
Copyright STAG Software Pvt. Ltd., 2011
3.4 Non-functional Testing
Specific Goals Specific Practices Guidelines to implement
SG 3 – Perform Non-Functional Test Analysis and Design
SP 3.1 Identify and prioritize non-functional test conditionsSP 3.2 Identify and prioritize non-functional test casesSP 3.3 Identify necessary specific test dataSP 3.4 Maintain horizontal traceability with non-functionalrequirements
-Identify use cases where NF requirements may not be fully met- Have scenarios and tests cases to replicate the identified use case- Ensure NF requirement to test assets traceability is established
SG 4 – Perform Non-Functional Test Implementation
SP 4.1 Develop and prioritize non-functional test requirementsSP 4.2 Create specific test data
-Scope and plan for NF testing-Create required scripts to produce the targeted use case scenario
SG 5 – Perform Non-Functional Test Execution
SP 5.1 Execute non-functional test casesSP 5.2 Report non-functional test incidentsSP 5.3 Write test log
-Execute planned NF tests-Report NF test issues and track till closure-Capture test outcome
Copyright STAG Software Pvt. Ltd., 2011
3.5 Peer Reviews
Specific Goals Specific Practices Guidelines to implement
SG 1 – Establish a Peer Review Approach
SP 1.1 Identify work products to be reviewed SP 1.2 Define Peer Review Criteria
-Agree upon work products which will go through peer review- Expectation defined for peer review
SG 2 – Perform Peer Reviews SP 2.1 Conduct peer reviewsSP 2.2 Testers review test basis documentsSP 2.3 Analyze peer review data
-Peer review happens as per plan- Test engineers are part work product reviews which impacts their deliverables-Analyze peer review outcome and improve organization checklist
Copyright STAG Software Pvt. Ltd., 2011
Copyright STAG Software Pvt. Ltd., 2011
What you can see in L3 organizations?
• There is an established test organization
• Testing is integrated to SDLC
• Test plan is developed, tracked and controlled (Integrated with project plan)
• Key work products go through peer reviews
• Organization review checklist for all documents evolve
• Test engineers drive test process improvement
• Users/clients attend milestone meeting
• User/clients support in developing usability test plans