Engineering, Operations & Technology
Approval for public release 15-00006 Copyright © 2015Boeing. All rights reserved.
Boeing Test & Evaluation
Shifting Left With Developmental Testing for Systems of Systems
Pete De Salvo – Systems/Systems of Systems, Test Program Manager David Allsop - Systems/Systems of Systems, Systems Engineering
Author, 1/29/2015, Filename.ppt | 1
ITEA SoS Conference January 27-29, 2015
Approval for public release 15-00006 Copyright © 2014 Boeing. All rights reserved.
Boeing Test & Evaluation | Project Name
Late Discovery is Driving Program Cost and Schedule
Author, 1/29/2015, Filename.ppt | 2
DT&E Assessments recommending to proceed to IOT&E
(2012 DT&E Annual Report)
Proceed
Do Not Proceed
IOT&E
Significant Issues No Issues
25%
50%
Only 41% (5 of 12) programs made it to & through a successful IOT&E
Num
ber o
f Iss
ues
Rea
lized
Issue actual discovered Earliest opportunity for discovery
Nominal cost for fault removal (NIST data) 1x 5x 16x 40x 110x
Notional
Late discoveries are considerably more costly!
Resulting in production delays
The Cost The Impact
Copyright © 2014 Boeing. All rights reserved.
Sources NDIA ICOTE, Chief Developmental Tester & Industry Test Lead – Partnering for success, August 2013 NIDS Planning report 02-3. The economic Impacts of Inadequate Infrastructure for Software Testing May 2002. D.Galin, Software Quality Assurance From Theory to Implementation, Pearson/Addison-Wesley (2004) B.W. Boehm, Software Engineering Economics, Prentice Hall (1981)
Approval for public release 15-00006 Copyright © 2014 Boeing. All rights reserved.
Boeing Test & Evaluation | Project Name
▫ Systems and products are becoming ever more complex with their internal and external design approach evolving into systems of systems.
▫ Declining budgets and poor cost and schedule performance suggest new methods are needed.
▫ Verifying and validating the design early can prevent late problem discovery during test or during customer operation.
▫ Early validation and developmental test is commonly called “shift left”.
The Test Community Needs to Shift Left to Partner With Systems and Program Engineering to Improve Program Performance
Shift Left
Approval for public release 15-00006 Copyright © 2014 Boeing. All rights reserved.
Boeing Test & Evaluation | Project Name
Model-Based Concurrent Engineering Processes
Requirements-Analysis Systems
Analysis & Design
HW/SW Design HW/SW Implementation
Module Integration &
Test
System Integration &
Test System
Acceptance
4
Performing requirements and design validation as early as possible, as opposed to waiting until ”composition” activities begin, reduces cost and schedule risks.
Increase design stability by requirements validation and systems analysis prior to implementation
Cost of Design Change
Time Source: Use of a Model-Based Approach to Minimize System Development Risk and Time-to-Field for New Systems, US Army AMRDEC presentation at the 2010 NDIA Systems Engineering Conference, October 2010
Approval for public release 15-00006 Copyright © 2014 Boeing. All rights reserved.
Boeing Test & Evaluation | Project Name
Challenge – testers need ‘real’ requirements; developers need to prototype requirements (via test) to define ‘real’ requirements
Shifting Test to the Left Side of the ‘V
Approval for public release 15-00006 Copyright © 2014 Boeing. All rights reserved.
Boeing Test & Evaluation | Project Name
Key Concepts
Author, 1/29/2015, Filename.ppt | 6
Validation Build the right system Fitness for purpose Quality assurance External – stakeholder evaluation (suitability and effectiveness) Operational Test End to end focus Top down Did we get what we want?
Verification Build the system right Satisfies requirements Quality control Internal – contractor testing (demo, inspection, analysis) Development test Traceability Bottom up Did we get what we asked for?
Model Based Systems Engineering “…is the formalized application of modeling to support system requirements, design,
analysis, verification and validation activities. It begins at conceptual design & continues throughout the development life cycle.”
INCOSE Vision 2020 (Reference: “International Council on Systems Engineering, Systems Engineering Vision 2020,” INCOSE-TP-2004-004-02,
Version/Revision: 2.03, Dated September 2007)
Perform Validation and Verification as soon as Possible to Identify, Mitigate, and Retire Program Risks Early
Approval for public release 15-00006 Copyright © 2014 Boeing. All rights reserved.
Boeing Test & Evaluation | Project Name
Methods/Types of shift left Validation Test
Author, 1/29/2015, Filename.ppt | 7
Developmental Testing Includes • Modeling and Simulation • Model Based Systems Engineering • Physical prototyping • Brass Board • Virtualization • Actual Hardware Test • Combinations of above elements, eg, LVC testing
Early Risk Discovery and Retirement is Key to Project Execution
Source NDIA Final Report – Model Based Engineering (MBE) Subcommittee, October 2011
Approval for public release 15-00006 Copyright © 2014 Boeing. All rights reserved.
Boeing Test & Evaluation | Project Name
Considerations for shift-left-testing
Author, 1/29/2015, Filename.ppt | 8
• Partnership between test, systems engineering, and design engineering essential throughout development process is essential
• Tailor fidelity of simulation to particular need • Tailor level of test process discipline to match desired outcome • Validate the system and the system of systems interactions (typically in
build-up fashion)
Approval for public release 15-00006 Copyright © 2014 Boeing. All rights reserved.
Boeing Test & Evaluation | Project Name
Potential Organizational Impediments Chief Developmental Test role being defined to elevate the importance of shift left test. Address organizational Stove piping
Author, 1/29/2015, Filename.ppt | 9
Approval for public release 15-00006 Copyright © 2014 Boeing. All rights reserved.
Boeing Test & Evaluation | Project Name
Conclusions
• Testing is Both a Left and Right Side Vee Activity • Discover and Retire Risk Early Through Early Validation • Verification and Validation are Different
Author, 1/29/2015, Filename.ppt | 10
Discover and Retire Risk Early
Approval for public release 15-00006 Copyright © 2014 Boeing. All rights reserved.