Post on 27-Dec-2015
transcript
1ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
Déjà vu - Integration and Test Challenges
April 2014
Tom Wissink, Lockheed Martin Senior FellowDirector of Integration, Test & Evaluation
2ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
A little bit about Me – an Experiment
My Test Journey started at STARWEST 1998– That was 25 years into my Career
• 5 Companies and over 20 consulting assignments• Programmer, Manager, Tech Lead, Integrator, Tester, etc.
– 15 Years ... a never ending Journey
Resulted in a sabbatical from and then a return to Lockheed Martin
– Worked for a small Software Testing Company; three years of consulting, training and learning
• With large Fortune 500, mid-size & small companies
– One of my new goals - ensure no one has to leave the company to learn the right way to test software!
3ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
An Important Caveat
• I will make many general comments about software and software testing:– These comments will come from:
• My 40 years experience • Consulting/Working in the aerospace & commercial
industry• Testing research with Industry and Academia • Supporting Government & Industry Test Associations,
Committees and Conferences
– The facts and figures used are publically available
• These are my personal views and comments!
4ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
The Past
Day 1 in Software Testing at least from my careers perspective– I started before the first important
book on software testing was published• “The Art of Software Testing”
By Glenford J. Myers
5ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
The Space Shuttle Program
• First Shuttle Launch:– http://www.youtube.com/watch?v=KwLlzUSOW_E
• Last Shuttle Launch:– http://www.youtube.com/watch?v=2EFuLap5Pgg
6ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
Software Testing Lessons from the Past
– Software systems getting larger and more complicated and test needs to increase methods and techniques• Not many specific software testing techniques utilized • But they were known at this point in time but not taught/used• Structured Programming methods/techniques
– Defects found during final testing • The need for multiple test phase became apparent
– Unit, functional, subsystem, system, user• The idea of a separate integration phase was not planned but quickly
became known as an area to specifically work– Was also talked in the literature by different terms, module
testing, interface testing, etc.
– Integration & troubleshooting skills realized as a key need
7ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
The Present
Software Testing Today– Many, many book on the market
and classes in Universities (typically at the graduate level)
– Many theses and dissertations on software testing exist
– Conferences, tools, consultants– Issues still exist in software testing
April2014
8ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
The F-35 Joint Strike Fighter
• The F-35 Flight Testing Video:– http://www.youtube.com/watch?v=8HvOFTIyQzc
9ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
F-4 (1960)
A-7 (1964)
F-111 (1970)
F-15A (1975)
F-16 (1982)
B-2 (1990)
F-22 (2000)
F-35 (2012)
8 10
20
35
45
65
8087
Percentage of Functionality in Software
The F-35 Joint Strike Fighter
60,000 LOC
2 mil LOC
8 mil + LOC
10ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
Current State of the Practice
Toyota faulty accelerator Cost impact over a billion dollars
2002 NIST Study indicates poor software testing infrastructure costs
the US economy as much as
$59 billion a year
Healthcare.govSignificant technical issues, cost increase and very late
Ariane 5 Basically one bad line of code in a reused module from Ariane 4
12ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
Recent Government Actions in T&E2006 – Testing of Defense Systems in an Evolutionary Acquisition Environment
- Results from several Hearing, Studies, etc
2008 – Defense Science Board on DT&E
2012 – Industrial Methods of the Effective Development and Testing of Defense Systems – Focus is on scientific testing approaches
and methods like DOE and combinatorial test design
National Defense Authorization Act 2012 includes:• New Government role: “Chief Developmental Tester”
- The Chief Developmental Tester drives T&E activities across the entire program life cycle, ensuring consistency with customer's T&E Strategy.
13ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
More Government Actions in T&E
National Defense Authorization Act 2013 includes:• Annual report to Congress on:
- Programs that don’t follow DT&E recommendations
- About technical maturity and integration risks of critical technologies
• Presidents budget shall include DT&E line item• Increase to DT&E staff to support
program oversight
14ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
Progression of Software Systems
Simple• Easy to understand• Repeatable processes to
create• No Expertise needed to
follow process• Completely testable
Complicated• Large Number of
components• Large # of connections• Integration of different
technologies • Difficult to have
knowledge across all elements
• Process to create is difficult but not impossible
• Incomplete testing
Complex• System evolves • Very large # of
connections• Feedback changes
behavior• Adaptive• Emergent• Social Behavior• Unexpected results
Test is Easy• Test Everything
Can’t test everything• Knowing what to test is
more important then how much
Non-predictable results• How to test what you don’t
know?
XWe are Here
15ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
Current Software Testing Lessons
– Déjà vu: Software systems getting much larger and more complicated and test needs to increase methods and techniques • Many specific software testing techniques known but not used • Object Oriented & Agile programming methods/techniques
– Déjà vu: Many defects found during final testing• Multiple test phase utilized but poor test coverage
– Unit, functional, subsystem, system, user• The idea of a separate integration phase is known but typically not
done until after the first release of software capability doesn’t work• No real improvement in the literature available on integration
– Déjà vu: Integration & troubleshooting skills realized as a key need
16ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
The Future
Testing Tomorrow – Ever increasing System and Software Complexity!
Model-based Test & Integration–Many if not most test cases
generated & executed automatically – less human execution time– However, Human testing will
become even more important– Integration still a human endeavor
with help from Models Unknown – Need Research
17ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
Is there a Test Technology Gap?
Singularity
The test community need to close the gap
Test Technology
Gap
TechnologyTest Research
18ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
Complicated Systems
System of Systems
Autonomous Platforms
Collaboration
Large number of connectionsFree-scale networks
Large set of sensors
Feedback loops modifying behavior
Non-predictable
Systems/Software Complexity
Evolving FutureToday
19ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
Simple Emergent Behavior Example
https://www.youtube.com/watch?v=1X-gtr4pEBU
20ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
Complex I&T Research Sources1. Brat, Guillaume, et al. "Verification of autonomous systems for space applications."
Aerospace Conference, 2006 IEEE. IEEE, 2006.2. Truszkowski, Walter F., et al. "Autonomous and autonomic systems: A paradigm for
future space exploration missions." Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on 36.3 (2006): 279-291.
3. Dixon, Clare, Alan Winfield, and Michael Fisher. "Towards temporal verification of emergent behaviours in swarm robotic systems." Towards Autonomous Robotic Systems. Springer Berlin Heidelberg, 2011. 336-347.
4. Gleizes, Marie-Pierre, and Jorge J. Gomez-Sanz. "Agent-Oriented Software Engineering X." (2009).
5. Strunk, Elisabeth A., M. Anthony Aiello, and John C. Knight. "A survey of tools for model checking and model-based development." University of Virginia (2006).
6. De Lemos, Rogério, et al. "Software engineering for self-adaptive systems: A second research roadmap." Software Engineering for Self-Adaptive Systems II. Springer Berlin Heidelberg, 2013. 1-32.
7. Gjondrekaj, Edmond, et al. "Towards a formal verification methodology for collective robotic systems." Formal Methods and Software Engineering. Springer Berlin Heidelberg, 2012. 54-70.
8. Choi, Jiyoung. Model checking for decision making behaviour of heterogeneous multi-agent autonomous system. Diss. Cranfield University, 2012.
21ICST 2014 Keynote: Déjà vu - Integration and Test Challenges
Summary
• Academia and Industry must work more closely to solve the software complexity issues– e.g. Model-based test case generation at multiple
levels of testing
• Goal: Software testing evolve along with technology and methodology discovery