Software Testing: does Academia
meet Industry?
A brief look into the interaction of the software
testing discipline and the academic world
Dani Almog – Ben Gurion university
of the Negev Israel
Ben-Gurion University
of the Negev Israel
1
Agenda
What is software testing (mapping the
terminology)
Education or research?
Around the world (stats and facts)
Highlights on new research thread (topics)
How does academia teach the new generation?
Closing the gap
2
Software Testing in WikipediaSoftware engineering
Software requirements:
Software design:
Software development:.
Software testing
◦ Topics
◦ Methods
◦ Levels
◦ NFT
◦ Processes
◦ Automation
◦ Measurements
◦ Artifacts
◦ certifications
Software maintenance:
Software configuration management:
Software engineering management:
Software development process:
Software engineering tools
Software quality
Scope
Functional vs. non-functional testing
Defects and failures
Finding faults early
Compatibility
Input combinations and preconditions
Static vs. dynamic testing
Software verification and validation
The software testing team
Software Quality Assurance (SQA)
Black box testing
White box testing
Grey Box Testing
Exploratory
MBT
TDD
SOA Testing
SBT
3
Software Testing in WikipediaSoftware engineering
Software requirements:
Software design:
Software development:.
Software testing
◦ Topics
◦ Methods
◦ Levels
◦ NFT
◦ Processes
◦ Automation
◦ Measurements
◦ Artifacts
◦ certifications
Software maintenance:
Software configuration management:
Software engineering management:
Software development process:
Software engineering tools
Software quality
Unit Testing
Component testing
Integration Testing
Product testing
System Testing
System Integration Testing
Regression Testing
Acceptance testing
Alpha testing
Beta testing
Tools
Gui / batch
Regression
maintenance
Reporting
waterfall
Spiral testing cycle
Agile/Extreme
CMMI
TPI
4
Software Testing in WikipediaSoftware engineering
Software requirements:
Software design:
Software development:.
Software testing
◦ Topics
◦ Methods
◦ Levels
◦ NFT
◦ Processes
◦ Automation
◦ Measurements
◦ Artifacts
◦ certifications
Software maintenance:
Software configuration management:
Software engineering management:
Software development process:
Software engineering tools
Software quality
Test Case
Test Plan
Tractability Matrix
Test suite
Test Harness
Test Data
ISO9000/…
CMM
CMMI
SPI
professional Certification
programs
5
?
Education or research?
What is really motivating academia (Publish or Parish)
In between CS and IT
Difference between ‘undergrad’ and ‘graduate’
Industry & Academy - this is not only a terminology gap
"Universities are producing IT graduate students, but whether the degrees fit with
business needs is another question. We need a wider conversation between business
and universities,” - Gareth Preece, skills specialist at UK Trade & Investment
"What we need is a qualification that is really fit for purpose, universities, business and
the government need to work together to create a worthwhile qualification”.
6
Remarks by Bill Gates17th Annual ACM Conference on Object-Oriented Programming, Seattle,
Washington, November 8, 2002
“… When you look at a big commercial software company like
Microsoft, there's actually as much testing that goes in as
development. We have as many testers as we have developers.
Testers basically test all the time, and developers basically
are involved in the testing process about half the time…
“… We've probably changed the industry we're in. We're not
in the software industry; we're in the testing industry, and
writing the software is the thing that keeps us busy doing all
that testing.”
“…The test cases are unbelievably expensive; in fact, there's
more lines of code in the test harness than there is in the
program itself. Often that's a ratio of about three to one.”
7
Tagged Questions from Stack
overflowNo. of Questions Subject
63563c#35648java33523.net29279asp.net 28944php
26054javascript23120c++19566jquery
18557python17140iphone15974sql12852mysql12745sql-server12576html9962c9933asp.net-mvc9875ruby-on-rails9484wpf
Stack Overflow is a programming Q & A
site that's free. Free to ask questions, free
to answer questions, free to read, free to
index, built with plain old HTML
http://stackoverflow.com/
8
Cem Kaner compares between
Academic and commercial education
9
TOP Ten CS Schools in USA
Schools
New
enrolment
2008
New
enrolment
2009 SW Quality or testing courses
University of Texas – Austin 225 237
• Introduction to Usability Testing
• EE379K Software Testing
• EE382C-3 Verification and Validation
Georgia Institute of
Technology 219 225
• CS 6330 Software Generation, Testing, and
Maintenance
• CS 4560 Verification of Systems
• CS6340 Software Analysis and Testing
University of Washington 175 175 • 4 Certification programs
Princeton University 200 225
• 466 Digital System Testing
• COS217 Scoping and Testing
Cornell University 67 76
University of Illinois at
Urbana-Champaign 170 215
• CS498DM: Software Testing
• CS598DM: Software Testing and Analysis
Carnegie Mellon University 130 130
15-414 Bug Catching: Automated Program
Verification and Testing
CS 119 - Reliable Software: Testing and
Monitoring
University of California-
Berkeley 232 263
• CS 294 Dynamic Program Analysis and
Software Testing
Stanford University 87 123
Massachusetts Institute of
Technology 159 167
1664 1836 10
Selected program from other schools
CAL Tec –
◦ CS 118 Formal software verification
◦ CS119 Reliable Software: Testing and Monitoring
◦ EE/MA/CS 127 Error correcting codes
Northeastern University (certificate)◦ COM 5383 Introduction to Software Testing
◦ COM5384 Introduction to Software Testing Automation
◦ COM5387 Certified Software Quality Engineer (CSQE) Exam Preparation
University of Massachusetts Amherst◦ Software Engineering: Analysis and Evaluation
◦ Introduction to Software Testing
◦ Advance topics in software testing
University of Limrick◦ CS 4157 software Quality
◦ CS 41703 software Qulaity (elective)
◦ CS 4004 Software testing and inspection
http://www.testingeducation.org/general/othertestingcourses.html11
school CourseUniversity of St. Thomas Software Quality Assurance/Quality Control (CSIS605T)
Southern Methodist University Software Testing and Quality Assurance (CSE7314)
George Mason University Software Testing and Quality Assurance (CSE637)
The University of Texas in Dallas Software Testing, Verification, Validation and Quality Assurance (CSE4367)
Mississippi State University Software Testing and QA (CSE 6283)
University of Montata-Missoula Software Science II: Design, Implementation and Testing (G542)
Texas A&M University Advanced Topics on Logic Synthesis (CPSC678)
Florida Institute of Technology Software Testing Methods (CSE4431)
Azusa Pacific University Software Engineering II (CS525)
Gannon University Software Engineering and Test (CIS315)
California National University Software Quality Assurance (QAS 207)
Christian Brothers University Quality Assurance (ENGM605)
Embry-Riddle Aeronautical University Quality Engineering and Assurance MSE 625
University of Missouri – Rolla Software Testing and Quality Assurance (CSc 307)
University of Kansas Software Quality Assurance (EECS 814)
Winthrop University Software Quality Assurance (626)
University of Maryland Principles of Quality and Reliability in Engineering (ENRE 400)
DalHousie University, Continuing Technical Education Quality Assurance & Testing of Software
Santa Clara University Software Quality Assurance and Testing (COEN 286)
University of Waterloo , Canada Software Testing, Quality Assurance, and Maintenance (CS 447)
Middle East Technical University, Informatics Institute Introduction to Software Testing (SM514)
http://blackbox.cs.fit.edu/general/othertestingcourses.html 12
A deeper look into selected
academic programs Are software developers exposed to testing
procedures (note the growing popularity of TDD)
Modern approach towards SW development and testing is not academically addressed
What about hand-on experience?
How the academic program influence career opportunities?
Only three declared SW testing programs at the Graduate level
13
PURDUE university programs
Careers:
Application developer
Consultant
Data analyst or administrator
Information analyst
IT analyst
Network engineer or administrator
Programmer
Programmer analyst
Project manager
Security specialist
Software developer, engineer, support or tester
Web developer
one course of Foundations of Software Testing14
Purdue Software Testing Group
The Software Vulnerabilities Testing Group is composed of currently four students
and one faculty within the COAST Laboratory at Purdue University who are
interested in methods of software security testing.
Purpose of the Group
The purpose of the group is to evaluate and develop methods of testing software
efficiently that aims on discovering security relevant software flaws before the final
product is deployed.
Near-Term Goals
Our first goal is to study and review the existing literature on software testing,
which describes a wide range of techniques including mutation testing, slicing, test
case coverage determination, etc. Many of the software testing techniques described
in the literature are very expensive and time consuming. Therefore, we want to
establish, which existing testing techniques are most effective for vulnerability
detection, in order to provide software engineers guidelines for the selection of
testing methods. Our second goal is to investigate the existing security flaws and
earlier work on classification of software faults, we want to find or design a suitable
classification scheme that we can use in our experiment later.
15
University of Oslo
INF4290 - Software Verification
and Validation
10 academic credit – graduate level
INF1000 -
Introduction to
object-oriented
programming
INF1010 -
Object
Oriented
Programming
Introduction to
programming
with scientific
applications
MAT-INF1100 -
Modeling and
computations
INF1050 -
Systems
Development
INF2100 -
Programming
project
16
Academic institutes (Florida Tech)
Center for Software Testing Education & Research
Our mission is to create effective, grounded, timely materials to support the teaching and self-study of software testing, software reliability, and quality-related software metrics.
Florida Tech’s Center for Software Testing Education & Research formed in November 2003, as a collaboration among ◦ Cem Kaner, Ph.D., J.D. (Professor) (Director),
◦ Walter P. Bond, Ph.D. (Associate Professor),
◦ Scott Tilley, Ph.D. (Associate Professor),
◦ Michael Andrews, Ph.D. (Assistant Professor),
◦ James Whittaker, Ph.D. (Professor)
17
Sample tour in Israel
Ort Brauda – Computer Testing &
Verification
Technion - Software Quality Assurance
Ariel University – Software Quality
Assurance
Bar Ilan – software verification
Hebrew university Jerusalem –
Software Testing
SCE - Testing and verification in SW
engineering
BGU – Software Quality engineering
- Advance topics in SW quality
18
Selected recent academic publications
0
50
100
150
200
250
Publication topics
0
10
20
30
40
50
60
70
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
Publication volume
Papers were collected by:
• key words
•Tittles
•Abstracts
DERIVED FROM
COLLECTION OF 441
PAPERS/BOOKS
ACCUMILATED 1985 - 2008
19
Research trajectories (threads)
Search into the gap between development
and testing (new software development/
testing paradigms )
Test case generation
SOA testing and testing as a service
Testing models for the cloud
Lean Testing
20
Implications
Too few programs
Too few graduates in all levels
A narrow topic coverage
Insufficient research
21
Conclusions
SW quality & Testing was almost ignored
by the academic world until 1999
SW Quality and Testing is currently
getting more emphasis in SW engineering
A gap between industry needs and
academic education and research
Academic Research is basically detached
from industry achievements
22
What’s next?
Get the professionals to contribute to
universities
Open communication channels
Expose students to industry real life cases
and reality
Generic research programs (jointly
initiated by industry and academia)
23
Software Testing: does Academia
meet Industry?
Questions?
Please call me if you want to be involved
Dani Almog – Ben Gurion university
of the Negev Israel
Ben-Gurion University
of the Negev Israel
24
