Computing and SE II
Chapter 2: A Brief History of SE
Er-Yu DingSoftware Institute, NJU
A View of 20th and 21st Century Software Engineering
Barry Boehm
ICSE 2006 Keynote Address
May 25, 2006
http://sunset.usc.edu
Modified by Er-Yu Ding
05/25/06 3
Outline
• Motivation and Context
• A 20th Century View
• A 21st Century View
• Conclusions
– Timeless principles and aging practices
05/25/06 4
Motivation and Context
• Importance of historical
perspective
• We are losing our history
05/25/06 5
Why Software Projects Fail- Average overrun: 89.9% on cost, 121% on schedule, with 61% of
content
05/25/06 6
Importance of Historical Perspective
• Santayana half-truth:– “Those who cannot remember the past are
condemned to repeat it”
• Don’t remember failures?– Likely to repeat them
• Don’t remember successes?– Not likely to repeat them– Our objectives– Understanding SE more deeply– Organizing important
techniques/methods/methodologies of SE systematically
05/25/06 7
A Hegelian View of Software Engineering Evolution
Autonomy; Bio-Computing
1990's 2010's2000's1970's 1980's1960's1950's
Engineer Software
like Hardware
Risk-Based Agile/Plan
-Driven Hybrids;
Model-Driven Development
Value-Based Methods;
Collaboration; Global
Development; Enterprise
Architectures
Software Differences,
Engineer Shortages
Scalability,Risk Mgmt.
Many defects
Compliance
Time to Market,Rapid Change
Software Value-Add
COTS
Process Overhead
Scalability
SoftSysE
Software as Craft
Formality, Waterfall
Productivity; Reuse;
Objects;Peopleware
Agile Methods
Plan-Driven
Software Maturity Models
Integrated Sw-Systems Engineering
GlobalSystems
ofSystems
Theses
Syntheses
Antitheses
Prototyping
Risk Mgmt.Domain Engr.
05/25/06 8
Outline
• Motivation and Context
• A 20th Century View
• A 21st Century View
• Conclusions
– Timeless principles and aging
practices
05/25/06 9
1950’s Thesis: Engineer Software Like Hardware
• Hardware-oriented:– Software applications: airplanes,
bridges, circuits– Economics: Boehm supervisor, 1955
• “We’re paying $600/hour for that computer, and $2/hour for you, and I want you to act accordingly.”
– Software Processes: SAGE (Semi-Automated Ground Environment)
• 1 MLOC air defense system, real-time, user-intensive• Successful development of highly unprecedented
system• Hardware-oriented waterfall-type process
05/25/06 10
OPERATIONAL PLAN
MACHINE SPECIFICATIONS OPERATIONAL SPECIFICATIONS
PROGRAM SPECIFICATIONS
CODING SPECIFICATIONS
CODING
PARAMETER TESTING (SPECIFICATIONS)
ASSEMBLY TESTING (SPECIFICATIONS)
SHAKEDOWN
SYSTEM EVALUATION
The SAGE Software Development Process - (Benington, 1956)
“We were successful because we were all engineers”.
05/25/06 11
1950’s Thesis: Engineer Software Like Hardware
• Software Techniques:– Machine-Centric Application– For special usage, research mainframes
• 1GL, 2GL – Organization software with statements
• Coding tips for a optimized program are important
05/25/06 12
1960’s, techniques progress
• Complier can optimizing statements well• Organizing program with function
– Function was derived from λ calculus
• software get more complex
05/25/06 13
1960’s Antithesis: Software Is Not Like Hardware
- Four Brooks factors plus two• Invisibility: like the Emperor’s Magic Cloth
– Visibility and controllability of project is always a topic of SE
• Complexity: we have see this last lecture• Conformity: executed by computers, not
people• Changeability: up to a point, then becomes
difficult• Doesn’t wear out: different reliability,
maintenance phenomena• Unconstrained: can program antigravity,
time travel, interpenetration, …
05/25/06 14
1960’s Antithesis: Software Crafting
• Flexible materials, frequent changes as above
• SW demand exceeded supply of engineers– Cowboy programmers as heroes– Code-and-fix process– Hacker culture (Levy, 1984)
• Judge programmers by the elegance of their code
05/25/06 15
1960’s Progress and Problems
• Better infrastructure: OS, compilers, utilities• Computer Science Departments• Product Families: OS-360, CAD/CAM,
math/statistics libraries• Some large successes: Apollo, ESS, BofA
check processing• Problems: 1968, 1969 NATO Reports
– Failure of most large systems– Unmaintainable spaghetti code– Unreliable, undiagnosable systems
05/25/06 16
1970’s Antithesis: Formal and Waterfall Approaches
• Structured Methods– Structured programming (Bohm-Jacopini: GO TO
unnecessary)• Formal programming calculus: Dijkstra, Hoare, Floyd• Formalized Top-Down SP: Mills, Baker
• Waterfall Methods– Code and fix too expensive (100:1 for large
systems)– Precede code by design (De Marco SD, Jackson
JSD/JSP)– Precede design by requirements (PSL/PSA, SA,
SREM)
05/25/06 17
1970’s Antithesis: Formal Methods
• “formal methods” branch that focused on program correctness, either by mathematical proof, or by construction via a “programming calculus”
• Over the decade of the '60s theoretical computer science achieved standing as a discipline recognized by both the mathematical and the computing communities, and it could point to both applications and mathematical elegance
05/25/06 18
1970’s: Formal Methods
• Organized programs with – Function
• Algorithm
– Three control logic– Type and type check
• When compiler deal with functions well, organized programs with– Modularity
• This things gets importance with modularity– Information hiding– Data abstraction
05/25/06 19
1970’s: Formalized Top-Down SP
• Function decomposition– divide and rule– Process modeling
• DFD
– Data modeling• ERD
05/25/06 20
Increase in Software Cost-to-fix vs. Phase (1976)
Phase in Which defect was fixed
10
20
50
100
200
500
1000
Rel
ativ
e co
st t
o f
ix d
efec
t
2
1
5
Requirements Design Code Development Acceptance Operationtest test
Smaller Software Projects
Larger Software Projects
• Median (TRW Survey)80%
20%SAFEGUARD
GTE
IBM-SSD
•
•
•
•
•
•
Phase in Which defect was fixed
10
20
50
100
200
500
1000
Rel
ativ
e co
st t
o f
ix d
efec
t
2
1
5
Requirements Design Code Development Acceptance Operationtest test
•
••
••
••
•
•
•
05/25/06 21
SYSTEM REQUIREMENTS
TESTING
CODING
PROGRAM DESIGN
ANALYSIS
PRELIMINARY PROGRAM
DESIGN
SOFTWARE REQUIREMENTS
OPERATIONS
PRELIMINARY DESIGN
ANALYSIS
PROGRAM DESIGN
CODING
TESTING
USAGE
The Royce Waterfall Model (1970)
- Explicit feedback- “Do it twice”
05/25/06 22
1970’s: Problems with Formal Methods
• Successful for small, critical programs• Largest proven programs around 10 KSLOC• Proofs show presence of defects, not
absence– Defects in specification, proofs happen
• Scalability of programmer community– Techniques require math expertise, $500/SLOC– Average coder in 1975 survey:
• 2 years of college, SW experience• Familiar with 2 languages, applications• Sloppy, inflexible, in over his head, and
undermanaged
05/25/06 23
Large-Organization HW/SW Cost Trends (1973)
100
80
60
40
20
0
1955 1970 1985
Hardware
Software
Year
% of total cost
05/25/06 24
1970’s: Problems with Waterfall Model• Overly literal interpretation of sequential
milestones– Prototyping is coding before Critical Design Review– Mismatch to user-intensive systems
• Heavyweight documentation hard to review, maintain– 7 year project with 318% requirements change– Milestones passed but not validated
• Mismatch to COTS, reuse, legacy SW– Bottom-up vs. top-down processes
05/25/06 25
A Hegelian View of Software Engineering Evolution
Autonomy; Bio-Computing
1990's 2010's2000's1970's 1980's1960's1950's
Engineer Software
like Hardware
Risk-Based Agile/Plan
-Driven Hybrids;
Model-Driven Development
Value-Based Methods;
Collaboration; Global
Development; Enterprise
Architectures
Software Differences,
Engineer Shortages
Scalability,Risk Mgmt.
Many defects
Compliance
Time to Market,Rapid Change
Software Value-Add
COTS
Process Overhead
Scalability
SoftSysE
Software as Craft
Formality, Waterfall
Productivity; Reuse;
Objects;Peopleware
Agile Methods
Plan-Driven
Software Maturity Models
Integrated Sw-Systems Engineering
GlobalSystems
ofSystems
Theses
Syntheses
Antitheses
Prototyping
Risk Mgmt.Domain Engr.
05/25/06 26
1980’s Synthesis: Productivity, Reuse, Objects
• Worldwide concern with productivity, competitiveness– Japanese example: autos, electronics, Toshiba
SW reuse
• Major SW productivity enhancers– Working faster: tools and environments– Working smarter: processes and methods– Work avoidance: reuse, simplicity; objects– Technology silver bullets: AI, transformations,
DWIM, PBE• Do what I mean; programming by example
05/25/06 27
05/25/06 28
Tools, Environments, and Process• 1970s, some requirements and design tools
developed, significant tool progress had been mode in such areas as test tools and SCM tools
• Major emphasis in 1980s was on integrating tools into support environments: IPSEs to CASEs– Integrated Project Support Environment, Computer Aided
Software Engineering – Requirements, design, planning and control, office support– Formatted vs. formal specifications
• 1980s: Resource estimations and middleware is underway
• Process-driven CASEs– Use process knowledge as tool integration framework– Some overkill in locking people into roles
05/25/06 29
Process: Industrial Engineering
• Frederick W. Taylor : scientific management– First. They develop a science for each element
of a man's work– Second. They scientifically select and then
train, teach, and develop the workman– Third. They heartily cooperate with the men so
as to insure all of the work is being done in accordance with the principle of the science which has been developed.
– Fourth. There is an almost equal division of the work and the responsibility between the management and the workmen.
05/25/06 30
Process
• Process execution support: “SW processes are SW too”– What’s good for products is good for processes– Reuse, prototyping, architecture, programming
• Process compliance support: Standards and CMMs
05/25/06 31
Reuse: Mechanical Engineering
• In modular software, clearly the "standardized units or dimensions" should be standards such that software modules meeting the standards may be conveniently fitted together (without "trimming") to realize large software systems. The reference to "variety of use" should mean that the range of module types available should be sufficient for the construction of a usefully large class of programs.
05/25/06 32
Reuse and Object Orientation
• 1950’s: Math routines, utilities• 1960’s: McIlroy component marketplace,
Simula – 67• 1970’s: Abstract data types, Parnas program
families• 1980’s: Smalltalk, Eiffel, C++, OO methods,
reuse libraries• 1990’s: Domain engineering, product lines,
UML, components, patterns, pub-sub architectures, etc.
• 2000’s: Model driven development, service oriented architectures
05/25/06 33
No Silver Bullet: Brooks• Automated solutions are good for
“accidental” software problems– Simple inconsistencies, noncompliance, inferences
• They do not do well on “essential” software problems– Changeability: adapting themselves to unanticipated
changes– Conformity: working out everything the computer
needs to “know”• Devoid of intuition, commonsense reasoning
– Complexity: integrating multiple already- complex programs
– Invisibility: communicating their likely behavior to humans
• Closest thing to silver bullet: great designers
05/25/06 34
People: The Most Important Factor- SW engineering is of the people, by the people, and for the
people• 1970’s: Weinberg Psychology of Computer
Programming• 1980’s: DeMarco-Lister Peopleware• 1990’s – 2000’s: Importance emphasized
in both Agile and CMM cultures– Individuals and interactions over process and
tools– People CMM, Personal Software Process,
Team Software Process
05/25/06 35
Dual 1990’s – Now: software progress
• 1950s-1960s: machine-centric software• 1960s-1980s: application-centric software• 1990s-now: enterprise-centric software
– Distributed, large-scale– Design patterns, Software architectures, UML– Requirement engineering– Middleware, Components, CDE, Product Line, COTS,
etc. • Collaborative development environment, Commercial off the
shelf
• Future?– domain-centric software
• Components, Product Line, COTS
05/25/06 36
Dual 1990’s – Early 2000’s: software techniques
• OO methods were strengthened through– OOP ( Java )OO Design (1995)OO
Analysis– UML– OO design principles– Design patterns
05/25/06 37
Dual 1990’s – Early 2000’s Antithesis:- Maturity Models and Agile Methods
• Predictability and Control: Maturity Models——CMMI– Reliance on explicit documented knowledge– Heavyweight but verifiable, scalable
• Time to Market and Rapid Change: Agile Methods– Reliance on interpersonal tacit knowledge– Lightweight, adaptable, not very scalable
05/25/06 38
Other 1990’s – Early 2000’s Developments
• Y2K and reverse engineering• Mature risk-driven software model: RUP
– Rational unified process
• Nature and importance of software architecture
• Usability and Human-Computer Interaction• COTS, open source, and legacy software• Software as the primary competitive
discriminator
05/25/06 39
Open source and legacy
• Open source– From its roots in the hacker culture of the
1960’s• Collective code ownership• Free software, data, computing access
– Free Software– Linux– Raymond’s “The Cathedral and the Bazaar”
• Legacy– EAI (enterprise application integration), BPR
(business process reengineering)
• Components and COTS are their future
05/25/06 40
COTS: The Future Is Here• Escalate COTS priorities for research, staffing, education
– Software is not “all about programming” anymore– New processes requiredCBA Growth Trend in USC e-Services Projects
0
10
20
30
40
50
60
70
80
1997 1998 1999 2000 2001 2002
Year
Perc
en
tag
e
*
• CBA: COTS-Based Application
* Standish Group CHAOS 2000 (54%)
05/25/06 41
Mid-2000’s Synthesis: Risk-Driven Hybrid Products and Process
• Increasing integration of systems engineering and SW engineering– Increasing trend toward “soft systems
engineering”
• Increasing focus on usability and value– Fit the software and hardware to the people,
not vice versa
• Model-driven development and service-oriented architectures
05/25/06 42
05/25/06 43
Outline
• Motivation and Context
• A 20th Century View
• A 21st Century View
• Conclusions
– Timeless principles and aging practices
05/25/06 44
The Future of Systems and Software
• Eight surprise-free trends1. Increasing integration of SysE and SwE2. User/Value focus3. Software Criticality and Dependability4. Rapid, Accelerating Change5. Distribution, Mobility, Interoperability,
Globalization6. Complex Systems of Systems7. COTS, Open Source, Reuse, Legacy Integration8. Computational Plenty
• Two wild-card trends9. Autonomy Software10.Combinations of Biology and Computing
05/25/06 45
Outline
• Motivation and Context
• A 20th Century View
• A 21st Century View
• Conclusions
– Timeless principles and aging practices
05/25/06 46
Timeless Principles (+) and Aging Practices (-)
• From the 1950’s+ Don’t neglect the sciences+ Look before you leap (avoid premature
commitments)- Avoid inflexible sequential processes
• From the 1960’s+ Think outside the box+ Respect software’s differences- Avoid cowboy programming
05/25/06 47
Timeless Principles (+) and Aging Practices (-)
• From the 1970’s+ Eliminate errors early+ Determine the system’s purpose- Avoid top-down development and reductionism
• From the 1980’s+ These are many roads to increased productivity+ What’s good for products is good for processes- Be skeptical about silver bullets
05/25/06 48
Timeless Principles (+) and Aging Practices (-)
• From the 1990’s+ Time is money and value to people+ Make software useful to people- Be quick, but don’t hurry
• From the 2000’s+ If change is rapid, adaptability trumps
repeatability+ Consider and satisfice all of your stakeholders’
value propositions- Avoid falling in love with your slogans
05/25/06 49
Timeless Principles (+) and Aging Practices (-)
• For the 2010’s+ Keep your reach within your grasp+ Have an exit strategy- Don’t believe everything you read
- “It’s true because I read it on the Internet”
05/25/06 50
The END
• Recommend papers– A View of 20th and 21st Century Software
Engineering– Fifty Years of Progress in Software Engineering– Finding a History for Software Engineering
• Next Lecture– Software developments activities