Agile New England July 2011
Copyright © 2011 Philippe Kruchten 1
Agility and Architecture or: What colours is your backlog?
Philippe Kruchten
July 7, 2011
Philippe Kruchten, Ph.D., P.Eng., CSDP Professor of So*ware Engineering NSERC Chair in Design Engineering Department of Electrical and Computer Engineering University of BriLsh Columbia Vancouver, BC Canada [email protected]
Founder and president Kruchten Engineering Services Ltd Vancouver, BC Canada [email protected]
Co founder and past-‐chair
2
Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 2
Outline
1. The frog and the octopus 2. Architecture and agility 3. Release planning 4. Technical debt 5. Architecture, agility,… revisited
Copyright © 2011 Philippe Kruchten 5
A Conceptual Model of SoYware Development
4 key concepts, 5 key aZributes
Intent Product Work People
Time Quality Risk
8 Copyright © 2011 Philippe Kruchten
Value Cost
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 3
Intent, Work, People, Product
9 Copyright © 2011 Philippe Kruchten
Copyright © 2011 Philippe Kruchten 10
Frog: “All projects are the same”
Value Value
Cost Cost
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 4
11 Copyright © 2011 Philippe Kruchten
Octopus: “All projects are different!”
Copyright © 2011 Philippe Kruchten 12
Context
Size CriLcality
Business model
Stable architecture Team
distribuLon
Governance
Rate of change
Age of the
system
Domain, Industry
Corporate & Na7onal Culture
Organiza7onal Maturity
Degree of Innova7on
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 5
SW Dev. Project: Tension between Intent and Product
Intent
Product
Work V
T 13 Copyright © 2011 Philippe Kruchten
IteraLons
14 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 6
Outline
1. The frog and the octopus 2. Architecture and agility 3. Release planning 4. Technical debt 5. Architecture, agility,… revisited
Copyright © 2011 Philippe Kruchten 15
Agile & Architecture? Oil & Water?
• Paradox • Oxymoron • Conflict • IncompaLbility
Copyright © 2011 Philippe Kruchten 16
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 7
What is Agility?
• Jim Highsmith (2002): – Agility is the ability to both create and respond to change in order to profit in a turbulent business environment.
• Sanjiv AugusLne (2004): – IteraLve and incremental – Small release – CollocaLon – Release plan/ feature backlog – IteraLon plan/task backlog
Copyright © 2011 Philippe Kruchten 17
Geqng at the Essence of Agility • SoYware development is a knowledge acLvity
– Not producLon, manufacturing, administraLon… • The “machines” are humans • Dealing with uncertainty, unknowns, fear, distrust • Feedback loops ➜
– reflect on business, requirements, risks, process, people, technology
• CommunicaLon and collaboraLon – Building trust ➜ rely on tacit informaLon ➜ reduce waste
Copyright © 2011 Philippe Kruchten 18
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 8
SoYware Architecture: A DefiniLon
" " " "“Itʼs the hard stuff.”"“Itʼs the stuff that will be hard to change”"
M.Fowler, cited by J. Highsmith Copyright © 2011 Philippe Kruchten
20
ISO/IEC 42010
Architecture: the fundamental concepts or properLes of a system in its environment embodied in its elements, their relaLonships, and in the principles of its design and evoluLon
21 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 9
SoYware Architecture SoYware architecture encompasses the set of significant decisions about
• the organizaLon of a soYware system, • the selecLon of the structural elements and their interfaces by which the system is composed together with their behavior as specified in the collaboraLon among those elements,
• the composiLon of these elements into progressively larger subsystems,
Grady Booch, Philippe Kruchten, Rich Reitman, Kurt BiCner; RaLonal, circa 1995 (derived from Mary Shaw) 22 Copyright © 2011 Philippe Kruchten
SoYware Architecture (cont.) …
• the architectural style that guides this organizaLon, these elements and their interfaces, their collaboraLons, and their composiLon.
• SoYware architecture is not only concerned with structure and behavior, but also with usage, funcLonality, performance, resilience, reuse, comprehensibility, economic and technological constraints and tradeoffs, and aestheLcs.
23 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 10
Perceived Tensions Agility-‐ Architecture
• Architecture = Big Up-‐Front Design • Architecture = massive documentaLon • Architects dictate form their ivory tower
• Low perceived or visible value of architecture • Loss of rigour, focus on details • Disenfranchisement • Quality aZribute not reducible to stories
Copyright © 2011 Philippe Kruchten 24
HazraL, 2008 Rendell, 2009 Blair et al. 2010, etc.
Perceived Tensions Agility-‐ Architecture
AdaptaLon versus AnLcipaLon
Copyright © 2011 Philippe Kruchten 25
Highsmith 2000
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 11
Issues 1. SemanLcs 2. Scope 3. Lifecycle 4. Role 5. DescripLon 6. Methods 7. Value & cost
Copyright © 2011 Philippe Kruchten 26
SemanLcs
• What do we mean by “architecture”?
• What do we mean by “soYware architecture”?
Copyright © 2011 Philippe Kruchten 28
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 12
Issues 1. SemanLcs 2. Scope 3. Lifecycle 4. Role 5. DescripLon 6. Methods 7. Value & cost
Copyright © 2011 Philippe Kruchten 29
Scope
• How much architecture “stuff” do you really need?
• It depends…
• It depends on your context
Copyright © 2011 Philippe Kruchten 30
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 13
Context aZributes
1. Size 2. CriLcality 3. Age of system 4. Rate of change 5. Business model 6. Domain 7. Team distribuLon 8. Governance
Age of System
Rate of change
Governance
Team Distribu7on
Domain
Business model
Cri7cality
Size
Context
31
Copyright © 2011 Philippe Kruchten
All soYware-‐intensive systems have an architecture
• How much effort should you put into it varies greatly
• 75% of the Lme, the architecture is implicit – Choice of technology, plauorm – SLll need to understand the architecture
• Novel systems: – Much more effort in creaLng and validaLng an architecture
• Key drivers are mostly non-‐funcLonal: – RunLme: Capacity, performance, availability, security – Non runLme: evolvability, regulatory, i18n/L10n…
32 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 14
Lifecycle
• When does architectural acLviLes take place? • The evil of “BUFD” = Big Up-‐Front Design • “Defer decisions to the last responsible moment”
• YAGNI = You Ain’t Gonna Need It • Refactor!
Copyright © 2011 Philippe Kruchten 34
Architectural Effort During the Lifecycle
time
Inception Elaboration Construction Transition
Majority of architectural design activities
Copyright © 2011 Philippe Kruchten 35
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 15
LiZle dedicated architectural effort
time
Inception Construction Transition
Minimal pure Architectural
Activities Ideal realm of agile
practices
Copyright © 2011 Philippe Kruchten 36
An architectural iteration focuses in putting in place major architectural elements, resulting in a baseline architectural prototype at the end of elaboration.
Iterations with focus on architecture
IteraLons and Phases
Iterations with main focus on features
Copyright © 2011 Philippe Kruchten 37
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 16
Team Structure over Time (Very Large)
Initial team Architecture
team
integration team
Architecture team
Feature team 2
Feature team 1
Feature team 3 Infrastructure team A
Infrastructure team B
Prototyping team
Management team Management team
Construction and Transition Elaboration Inception
Copyright © 2011 Philippe Kruchten 38
Teams using agile development pracLces
Initial team Architecture
team
integration team
Architecture team
Feature team 2
Feature team 1
Feature team 3 Infrastructure team A
Infrastructure team B
Prototyping team
Management team Management team
Construction and Transition Elaboration Inception
Copyright © 2011 Philippe Kruchten 39
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 17
Issues 1. SemanLcs 2. Scope 3. Lifecycle 4. Role 5. DescripLon 6. Methods 7. Value & cost
Copyright © 2011 Philippe Kruchten 40
New Role – Agile Architect ?
• A. Johnston defines the agile architect, but it does not seems to be any different from a soYware architect before agile methods came in.
• CombinaLon of – Visionary -‐ Shaper – Designer – making choices – Communicator – between mulLple parLes – Troubleshooter – Herald – window of the project – Janitor – cleaning up behind the PM and the developers
Copyright © 2011 Philippe Kruchten 42
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 18
FuncLons of the soYware architect
Defini7on of the architecture
• Architecture definiLon • Technology selecLon • Architectural evaluaLon
• Management of non funcLonal requirements
• Architecture collaboraLon
Delivery of the architecture
• Ownership of the big picture • Leadership • Coaching and mentoring • Design, development and
TesLng
• Quality assurance
Copyright © 2011 Philippe Kruchten 43
Brown 2010
Two styles of soYware/system architects
• Maker and Keeper of Big decisions – Bring in technological changes
– External collaboraLon – More requirements-‐facing – Gatekeeper – Fowler: Architectus reloadus
• Mentor, Troubleshooter, and Prototyper – Implements and try architecture
– Intense internal collaboraLon – More code-‐facing
– Fowler: Architectus oryzus
Only big new projects need both or separate people Copyright © 2011 Philippe Kruchten 44
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 19
Team Structure over Time (Very Large)
Initial team Architecture
team
integration team
Architecture team
Feature team 2
Feature team 1
Feature team 3 Infrastructure team A
Infrastructure team B
Prototyping team
Management team Management team
Construction and Transition Elaboration Inception
Copyright © 2011 Philippe Kruchten 45
A. Reloadus and A. Oryzus ecological niches
Initial team Architecture
team
integration team
Architecture team
Feature team 2
Feature team 1
Feature team 3 Infrastructure team A
Infrastructure team B
Prototyping team
Management team Management team
Construction and Transition Elaboration Inception
A. Reloadus
A. Oryzus Copyright © 2011 Philippe Kruchten 46
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 20
A. Reloadus and A. Oryzus ecological niches
Initial team Architecture
team
integration team
Architecture team
Feature team 2
Feature team 1
Feature team 3 Infrastructure team A
Infrastructure team B
Prototyping team
Management team Management team
Construction and Transition Elaboration Inception
A. Reloadus
A. Aryzus A. Aryzus A. Oryzus Copyright © 2011 Philippe Kruchten 47
Role
Architecture owner
Copyright © 2011 Philippe Kruchten 48
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 21
Issues 1. SemanLcs 2. Scope 3. Lifecycle 4. Role 5. DescripLon 6. Methods 7. Value & cost
Copyright © 2011 Philippe Kruchten 49
Architectural descripLon
• Metaphor (XP) • Prototype • SoYware architecture document
• Use of UML? • UML-‐based tools? • Code?
Logical View Implementation View
Process View
Deployment View
Use Case View
Copyright © 2011 Philippe Kruchten 50
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 22
51 Copyright © 2011 Philippe Kruchten
UML 2.0
• A notaLon • BeZer “box and arrows” • Crisper semanLcs • Almost an ADL ?
• Model-‐driven design, • Model-‐driven architecture.
52 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 23
Again, it depends on the context
1. Size 2. CriLcality 3. Age of system 4. Rate of change 5. Business model 6. Stable architecture 7. Team distribuLon 8. Governance
Age of System
Rate of change
Governance
Team Distribu7on
Stable Architecture
Business model
Cri7cality
Size
Context
53
Copyright © 2011 Philippe Kruchten
AdaptaLon versus AnLcipaLon
Copyright © 2011 Philippe Kruchten 54
Highsmith 2000
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 24
Issues 1. SemanLcs 2. Scope 3. Lifecycle 4. Role 5. DescripLon 6. Methods 7. Value & cost
Copyright © 2011 Philippe Kruchten 55
Architectural design methods
• Many agile developers do not know (much) about architectural design
• Agile methods have no explicit guidance for architecture – Metaphor in XP – Technical acLviLes in scrum
• Relate this to SemanLcs and Scope issue
• May have to get above the code level Copyright © 2011 Philippe Kruchten 56
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 25
Architectural Methods
• ADD, ATAM, QAW (SEI) • RUP (IBM) • SAV,… (Siemens) • BAPO/CAFR (Philips) • Etc. ….
• SoYware Architecture Review and Assessment (SARA) handbook
57 Copyright © 2011 Philippe Kruchten
Architectural Design
Ideas
Architecture
Architectural Evalua5on
Architectural Synthesis
Ideas
Context, Constraints
Evalua7on results
Architectural Assets
Architecturally
Requirements
Architecture
Backlog
Architectural Evalua5on
Architectural Synthesis
Architectural Analysis
Significant
58 Copyright © 2011 Philippe
Kruchten Source: Hofmeister, Kruchten, et al., 2005, 2007
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 26
IteraLve Architecture Refinement
• There are no fixed prescripLons for systemaLcally deriving architecture from requirements; there are only guidelines.
• Architecture designs can be reviewed. • Architectural prototypes can be thoroughly tested.
• IteraLve refinement is the only feasible approach to developing architectures for complex systems.
59 Copyright © 2011 Philippe Kruchten
Issues 1. SemanLcs 2. Scope 3. Lifecycle 4. Role 5. DescripLon 6. Methods 7. Value & cost
Copyright © 2011 Philippe Kruchten 60
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 27
Value and Cost
• Value: to the business (the users, the customers, the public, etc.)
• Cost: to design, develop, manufacture, deploy, maintain
• Simple system, stable architecture, many small features: – StaLsLcally value aligns to cost
• Large, complex, novel systems ? 61 Copyright © 2011 Philippe Kruchten
Outline
1. The frog and the octopus 2. Architecture and agility 3. Release planning 4. Technical debt 5. Architecture, agility,… revisited
Copyright © 2011 Philippe Kruchten 63
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 29
Feature Requests
Priority Sprint Backlog
66 Steve Adolph 2008 Copyright © 2011 Philippe Kruchten
Time-‐box
Staff
Time
Work (≈Cost)
69 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 30
Time-‐boxes: Releases
Time
Release 1 R2 R3 R4
71 Copyright © 2011 Philippe Kruchten
Time-‐boxes: IteraLons (sprints)
Time
Release N
ITERATION 1 It. 2 It. 3
72 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 31
Features
Rn
75 Copyright © 2011 Philippe Kruchten
Work and Cost
• How much work is associated to a feature? • Work is strongly related to cost in soYware development (a human-‐intensive acLvity)
• Overall budget is roughly the size of the Lme-‐box(es)
• Time-‐box = budget • Features must fit in budget • Q: How do we select what goes in the box?
76 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 32
Maximizing value
4
4
6
5
2
7
3
12
3
5
Highest value first Ignore Lme
79 Copyright © 2011 Philippe Kruchten
Value = Cost?
4
4
6
5 2
7 3
12
3 5
$8
$15
$5
$5
$2
$5
$5
$5
$5
$1
80 Only for simplest cases Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 33
Value /= Cost
6 $8
12 $15
3 $5
3 $5
7 $2
4 $5
2 $5
5 $5
4 $5 5 $1
Value
Cost 81 Copyright © 2011 Philippe Kruchten
Value and Cost
• Value: to the business (the users, the customers, the public, etc.)
• Cost: to design, develop, manufacture, deploy, maintain
• Simple system, stable architecture, many small features: – StaLsLcally value aligns to cost
• Large, complex, novel systems ? 82 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 34
Value
Cost 83 Copyright © 2011 Philippe Kruchten
Efficiency vs. EffecLveness
Efficiency
• relaLonship between the output in terms of goods, services or other results and the resources used to produce them
Effec7veness
• relaLonship between the intended impact and the actual impact of an acLvity
84
Cost Value
Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 35
“Spent-‐Cost” System
86
Actual COST of Work Performed
Budgeted COST of Work Schedule
d
Budgeted COST of Work Performe
d
Copyright © 2011 Philippe Kruchten
Invisible Features
12 12
?
$10 $15
$5
88 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 36
Invisible Features
• Architecture • Infrastructure • Common elements • Framework • Libraries • Reuse • DSL • Product line
89 Copyright © 2011 Philippe Kruchten
Features
90 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 37
Dependencies
91 Copyright © 2011 Philippe Kruchten
Release Planning
• Time-‐box = budget • Fill the Lme-‐box with a combinaLon of visible and invisible features
• … while maximizing value
• Easy, no?
92 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 38
Tension
• Product manager: maximize value (green stuff)
• Project manager: maximize budget uLlizaLon – i.e., minimize cost
• Techie: maximize the fun stuff (yellow) ?
93 Copyright © 2011 Philippe Kruchten
Value for the yellow stuff: HeurisLcs
• Value of invisible feature = Max (value of all dependents)
• Value of invisible feature = Max + f(number of dependents)
• Value of invisible feature = total value achievable if implemenLng it – total value achievable without implemenLng it
• … (Not there yet, more research need to happen)
99 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 39
More on value & cost
• CBAM = Cost Benefit Analysis Method – Chap 12 in Bass, Clements, Kazman 2003
• IMF: Incremental Funding Method – Denne & Cleland-‐Huang, 2004 So*ware by numbers
• AnalyLc Hierarchy Process (AHP) Saaty, 1990 • Evolve* -‐ Hybrid
– Günther Ruhe & D. Greer 2003, etc…
100 Copyright © 2011 Philippe Kruchten
IFM: Incremental Funding Method
• MMF = Minimum Marketable Features • AE = Architectural elements • Cost • MMF depends on AE • Time and NPV = Net Present Value • Strands = Sequences of dependent MMFs • HeurisLc
101
Denne & Huang, www.soYwarebynumbers.org
Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 40
Visible & Invisible Features
103 Copyright © 2011 Philippe Kruchten
EsLmaLon
• Cost esLmaLon • Work • EsLmate
– Ideal case? • Things go wrong
– Worse case? • ∑ all worse cases = impossible implementaLon
104 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 41
Buffers
• E. GoldraZ: Theory of constraints • D. Anderson: Agile Project Management
• Buffer: unallocated effort (work) • Shared by all staff members and all explicit work
105 Copyright © 2011 Philippe Kruchten
Time-‐box with Buffer
107 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 42
Defects
• Defect = Feature with negaLve value
• Fixing a defect has a posiLve cost (work)
• Time/place of discovery – Inside development (in-‐house, in process) – Outside development (out-‐house?) in a released product (escaped defects)
109 Copyright © 2011 Philippe Kruchten
Escaped Defect has Value
Perfect product Imperfect product Defect
110 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 43
Fixing a Defect has a Cost
• Defects have both value and cost • Value of fixing a defect = −Value of the defect • Cost of fixing a defect (esLmated, actual)
• Defects have dependencies – Defect fix depend on invisible feature – Visible feature depending on a fix
111 Copyright © 2011 Philippe Kruchten
Visible & Invisible Features + Defects fixing
113 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 44
Outline
1. The frog and the octopus 2. Architecture and agility 3. Release planning 4. Technical debt 5. Architecture, agility,… revisited
Copyright © 2011 Philippe Kruchten 116
Technical Debt
• Concept introduced by Ward Cunningham • OYen menLoned, rarely studied • All experienced SW developers “feel” it. • Drags long-‐lived projects and products down
117 Cunningham, OOPSLA 1992 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 45
Origin of the metaphor
• Ward Cunningham, at OOPSLA 1992
“Shipping first Lme code is like going into debt. A liZle debt speeds development so long as it is paid back promptly with a rewrite… The danger occurs when the debt is not repaid. Every minute spent on not-‐quite-‐right code counts as interest on that debt. EnLre engineering organizaLons can be brought to a stand-‐sLll under the debt load of an unconsolidated implementaLon, object-‐oriented or otherwise.”
Cunningham, OOPSLA 1992
118 Copyright © 2011 Philippe Kruchten
Technical Debt (S. McConnell)
• Implemented features (visible and invisible) = assets = non-‐debt
• Type 1: unintenLonal, non-‐strategic; poor design decisions, poor coding
• Type 2: intenLonal and strategic: opLmize for the present, not for the future. – 2.A short-‐term: paid off quickly (refactorings, etc.)
• Large chunks: easy to track • Many small bits: cannot track
– 2.B long-‐term McConnell 2007
119 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 46
Technical Debt (M. Fowler)
Fowler 2009, 2010
120 Copyright © 2011 Philippe Kruchten
First more capabiliLes
First more infrastructure
Then, more infrastructure
Then, more capabiliLes
underesLmated re-‐architecLng costs
neglected cost of delay to market
need to monitor technical debt to gain insight into life-‐cycle efficiency
Example
Ozkaya, SEI,2010 121 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 47
Technical Debt (Chris Sterling)
• Technical Debt: issues found in the code that will affect future development but not those dealing with feature completeness.
Or
• Technical Debt is the decay of component and intercomponent behaviour when the applicaLon funcLonality meets a minimum standard of saLsfacLon for the customer.
122 Copyright © 2011 Philippe Kruchten
Time is Money (I. Gat)
• Time is money: Think of the amount of money the borrowed Lme represents – the grand total required to eliminate all issues found in the code
Gat 2010
123 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 48
TD is the sum of…
• Code smells 167 person days • Missing test 298 person days • Design 670 person days • DocumentaLon 67 person days
Totals Work 1,202 person x days Cost $577,000
124 Copyright © 2011 Philippe Kruchten
Israel Gat, 2010 hZp://theagileexecuLve.com/2010/09/20/how-‐to-‐break-‐the-‐vicious-‐cycle-‐of-‐technical-‐debt/
(more) Relentless Pressure
Take Technical Debt
Fail to Pay back
Technical debt
Technical Debt Accrues
Reduced Development
Team Velocity
125 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 49
Tech Debt (Jim Highsmith)
Source: Highsmith, 2009 126 Copyright © 2011 Philippe Kruchten
Technical Debt (1)
12
12
a
$15
$5
12
b
$16
$3
12 $18
$20 $19 $18
127 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 50
Technical Debt (2)
12
12
a
$15
$5
12
b
$16
$3
12 $18
8 8 $5 8 $8 8 $10
$25 $27 $28
128 Copyright © 2011 Philippe Kruchten
Technical Debt (3)
12
12
a
+$2
$5
12 $18
8 8 $5
$30
129 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 51
Technical Debt
• Defect = Visible feature with negaLve value • Technical debt = Invisible “feature” with negaLve value
• Cost …. of fixing • Value …. of repaying technical debt ???
130 Copyright © 2011 Philippe Kruchten
“Interests” ?
• In presence of technical debt: Cost of adding new features is higher
• When repaying (fixing), addiLonal cost for retrofiqng already implemented features
• Technical debt not repaid => lead to increased cost, forever
• Cost of fixing increases over Lme
131 M. Fowler Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 52
TD and Real OpLons
P1: S0
Market loves it + $4M
Market hates it + $1M
S1
NPV (P1) = -‐2M + 0.5x4M + 0.5x1M = 0.5M
-‐2M
p=0.5
p=0.5
Source: K. Sullivan, 2010 at TD Workshop SEI 6/2-‐3
132 Copyright © 2011 Philippe Kruchten
TD and Real OpLons (2)
P2: S0
Market loves it
Market hates it + $1M
Sd
NPV (P2) = -‐1M + 0.5x3M + 0.5x1M = 1M
-‐1M
Source: K. Sullivan, 2010
p=0.5
p=0.5
-‐1M S1 +4M
Taking Technical Debt has increased system value.
133 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 53
TD and Real OpLons (3)
P2: S0
Market loves it
Market hates it + $1M
Sd
NPV (P3) = -‐1M + 0.67 x 2.5M + 0.33 x 1M = 1M
-‐1M
p=0.33
p=0.67
-‐1.5M S1 +4M
More realisLcally: Debt + interest High chances of success
Higher chance of success
Repay debt + 50% interest
135 Copyright © 2011 Philippe Kruchten
TD and Real OpLons (4)
S0
Favourable
Unfavourable
Sd
p=?
p=?
S1 S2
S2d
…..
…..
Not debt really, but op7ons with different values… Do we want to invest in architecture, in test, etc…
Refactor
Add feature
Add feature
?
Source: K. Sullivan, 2010
136 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 54
• Technical debt is more a rhetorical category than a technical or ontological category. – The concept resonates well with the development community and the business community
– Both sides “get” the metaphor.
• Technical debt is a concept that bridges the gap between: – Business decisions makers – SoYware designers/developers
137 Copyright © 2011 Philippe Kruchten
Buffer for debt repayment
Simple work EsLmate uncertainLes
Defect correcLon
Debt Repayment
138 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 55
Colours in your Backlog
Visible Feature
Hidden, architectural feature
Visible defect
Technical Debt
Visible Invisible
PosiLve Value
NegaLve Value
139 Copyright © 2011 Philippe Kruchten
• YAGNI = You Ain’t Gonna Need It – But when you do, it is technical debt – Technical debt oYen is the accumulaLon of too many YAGNI decisions
• Again the tension between the yellow stuff and the green stuff.
140 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 56
Visible & Invisible Features + Defects fixing + Technical Debt payment
148 Copyright © 2011 Philippe Kruchten
Time and depreciaLon
Denne & Huang 2004 149 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 57
Net Present Value
150 Copyright © 2011 Philippe Kruchten
Value decreases
Time
R1 R2 R3 R4
8
152
8 7.5 7 6
Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 58
Technical debt: increase (?)
Time
R1 R2 R3 R4
8
153
8 8.5 9 10
Copyright © 2011 Philippe Kruchten
155
BASED on by Nicolas Kruchten & co
Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 59
156 Based on Redmine, by Chris Nicola Copyright © 2011 Philippe Kruchten
Risks & UncertainLes
Rule of thumb: • Green stuff: move up
– defer • Yellow stuff: move down
– Experiment now
158
Karl Wiegers, 1999 RUP, 1998
Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 60
Outline
1. The frog and the octopus 2. Architecture and agility 3. Release planning 4. Technical debt 5. Architecture, agility,… revisited
Copyright © 2011 Philippe Kruchten 159
Copyright © 2011 Philippe Kruchten 160
What would Frog say?
Value Value
Cost Cost
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 61
What would Octopus say?
Copyright © 2011 Philippe Kruchten 161
Context
Size CriLcality
Business model
Stable architecture Team
distribuLon
Governance
Rate of change
Age of the
system
Domain, Industry
Corporate & Na7onal Culture
Organiza7onal Maturity
Degree of Innova7on
Architecture: Value and Cost
• Architecture has no (or liZle) externally visible “customer value”
• IteraLon planning (backlog) is driven solely by “customer value”
• YAGNI, BUFD, Metaphor… • “Last responsible moment!” & Refactor! • Ergo: architectural acLviLes are not given proper aZenLon
• Ergo: large technical debts
162 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 62
Role of Architecture
• Novel system • Gradual emergence of architecture • ValidaLon of architecture with actual funcLonality
• Early enough to support development
Zipper model… • Not just BUFD • No YAGNI effect
163 Copyright © 2011 Philippe Kruchten
Planning
• From requirements derive: – Architectural requirements – FuncLonal requirements
• Establish – Dependencies – Cost
• Plan interleaving: – FuncLonal increments – Architectural increments
Copyright © 2011 Philippe Kruchten 164
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 63
Zipper: Weaving the funcLonal and architectural work items
165 Copyright © 2011 Philippe Kruchten
Benefits
• Gradual emergence of architecture • ValidaLon of architecture with actual funcLonality
• Early enough to support development
• Not just BUFD • No YAGNI effect
Copyright © 2011 Philippe Kruchten 166
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 64
SuggesLons for project management
• Separate the processes for esLmaLon of cost and value
• Avoid monetary value (points & uLls) • IdenLfy invisible features and make them more visible to more stakeholders
• Allocate value to invisible feature • Use nominal and worse case esLmates for cost (effort); create shared buffers
167 Copyright © 2011 Philippe Kruchten
SuggesLons (cont.)
• Manage all elements together
• Make technical debt visible – Large chunks (McConnell type 2)
• Assign some value to technical debt type 2.B and include in backlog
• Exploit different type of buffers 168 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 65
Manage all colours in your backlog!
169 Copyright © 2011 Philippe Kruchten
3 Kinds of Buffers
Straighforward work
EsLmate uncertainLes
Defect correcLon
Debt Repayment
171
To care of:
Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 66
Toward the 1st release
172 Copyright © 2011 Philippe Kruchten
A later release
173 Copyright © 2011 Philippe Kruchten
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 67
AdaptaLon versus AnLcipaLon
Copyright © 2011 Philippe Kruchten 174
Highsmith 2000
Four take-‐aways
• Put it in context
• DisLnguish value and cost
• Define an “Architecture owner”
• Expose & manage technical debt
Copyright © 2011 Philippe Kruchten 175
$ vs $
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 68
Game to Introduce Tech Debt www.sei.cmu.edu/architecture/tools/hardchoices/
177 Copyright © 2011 Philippe Kruchten
Mission to mars A Release Planning game
Copyright © 2011 Philippe Kruchten 178
philippe.kruchten.com/mtm
with J. King SoYEd, Aus.
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 69
StarLng with soYware architecture Gorton, I. (2006). EssenVal so*ware architecture. Berlin: Springer. Rozanski, N., & Woods, E. (2005). So*ware Systems Architecture:
Working With Stakeholders Using Viewpoints and PerspecVves. Boston: Addison-‐Wesley.
Bass, L., Clements, P., & Kazman, R. (2003). So*ware Architecture in PracVce (2nd ed.). Reading, MA: Addison-‐Wesley.
Fairbanks, G. (2010). Just enough so*ware architecture. Boulder, Co: Marshall and Brainerd.
Kruchten, P., Obbink, H., & Stafford, J. (2006). The past, present and future of soYware architecture. IEEE So*ware, 23(2), 22-‐30.
Brown, S. (Feb. 9, 2010) Are you an architect?, InfoQ hZp://www.infoq.com/arLcles/brown-‐are-‐you-‐a-‐soYware-‐architect.
Fowler, M. (2003) Who needs an architect?, IEEE So*ware, 20(4), 2-‐4.
179 Copyright © 2011 Philippe Kruchten
Agility & architecture • Abrahamsson, P., Ali Babar, M., & Kruchten, P. (2010). Agility and Architecture: Can they Coexist?
IEEE So*ware, 27(2), 16-‐22. • Ambler, S. W. (2006). Scaling Agile Development Via Architecture [Electronic Version]. Agile
Journal, from hZp://www.agilejournal.com/content/view/146/ • Blair, S., WaZ, R., & Cull, T. (2010). Responsibility-‐Driven Architecture. IEEE So*ware, 27(2), 26-‐32. • Brown, S. (2010), "Are you an architect?," InfoQ, hZp://www.infoq.com/arLcles/brown-‐are-‐you-‐a-‐
soYware-‐architect • Brown, N., Nord, R., Ozkaya, I. 2010. Enabling Agility through Architecture, Crosstalk, Nov/Dec
2010. • Clements, P., Ivers, J., LiZle, R., Nord, R., & Stafford, J. (2003). DocumenVng So*ware Architectures
in an Agile World (Report CMU/SEI-‐2003-‐TN-‐023). PiZsburgh: SoYware Engineering InsLtute. • HazraL, V. (2008, Jan.6) "The Shiny New Agile Architect," in Agile Journal.
hZp://www.agilejournal.com/arLcles/columns/column-‐arLcles/739-‐the-‐shiny-‐new-‐agile-‐architect • Johnston, A., The Agile Architect, hZp://www.agilearchitect.org/ • Madison, J. (2010). Agile-‐Architecture InteracLons. IEEE So*ware, 27(2), 41-‐47. • Nord, R. L., & Tomayko, J. E. (2006). SoYware Architecture-‐Centric Methods and Agile
Development. IEEE So*ware, 23(2), 47-‐53. • Parsons, R. (2008). Architecture and Agile Methodologies—How to Get Along. Tutorial At WICSA
2008, Vancouver, BC. • Rendell, A. (2009) "Descending from the Architect's Ivory Tower," in Agile 2009 Conference, A.
Sidky, et al., eds. IEEE Computer Society, pp. 180-‐185. • Woods, E. (2010). Agile Principles and So*ware Architecture, presentaLon at OOP 2010 Conf.,
Munich, Jan 26. Copyright © 2011 Philippe Kruchten 180
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 70
References (1) • Agile Alliance (2001), "Manifesto for Agile SoYware Development," Retrieved May 1st, 2007 from
hZp://agilemanifesto.org/ • Abrahamsson, P., Ali Babar, M., & Kruchten, P. (2010). Agility and Architecture: Can they Coexist?
IEEE So*ware, 27(2), 16-‐22. • Ambler, S. W. (2006). Scaling Agile Development Via Architecture [Electronic Version]. Agile
Journal, from hZp://www.agilejournal.com/content/view/146/ • AugusLne, S. (2004), Agile Project Management, Addison Wesley Longman • Bass, L., Clements, P., & Kazman, R. (2003). So*ware Architecture in PracVce (2nd ed.). Reading,
MA: Addison-‐Wesley. • Beck, K., & Fowler, M. (2001). Planning Extreme Programming. Boston: Addison-‐Wesley. • Blair, S., WaZ, R., & Cull, T. (2010). Responsibility-‐Driven Architecture. IEEE So*ware, 27(2), 26-‐32. • Brown, S. (2010), "Are you an architect?," InfoQ, hZp://www.infoq.com/arLcles/brown-‐are-‐you-‐a-‐
soYware-‐architect • Brooks, F. (1975) The mythical man-‐month, Reading, MA: Addison-‐Wesley. • Brown, N., Cai, Y., Guo, Y., Kazman, R., Kim, M., Kruchten, P., et al. (2010). Managing Technical Debt
in SoYware-‐Intensive Systems. Paper presented at the Future of soYware engineering research (FoSER) workshop, part of FoundaLons of SoYware Engineering (FSE 2010) conference
• Brown, N., Nord, R., Ozkaya, I. 2010. Enabling Agility through Architecture, Crosstalk, Nov/Dec 2010.
• Cohn, M. (2006) Agile EsVmaVng and Planning. Upper Saddle River, N.J.: PrenLce-‐Hall.
Copyright © 2011 Philippe Kruchten 181
References (2) • Clements et al. (2005). DocumenVng So*ware Architecture, Addison-‐Wesley. • Clements, P., Ivers, J., LiZle, R., Nord, R., & Stafford, J. (2003). DocumenVng So*ware Architectures
in an Agile World (Report CMU/SEI-‐2003-‐TN-‐023). PiZsburgh: SoYware Engineering InsLtute. • Cunningham, W. 1992. The WyCash Poruolio Management System. OOPSLA '92 Experience Report.
hZp://c2.com/doc/oopsla92.html. • Denne, M., & Cleland-‐Huang, J. (2004). SoYware by Numbers: Low-‐Risk, High-‐Return Development,
PrenLce Hall. • Faber, R. (2010). Architects as Service Providers. IEEE So*ware, 27(2), 33-‐40. • Fowler, M. (2003). Who needs an architect? IEEE So*ware, 20(4), 2-‐4. • Fowler, M. (2004) Is design dead? At hZp://marLnfowler.com/arLcles/designDead.html • Fowler, M.( 2009) Technical debt quadrant, Blog post at:
hZp://marLnfowler.com/bliki/TechnicalDebtQuadrant.html. • Gat, I., Heintz, J. (Aug. 19, 2010) Webinar: Reining in Technical Debt, CuZer ConsorLum. • HazraL, V. (2008, Jan.6) "The Shiny New Agile Architect," in Agile Journal.
hZp://www.agilejournal.com/arLcles/columns/column-‐arLcles/739-‐the-‐shiny-‐new-‐agile-‐architect • Johnston, A., The Agile Architect, hZp://www.agilearchitect.org/ • Karlsson, J. & Ryan, K. (1997). A Cost-‐Value Approach for PrioriLzing Requirements, IEEE SoYware,
14 (5) 67-‐74. • Kniberg, H. (2008) Technical debt-‐How not to ignore it, at Agile 2008 conference
Copyright © 2011 Philippe Kruchten 182
Agile New England July 2011
Copyright © 2011 Philippe Kruchten 71
References (3) • Kruchten, P. (1995). The 4+1 View Model of Architecture. IEEE So*ware, 12(6), 45-‐50. • Kruchten, P. (1999). The SoYware Architect, and the SoYware Architecture Team. In P.
Donohue (Ed.), So*ware Architecture (pp. 565-‐583). Boston: Kluwer Academic Publishers.
• Kruchten, P. (March 2001). The Tao of the SoYware Architect. The RaVonal Edge. At hZp://www-‐106.ibm.com/developerworks/raLonal/library/4032.html
• Kruchten, P. (2003). The RaVonal Unified Process: An IntroducVon (3rd ed.). Boston: Addison-‐Wesley.
• Kruchten, P. (2004). Scaling down projects to meet the Agile sweet spot. The RaVonal Edge. hZp://www-‐106.ibm.com/developerworks/ raLonal/library/content/RaLonalEdge/aug04/5558.html
• Kruchten, P. (2008). What do soYware architects really do? Journal of Systems & So*ware, 81(12), 2413-‐2416.
• Madison, J. (2010). Agile-‐Architecture InteracLons. IEEE So*ware, 27(2), 41-‐47. • McConnell, S. (2007). Technical Debt. 10x SoYware Development [cited 2010 June 14];
hZp://blogs.construx.com/blogs/stevemcc/archive/2007/11/01/technical-‐debt-‐2.aspx. • Mills, J. A. (1985). A PragmaLc View of the System Architect. Comm. ACM, 28(7),
708-‐717. • Nord, R. L., & Tomayko, J. E. (2006). SoYware Architecture-‐Centric Methods and Agile
Development. IEEE So*ware, 23(2), 47-‐53. Copyright © 2011 Philippe Kruchten 183
References (4) • McConnell, S. (20087) Notes on Technical Debt, Blog post at:
hZp://blogs.construx.com/blogs/stevemcc/archive/2007/11/01/technical-‐debt-‐2.aspx • Parsons, R. (2008). Architecture and Agile Methodologies—How to Get Along. Tutorial
At WICSA 2008, Vancouver, BC. • Qumer, A., & Henderson-‐Sellers, B. (2008). An evaluaLon of the degree of agility in six
agile methods and its applicability for method engineering. InformaVon and So*ware Technology, 50(4), 280-‐295.
• Rendell, A. (2009) "Descending from the Architect's Ivory Tower," in Agile 2009 Conference, A. Sidky, et al., eds. IEEE Computer Society, pp. 180-‐185.
• Rozanski, N., & Woods, E. (2005). So*ware Systems Architecture: Working With Stakeholders Using Viewpoints and PerspecVves. Addison-‐Wesley.
• Special issue of CuZer IT Journal, edited by I. Gat (October 2010) CuZer IT Journal, 23 (10).
• Sterling, C. (2010) Managing So*ware Debt, Addison-‐Wesley. • Wiegers, K. (1999). First Things First: PrioriLzing Requirements. So*ware Development
Magazine, 7(9), 48-‐53. • Woods, E. (2010). Agile Principles and So*ware Architecture, presentaLon at OOP 2010
Conf., Munich, Jan 26. • Saaty, T. (1990). How to make a decision: The analyLc hierarchy process. European
journal of operaLonal research, 48(1), 9-‐26. Copyright © 2011 Philippe Kruchten 184