Date post: | 27-Dec-2015 |
Category: |
Documents |
Upload: | lee-marshall |
View: | 213 times |
Download: | 0 times |
Working in Chaos & Complexity for Success
May 11-14, 2009
Richard M. Wallace
Agile Development Methods
in Current Environments
Outline
• Agile does work– Proven in commercial industry, very late to government
procurement
• Software is not manufactured– Government sees software development as a manufacturing
process– Procurement treats software as a "mature technology" item
• Chaos is always present – get used to it– In all development chaos and complexities arise preventing
wanted functionality
• Success is possible with simple steps– Agile development, a framework for traversing organizational
social complexity, and efficient processes equals delivering wanted functionality
Agile Accepted in Commercial Industry
The 3rd Annual "State of Agile Development" survey was conducted and sponsored by VersionOne. This global survey ran in June and July of 2008 and received 2,319 completed surveys with 3,061 total respondents from 80 countries.
Percentage of Successful Agile Projects1
1: 3rd Annual "State of Agile Development" survey
3.8%
11.5%
21.2%
55.0%
3.7%4.8%
0
10
25
50
75
90-100
Agile is Fully Adopted in Commercial Industry
Crossing the Chasm, Geoffrey A. Moore, Collins Business; Revised edition (August 20, 2002) ISBN-10: 0060517123
The chasm is between the early adopters of the product (the technology enthusiasts and visionaries) and the early majority (the pragmatists).
Agile has crossed the chasm and is now part of mainstream software development. Mary Poppendieck1 used Geoffrey Moore’s classic product adoption diagram to illustrate where agile practices are in the general software development adoption curve.1: http://www.poppendieck.com/
Accepted Agile Methodologies
• Multiple development methodologies:– Agile Unified Process – AUP (simplified RUP)– eXtreme Programming – XP
– Scrum – Very shorttime frame
• Methodologies not accepted as “Agile”– Waterfall– Incremental– Spiral– Evolutionary (Deming)
[1]
What Northrop Grumman normally uses
Software Is Not Manufactured
Government sees software development as a manufacturing process
Treats software as a "mature technology"
DoD Views Software As A Manufactured Item
Advancing Producibility for Software-Intensive Systems-- Grady H. Campbell, Jr., SEI
(12/2008)
What Engineering Has in Common With Manufacturing and Why It Matters
-- Dr. Alistair Cockburn (4/2007)
Production Planning for a Software Product Line-- Gary J. Chastek, Ph.D., SEI Linda M. Northrop, SEI John D. McGregor, Ph.D., Clemson (1/2009)
DoD Instruction DODI 5000.2 views software as a "part" like hardware
5 Reasons Why the ManufacturingMetaphor Doesn’t Work
1. Software development is much less predictable than manufacturing a product
2. Software is not mass-produced on the same scale as a physical product
3. Not all software faults result in failures
4. Software doesn’t wear out
5. Software is not bound by physics"The creation of software is an intellectual human endeavor. Creating good software relies on the personalities and the intellects of the members of the teams that create it. When applied to a different team of developers a process that delivers great software for one team of developers may fail to deliver anything at all for another team."
-- The Practical Guide to S/W Arch.
What Software is Not
"These manufacturing focused models tell us nothing about the typical back-and-forth activities that lead to creating a final product. In particular, creation usually involves trying a little of this-or-that, developing and evaluating prototypes, assessing the feasibility of requirements, contrasting several design, learning from failure, and eventually settling on a satisfactory solution to the problem at hand."
-- Shari Lawrence Pfleeger, Joanne M. Atlee (7/2005)
"Software development is not a manufacturing process ... it is largely an intellectual and sociological activity"
-- Martyn Ould (10/1999)
Very Significant Paradigm Shift
Chaos is Always Present – Get Used To It
In all development chaos and complexities arise preventing wanted functionality
Development is Chaotic
• Development, by its very nature, has historically been chaotic
• System Engineering treats this chaotic phase as if it were a production phase which leads to overruns as shown by the historical record– This is a root problem with “agile” methods– This has been noted as Fr-”agile” in the literature
• Agile Software Development is done using a defined process framework– Some Agile Methodologies, like Scrum, have been very
prescriptive and lay down rigid practices as “rules of the game”
Complexity Interrelationships1
Scale-Free Networks
Autopoiesis (self-organized
criticality, self-persistence)
Complex Adaptive Systems
form via growth and preferential attachment
Lead toSynchronization• Physical and animate• Phase transitions
Chaos(“Butterfly
effect”)
ComplexityEdge of Chaos = Life
Small Worlds • Small avg # steps• High local clustering• Hubs• Resistance to failure• Security weak points• Tipping points Indicate
Power Law Distributions
FractalsStrange AttractorsSmall changes Big Effects Movement from Order through
Oscillation to chaosEntropy as a measure
Chaos & Complexities
Regardless of method, chaos and complexities arise in many forms:
• Changing Staff
• Changing Budget
• Requirements creep
• Product morphing (outside of budget)
• Intended – Eventual end user directs or wants the change
• Unintended – Difference in vision between development and end user
• Customer satisfaction by rapid, continuous delivery of useful software
• Working software is delivered frequently (weeks rather than months)
• Working software is the principal measure of progress
• Even late changes in requirements are welcomed
• Close, daily cooperation between business people and developers • Face-to-face conversation is the best form of communication
• Projects are built around motivated individuals, who should be trusted
• Continuous attention to technical excellence and good design
• Simplicity
• Self-organizing teams
• Regular adaptation to changing circumstances
Agile Manifesto – Some Chaos Applied
Chaos Factor #1EVMSChaos Factor #2
Slow Contract Changes
Chaos Factor #3Culture Gulf & “Arms Length”
Relationships [1]
[1] Short-term benefits of concurrent engineering
Agile Limitations… or not [1]
• Large scale development efforts (>20 developers), though scaling strategies [2] and evidence to the contrary [3] have been described
• Distributed development efforts (non-co-located teams). Strategies have been described in Bridging the Distance [4] and Using an Agile Software Process with Offshore Development [5]
• Mission- and life-critical efforts
• Chain of Command company cultures force Agile teams to fail
• Development teams need extremely high levels of creativity, innovation, commitment and discipline and cannot bloom in a strictly defined process environment due to the high degree of interaction, informal face to face communication with stakeholders, ability to change directions midway, and learning along the way what is needed to build the best possible solution
Cynefin + Agile
• Cynefin (Cun-ev-in): Welsh
– Has no direct equivalent in English. It is translated as familiar habitat
– Thus, in general, if a community is not physically, temporally and spiritually rooted, it is alienated from its environment and will focus on survival rather than creativity and collaboration
• The use of methods such as eXtreme Programming and Scrum has helped many software projects to success, although it’s not always clear why and how it is accomplished
Cynefin Framework Use
• Using the Cynefin framework to move between domains requires a shift to a different model of understanding and interpretation as well as a different leadership style.
• Understanding differences among the different movements in the framework increases the sophistication of the response of a decision-making group to rapid change.
• This understanding allows Agile development teams to anticipate the pattern of social complexity, creating change or chaos [1]
Four named domains and a fifth in grey that is “disorder”
Chaos Theory and Cynefin
• A system is classified as chaotic if it has the following properties:[1]
– It must be sensitive to initial conditions, – It must be topologically mixing, and – its periodic orbits must be dense
This framework originated in the practice of knowledge management as a means of distinguishing between formal and informal communities, and as a means of talking about the interaction of bothwith structured processes and uncertain conditions.
The Cynefin [2] framework:
Success is Possible with Simple Steps
• Agile development
• A framework for traversingorganizational social complexity
• Efficient processes
Equals delivering wanted functionality
Agile + Chaos
• An understanding of the roots of agility, though, is a requirement for adopting and adapting agile processes in organizations
• New research in social complexity theory provides an explanation for when and how agile methods work
• Using techniques from this relatively new field, developers and managers can more easily adapt agile methods to the idiosyncrasies of specific organizations and projects, are more aware of weak signals that can lead to problems, and can design interventions to correct them [1]
The Human Element in Agile
• The Cynefin framework is based on three ontological states juxtapositioned– Order (Known & Knowable)– Complexity– Chaos
• Success is not getting lost on your way from CONOPS to product instantiation– Relaxing the assumption of order and follow the pattern– Relaxing the assumption of rational choice from rigid formalism
and use context and perspective– Relaxing the assumption of intentional actions allowing for
accident of interaction
Navigation on the Cynefin Framework
• Movement at the known-chaos boundary – (1)Asymmetric collapse – disastrous
movement from known to chaotic– (2) Imposition – forceful movement
from chaotic to known (Draconian)• Movement at the known-knowable
boundary– (3) Incremental improvement –
engine of tech growth, but can be pathological
• Movement at the knowable-complex boundary– (4) Exploration – selective
movement to complexity with trust– (5) Just-in-time transfer – choice of
stable patterns in complex space• Movement at the complex-chaotic
boundary– (6) Swarming – mass movement to
an attractor, some may not survive– (7) Divergence-convergence – Able
to handle disruption
Navigation on the Cynefin Framework Cont.
• Visiting chaos– (8) Entrainment
breaking – Create and validate new sources “breaking out of the groove”
– (9) Liberation – casting idea “seeds” for growth and quick exploitation of “seeds” that grow
– (10) Immunization – Inure people to the devastation of chaos to be better prepared for it in the future
Assuring What Is Wanted By The Customer Is Delivered
• Know that software development is highly subject to– Chaos and Complexity– Changing requirements or "desire-ments"– Human social structure changes
• Assure delivering wanted functionality by use of Agile development as– Agile development is the natural form of human grouping– Agile has demonstrated world-wide acceptance and proof-of-performance
• Scrap the "manufacturing" mentality– Understanding that the "manufacturing" mentality in procuring software is a
failure and that software is an intellectual and sociological activity allows better delivery of software product
– Understanding that what appears to be a process out of control is, in fact, a process in control if complexity and chaos patterns are known
• Embrace the fact that social structures are dynamic and using the Cynefin framework allows informed navigation between styles for successful creation of software products
Schools of Thought for Chaos/Complexity
According to De Wolf and Holvoet (2005), there are actually four central schools of research that influence the way complex behavior of complex systems is studied:
• Complex adaptive systems theory (Santa Fe Institute (SFI) 2006): Complex systems are seen as having similarities that can be studied and exploited in order to ease finding underlying principles of a unified complexity theory (Holland, 1996). The SFI often call complex systems complex adaptive systems (CAS)
• Nonlinear dynamical systems theory and chaos theory: Promulgates the central concept of attractors
• Synergetics: Study of emergence in complex systems with the idea of an order parameter that influences which macro-level coherent phenomena a system exhibits (Haken, 1981)
• Far-from-equilibrium thermodynamics: Based on nonlinear dynamics and far-from-equilibrium thermodynamics. Disciplines include catastrophe theory, chaos theory, and complexity theory. These theories push beyond some of the limitations of classical physics, and explore classes of phenomena outside the traditional linear realm