© 2014 Israel Gat
The Macro Process Is the
Micro Process Israel Gat, Director and Fellow
(With many thanks to Murray Cantor,
Tom Grant and Paul Ryan)
IEEE Computer Society Symposium
November 12, 2014
© 2014 Israel Gat
Bio Areas of research & consulting:
• Agile methods
• Devops
• Software governance
• Technical debt
• Technical Due Diligence
Major products delivered:
• BMC Performance Manager/PATROL
• Microsoft Operations Manager
• Tivoli Smart Handheld Device
Manager
• EMC Cellera
• Digital‟s NetView
• Nixdorf 8890
Books:
• The Concise Executive Guide to Agile
Sample accolades:
• Winner, 2006 Innovator Award
[Application Development Trends, May 2006]
• "When I deal with technical debt
issues, I refer to Israel Gat regularly.
His approach is the only one I've
found that actually works…” [Director,
Verisk Health]
• “Nearly three times faster time to
market than industry average…
… one quarter the expected number
of defects based on team sizes and
schedules.” [QSM Study, August 2007]
• “The change you bought to BMC
with Agile is the single largest
change to the development model
that I have ever witnessed in my
almost 20 years at BMC.” [Director, BMC
Software]
2
© 2014 Israel Gat
Agenda
Faking It
Single-Piece Batch
The New Semantics
Recipe for Software
Development in our Era
3
© 2014 Israel Gat
Part I:
Faking It
4
© 2014 Israel Gat
Faking It
Parnas and Clements in their 1986 paper „A Rational Design
Process: How and Why to Fake It‟
• “we will never find a process that allows us to design software in
a perfectly rational way”
• “the good news is that we can fake it”
• “if we agree on an ideal process, it becomes much easier to
measure the progress that the process is making.”
5
© 2014 Israel Gat
Why is the Software Development Process so
Difficult that We Need to Fake It?! “When an individual or a team solves a problem, they engage in
four activities:
• Scoping. Ensuring they fully understand the problem.
• Designing. Developing an approach to solving the problem, usually
using some sort of sketch or diagram.
• Implementing. Carrying out the design.
• Verifying. Confirming that the solution actually solves the original
problem.”
“It is important to understand that these are activities, not phases.
An activity is something you do to reach an outcome. Phases are
the steps in the lifecycle that mark the project‟s progress.”
“Phases are not strictly tied to problem-solving activities since the
activities often span the phases.” [Murray Cantor]
6
© 2014 Israel Gat
Grady Booch’s Two-Tier Process Structure
Management
Environment
Business Modeling
Implementation
Test
Analysis & Design
Preliminary Iteration(s)
Iter. #1
Phases
Process Workflows
Iterations
Supporting Workflows
Iter. #2
Iter. #n
Iter. #n+1
Iter. #n+2
Iter. #m
Iter. #m+1
Deployment
Configuration Mgmt
Requirements
Elaboration Transition Inception Construction
Source: Grady Booch, Rational Software
© 2014 Israel Gat
The Macro Process vis-à-vis the Micro Process
The macro process represents the activities of the entire
development team/value stream in terms of content and time
• Content:
– Requirements
– Analysis & design
– Implementation
– Test
– Deployment
• Time:
– Milestones to ensure quality or maturity, not dates per se
The Micro Process represents the technical activities of the
development team – Traditionally analysis and design, e,g, in the Booch method
8
© 2014 Israel Gat
The Macro Process vis-à-vis the Micro Process
Strong separation of concern between the two processes
• Macro Process: choice of lifecycle style: Waterfall, Iterative, Agile, etc.
• Micro Process: choice of technique, e.g. Object Oriented
The Macro Process drives the Micro Process by prescribing
products and activities that enable:
• Assessing risk
• Early corrections to the Micro Process
9
© 2014 Israel Gat
The Flow Problem: Handoff from One Phase to the
Next One
To represent reality, the macro process must capture wait periods
between phases – Like it or not, your artifacts are waitinggggggggggggggggggggggggggggggggg
VSM has been used as a supplementary tool to that purpose, but
in general it has not been explicitly incorporated in the macro
process
• Only way to do so is by incorporating artifact wait states in the process
definition
– See http://www.cutter.com/offers/devintell.html
As various studies by Don Reinertsen have illustrated, the waiting
periods between phases might be more important (in terms of flow
than) the phases themselves
10
© 2014 Israel Gat
Part II: Single-Piece Batch
11
© 2014 Israel Gat
Single-Piece Batch Enables Optimizing Flow
12
© 2014 Israel Gat
The Process is Managed through Software
Development Analytics
13
Source: Murray Cantor
Before After
© 2014 Israel Gat
By the Way, It All Applies to Devops
The whole “business” of deployment as a separate
activity/discipline is an extremely unfortunate consequence of
archaic organizational structures
All you need to do is handle deployment as ordinary steps/tasks in
your single-piece batch system
14
© 2014 Israel Gat
Part III: The New
Semantics
15
© 2014 Israel Gat
You Win on Flow, You Lose on Grouping
Substance
The very nature of the phases concept is transformed:
• Phase gates Flow optimization levers (WIP limits, etc.)
Question: How do we to capture the user mental model? – Think of the Table construct in a text editor
– Not understanding it in an early phase has a cost for flow, value delivered
– By analyzing the amount of investment in understanding the user mental model
needed to increase the reliability of value delivered, analyzed at the micro level, we
can reach macro level conclusions
– Think of what it means: we de-facto reintroduced the phase construct through new
techniques
16
© 2014 Israel Gat
Question: When is Conception Completed in the
Era of Continuous Deployment?
Management
Environment
Business Modeling
Implementation
Test
Analysis & Design
Preliminary Iteration(s)
Iter. #1
Phases
Process Workflows
Iterations
Supporting Workflows
Iter. #2
Iter. #n
Iter. #n+1
Iter. #n+2
Iter. #m
Iter. #m+1
Deployment
Configuration Mgmt
Requirements
Elaboration Transition Inception Construction
Source: Grady Booch, Rational Software
© 2014 Israel Gat
Answer: Three Kinds of Development
More
Descriptive
More
Predictive
Source: Murray Cantor
© 2014 Israel Gat
Part IV: Recipe for
Software Development in
our Era
19
© 2014 Israel Gat
First Step: What Kind of Project is on Your Hands?
Are you doing:
• A new platform?
• New features on existing platform?
• Maintenance and small change requests on existing platform?
Each kind requires different kind of analytics
Moreover, for a new platform you will need to resurrect some form
of the Elaboration phase to ensure just enough architectural
runway to enable future small changes and new features will add
up
• Appropriately enough, we call it “Agile RUP”
20
© 2014 Israel Gat 21
End User The
Business
Customers Domain
Experts
Developers
& Testers
End User
Feature
priorities,
scope
Purchase
convenience
Product/
feature
feasibility
Quality and
proper
functionality
The
Business
Feasibility Create
standards
Process
requirements
Feasibility Source of
revenue
Customers
A market Products and
services
Create
standards
Compliance
with
standards
Source of
revenue
Domain
Experts
Range of
product
variation
Workplace
well-being
Need for
standards
Domain
synergies
and conflicts
Constraints
on
technology
Developers
& Testers
Requirement
clarification
Workplace
well-being
Advice on
delivery
process
Guidance,
APIs,
poka-yoke
Clarification
of how
existing code
works
Read down the columns to see what the roles contribute to the value stream; rows indicate the roles to whom
value is provided. Source: Coplien & Bjornvig, Lean Architecture for Agile Software Development, Wiley , 2010.
Second Step: Minimize Number of Roles; Maximize Flexibility
© 2014 Israel Gat
Alignment vs. Autonomy is a false dichotomy
Alignment leads to autonomy
My job is to provide the alignment:
Overall team goals
Overall business goals
Top Level product strategy
Your jobs are to creatively work to achieve these goals by innovative problem solving
Autonomy Low Autonomy High
Alig
nm
en
t L
ow
A
lign
me
nt H
igh
Silos Everyone gets „their‟
thing done, not necessarily overall
success.
Innovation
Everone knows all of the goals and works
to achieve them though problem
solving.
Salt Mines
MIcromanagement.
Anarchy This is not the
meaning of „agile‟
Third Step: Build on the First Two Steps to Attain Autonomy through Alignment
High Autonomy w/High Alignment is Possible
Slide used with the kind permission of Mr. Paul Ryan 22
(stolen from http://vimeo.com/85490944)
© 2014 Israel Gat 23
QUESTIONS?
