Post on 13-Sep-2014
description
transcript
Getting Some Respect:
How to Measure and
Communicate Your EA SuccessDavid Baker
Michael Janiszewski
Diamond Management and Technology Consultants
david.baker@diamondconsultants.com,
michael.janiszewski@diamondconsultants.com
October 26, 2007
Page 1
Introductions
We both have extensive experience practicing Enterprise Architecture
as well as developing Enterprise Architecture organizations,
processes, and measurement techniques
David Baker
Partner and chief architect for
Diamond Management &
Technology Consultants
Mike Janiszewski
Manager for Diamond
Management & Technology
Consultants
Page 2
Introduction – Ice Breaker
Split up into groups and discuss your expectations for this workshop
Before you begin the discussion, assign a person who will take notes and share the highlights of the discussion with rest of the team
Upon completion of the assignment, share your group’s thoughts
Intro – 5 minutes
Exercise – 10 minutes
Debrief – 15 minutes
Page 3
Introduction – Ice Breaker
Take the next 10 minutes to discuss these three questions…
1. What do you hope to learn from this workshop?
2. Why are EA metrics important to your organization?
3. What emerging trends are shaping the future of EA measurement?
Page 4
Share your group’s discussion
Page 5
By the end of this course, you should be able to…
Explain the importance and classification of EA metrics
Include appropriate resources in your EA metrics program
Analyze EA measurements to improve your EA program
Integrate EA metrics into your organization’s IT lifecycle
Identify training and management issues associated with launching
an EA program
Identify organizational problems that may limit your EA metrics
program
Design your own EA metrics deliverables and reporting templates
Page 6
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Agenda
Page 7
What Problem Are We Solving?
We know EA is valuable, but our stakeholders often do not
Business
“I just can’t see the benefit of our EA spend. Do we really
need this EA thing around?”
IT Managers
“We know doing EA is a best practice, but we can’t really
quantify how good we are at it. You [the business] can’t
measure it that way”
Technical Staff
“Our EA is so conceptual, we don’t know if we’re really doing
what you want us to do with it”
Page 8
What Problem Are We Solving?
The reality is…
EA offers significant benefits
Architecture and IT are cost organizations
EA groups don’t have significant budgets
We need to demonstrate the EA value associated with our business
impact
If we can’t quantify our value, we shouldn’t be in business
Metrics are a key part of marketing the value of EA
Page 9
What Problem Are We Solving?
Discussion: How do you market your EA
What do you measure?
EA operations?
EA compliance?
Business value?
How do you communicate those measures?
Page 10
What Problem Are We Solving?
We can market EA by focusing on results
Is my IT portfolio aligned with
the business?
Is IT meeting expectations
efficiently?
What risks am I taking?
Business
wants to know
IT
Managers
want to know
Is my strategy being realized?
Is my EA delivering business
value?
Technical
Staff
wants to know
Am I making efficient use of EA assets?
How well am I aligning with our EA?
What things should I NOT be developing?
EA Metrics
Page 11
What Problem Are We Solving?
Why are EA metrics important?
Metrics quantify what we do in a useful way
Metrics allow stakeholders to make informed decisions
Manage change
Set targets and challenges
Manage risk
Celebrate what goes well
Understand and address what does not
The capability to measure, track, and evaluate drives continual
improvement
Page 12
What Problem Are We Solving?
EA metric capture and reporting is difficult for a variety of
reasons
Scope – The Enterprise
Dynamic - EA is a moving target
Skills - Lack of development within the discipline
Vague - Success and failure are sometimes difficult to define
Measurability - Actionable EA deliverables are not always
discrete/easily measurable
Priorities - Stakeholders value other disciplines to assess
performance (e.g., program management, development practices)
Page 13
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Agenda
Page 14
Context for Measuring EA
EA Operates within the context of the IT Lifecycle…
Business
Strategic
Planning
IT Strategic
Planning
Release Planning(Portfolio
Mgmt)
ProjectExecution
(SDLC)
Business
Operations
Use enterprise and business unit
direction and goals to drive IT plans
Develop projects that
support businesses’
annual and strategic
plans
Prioritize the allocation of
IT resources to achieve
business strategy, in
alignment with enterprise
architecture
Run the business
Portfolio 1
Blueprints
Portfolio 2
Blueprints
Portfolio 3
Blueprints
Enterprise
Blueprints
Filte
r
Project
Project
Project
Project
Project
Multi-Year Plan Budget Cycle Project Cycle Continuous
Page 15
Context for Measuring EA
…with EA involvement at every stage
Business
Strategic
Planning
IT Strategic
Planning
Release Planning(Portfolio
Mgmt)
ProjectExecution
(SDLC)
Business
Operations
Strategic Architecture Innovation
Blueprinting
Standards Definition
and Management
Reference Architecture
Application Portfolio
Management Application Portfolio
Rationalization
Inventory Refresh
Annual Budgeting Cycle Impact Analysis
Risk Analysis
QA of IT Financials
RFP Process Support Content
Vendor Support
Assessment and
Selection
EA Oversight Architecture Reviews
Architecture Issue
Management Issue Management
Corrective Action Plan
Escalation Management
Exception Management
Architecture Support Short Term Issue
Resolution
Long Term Engagement
Needs Assessment
Page 16
Context for Measuring EA
EA activities during strategic planning
Associated Processes
Participants
Things to measure
Innovation
Architecture Planning
Standards Definition and Management
Reference Architecture
Domain architects responsible for Architecture Planning
Enterprise architects responsible for other Strategy processes
Business owners and SMEs
Guidelines (tech standards, blueprints, reference architecture)
used in other Arch. functions, (e.g. Project Execution)
Objective Define Enterprise Architecture guidelines
Business
Strategic
Planning
IT Strategic
Planning
Page 17
Context for Measuring EA
EA activities during release planning
Business
Strategic
Planning
IT Strategic
Planning
Release Planning(Portfolio
Mgmt)
Associated Processes
Participants
Things to measure
Application Inventory
Management
Inventory Refresh
Domain architects responsible for these portfolio activities
Enterprise architects provide oversight and enterprise view
Objective Review the application portfolio
Facilitate financial review of the application portfolio
Review content of RFPs, RFIs; assist in vendor evaluation
Impact Analysis
Risk Analysis
QA of IT Financials
Content
Vendor Support
Assessment and
Selection
Updated application
inventory
Inventory assessments
Dependencies
Tech, bus. & delivery
risks
Updated costs
RFP, RFI
RFP/proposal
assessment/scoring
Page 18
Context for Measuring EA
EA activities during project execution and operations
Business
Strategic
Planning
IT Strategic
Planning
Release
Planning(Portfolio
Mgmt)
ProjectExecution
(SDLC)
Business
Operations
Associated Processes
Participants
Things to measure
Issue Management
and Gate Reviews
Vendor Assessment
Oversight
Domain architects responsible for these delivery activities
Enterprise architects provide oversight
CAPs, issues, exceptions, deferrals
generated during reviews
Objective Monitor adherence to standards, blueprints, reference architectures
Manage architecture-related issue resolution/escalation
Provide support for requests for architecture assistance
Issue Management
Corrective Action Plan
Escalation Mgmt
Exception Mgmt
Short Term Issue
Resolution
Long Term Support
Needs Assessment
Resolution of CAP
Project support
Page 19
Context for Measuring EA
Discussion: How many of these processes do you have in
place?
How mature is your EA practice?
Is there a relationship between the maturity of your EA practice and
your ability to measure it’s impact?
Page 20
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Agenda
Page 21
Team Exercise 1
Brainstorm EA metrics
Brainstorm EA metrics by stakeholders
Break into teams, teams will brainstorm metrics in different topic areas
Before you begin the discussion, assign a person who will take notes and share the highlights of the discussion with rest of the team
Upon completion of the assignment, debrief rest of the team about list of EA metrics required by various stakeholders
Intro – 5 minutes
Exercise – 15 minutes
Debrief – 15 minutes
Page 22
Team Exercise 1
Brainstorm EA metrics
Topic 2 - Brainstorm a list of
EA metrics necessary for IT
Managers
Business
wants to know
IT
Managers
want to know
Topic 1 - Brainstorm a list of EA
metrics necessary for Business
Technical
Staff
wants to know
Topic 3 - Brainstorm a list of EA
metrics required by Technical Staff
EA Metrics
Topics for discussion
Page 23
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Agenda
Page 24
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Agenda
Page 25
EA Metrics – What to Measure
EA not only takes place at different points in the Lifecycle, it also
takes place at different levels of detail
The Enterprise
Business Domain
(line of business)
Business Domain
(line of business)
Business Domain
(line of business)
Business Domain
(line of business)
Scope:
Enterprise
One (or more)
lines of
business
One (or more)
individual
projects
Page 26
EA Metrics – What to Measure
It is helpful to group EA metrics into a small number of
categories
Value Creation Measures benefits delivered as result of architecture
processes
IT/Business
Alignment
Measures how aligned a technical project is with respect to
the underlying business strategic requirements
Risk Management Measures architectural risks inherent in project activities
(e.g. new platform risks, sunsetting, upgrades)
Operational
Effectiveness
Measures compliance accuracy with respect to enterprise
architecture
Measures effectiveness of architecture processes such as
governance
Page 27
EA Metrics – What to Measure
Measuring EA is different from program management
Enterprise Architecture
Cross-LOB and cross-program
measures
On the path to the future state
Alignment focused
Blueprint earned value
Business operations
e.g. compliance with SLAs
Program Management
Cross-project and project
measures
On-time, on-budget
Execution focused
Project earned value
System operations
e.g. system uptime
Page 28
EA Metrics – What to Measure
Supporting that difference requires a linked architecture…
Solution Architecture
Business Architecture
Technology Architecture
Desired Business Capabilities
Business
Strategy
Infrastructure Model
Interface
ModelInformation
Model
Application
Model
Data
Models
App
Models
Development
Models
Execution
Models
Operations
Models
Network
Models
Security
Models
Business
Operations
Blueprint &
RoadmapService Model
Page 29
EA Metrics – What to Measure
…with sufficient detail to allow the relationships to be measured
Solution Architecture
Business Architecture
Business Services
Mission
Vision
Goals
Objectives
Infrastructure Model
InterfacesMaster
DataApplications
Organization
Stakeholders
Functions
Data / Application / Common Services
Business
Metrics
Linked business metrics
Supported business functions
Am I building the right things?
Reuse things that are of
interest to the enterprise
Page 30
EA Metrics – What to Measure
Value creation metrics
Metric Definition Calculation
Blueprint Value
Measurement
Metric that provides quantitative view of
advancement towards a blueprint. Metric
is created by tracing back from initiative -
> capability -> objective -> corresponding
performance indicator (yearly target)
IT initiatives must have assigned business capabilities
Calculate project advancement based upon blueprint's earned value measurements
Architecture
Coverage
Comparison of the top 10 architecturally
complex projects versus the top 10 high
business priority projects
Also applies to Linkage metrics category
Estimated architecture complexity* for each initiative
Obtain top 10 architecturally complex projects
Obtain top 10 projects with the highest business priority
Count the number of projects that intersect
% of Innovation
Ideas
Converted into
Production
Captures the success rate of ideas
explored by architecture
# of ideas explored that ended up in production divided by the total # of ideas explored
* Discussed during “project typing” in “Metrics Process – How to Use Metrics”
Page 31
EA Metrics – What to Measure
Value creation metrics (continued)
Metric Analysis
Blueprint Value
Measurement
The business should ensure that IT is making continual progress on delivery of the
defined business capabilities
IT should be able to explain any acceleration / deceleration in delivery
The business should be able to clearly see how infrastructure projects
(investments) support desired business capabilities
Architecture
Coverage
There is no right or wrong answer here, but it is valuable to show how many of the
business priorities depend upon solutions to architecturally complex problems. This
demonstration is valuable to the EA community.
# of Innovation
Ideas Converted
into Production
A high percentage is good, but does too high mean there’s not enough innovation
(no risk being taken)?
Page 32
EA Metrics – What to Measure
Additional value creation metrics (continued)
% of Blueprints Defined
# of Shared Technology Services Identified
# of Shared Technology Services Delivered
# of Reference Implementations Developed
Savings from reuse of Standard Patterns
TCO Savings from Application Retirement
Page 33
EA Metrics – What to Measure
IT/Business alignment metrics
Metric Definition Calculation
Spend amount
per business
goal
Cost of all IT projects associated with
delivery of a business goal. Can be at the
enterprise or LOB level.
IT initiatives must have assigned
business goals
Sum the initiative costs for each goal
Number of
projects
involved with
each of the
business goals
Can be at the enterprise and LOB level
IT initiatives must have assigned
business goals
Count the projects under each initiative
Change
Agenda
Identifies the traceability between
business strategy, objectives and
strategic technology investments.
IT initiatives must be traceable back to
business capabilities and capabilities
traceable back to objectives and goals
Page 34
EA Metrics – What to Measure
IT/Business alignment metrics (continued)
Metric Analysis
Spend amount
per business
goal
Look for cases where the spend is not commensurate with the business priority
High priority business goals with low spend (good thing)
Low priority business goals with high spend (bad thing)
Number of
projects
involved with
each of the
business goals
Helps identify high risk business goals
High number of projects may reflect complex implementation and therefore
high risk
Change Agenda
Can be used to conduct impact analysis of project delivery on advancing the
business strategy (and vice versa)
What is the business impact of NOT spending on an IT project?
What is the IT impact of changing a business strategy?
Page 35
EA Metrics – What to Measure
Risk management metrics
Metric Definition Calculation
Technology
Lifecycle Risk
Risks associated with project
architectural component maturity.
Measured in relation to product
technology lifecycle
Calculate percentage of lifecycle risks
attributed to each piece of the
technology lifecycle (e.g. emerging
technology, mature outside your
company, obsolete technology)
Architecture
risk count
An architecture risk is a technical
characteristic of a solution that has a
potential negative impact on that
solution’s chances of a success (e.g.
vendor risk, new technology risk, security
risk, supportability risk, . . . )
Accumulate risks as they are identified.
Classify both their likelihood (high,
medium, low) and severity (severe,
moderate, low)
Segment by architecture domain
Severe
Architecture
Risk Ratio
Measures the number of architecture
risks across technical domains
Calculate ratio of high probability and
severe risks to total architectural risks
per project.
Segment by architectural domain
Page 36
EA Metrics – What to Measure
Risk management metrics (continued)
Metric Analysis
Technology
Lifecycle Risk
Provides insight into technology lifecycle root causes for projects and articulates
where the enterprise is with respect to technology early adopter, late adopter, etc.
This information can be used to ensure that projects use the right strategy for
implementing the EA vision (e.g. due to high importance of integration efforts, such
projects should use DB2 (mature) as opposed to IDMS (sunsetting))
Architecture
risk count
Can help identify common architecture elements that are at risk.
Can help identify weaknesses in architecture skills (architecture domain disciplines)
Severe
Architecture
Risk Ratio
Provides insight into the overall severity of architectural risks encountered by
projects. Segmentation by architectural domain allows a starting point to take
action to remedy at-risk projects (and at-risk reference architectures)
Page 37
EA Metrics – What to Measure
Operational effectiveness metrics - compliance
Metric Definition Calculation
Compliance Percentage of project using / reusing
standards, services, reference
architectures
% of projects using standard products
% reuse of standard services, patterns
EA Compliance
Efficiency
Measures the percentage of projects
with architecture signoff on first try
Calculate number of projects that
receive governance signoff on first try
divided by total number of reviewed
projects
EA Compliance
Ratio
Measures the percentage of projects
fully compliant with standards by
business domain and/or technology
architecture domain
Calculate number of reviewed projects
found to be fully compliant with EA
standards divided by total number of
reviewed projects
Exception/
Deferrals
frequency
Measures the frequency of
exceptions/deferrals in information,
application, technical and business
architecture as well as standards
Calculate the total number of
exceptions/deferrals per architecture
domain
Reference
Recommendation
Revision Count
Number of reference material revision
suggestions submitted by business
domain and/or technology architecture
domain
Calculate total number of suggestions
against a given reference material (e.g.
business domain blueprint)
Page 38
EA Metrics – What to Measure
Operational effectiveness metrics - compliance (continued)
Metric Analysis
Compliance
A low percentage could indicate an incomplete set of guidelines, or poorly
communicated guidelines, or just the wrong set of guidelines
A high percentage shoud be correlated with better program management metrics
EA Compliance
Efficiency
Provides insight into accuracy of projects' delivery in compliance with your
company’s EA standards
EA Compliance
Ratio
Allows the business to understand the level of EA compliance within its project
portfolio. Provides confidence level in delivered architecture.
Exception/
Deferral
frequency
Too many exceptions and deferrals may indicate an unachievable architecture
vision or possibly an unachievable project schedule
Exceptions and deferrals are and important and valuable tool for IT governance,
yet deferrals should be tracked to ultimate remediation
Reference
Recommendation
Revision Count
Provides insight into use of reference materials. Also provides insight into
reference material needed by the enterprise.
Page 39
EA Metrics – What to Measure
Additional operational effectiveness (compliance) metrics
Number of exceptions (by level)
Reference Recommendation Revision Approval Ratio
Exception/Deferrals frequency by Blueprints across domains
% of Projects Rejected
% of Applications that are in Obsolescence Stage
Page 40
EA Metrics – What to Measure
Operational effectiveness metrics - governance
Metric Definition Calculation
Oversight
Frequency
Measures the number of architecture
oversights conducted per week
Calculate the total number of oversights
Find the average over a week
Oversight
Efficiency
Measures the amount of time / level of
difficulty to complete the architecture
reviews for a project
Calculate the number of days and
sessions required to complete signoff
per project per gate (also average)
Issues,
exceptions,
deferrals,
corrective action
plans
Totals and averages for the various
dispositions of architecture issues
accumulated during reviews
Track, count, and average totals for
each
Provide totals per project, per gate
Trending A complete view of architecture health
includes the total counts as well as their
week-over-week trend
Week-to-week comparison of key
metrics (issues, CAPs, exceptions,
deferrals)
Indicate week-over-week trend
Page 41
EA Metrics – What to Measure
Operational effectiveness metrics – governance (continued)
Metric Analysis
Oversight
Frequency Ensures governance processes are occurring according to targets
Oversight
Efficiency
Ensures that governance processes remain lightweight and do not impact project
delivery schedules
Issues,
exceptions,
deferrals,
corrective action
plans
Low numbers indicate maturity level of project compliance
Compare these historical numbers with problems during development and
operations to identify value of architecture compliance
Trending Trend combined with phase can warn of impending problems
Upward trend can signal the need to deploy senior architects
Page 42
EA Metrics – What to Measure
Additional operational effectiveness (governance) metrics
Issue Closure Efficiency
Oversight Capacity Sufficiency
Page 43
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Agenda
Page 44
Organizational Impacts
Discussion: Architecture Organization
Do you identify different types of architects?
Or is everyone becoming an architect?
How are your architects organized?
Highly distributed?
Centralized?
Virtual?
Who develops standards and reference architectures?
If you do blueprints, who does them?
What role does architecture play in project delivery?
Page 45
Organizational Impacts
Enterprise architecture is delivered via several different types of
architects
Type Role
Enterprise Architect
Oversee the blueprinting process to translate enterprise and business unit
strategies into the technology roadmap
Ensure interoperability across domains
Ensure compliance with architectural blueprints, roadmaps, and standards
Domain Architect
Translate business unit strategy into functional domain (such as Sales) blueprints
Provide deep business and technology knowledge
Guide project architects to ensure project compliance with enterprise and domain
architectural blueprints
Project Architect
Provide architectural expertise to lead the solution delivery projects
Responsible for overall system architecture and integration with other domains
and systems
Systems Engineer
“Internal” architecture
Requirements analysis, detailed software design
Process planning and control
Verification, validation
Software Developer Code, unit testing
Page 46
Organizational Impacts
Architects are social, community oriented creatures, found in
groups
Infrastructure
Infra
architect
Solutions
Delivery
Project
Architect
Systems
Engineer
Software
Developers
Enterprise
Architecture
Business
Architect
Application
Architect
Information
Architect
Business Unit
C
Domain
Architect
Business Unit
B
Domain
Architect
Business Unit
A
Domain
Architect
Enterprise Architecture Virtual Community
CEO
Operations CIO
Diagram assumes a large
organization, same principles
apply to smaller organizations
but structure will be different
Page 47
Organizational Impacts
The various architects engage at different levels to deliver
projects
Enterprise Architecture
Services Services Services
Sample Application Domain
Application Architecture
Infrastructure Architecture
Business Architects
Domain Manager
Systems Engineers
Domain Architects
Project Manager
Project Architects
Software Developers
Solutions Delivery
Services Services Services
Application Development Services
Quality Engineering Services
Program Management Office
Emerging Technologies
Business Strategy
IT Strategy
Business Environment
(regulation, compliance,
market factors)
Architecture
Community
(updates, exceptions)
Release Planning
Information Architecture Blueprints
(Enterprise &
Domain)
Page 48
Organizational Impacts
EA requires the attention of a small group focused on
Architecture project management
The Enterprise Architecture Group should be a small, high powered
group of Enterprise Architects
One per architecture discipline
Domain architects may also be in this organization (or in the
business)
The Enterprise Architecture Group should also contain a small
number of project management resources
Oversees the operation of the virtual community and its
working groups
Manages and tracks oversight and issue management
This is the group that collects and reports the EA metrics
Page 49
Organizational Impacts
Transformation to / creation of a centralized enterprise
architecture group requires:
Identify the true architects
Be sure to look in the business groups as well
Software engineers / programmers don’t always make the bets
architects
Establish career path for architects
Application -> project -> domain -> enterprise
Assess architect skills to identify gaps
Define standard architecture processes so that standard metrics can
be measured and reported across the enterprise
Innovation, blueprinting, portfolio assessment, governance,
issue management, etc
Page 50
Organizational Impacts
Training and change management
Establish a small enterprise architecture group, initial focus on
establishing the virtual community and defining the architecture
processes (including metric definition)
Provide adequate training and learning opportunities to teach the
standard architecture processes to both architect and non-architect
resources
Implement these processes one domain at a time, gaining
experience as you go
Start with the operational metrics (governance and compliance)
Add risk management metrics as your processes mature
Then add the alignment and business value metrics
Page 51
Organizational Impacts
Training metrics
Metric Definition Calculation% completion of
Architecture
Training
Curriculum
Measures progress towards creation
and delivery of architecture training
Establish a curriculum
Use curriculum project plan to estimate
completion
# of Skills Gaps
addressed
Identification of sub-par skills among
the architects and specific plans to
address each
Skills Assessments
Count of skills gaps
Plans to address each gap
% of Architects
trained
To understand the level of training
activity occurring in EA
# of architects trained divided by total
number of architects
Page 52
Organizational Impacts
Training metrics (continued)
Metric Analysisx% completion of
Architecture
Training
Curriculum
A high percentage indicates that the Architecture training curriculum is complete
and available for use
# of Skills Gaps
addressed
Ensure any identified skills gap is being addressed either via training or on job
experience
% of Project
Architects trained Low is bad, high is good
Page 53
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Agenda
Page 54
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Agenda
Page 55
Metrics Process – How to Use Metrics
Establish the reporting process
Establish the Architecture program management office
Establish and communicate targets
Track trends
Integrate metrics with
Business & IT Strategy planning (feedback loop)
Release Planning
Project Execution and Operations
Page 56
Metrics Process – How to Use Metrics
Remember the context within which EA takes place
Business
Strategic
Planning
IT Strategic
Planning
Release Planning(Portfolio
Mgmt)
ProjectExecution
(SDLC)
Business
Operations
Strategic Architecture Innovation
Blueprinting
Standards Definition
and Management
Reference Architecture
Application Portfolio
Management Application Portfolio
Rationalization
Inventory Refresh
Annual Budgeting Cycle Impact Analysis
Risk Analysis
QA of IT Financials
RFP Process Support Content
Vendor Support
Assessment and
Selection
EA Oversight Architecture Reviews
Architecture Issue
Management Issue Management
Corrective Action Plan
Escalation Management
Exception Management
Architecture Support Short Term Issue
Resolution
Long Term Engagement
Needs Assessment
Page 57
Metrics Process – How to Use Metrics
Responsibility for collecting and using metrics varies by
lifecycle stage
Business
Strategic
Planning
IT Strategic
Planning
Release Planning(Portfolio
Mgmt)
ProjectExecution
(SDLC)
Business
Operations
Strategic Architecture Enterprise Architects
Domain Architects
Application Portfolio
Management Domain Architects
Annual Budgeting Cycle Domain Architects
RFP Process Support Domain Architects
EA Oversight Domain Architects
(primary)
Enterprise Architects
Architecture Issue
Management Domain Architects
Enterprise Architects
Architecture Support Domain Architects
Page 58
Metrics Process – How to Use Metrics
Follow a goal -> objective -> metric framework to establish
targets
This approach should be built into
the your business architecture
Generate domain specific
business metrics for
measuring value and
alignment
For each objective, set
progress goals by establishing
yearly targets
Approach can also be applied to
generating targets for the risk and
operational metric categories
Strategic Business Architecture
KE
Y D
RIV
ER
S &
GU
IDIN
G P
RIN
CIP
LE
S
MISSION
VISION
GOAL GOAL GOALGOAL
BUSINESS
METRICS
CAPABILITIES
OBJECTIVE
Page 59
Example capabilities form a financial management system
Maintain common processes centrally
across the Department to support
centralized administration and
standardization
Support ad hoc data access across all
LOBs. This will provide a simplified,
single source for report information in a
timely fashion
Generate performance and
accountability reports, within OMB
specified timeline
Capability for drill-down to transaction
level information
Improve financial performance
Improve operating efficiency of financial
management and procurement functions
Consistently comply with federal,
accounting, and system standards
Improve financial performance
Capability Linked Goal
Page 60
Metrics Process – How to Use Metrics
Using metrics in the Business & IT Strategy phases
Technology
Assessment• Review current architecture
and plans
• Document current technology environment and pain points
• Confirm enterprise architecture strategy and initiatives
Business Alignment• Confirm business strategy
and goals
• Refine understanding of
requirements driven by
recently defined business
strategy
Technology Future
State Definition• Define key technology
capabilities to enable the
business strategy
• Define future state IT vision
and models
Roadmap• Develop and prioritize
projects and required
investments
• Define implementation plan
and analyze economics
• Define risk mitigation strategy
Business Future State
Definition• Define key business
capabilities to enable the
business strategy
• Define future state business
vision and models
Generate future state
alignment & value
metrics
Apply future state
alignment, value and risk
metrics
Apply current state
alignment and value metrics
Apply current state risk &
operational metrics
Page 61
Metrics Process – How to Use Metrics
Using metrics in the release planning phase
Initiate Release
Plan
Budget & Risk Assessment Update Release
Plan
Transition
Initial
Release
Plan
Release
Schedule
Initiative
Risks
Initiative
Resource
Reqs
Initiative
Depend-
encies
Initiative’s
Budget
Updated
Release
Plan
Initiative-focused Domain-wide
Initiative/
Portfolio-focused
Initiative
Assessment
Solution Design
Project Initiation
Blueprint
Phase
Roadmap
Project
focused
Domain - wide
Sequenced
Initiatives
Project
Charter
Approved &
Funded
Apply risk
metrics
Apply compliance
metricsApply alignment
metrics
Page 62
Metrics Process – How to Use Metrics
Metrics can be used to determine a project’s complexity rating
Are new technologies going to be introduced into the
environment?
How mature is the new technology?
New
Technologies
Are the service levels going to be significantly changed (e.g.
number of users, performances, availability levels)?Service Level
Changes
Are there significant impacts to applications outside of the
business domains?
Are there significant impacts to shared technology services?
Do the impacts require new model of integration?
Does the project proposal match overall architecture
planning? Is an implementation out of sequence with
proposed roadmap?
Are applications to be sunset being significantly enhanced?
Is data redundancy being introduced?
Are there significant exceptions to existing technology
standards and reference architectures?
Are products to be sunset used?
Key Considerations
Cross-Domain
Impacts
Architecture
Planning
Compliance
Technology &
Information
Standards
AttributeLow
More than one
exception
No standard
exception
Significant
deviations
No standard
exception
Significant
impacts
No impact
Significant new
technologies
No new
technologies
Significant service
level changes
No service
level changes
Are we enabling new / modifying existing business
capabilities?
Are new user types being introduced?
Are there compliance issues?
Significant impact
to business
No impact to
business
Business
Impact
HighMedium
Page 63
Metrics Process – How to Use Metrics
Project typing drives the level of governance during the project
execution phase
Basic:
Operational metrics such as Issue,
Exception, Deferral Mgmt
Moderate:
Domain level business
value, blueprint alignment,
EA compliance
Maximum:
Enterprise & Domain
Architect Involvement
Category Characteristics
• Fewest # of Gate Reviews
• “Low” metrics indicators
• Driven by Project Architect with
oversight by Domain Architects
• Moderate # of Gate Reviews
• “Medium” metrics indicators
• Domain Architect involved,
Enterprise Architect involvement as
needed
• Highest # of Gate reviews
• “High” metrics indicators
• Enterprise and Domain Architects
involved
Level of EA Metric Reporting
Type 3
Projects
Type 2
Projects
Type 1
All Projects
Page 64
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Agenda
Page 65
Deliverables and Templates (Case Examples)
Case #1 – Healthcare Payer
The Company
~$20bn multi-line healthcare insurance company employing over 27,000 people with over
$19bn in revenue managing over 10 million medical members in all 50 states
The Challenge
Lack of integration across business units presents fragmented view of constituents
Second highest cost structure amongst direct competitors
Slow and expensive product development inhibits ability to anticipate or lead in the market
Corporate vision and strategy in place without an execution plan
The EA program
Phase I: Establish a program level architecture governance & design board to ensure the
3-Year program adheres to architecture goals & principles
Phase II: Establish an architecture governance process & metrics/measurement model for
the entire I.T. organization
Page 66
Deliverables and Templates (Case Examples)
Sample Report: Value creation report
Purpose: To provide visibility into the alignment of the actual release state with original target
release goals
Description: The Value Creation Report illustrates the each domain’s alignment with functional
and technical business capabilities. This report takes into account the impact of
exceptions and deferrals on the intended release state of each domain and exposes
any gaps that have formed between target and actual end-states.
Frequency: Monthly/Quarterly
Audience: CTO, Business Owners
Empty circle - Not addressed in releaseHalf circle - Core functionality met, high exception score3/2 circle - Core functionality met, low exception scoreFull Circle - On target with goal
Page 67
Deliverables and Templates (Case Examples)
Sample Report: Architecture health status report
Purpose: To provide a detailed illustration of the architectural health of each domain.
Description: The architecture health status report is a synthesis of key metric data provided by
the architecture dashboard and textual highlights of key issues and milestones.
Frequency: Bi-weekly
Audience: CTO, Program Directors
Page 68
Deliverables and Templates (Case Examples)
Sample Report: EA governance operational effectiveness report
Purpose: To illustrate project compliance with blueprints and standards and to measure
reference material effectiveness
Description: The Enterprise Operational Effectiveness Report will measure project compliance
with domain architecture standards, enterprise blueprints, and EA standards. It will
also measure the usage and effectiveness of EAG reference material.
Frequency: Bi-weekly
Audience: CTO, EA Core Team, Domain Architects, Program Leads, project Architects
Page 69
Deliverables and Templates (Case Examples)
Process example: Weekly project architecture health reporting
EA Manager
EA Admin
Domain admin
Project Lead
Enterprise
Architecture
PMO
Domain Lead
Project Team
Determine project
health status
Consolidate all
TSMs health status
information
Create health
report overview
Request Architecture
health status
overview
Submit
aggregate
health status
report
1Project Architect
Submit Weekly
project health
status report
Triggering
Event
End
ResultProcess GateOrg Role Package Process Break
Legend
Page 70
Deliverables and Templates (Case Examples)
Case #2 – Pharmaceutical Company
The Company
A leading pharmaceutical company, with over 100,000 employees and $19 billion in reported net income for 2006
The Challenge
Having launched an aggressive enterprise-wide modernization program, the company was looking for a way to ensure the program delivered the promised capabilities and economic improvements
The EA program
Improve enterprise architecture service delivery through new organization and governance structures and strong alignment with business value
Provide the mechanisms to more effectively link architecture to concerns of the business as well as to the updated software Delivery Model
Create meaningful and objective measures which can be used to assess the success of architectural efforts in business terms
Page 71
Deliverables and Templates (Case Examples)
Sample Report – Local Portfolio Architecture Health
Page 72
Deliverables and Templates (Case Examples)
Sample Report – Business Unit Portfolio Architecture Health
Page 73
Deliverables and Templates (Case Examples)
Sample Report – EA Operational Effectiveness Dashboard
Page 74
Deliverables and Templates (Case Examples)
Sample Report – Business Alignment and Value Creation
Page 75
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Agenda
Page 76
Team Exercise 2
What will you do differently when you get back to the office?
Consider metrics that you can begin to implement in your organization
This is an individual exercise using the worksheets in your handouts
Identify one or two metrics in each category that would benefit your organization
Jot down some notes about what you might do to get a metrics project underway
The instructors will be around to help you think about what you can do
Intro – 10 minutes
Exercise – 20 minutes
Page 77
Team Exercise 2
EA Metrics Worksheet (1 of 3)
Oversight Frequency
Oversight Efficiency
Issues, Exceptions, Deferrals, CAPs
Trending
Issue Closure Efficiency
Oversight Capacity Sufficiency
__________________________________
Operational Effectiveness (Governance) What will do you do?_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
Spend per Business Goal
Number of Projects per Business Goal
Change Agenda Metrics
__________________________________
__________________________________
IT/Business Alignment Metrics What will do you do?_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
Page 78
Team Exercise 2
EA Metrics Worksheet (2 of 3)
Compliance
EA Compliance Efficiency
EA Compliance Ratio
Exception/ Deferrals frequency
Reference Recommendation Revision Count
Number of exceptions (by level)
Reference Revision Approval Ratio
Exception/Deferrals frequency by Blueprints
% of Projects Rejected
% of Applications in Obsolescence Stage
__________________________________
Operational Effectiveness (Compliance) What will do you do?_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
Technology Lifecycle Risk
Architecture risk count
Severe Architecture Risk Ratio
Risk Management Metrics What will do you do?
_________________________________________________
_________________________________________________
_________________________________________________
Page 79
Team Exercise 2
EA Metrics Worksheet (3 of 3)
Blueprint value
Architecture Coverage
% of Innovation Idea converted
Shared Technology Services
% of Blueprints Defined
# of Shared Technology Services Identified
# of Shared Technology Services Delivered
# of Reference Implementations Developed
Savings from reuse of Standard Patterns
TCO Savings from Application Retirement
__________________________________
Value Creation Metrics What will do you do?_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
Page 80
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Agenda
Page 81
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Agenda
Page 82
Metrics Frameworks
Federal Enterprise Architecture – Performance Reference Model
Source: “How to Use the Performance Reference Model”, Version 1
Page 83
Metrics Frameworks
The PRM structure is designed to clearly articulate the cause
and effect relationship between inputs, outputs and outcomes
Source: “How to Use the Performance Reference Model”, Version 1
Page 84
Metrics Frameworks
Key intersections of the PRM and other management processes
Budget Assess progress and inform subsequent budget decisions
GPRA Progress towards relevant PRM Measurement Indicators informs
evaluations to inform subsequent strategic plans
Program
Assessment
Rating Tool (PART)
Progress towards Measurement Indicators can facilitate cross-agency
evaluation of PART programs within a BRM Sub-function. Higher-
scoring programs can share best practices with lower-scoring
programs with similar missions.
IT Capital Planning
& Investment
Control (CPIC)
Progress towards Measurement Indicators inform updated Exhibit
300s (business cases) and Agency Post-implementation Reviews
Enterprise
Architecture
Progress toward indicators can determine changes needed in
Migration Plans
Source: “How to Use the Performance Reference Model”, Version 1
Page 85
Metrics Frameworks
Example: Using PRM measures to track business case (Exhibit
300) progress
Source: “How to Use the Performance Reference Model”, Version 1
Page 86
Metrics Frameworks
COBIT – The overall framework is a form of IT lifecycle
Source: “CobiT 4.0”, www.itasca.org
Page 87
Metrics Frameworks
COBIT – Interrelationship of components
Source: “CobiT 4.0”, www.itasca.org
Page 88
Metrics Frameworks
COBIT – IT Metrics and indicators
Source: “CobiT 4.0”, www.itasca.org
Page 89
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Agenda
Page 90
Deliverables and Templates (Case Examples)
An integrated EA toolset strategy is critical to EA success
Enterprise Architecture Tools
Architecture Deliverables Support
Support tools such as repository tools help provide a centralized store that can help the enterprise gain access to,
learn and implement standards and blueprints. EA modeling tools can also help deliver information in a reusable
and consistent manner
EA Governance Support
Tools
Governance support tools
such as scheduling tools
allows the EA PMO to
operate and function
effectively and efficiently.
Such tools can optimize
performance of the EA PMO
Project Implementation
Tools
Project Implementation tools
enable the sharing of
important knowledge across
the enterprise leading to
fewer redundancies and
increased visibility and
clarity. Issue/ exception/
deferral tracking and
management, approval
tracking are some examples
Strategic Planning
Tools
Strategic planning tools such
as Metrics/Dashboard tools
enables visualization of ties
between the business,
vision, goals, and objectives.
This feedback will help
escalate visibility of projects
to upper management
isolating key issues and
common inadequacies
Page 91
Deliverables and Templates (Case Examples)
Tools must operate across governance levels and easily scale to
address enterprise-wide number of projects
EA Governance
Support Tools
Project Implementation ToolsStrategic Planning
Tools
Architecture
Deliverables
Support
LEVEL 2
LEVEL 1
LEVEL 3
Decision
Log
Level 1 EA
Issue, Exceptions &
Deferral
Tracking and
Reporting DB
EA Modeling
Tools
EA Repository
Tools
Level 2 & 3 EA
Issue, Exceptions &
Deferral
Tracking and
Reporting DB
CAP
Mgmt
EA Governance
Dashboard
Status Reporting Scheduling
Tools
Document
Management
Tools
Page 92
8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Agenda
Page 93
Wrap-up
Key success factors for an EA metrics program
Get your architects working in unison via common architecture
processes
Innovation, blueprinting, portfolio management, governance, issue
management
Engage your architects in all phases of the IT Lifecycle
Business & IT Strategy
Release Planning (Budgeting/Funding)
Project Execution
Operations
Use a virtual community to report EA progress
Business value, Alignment, Risk, Compliance/Governance
Establish an EA program management office to aggregate and
report EA progress and value
Page 94
Thank you and good luck with your metrics efforts!