Lean Integration:Lean Integration:Translating an Innovative Agile Approach into Business Valuepp
MIS Research Center SeminarMarch 25 , 2011
John SchmidtVP, Global Integration Services
1
InformaticaThe #1 independent leader in Data Integration
• 2009 Revenue: $500 million $500
$550
• 5-year AverageGrowth Rate: 18% per year
• Customers: 3,931$400
$450
$500
• 84 of Fortune 100
• 87%+ of Dow Jones
• Government organizations $350
$400
gin 20 countries
• Partners: 400+
• Major SI, ISV, OEM and $250
$300
On-Demand Leaders
• Employees: 1,755
• Direct in 28 Countries $150
$200
2005 2006 2007 2008 2009
2
2005 2006 2007 2008 2009
Our Singular MissionEnabling The Information EconomyEnabling The Information Economy
We enable organizations to reduce IT costs and gain a competitive advantage in today’sand gain a competitive advantage in today s global information economy by empowering them to access, integrate and trust all theirthem to access, integrate and trust all their
information assets.
3
The Role of IT in Business
“CEO’s, CFO’s and CIO’s all recognize there is zero separation between business strategy and IT execution today. There is pretty much nothing you can do, from cutting costs or growth without the enablement of IT in one way shape or form ”the enablement of IT in one way, shape or form.
Steve Schuckenbrock, President, Dell Large Enterprise Business
4
Reference: Financial Times Special Report, Wednesday, October 27, 2010
The Data Driven EnterpriseThe Data Driven Enterprise
Partner Trading Network(B2B)
Cloud Computing TraditionalEnterprise (B2B)Enterprise
SEPA
HIPAA
NACHA
5
Problem: Data chaos results from unnecessary complexityunnecessary complexity
Root Cause - Integrations manufactured as custom point solutions
6
Craftsman Approach – Custom Development
Manufacturing WorldSkilled manual workersManufacture by hand
Custom products (works of art)
Integration WorldSkilled IT Specialists
Hand codingCustom integration points
7
Custom products (works of art) Custom integration points
Assembly Line Approach – This is ICCs now
BenefitsConsistent Quality
Lower Cost
Integration WorldDedicated skilled labor
Standard ToolsPeriodic Large Releases
Manufacturing WorldDedicated skilled labor
Standard ToolsMass production
8
Periodic Large ReleasesMass production
Modern Factory Approach – Automated Flow of Materials and Information
BenefitsLow cost AND high qualityExactly the way you want it
Flow of Materials
Flow of Information
M f t i W ld I t ti W ldManufacturing WorldAutomation – new roles
JIT WorkflowMass-Customization
Integration WorldTemplate-driven self-service solutions
Agile/iterative DevelopmentConfigurable re-useable objects
9
g j
It is Critical That You Have an Expert and S t i bl I t ti C bilitSustainable Integration Capability
Efficiency Spectrum
Operational EfficiencyLow High
y p
Hand codingIntegration
Tools (EAI,...)
IntegrationTechnology
Integration Platform
Integration Factory Platform
Projects& Ad hoc
integration
ProgramManagement
ManagementPractices
Competency Center
LeanIntegration
Fast(quick & dirty)
ImprovedQuality
CostEfficiency
Faster, Better AND Cheaper
BusinessBenefits
1990’s 2000’s 2010’s
10
Point-in-time Methods Sustaining Methods
Transform complexity and chaos to an Agile, Data-driven Enterprise
Complexity Agility
Competitive Weapon
ata
No Compromise
Complexity Agility
Shared Resource
ion
of D
a
Controlled cost
Perc
epti Business
EmpowermentAND
IT Control
Unmanaged silos
StandardizationAND
Innovation
Technology Hand coding IntegrationTools
Integration Platform
Integration Factory Platform
Methodology
Independent to Tactical to Managed to
Project Program CompetencyCenter
LeanIntegration
12
Transformationsp
Interdependent Strategicg
Cultural
Analyst Perspective
“Taking a production line approach to integrations makes a lot of sense…they should be more widely considered and i l t d ”implemented.”
Philip Howard, Research Director - Data ManagementBloor Research, January 7, 2009Bloor Research, January 7, 2009
13
Lean Integration Principles
Eli i t W t Continuous ImprovementEliminate Waste Continuous Improvement
Automate Processes Empower the Team
Build Quality InPlan for Change
14
Optimize the Whole
g
1. Eliminate WasteExample: Value Added Ratio – Before Lean
Scenario: Simple Change Request to the Data Warehouse
5 min 5 min 2 hours 1 hour 1 hour 50 min
1 k 2 k 2 k 1 k 2 k
Value-Add
NonValue-Add 1 week 2 weeks 2 weeks 1 week 2 weeks
15
1. Eliminate WasteExample: Most non-value added activities eliminated
5 min 5 min 2 hours 1 hour 1 hour 50 minValue-Add
4 hours 8 hours 0 weeks 4 hours 16 hours
NonValue-Add
Lead Time Reduction = = = 97%Original Lead Time – Current Lead Time 1344 - 32
Lead Time Reduced from 2 months to <1.5 days
16
Lead Time Reduction 97% Original Lead Time 1344
1. Eliminate WasteKeeping data in production that no-one uses is waste
Archiving reduces datafile size (GB)
1 200
1,400
Without archiving With archiving …and truncating temp tables
400
600
800
1,000
1,200
0
200
400
Current Year 1 Year 2 Year 3
Data that is not used still costs a lot:• Disk capacity (production, test, DW)• Backup costs (tapes, labor, storage)• Business slow down by end of period jobs
Projected savings by eliminating data waste
$12,907,437
• Business slow-down by end-of-period jobs• Application upgrade costs• Impact on real-time transactions• Compliance costs
$1,603,277
$4,925,769$6,378,391
Year 1 Year 2* Year 3 3-year Total
17
• Data quality analysis and resolution timeYear 1 Year 2 Year 3 3 year Total
Take Away: Eliminating Waste is not about cutting out fatg
“Don’t think of lean as requiring a diet to lose weight; think of it as focusing the people, process, technology, and architecture that you possess on the projects that have the most impact for the business.”
Mike Gualtieri, John Rymer, Jeffrey Hammond, Top Five Changes For Application Development
Forrester, January 4, 2010
18
2. Automate ProcessCreate assembly lines for rapid customization
• Generating custom wizards for templates to maximize productivity for end users– developers and/or analysts
IT Designs Templates and Patterns – Users create custom solutions fast
• Ability to build your own wizards based on patterns without coding
• Common data integration and event processing patterns
2. Design Template1. Understand and Define Common Patterns
3. End Users Quickly Configure and Customize in Assembly Line-Fashion
Examples:• Slowly changing dimension in
warehousing• Data file interchange with supply
chain partnersTi b d t l ti
Mapping Template Example: Slowly Changing Dimension Wizard
• Time-based event correlation
19
Huge Productivity Gains for End Users
Take Away: Automation is a key to simplificationsimplification
“Factory IT couples lean management techniques and process improvements with advances in cloud computing and software development to simplify thecomputing and software development to simplify the operating environment and improve productivity and cost performance.”
Reshaping IT management for turbulent times,Roger Roberts,Hugo Sarrazin, Johnson Sikes, McKinsey
Quarterly, November, 2010
20
3. Empower the TeamProblem: IT is the bottleneck for data requestsSolution: Give the business self-service capabilities
Business Self-Service
Enables business self-service, improving
business agility
S
IT maintains governance and oversight
IT personnelSupport
21
Take Away: Self-service improves business agility AND gives IT better governance
“Lean is about ensuring IT is more closely aligned to the Business.”
Stephen Pritchard Financial TimesStephen Pritchard, Financial Times
22
Source: Financial Times Special Report, Wednesday, October 27, 2010
4. Continuously ImproveProblem: How can groups improve themselves?Solution: Use visual management of the entire integration lifecycle.Solution: Use visual management of the entire integration lifecycle.
Data Quality MetricsData Volumes by ProjectDevelopment Activity
Object reuse Project X Development Project Y Development
23
Take Away: Don’t use Lean to cut staff
“Best-practice executives view lean as aBest practice executives view lean as a performance improvement strategy rather than merely a cost-cutting exercise.”
Alexander Peters, Forrester Research
24
Source: Financial Times Special Report, Wednesday, October 27, 2010
5. Build Quality InProblem: Quality issues accumulate without business involvementSolution: Involve business in fixing highest priority issuesSolution: Involve business in fixing highest priority issues
Line of business managermanager
DataData Steward
Data Quality
ITDeveloper
25
Take Away: Metadata is an enabler for IT automation
“Metadata allow systems to adapt more quickly to changing requirements by pushing power into the g g q y p g pdata, and out onto users.”
Brian Foote and Joseph Yoder Big Ball of MudBrian Foote and Joseph Yoder, Big Ball of Mud, Addison-Wesley Software Patterns Series, 2000
26
Reference: Financial Times Special Report, Wednesday, October 27, 2010
6. Plan for ChangeInsulate consuming applications from change
Applications PortalsAnalytics
S i S iS i Service ServiceService
Service
ServiceServiceServiceService
Service
ServiceService Service
ServiceService ServiceService
ServiceService Service
ServiceService
CUSTOMER ORDER PRODUCT INVOICE
27
Lean IntegrationProven Value Proposition: Better, Cheaper & Faster
Build Quality
I
ContinuouslyImprove
Optimize The
Wh l
PlanFor
Ch
Focus on Customer Value and
p , p
AutomateProcesses
Empower the Team InImprove WholeChangeEliminate
Waste
Processes the Team
Implemented data lit t i d
Cloud Services The InformaticaPlatform
Mapping ArchitectFor Visio
PCAE - Metadata Manager
Saved $1.4M in first 6
Data Quality
Centrally managed
V9 Data ServicesDatabase Archiving
Business analysts 4 month payback from After re-working their Reduced impact quality metrics and monitoring system;
Saved est.$3M in development costs
and avoided $20M in AML regulatory fines
months because of improved customer master data quality; Reduced SKU’s by
40% by reducing old or obsolete parts
virtual views speed up time to add products to
DW portfolio from 1700 hours to 40
hours
use Informatica Cloud Services to self-serve data for their own analysis, cutting lead-times to information by
90%
p ystorage reduction,
and rapid retrieval of archived claims for improved customer
service
ETL processes to always use design patterns, achieved 8x efficiencies on
all new projects and re-work on old
projects
analysis time between 85-95%, reduced time for business analyst inquiries by 50%, reduced end-user support time by
65%
28
over 65%
Informatica Confidential and Proprietary
Leading Practice: Integration Systems are Managed Like First Class Business SystemsLike First-Class Business Systems
Business SystemsIntegration Systems
Business Systems
Enterprise Resource Planning Systems
Sales & Marketing Systems
Customer Product and Order M t
Human Resource Systems
Finance & A ti
IntegrationHub
TransformationHub
Application Integration
DataWarehouse
Business Analytics
Extract Transform Load
Data Services Enterprise Service Bus
CloudServices
CustomerSystems Management
SystemsAccounting
Systems
Distribution and Inventory Systems
External Systems (Software as a
Service)
Engineering & Manufacturing
Systems
IntegrationPortal Process
Master Data Management
Managed File Transfer
Information Security
Enterprise Messaging
Business to BusinessData Quality
Integration Systems
IntegrationHub
TransformationHub
Application Integration
Integration Repository
Portal Process Management
InformationPortal
Directory &Single Sign-on
Metadata Manager
Business Glossary
Process State Engine
ActivityMonitoring
Integration RepositoryPortal Process
Management
Content Management Service Registry Workflow
Search Complex Event Processing
Data Stewardship
30
Informatica Platform Mapping
Integration SystemsIntegration
HubTransformation
HubApplication Integration
D t E t t Cl d
UltraMessaging
CEPDataWarehouse
Business Analytics
Master Data
Extract Transform Load
Data Services
Managed File
Enterprise Service Bus
Information
CloudServices
CEP
B2B DataExchangeB2B DataExchangeB2B DataExchangeB2B DataExchange
Integration R it
Portal Process M t
Management Transfer Security
Enterprise Messaging
Business to BusinessData Quality
ILM
CloudServices
Repository Management
InformationPortal
Directory &Single Sign-on
Metadata Manager
Business Glossary
Process State Engine
ActivityMonitoring
PowerCenter
ILM
PowerCenterPowerCenterPowerCenterPowerCenter
Content Management Service Registry Workflow
Search Complex Event ProcessingData Stewardship
Data QualityData Quality
31
MDM
Informatica Factory Platform Products in Eight Growth Categories
Data Quality Assessment
Informatica Cloud PlatformData Loader
Data Synchronization
Data Replication
Data Quality Global Address
Verification
IdentityResolutionData ArchiveRulePoint PowerExchange MDMAdvanced XMLLBM Streaming
Data Explorer
Data QualityData Transformation
Data
Resolution
Data Services
PowerCenter Data Subset
Data Privacy
RuleCast
Real-Time
gg
UME Persistence
UME Queueing
32
Data ExplorerExchange Data Services Data PrivacyAlert ManagerUME Queueing
Production Line for Application Modernization
Data Quality Assessment
Data Loader
Data Synchronization
Data Replication
Data Quality Global Address
VerificationInformatica Cloud Platform
IdentityResolutionData ArchiveRulePoint PowerExchange MDMAdvanced XMLLBM Streaming
Data Explorer
Data QualityData Transformation
Data
Resolution
Data Services
PowerCenter Data Subset
Data Privacy
RuleCast
Real-Time
gg
UME Persistence
UME Queueing
33
Data ExplorerExchange Data Services Data PrivacyAlert ManagerUME Queueing
Production Line for Supply Chain Integration
Data Quality Assessment
Data Loader
Data Synchronization
Data Replication
Data Quality Global Address
VerificationInformatica Cloud Platform
IdentityResolutionData Archive RulePoint PowerExchange MDMAdvanced XMLLBM Streaming
Data Explorer
Data QualityData Transformation
Data
Resolution
Data Services
PowerCenter Data Subset
Data Privacy
RuleCast
Real-Time
gg
UME Persistence
UME Queueing
34
Data ExplorerExchange Data Services Data PrivacyAlert ManagerUME Queueing
Production Line for Channel Integration (ESB)
Data Quality Assessment
Data Loader
Data Synchronization
Data Replication
Data Quality Global Address
VerificationInformatica Cloud Platform
IdentityResolutionData Archive RulePoint PowerExchange MDMAdvanced XMLLBM Streaming
Data Explorer
Data QualityData Transformation
Data
Resolution
Data Services
PowerCenter Data Subset
Data Privacy
RuleCast
Real-Time
gg
UME Persistence
UME Queueing
35
Data ExplorerExchange Data Services Data PrivacyAlert Manager UME Queueing
Production Line for Business Intelligence COE
Data Loader
Data Synchronization
Data Replication
Informatica Cloud PlatformData Quality Assessment
Data Quality Global Address
Verification
IdentityResolutionData Archive RulePoint PowerExchange MDMAdvanced XMLLBM Streaming
Data Explorer
Data QualityData Transformation
Data
Resolution
Data Services
PowerCenter Data Subset
Data Privacy
RuleCast
Real-Time
gg
UME Persistence
UME Queueing
36
Data ExplorerExchange Data Services Data PrivacyAlert Manager UME Queueing
Good technology alone does not guarantee success
“Deploying technology, by itself, is a poor strategy for improving the implementation of interfaces. An effective strategy for p gyintegration is established by using these critical success factors: • Establish an ICC. • Develop an integration "city plan" consisting of: p g y p g
• Policies that implement the necessary governance • A set of best practices that is continually refined by measuring
the result of applying the best practices in integration projects pp y g p g p j• A technology architecture that establishes standards for the
technologies to be used during application integration • Have integration competency staff drive the selection of the g y
integration technologies that are to be deployed. “
The Seven Critical Success Factors for Application Integration,
37
pp g ,Jess Thompson, Gartner, November 10, 2010
As-Is Value Stream MapDOIT Corporation: Value Stream Map (AS-IS) for Change Request Process
Monday, December 20, 2010
GMNA Applications Team(Irina)
Data Warehouse Team
Change Request
Confirmation Request
Telephone Tag
Status Request
Status Update
Automated Workflow/Tracking (Cust satisfaction)
Status Request
Semi-Weekly Review
CR Review Committee
Production CR ApprovedChanges
Clarify Requirements
Requirements Clarification
Status Request
Status Update
Data Dictionary to clarify rqmnts
(13 days)
Bypass Council for simple CR’s
(26 d )
Notify Customer(Cust Satisfaction)
(5 days)
Status Update
Integration Team Manager
Assign Resource
Test Team Manager
Test SchedulingData Warehouse Team
SubmissionAdd CRTo List
g
ForwardCR Request
Change Management Board
CR Approval
Daily ETL Batch Run
Test Results Distribution
Charge Request
Bypass Committee for
simple changes (8 days) Design
DocumentApproved Designs
(26 days)
Automated
Architecture Review Council
Design Approval
Development Team
1
Requirements Review
Infrastructure Team
Production Deployment
Test Team
Test Execution
Development Team
Testing Handoff
Development Team
Design & Development
CR’sP1 12
Automatic Daily ETL Batch Run
Production Execution
CR RequestTo Developer Approved CR
& Design DocsDesign Docs& Schedule
Test Team
Test Case Development
CR Approval& Schedule
Automated Regression
Testing(21 days - requires
investment)
P1x12P2x35
P3x124
30 Minutes
8.8 Days
15 Minutes
8.5 Days
90 Minutes
12.8 Days
180 Minutes
1 Day
15 Minutes
26 Days
180 Minutes
13.3 Days
Work Time = 510 Minutes (8.5 hrs or 0.35 Days)
Lead Time = 75.6 Days0.3 Days
Value Ratio: Work Time / Lead Time = 0.5%Notes:
40
(1) Lead Time includes 5 delay in customer notification(2) Lead Time could be reduced to 24 days with just process changes and using existing tools(3) Lead Time could be reduced to 3 days with a capital investment for automated testing
Helpful resources: Best practice referencesAvailable at amazon.com or integrationfactory.com
• Integration Competency Center (2005): A permanent cross-functional team operatingpermanent cross functional team operating as a shared service function supporting multiple organizational units and sustaining integration in a coordinated manner. g
• Lean Integration (2010): A teachable, sustainable management system that emphasizes creating value for customers, gcontinuous improvement, and eliminating waste as a sustainable data management and integration practice.
42
CIO Magazine, October 2010Staff Pick for IT Management & LeadershipStaff Pick for IT Management & Leadership
44
Sharing Knowledge Among Users
The go-to destination to buy and sell proven The go-to destination to buy and sell proven data integration, data quality, and data management solutions (Blocks).
G t t d! R d ti t k t ithGet promoted! Reduce your time to market with pre-built solutions
Get ahead of the competition! Increase yourGet ahead of the competition! Increase your sales reach with Informatica Marketplace
Maximize your Informatica investment withMaximize your Informatica investment with Informatica Marketplace - www.informaticamarketplace.com
45
Integration Maturity Assessment Survey
Integration Maturity Assessment
60 0
Integration Maturity Assessment
60 0
30.0
40.0
50.0
60.0
MAX
- Av
erag
e Lean
Sustaining
Lean
Sustaining30.0
40.0
50.0
60.0
MAX
- Av
erag
e
0.0
10.0
20.0
agem
ent
hodolo
gy
agemen
tch
itectu
re
Process
agement
System
s
MIN
- M
Program
Project
Program
Project0.0
10.0
20.0
agem
ent
hodolo
gy
agemen
tch
itectu
re
Process
agement
System
s
MIN
- M
Financia
l Man
agInt
egratio
n Meth
oMeta
Data Man
agInf
ormati
on Arch
Busines
s PMod
eling M
anag
Integ
ration Sy
As of September 1, 2010Fina
ncial M
anag
Integ
ration M
etho
MetaData
Manag
Inform
ation
ArchBus
iness P
Modelin
g Mana
gInt
egratio
n Sy
46
http://vip.informatica.com/?elqPURLPage=7960
Takeaways
• Lean is Easy – don’t try to be “perfect”
• Learn a Lean technique and start using it
• Invest in Factory tools• Invest in Factory tools
48
Reference Models are an essential element of an effective Business FrameworkBusiness Framework
How does a framework help?
• Faster projects
Solutions defined in terms of• Higher quality solutions
• Decreased cost through reuse of technology assets
Tighter integration of
Solutions defined in terms of
• Tighter integration of business and IT – more
flexible and agile organization
C B i
IFW FOUNDATION MODELS
FSDM FSFM FSWM
IFW PROCESS & INTEGRATION MODELS
IFW DATA MODELS
• Common Business Language
• Up-to-date Risk and Compliance requirements
i t t d th h t
IBM’s IFW
IFW PROCESS MODELS
FS-BOM
FS-IDM
BDWM
BSTs
IFW ODS
Banking
Solutions defined in terms of integrated throughout
50
Information Architecture Framework
Layer 4 – Enterprise ViewOverarching consistent context for Vi
ews
Reference Models
Information ModelsBusiness GlossaryCanonical Model
Process ModelsBusiness Event Model
Operational Workflow Model
Overarching consistent context for executives & information stewards
Layer 3 - Business ViewDomain models for business owners and project sponsors 3R 3M
4
Bus
ines
s V
Business Context DiagramFunction/Information Matrix
Master Data List4
Metadata(federated
repository of relevant models)Physical Data Models
Logical Data ModelsEntity Relationship Diagram
Transformation ModelsI t f S ifi ti
Interaction ModelsLogical Data Map
Sequence diagrams
grat
ion
View
s
owners and project sponsors
Layer 2 – Solution ViewArchitecture models for specific
systems and solutions 2R 2M
3R 3M
Layer 1 Technology View models)yFile/Table PropertiesStorage Information
Interface SpecificationsTransformation rulesProtocol Definitions In
teg
1R 1M
Layer 1 –Technology ViewTechnical models for developers,
engineers and operations staff
Data at Rest Data in Motion
A layered approach is needed to manage data complexity, provide multiple stakeholder views and maintain traceability to business requirements
52
stakeholder views, and maintain traceability to business requirements.
Data Governance Methodology
One-Time Effort Repeat for every program
2Define
Governance Framework
6Assess
Baseline & Data Quality
10Implement Projects
10Implement Projects
1Organize
Governance Committee
5Scope
Program
3Develop
Enterprise Models
Data Quality
8Plan Migration
Roadmap
10Implement Projects
7
9Develop Program Models
4Assign
OrganizationalRoles
7Develop Target
Architecture
Entry Points1. Enterprise Initiative (Large-scale program with defined ROI – e.g. Merger or major business transformation)2. Top-level Directive (C-level or higher – e.g. one view of customer/business, control IP, data security, etc.)3. Scale-up DQ and MDM Projects (Leverage successful solutions that demonstrate the value for a specific
53
p j ( g pbusiness area and scale them up to cross-functional enterprise-wide initiatives)