Datawarehouse AutomationBest practices and customer cases
20 september 2012Amsterdam
Erik [email protected]@erikfransen
Best practices & customer cases2
Agenda
• BI Challenges of today• BI Ability Model & DWH Automation• Best practices• Cases: Rotterdam & KAS BANK• Q&A
Best practices & customer cases3
Centennium BI expertisehuis• Independent knowledge partner• Provides clients with the right skills, at the right
time and the right way to maximize BI results• Consultancy, ad-interim support, project
implementation and training services• We take or share responsibility for the
execution and management and support of BI and DWH projects
• We offer an extensive portfolio of courses and training services: www.bi-opleidingen.nl
• We provide our customers with the knowledge and practical insights required to be(come) self-sufficient in maintaining and expanding their BI-environments
Best practices & customer cases4
What are (y)our biggest BI-challenges today?
• Create more business value• Empower users• Lower the overall cost• Deliver high quality BI products• Reduce complexity• Organize Business Intelligence to
become more effective
How can DWH automation contribute?
Best practices & customer cases5
Ability to Benefit
Ability to Specify
Ability to Implemen
t
Ability to Execute
DEMAND
SUPPLY
DWH Automation – in perspective
BI Ability Model
Best practices & customer cases6
Automation – in perspective
• Supports your ability to execute– Cuts complexity and resources
• Requires a strategy to change focus from coding to (data) modeling
But….• You’re the solution – Automation is your
instrument…
Best practices & customer cases7
The solution?
• Automation vs. BI Professionals or….• The tool is the tool, the methodology is
the methodology, etc… • Team up to optimize our profession in
helping organizations to reach ‘infinity and beyond…’
Methodologies Automation tools
ProfessionalsCommunity
Best practices & customer cases8
Back to the Challenges
• How does DWH automation help face the challenges?
• Directly and indirectly.
• In short… DWH automation is great (!) if handled by a skilled and knowledgeable professional (you!)
Best practices & customer cases9
CDM: set of best practices• Best practice methodology for BI and DWH
automation• Create datawarehouse and BI products fast, with high
and constant quality and low cost• CDM includes:
– Tooling (free, open source or licensed)– Modeling paradigms like, DV, 3NF, DM– Quality control mechanism– Agile development– Knowledge partnership, training, coaching
• Quality control mechanism: extensive checklists and documentation
• Knowledge transfer by training, certification and learning on-the-job
Best practices & customer cases10
CDM
Structure
Organize
Knowledge Partnership
Model PresentGenerat
e Benefit
Best practices & customer cases11
Your datawarehouse is like a diner in a 3 star restaurant…
Create cook book
Manage the kitchen
Suppliers
Select Ingredient
sServeCook Eat
Best practices & customer cases12
Knowledge partnership (suppliers)
• Goal: self-supporting clients• On-the-job coaching, learning by doing• Training
– BI fundamentals– Data Vault certification – Dimensional modelling– Tool training (partners)
• Centennium supports it’s customers by (management) consultancy, assessments, projects, training and sourcing
Best practices & customer cases13
Structure (create cookbook)
• Establish a common vision on DWH and BI and the role of automation
• Identify needs, pain, benefits, goals• Create a roadmap: what and how?• Develop teams, knowledge, new roles:
prepare organization for automation• Design automation architecture • Create short list tools• Select agile approach
Best practices & customer cases14
Model (select ingredients)• Information & requirements analysis
– Business needs like KPIs, reports, cubes– Identify semantic gaps, business rules– MDM, MTM, Reference data– Business keys
• Analyze & model source extraction to support automation
• Model the staging, datawarehouse, data marts, meta layers, cubes
• Select tool(s) for generation– DWH Automation tools– Combination of “classic” ETL tools and automation
tools
• Should fit the structure!
Best practices & customer cases15
Generate (Cook)
• Create system setup with respect to automation architecture
• Develop test scenarios– Generic testing of automation process
(completeness and correctness)– Test compliance to architecture
• Generate components and objects– Focus on understanding data, data
modeling & business rules, not on coding
• Test and implement
Best practices & customer cases16
Present (serve)
• Deliver information products to users, fast and of high quality
• Automation leverages BI self-service– Model and generate cubes– Model and generate business rules – Adapt quickly to changing source data and
information needs
– Reduced technical complexity– Speed up of the overall BI process
Best practices & customer cases17
Benefit (eat)
• The business users can focus on creating value adding information products
• Constant and predictable quality• Short time to market• IT can quickly adapt to changing
business needs/focus• More time and resources for creating
value-adding BI• In the end: lower cost, higher value• But…
Best practices & customer cases18
Organize (manage the kitchen)
• Automation may have high impact on existing datawarehouse and BI teams– Elimination of tools, data manipulation processes
and people… (cutting complexity and resources)– Also: new roles like source analyst, data model
specialist, automation architect are introduced– Resistance by traditional vendors and suppliers
• Automation can create high business value, but not on it’s own– A knowledgeable team is essential
• We believe in self service: organize knowledge transfer a.s.a.p. and coach teams to be self supporting
Best practices & customer cases19
Rotterdam• Challenge: service team as a central and flexible point
for management information delivery for employees and partners of Rotterdam
• Structure:– As a knowledge partner we combined Rotterdam’s knowledge
on Oracle eBS with DWH automation– Automation architecture “forces” Rotterdam to comply to the
rules
• Model & generate– DWH automation optimized for Oracle eBS– Reusable methods for other eBS customers
• Benefit– Cuts complexity and lowers licensing & consultancy fees
significantly– DWH is now of strategic importance and acts as a central data
hub for Rotterdam– Self service for users
Best practices & customer cases20
• Challenge– Deliver integrated information instead of stovepipe data– Minimize development cost, maximize business value– Make KAS BANK self supporting as much as possible
• Structure– Create a shared view on datawarehousing and automation benefits
and proof it– Data Vault Methodology, near real-time and high volume– Strong focus on team training and coaching of architects
• Model & generate– Re-usable extraction of complex Oracle and Adabas data stores– Business driven Data Vault and Dimensional models– Generated data integration, data distribution, EDW code, without
ETL tools– Transparent and self supporting
• Benefit– Significantly shorter time to market of information products– 100% reliability, auditability and predictability– Business is eager for more– Next step: introduce self-service at business level
Best practices & customer cases21
Automation – do’s
• Do’s– Define an automation vision and strategy
• end-to-end or step-by-step – Take your time and involve all stakeholders– Explain concepts, (business) benefits and
potential risks– Consider a two-step approach: pilot -
project– Align and train the development- and
maintenance teams
Best practices & customer cases22
Automation –don ‘ts
• Don’ts– Deviate from the Structure– Generate or automate the DWH as a goal,
not a means to an end– Underestimate the need for presenting
and benefiting from information…
The proof of the pudding is in the eating!
Best practices & customer cases23
Experience with CDM and DWH Automation
CDM is an evolving set of best practices Introducing additional modelling approaches We are partnering with DWH automation vendors Research topic: generating datawarehouse models directly from business process models
Best practices & customer cases
Next data vault certification seminar: November 1-2, 2012 Amsterdam
www.data-vault.nlwww. geneseeacademy.com
24
25 october, the HagueCrash Course Datawarehouse
Automationwww.bi-opleidingen.nl
Best practices & customer cases25
ABOUT CENTENNIUM
26
Centennium BI expertisehuis houses all the experts under one roof, hereby offering all knowledge and expertise to address the complex business intelligence issues facing our clients today
Services overview:• Consultancy• Projects • Resourcing • Education
Some of our clients: Woonbron, Albron, NZa, CAK, OBR, Vopak, several Dutch Municipalities, Aegon, Nutreco, TNO, Genzyme, Tata Steel, KPN, DELTA, IKEA, Accell, TomTom, KAS BANK, LeasePlan, Brabant Water
Facts and figures:• Founded: 1998• 45+ business intelligence
consultants
Core values:• Human Capital• In close collaboration• Objective and Independent
Expertise:• Business intelligence• Strategic, tactic and operational• Vision based on “effective BI”
27
Centennium BI expertisehuisLange Voorhout 432514 EC 's-Gravenhage Telefoon 070 31 20 370Fax 070 31 20 371URL www.centennium.nl
www.bi-opleidingen.nlwww.data-vault.nl