Top 5 Data Architecture Challenges with Ron Huizenga

transcript

EMBARCADERO TECHNOLOGIESEMBARCADERO TECHNOLOGIES

Top 5 Data Architecture

Challenges

Ron HuizengaProduct Manager – ER/StudioRon.Huizenga@embarcadero.com

EMBARCADERO TECHNOLOGIES

5 Challenges

• Evolution of methodologies & culture

• Adapting to changing architecture

• Complex data environments

• Data quality

• Business focus

Evolution of Methodologies & Culture

• Traditional/Waterfall (up to 1999)– Big design up front– Rigid organizational structure

• Agile – The Rebellion? (1996 – present)– Incremental, time boxed delivery– Self organizing team (who needs roles?)

• The Hybrid (200x? – present)– Agile operational approach– Leverage individual roles/skill sets– Importance of architecture

Agile Process

ER/Studio – Agility Support• True multi-level sub-models• Repository

– Check-out/Check-in of sub-models– Individual data objects– Named versions– Annotation

• Reverse Engineering• Compare and Merge• Naming standards capabilities• Macros• Collaboration & publishing• Data interchange bridges

Adapting to Changing Architecture

• Relational database

• Object oriented

• Unstructured data

• Document management

• Service oriented architecture

• Big data

Service Oriented Architecture (SOA)

Publish & Subscribe

Federated Queries

Operational Data Store

Data Warehouse

Unstructured Data & DocumentsMetadata

Repository

Authentication& Security Services

Application ServerBusiness Process

ManagementBusiness Rule Management

Complex Event Processing Lifecycle Management Policy EnforcementBusiness Activity

MonitoringAdapters Communications Messaging

Metadata ManagementMaster Data Management

Transformation Canonical Model Canonical Messaging Data Integration

Data Federation Data Access Services Document ServicesContent Management

Services

Enterprise Service Bus

Mapping & Transformation

Business Objects

Big Data

• Volume

• Velocity

• Variety

• Veracity

ER/Studio – Native Big Data Support

• MongoDB– Versions 2.4 and 2.6

• Hadoop Hive– Versions 0.12 and 0.13

• Capabilities– Diagramming

– Reverse Engineering (JSON, BSON)

– DDL supported for Hive

ER/Studio – Big Data Notation Enhancement

• Physical Model

– Objects instead of Tables

• Nested Objects

– “Is Contained In” relationship type

Complex Data EnvironmentsEvolution:

• 38 years of construction

• 147 builders

• No Blueprints

• No Planning

Result:

• 7 stories

• 65 doors to blank walls

• 13 staircases abandoned

• 24 skylights in floors

• 160 rooms, 950 doors

• 47 fireplaces, 17 chimneys

• Miles of hallways

• Secret passages in walls

• 10,000 window panes (all bathrooms are fitted with windows)

Complex Data Landscape

• Proliferation of disparate systems

• ERP, mismatched departmental solutions

• SAAS (externally controlled and managed)

• Obsolete legacy systems

• Poor decommissioning strategy

• Point-to-point interfaces

• Data warehouse, data marts, ETL …

ER/Studio – Conquering Landscape Complexity• True multi-level sub-models (hierarchy)

• Reverse engineering (extensive list of platforms)

• Comprehensive metadata extensions (attachments)

• Naming standards capabilities

• Universal mappings

• Macros

• Data lineage

• Business Architect – data context (processes)

• Repository, collaboration & publishing

ER/Studio – Compare and Merge example

C&M cont.

Data Quality

• Accuracy

• Timeliness

• Completeness

• Consistency

• Relevance

• Fitness For Use

Poor Data Quality Implications

• Costs a typical company the equivalent of 15% to 20% of revenue– Estimated by US Insurance Data Management Association

• Low Quality = Low Efficiency

• It is insidious – most data quality issues are hidden in day to day work

• From time to time, a small amount of bad data leads to a disaster of epic proportions

Space Shuttle

Challenger

QueCreek

Mine Flooding

When Data Flaws Happen…

Poor data quality isn’t a new problem

Mitigation Best Practice

• Adopt the philosophy of prevention

• Show thought leadership

• Be accountable at the points of data creation

• Measure, control, improve

• Establish data culture

Technique: Attachments (Metadata extensions)

Business Focus

• Governance

• Champion for data value, data quality

• Consultative approach – business acumen

• Establish a data culture

• Business collaboration

Addressing Governance through Models

Data Governance

Data Architecture Management

Data Development

Database Operations

Management

Data Security Management

Reference & Master Data Management

Data Warehousing

& Business Intelligence

Management

Document & Content

Management

Metadata Management

Data Quality Management

Data Management

• Shared responsibility and collaboration

– Data Management Professionals

– Business Data Stewards

• Cultural change leadership

– Adopting the principles and practices of data management requires leadership from change agents at all levels

• Powerful enterprise glossary, model & metadata collaboration

• Integrate key business terms and definitions with business systems

• View, store, and manage a single source of business definitions

• Attach business policies to daily workflows with contextual alerts and tips

Business Collaboration

Concluding Remarks

Thank you for attending!

• Learn more about the ER/Studio product family: http://www.embarcadero.com/data-modeling

• Trial Downloads: http://www.embarcadero.com/downloads

• To arrange a demo, please contact Embarcadero Sales: sales@embarcadero.com 1 (888) 233-2224

Top 5 Data Architecture Challenges with Ron Huizenga

Software